Azure api change - image search Endpoint change in Azure - 'PermissionDenied' Error Lesson 2

[bkaCodes]

Hi,

I can't complete lesson 2 - I have an API key for Bing Images search v7 but it seems this resource changed at the end of October 2020 and this change isn't yet incorporated into the code (Azure update). Azure does say that it will support the older Cognitive Services API for the next 3 years, but unfortunately it appears that is only for pre-existing users as I cannot find a way to subscribe as a new Azure user.

Fundamentally from a code perspective, it appears that the endpoint changed from that used in the code (fastbook/utils.py):

def search_images_bing(key, term, min_sz=128):
    client = api('https://api.cognitive.microsoft.com', auth(key))
    return L(client.images.search(query=term, count=150, min_height=min_sz, min_width=min_sz).value)

When I try to run this in the notebook I get a 'PermissionDenied' error which I think is due to this change in APIs from Azure - if so, my theory on the error is this will likely need to be refactored from using the older https://api.cognitive.microsoft.com endpoint to use the new endpoint - https://api.bing.microsoft.com/.. I did attempt a naïve endpoint replacement (wishful thinking I know!) but that unfortunately didn't work.

I've tried on Collab, Paperspace, attempted a local install (hit other issues there) and tried a million different ways to get a Bing search API key via Cognitive Services but it does appear the guidance included on the forum here (referenced on the main fast.ai course page here) is no longer applicable (or I've done something silly - hopefully!).

Unsure how to continue in this lesson without this resource - safe enough to move onto the next? Or perhaps I've done something very incorrect and feedback would be appreciated.

Thanks for any advice/guidance/fixes!
Rebecca

[bkaCodes]

Potentially a hacky workaround, but looks like redefining the function search_images_bing to the following will work temporarily (at least to extract images, still working through the rest of content, will update with further issues)

def search_images_bing(key, term, min_sz=128):
        search_headers = {"Ocp-Apim-Subscription-Key": key}
        search_params  = {"q": search_term, "license": "public", "imageType": "photo", "count": 150,
                            "minHeight": min_sz, "minWidth": min_sz}
        search_ep = "https://api.bing.microsoft.com/v7.0/images/search"
        response = requests.get(search_ep, headers=search_headers, params=search_params)
        response.raise_for_status()
        search_results = response.json()
        thumbnail_urls = [img["thumbnailUrl"] for img in search_results["value"]]
        
        return L(thumbnail_urls)

ims = search_images_bing(key, 'grizzly bear')
len(ims)

[vrao423]

Potentially a hacky workaround, but looks like redefining the function search_images_bing to the following will work temporarily (at least to extract images, still working through the rest of content, will update with further issues)

def search_images_bing(key, term, min_sz=128):
        search_headers = {"Ocp-Apim-Subscription-Key": key}
        search_params  = {"q": search_term, "license": "public", "imageType": "photo", "count": 150,
                            "minHeight": min_sz, "minWidth": min_sz}
        search_ep = "https://api.bing.microsoft.com/v7.0/images/search"
        response = requests.get(search_ep, headers=search_headers, params=search_params)
        response.raise_for_status()
        search_results = response.json()
        thumbnail_urls = [img["thumbnailUrl"] for img in search_results["value"]]
        
        return L(thumbnail_urls)

ims = search_images_bing(key, 'grizzly bear')
len(ims)

This works! But is anyone looking into this? Kinda disappointing to get stuck in Ch. 2.

[bkaCodes]

So full list of changes to make to have this work for the grizzlys are the two below

1. Redefine the image search

def search_images_bing(key, search_term, min_sz=128):
        search_headers = {"Ocp-Apim-Subscription-Key": key}
        search_params  = {"q": search_term, "license": "public", "imageType": "photo", "count": 150,
                            "minHeight": min_sz, "minWidth": min_sz}
        search_ep = "https://api.bing.microsoft.com/v7.0/images/search"
        response = requests.get(search_ep, headers=search_headers, params=search_params)
        response.raise_for_status()
        search_results = response.json()
        thumbnail_urls = [img["thumbnailUrl"] for img in search_results["value"]]
        
        return L(thumbnail_urls)

2. Change the cell that runs the download to the below:

if not path.exists():
    path.mkdir()
    for o in bear_types:
        dest = (path/o)
        dest.mkdir(exist_ok=True)
        results = search_images_bing(key, f'{o} bear')
        download_images(dest, urls=results)
        # renaming from the weird Bing return to be jpgs and picked up by the next get_image_files function
        fs=os.listdir(dest)
        for f in fs:
            p=str(dest)+'/'+f
            n=str(dest)+'/'+Path(f).stem+'.jpg'
            os.rename(p,n)

It's also just been Thanksgiving and the old API is still supported so only people who were making new accounts from scratch would have caught this error - it's likely the CI test harness is still working with the current version!

Hope this helps.

[2-en]

@bkaCodes Can confirm this works, thank you!

[jph00]

Fixed now