AU-5 data issue

[siemvaessen]

Both Datastore and D-Portal are unable to retrieve AU-5 data from the originating AU-5 source. Humans are able to download the files, but it's very likely some machine crawling not-allowed of kinds is in place on the originating location making it impossible to download files in an automated fashion. As this is not a Datastore nor D-Portal issue, we have closed the issue on the Datastore end have added it to this issue tracker as it seems a more appropriate place to log this issue.

[andylolz]

Same problem raised here: codeforIATI/iati-data-bugtracker#9

I contacted the email addresses I found in DFAT’s metadata, but didn’t hear back.

[andylolz]

The registry archiver is now also showing issues fetching ausgov data:

[ss-bhat]

@andylolz

This is giving timeout

        with requests.get(url, headers=req_headers, timeout=url_timeout, verify=False, stream=True) as res:
            resp_headers = res.headers

requests.get(url) works not sure if it has to do with headers or is a bug in the requests package.

It's a header. They do not allow the user agent to 'Mozilla/5.0' (this is the user-agent we are using in the archiver)

[ss-bhat]

@andreaszenasidi @andylolz

I have changed the user agent, theoretically, this should be fixed when the archiver runs for the next time.

[andylolz]

I have changed the user agent, theoretically, this should be fixed when the archiver runs for the next time.

Nice – looks like it worked! 👏

[andreaszenasidi]

Looks good.