[Automation] Remote server SSL verify failed.

[AcckiyGerman]

The datapackage-pipeline cannot download and process the source.

Acceptance criteria

• datapackage is processed after data push-flow

Analysis

source link: https://www.nationwide.co.uk/~/media/MainSite/documents/about/house-price-index/downloads/uk-house-price-since-1952.xls

automation log: https://testing.datahub.io/AcckiyGerman/house-prices-uk/v/8

tabulator.exceptions.HTTPError: HTTPSConnectionPool(host='www.nationwide.co.uk', port=443): Max retries exceeded with url: /~/media/MainSite/documents/about/house-price-index/downloads/uk-house-price-since-1952.xls (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))

File could be downloaded via browser, but I failed to download it with curl
Definitely the bug is on the remote site. (I just wonder why could I get the file via browser without any security warnings)

[rufuspollock]

@AcckiyGerman this may be blocking bots. Have you replicated with data cat url or simple python script.

[AcckiyGerman]

Python "urlretreave" failed as well as curl.

…

On Jan 15, 2018 6:10 PM, "Rufus Pollock" ***@***.***> wrote: Reopened #6 <#6>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALKlIshLoNpPwwAlJCPf3itt0_e4fVbXks5tK4aCgaJpZM4ReEJ3> .

[AcckiyGerman]

"data cat <url>" fails too

…

On Jan 15, 2018 6:33 PM, "Dmitry German" ***@***.***> wrote: Python "urlretreave" failed as well as curl. On Jan 15, 2018 6:10 PM, "Rufus Pollock" ***@***.***> wrote: > Reopened #6 <#6>. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#6 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ALKlIshLoNpPwwAlJCPf3itt0_e4fVbXks5tK4aCgaJpZM4ReEJ3> > . >

[rufuspollock]

@AcckiyGerman have you tried sending a non-bot header with url retrieve or similar?

[AcckiyGerman]

@rufuspollock no, I haven't and reasons are:

• it will not help to automate the dataset, coz assembler still not able to retrieve the source
• it is only one dataset with such an error, I don't see it is worth fixing in the assembler code now
• for updating the datset itself I downloaded data manually 😄