Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http downloaders: Provide custom informative User-Agent, do not claim to be "Authenticated access" #5802

merged 2 commits into from
Jul 21, 2021


Copy link

Closes #5801 although I believe it is no longer an issue -- wikipedia seems to no longer mind any bad User-Agent (at least for me locally as of a few minutes back). But I think it would be nice/proper to provide detailed user-agent string.

See individual commits for more information

Christian M. boiled it down to having "-" in the python-requests
which started to throw wikipedia off.  If we are to change User-Agent,
I guess we better make it more informative to start with.

Closes datalad#5801
Logic was assuming that any AccessDeniedError which is not AnonymousAccessDeniedError
but as the case with wikipedia showed, it is not necessarily the case.  So
we will instead report "Anonymous" access issue whenever we know it was such.

Unfortunately I also found no way to get access to that informative detail
wikipedia returns along with its 403, so we will not report more on that
@yarikoptic yarikoptic added UX user experience semver-patch Increment the patch version when merged labels Jul 20, 2021
Copy link

codecov bot commented Jul 20, 2021

Codecov Report

Merging #5802 (5e15f55) into maint (612375e) will decrease coverage by 5.58%.
The diff coverage is 100.00%.

❗ Current head 5e15f55 differs from pull request most recent head 08f6de4. Consider uploading reports for the commit 08f6de4 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##            maint    #5802      +/-   ##
- Coverage   90.33%   84.75%   -5.59%     
  Files         300      297       -3     
  Lines       42349    42317      -32     
- Hits        38258    35865    -2393     
- Misses       4091     6452    +2361     
Impacted Files Coverage Δ
datalad/downloaders/ 77.73% <100.00%> (-0.51%) ⬇️
datalad/downloaders/ 79.71% <100.00%> (-0.44%) ⬇️
datalad/support/tests/ 0.00% <0.00%> (-100.00%) ⬇️
datalad/metadata/extractors/tests/ 19.35% <0.00%> (-80.65%) ⬇️
datalad/metadata/extractors/ 12.96% <0.00%> (-79.63%) ⬇️
datalad/metadata/extractors/ 18.75% <0.00%> (-78.13%) ⬇️
datalad/metadata/extractors/ 19.35% <0.00%> (-77.42%) ⬇️
datalad/metadata/extractors/ 20.00% <0.00%> (-77.15%) ⬇️
datalad/metadata/extractors/tests/ 24.00% <0.00%> (-76.00%) ⬇️
datalad/metadata/extractors/tests/ 26.92% <0.00%> (-73.08%) ⬇️
... and 103 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 612375e...08f6de4. Read the comment docs.

Copy link

adswa commented Jul 20, 2021

This fix works for me! Thanks!

wikipedia seems to no longer mind any bad User-Agent (at least for me locally as of a few minutes back).

For me it still fails

Copy link
Member Author

oh... may be some nice cookie got stored/reused and thus it didn't fail for me? ... odd -- anyways, great to hear that the fix works.
FTR: appveyor singular fail is error to instantiate mac env. not my fault ;) strange that there is no codecov overall status.

@yarikoptic yarikoptic added the merge-if-ok OP considers this work done, and requests review/merge label Jul 20, 2021
Copy link

@mih mih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Thx for the quick fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
merge-if-ok OP considers this work done, and requests review/merge semver-patch Increment the patch version when merged UX user experience
None yet

Successfully merging this pull request may close these issues.

None yet

3 participants