-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Zenodo server changes require user-agent #84
Comments
Just an update, when I access the Zenodo URL, in my Google Chrome browser, I now receive a valid JSON response (see below at end of post). But I still see the internal server error when using zen4R.
|
By evidence, this is due to a change Zenodo server-side. I've seen your new user agent, I will try to create a specific user agent for zen4R as I use to apply for other similar packages. |
@jeffreyhanson I've added a User-Agent systematically to all Zenodo requests. |
i'm going to send a revision to CRAN |
@jeffreyhanson From CRAN: "CRAN submissions will be offline from Jul 22, 2022 to Aug 5, 2022 -CRAN team vacation and maintenance work)" . I will send a revision to CRAN in August |
Brilliant - thank you very much @eblondel! |
I'm sorry, I'm trying the latest developmental version on GitHub (version 0.6-2) and I'm still experiencing this issue. Here's the same reprex as before (copied below) and my session information (with the GitHub version of the package). Please let me know if there's any further details I can provide to help resolve this?
|
Make sure to re-install zen4R from a clean R session, if you use R studio, they are caching issues that prevent from using the latest version. This occurs often if you don't restart R or if you have multiple rstudio open. |
This is what I get
|
Thanks for the suggestion! Although I don't use Rstudio (just plain R from the linux command line), I re-installed the package from GitHub and updated all packages on my system. Unfortunately, I'm still receiving the error. Do you have any other ideas? Would it be possible for me to override the user agent? |
May I ask what OS you're using? If not Linux, I could submit a PR to add a GitHub Action to enable automatic testing on Ubuntu to help reproduce the error? |
Tests are configured on Ubuntu already. I've added a unit test just know, let's see what we get to help troubleshooting the issue. |
From what I can see, it failed on Ubuntu-latest. I have to dig if it's actually related to the User-Agent, i still suspect Zenodo is experimenting issues and is not fully available. |
@jeffreyhanson I've set-up a browser-like agent to reproduce your tests. I've set-up a unit test that is run on 2 versions of R (latest, devel). The first attempt passed on one dist but not the other. I've re-run the failed job and it passed successfully. All this let me think that Zenodo is actually experimienting issues of availability and unexpectedly you get Internal Server errors. |
Thank you very much for looking into this further - I really appreciate it! To see if it's just the case that I'm occaisionally getting unexpected Internal server errors, I wrote a short script to try querying the Zenodo API multiple times to see what the success rate would be. If the issue was due to occaisional/unexpected errors, then I would expect to see some success and some failures. After running this script (see below for reference, along with my session Information), I found that none of attempts were succesful out 50 attempts. This would suggest to me that the underlying issue is not due to occaisional failures. What do you think?
|
For comparison, I'll re-run these benchmarks using the PR I posted earlier to what the success rate for that would be given that user agent header. Please give me a while for these runs to complete. |
Here's the results for running the same code with the PR I posted earlier (see below). Since there's a 100% success rate when using a different user agent header, this would indicate to me that the issue could be resolved by using a different user agent header. It would also further suggest that the issue is unlikely to be due to intermittent/occaissional issues with the Zenodo API. Does that sound right to you? Or maybe I am missing some important detail? Assuming that I am not misunderstanding something, would it be possible to update the package so that users could manually supply their own user agent header? This would allow me to continue using the official version of the package, since I could supply a user agent header that appears to work on my system. This could potentially be achieved using an environmental variable (similar to
|
@jeffreyhanson I've harcoded the same user agent that you put in your PR, but even with this I got Internal server errors randomly through continous integration tests done on R release and devel, without any reproducible pattern. Now the tests appear to pass on Github, but it's because i went there manually to restart the failed tests until they pass. What sounds correct is that now Zenodo requires a user agent, but I can't explain the internal server errors. I will contact Zenodo team and see what they think about it. Meanwhile you can use the current github master which is using the user agent you were suggesting. |
Awesome - thank you very much, I really appreciate it! I've just tested it now, and it works perfectly for me. Thank you for following up with the Zenodo team as well. |
Hi @eblondel,
Thank you very much for developing this package. I've been using it for a while to access data from Zenodo, and it works brilliantly! Recently, however, I've been encountering errors when trying to query records using DOIs. Specifically, here's a reprex (see end of post for session information).
Also, when inspecting the URL (i.e., https://zenodo.org/api/records/?q=doi:10.5281//zenodo.3378733&size=10&page=1&all_versions=1), I see the following text:
Do you have any ideas on what might be causing this? I've tried both the CRAN and GitHub version, but I receive this error on both versions. I also tried with and without supplying my own personal access token -- in case the issue was due to rate limiting -- but that doesn't appear to fix it either. Please let me know if there's any further information I can provide?
Session Information (based on GitHub version)
The text was updated successfully, but these errors were encountered: