Problem installing addons

[lucadelu]

Describe the bug
Impossible to install addon from g.extension

To Reproduce
Steps to reproduce the behavior:

Running g.extension on Singularity file have an error on github server, it seems something similar to #2137

grass -c EPSG:4326 --tmp-location --exec g.extension -s ext=i.modis
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
Cleaning up temporary files...
Executing <g.extension -s ext=i.modis> ...
WARNING: The download of the json file with add-ons paths from the github
         server wasn't successful, rate limit exceeded. The previous
         downloaded json file will be used if exists.
Fetching <i.modis> from GRASS GIS Addons repository (be patient)...
Compiling...
WARNING: The download of the commit from the GitHub API server wasn't
         successful, <rate limit exceeded>. Commit and commit date will not
         be included in the <i.modis.download> addon html manual page.
WARNING: The download of the commit from the GitHub API server wasn't
         successful, <rate limit exceeded>. Commit and commit date will not
         be included in the <i.modis.import> addon html manual page.
Installing...
Updating extensions metadata file...
Updating extension modules metadata file...
WARNING: The download of the json file with add-ons paths from the github
         server wasn't successful, rate limit exceeded. The previous
         downloaded json file will be used if exists.
Traceback (most recent call last):
  File "/usr/local/grass78/scripts/g.extension", line 2679, in <module>
    sys.exit(main())
  File "/usr/local/grass78/scripts/g.extension", line 2657, in main
    install_extension(source=source, url=url, xmlurl=xmlurl, branch=branch)
  File "/usr/local/grass78/scripts/g.extension", line 1032, in install_extension
    install_module_xml(new_modules)
  File "/usr/local/grass78/scripts/g.extension", line 1343, in install_module_xml
    mlist = filter_multi_addon_addons(
  File "/usr/local/grass78/scripts/g.extension", line 1321, in filter_multi_addon_addons
    for addon in get_multi_addon_addons_which_install_only_html_man_page():
  File "/usr/local/grass78/scripts/g.extension", line 1283, in get_multi_addon_addons_which_install_only_html_man_page
    with open(addons_paths_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/grass78/addons_paths.json'
Execution of <g.extension -s ext=i.modis> finished.
Cleaning up temporary files...

I'm not able to reproduce on my own laptop

Expected behavior
The addons is installed correctly

System description (please complete the following information):

• Debian Bullseyes using Singularity file
• GRASS GIS version release branch 7.8

Additional context
Add any other context about the problem here.

[tmszi]

During addon installation is addons_paths.json file downloaded from GitHub API server, if you hit rate limit exceeded error (For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the person making requests.), it not possible to continue the addon installation, because addons_paths.json file doesn't exists. You can wait if limit is freed.

This warning message is important and I will change message severity from WARNING to FATAL.

WARNING: The download of the json file with add-ons paths from the github
         server wasn't successful, rate limit exceeded. The previous
         downloaded json file will be used if exists.

[lucadelu]

@tmszi thanks for explanation. I don't understand why I exceeded the number of request, there is still some problem related to #2137 ?

I try to install addons immediately after the compilation.

I agree to switch WARNING to FATAL since addons installation will fail. Is it possible to use standard http request instead GitHub API server?

[tmszi]

Is it possible to use standard http request instead GitHub API server?

Rate limit exceeded error, is GitHub REST API server limitation (number of allowed request per hour per your computer ip address), it is not bug in our code. The solution is to pay money GitHub to increase the limit, or PR #2140.

1. You hit this error, because you compile GRASS GIS from release archive source code (without Git repo e.g. https://github.com/OSGeo/grass/archive/refs/tags/8.0.0.tar.gz), see explanation diagram bellow.

2. Then you want install addon, and this fail because it is not possible get response from GitHub REST API server (rate limit was exceeded for your computer ip address, as a result of point 1.) and make addons_paths.json file from GitHub REST API response JSON is not possible too, which is required for correct core module manual page build process.

[lucadelu]

@tmszi thanks a lot for the clear explanation. Now I have fully understand! I tried with grass-7.8.7RC1 tag but it doesn't work, however I tested #2140 (#2140 (comment)_ ) and it work properly, now the problem remain addons_paths.json, is it possible to find it?

[tmszi]

I tried with grass-7.8.7RC1 tag but it doesn't work

Yes you are right. I backported this PR #2139 into Git dev branch releasebranch_7_8, releasebranch_8_0, which means this bug is solved (if you hit GitHub REST API server rate limit during compilation process, it will be handled correctly) with PR #2139 and a fix will be included in the next official grass-7.8.7RC2 and grass-8.0.1 release.

Actually PR #2140 is only for GRASS GIS 8.1 dev release.

Is there a chance that you would compile GRASS GIS 7.8 from the Git dev branch releasebranch_7_8 and GRASS GIS 8.0 from the Git dev branch releasebranch_8_0?

Or if you want use official release grass-7.8.7RC1, or grass-8.0.0 I prepare patch for you.

now the problem remain addons_paths.json, is it possible to find it

Could you wait if GitHub REST API rate limit is freed, please?

I plan to change this message

WARNING: The download of the json file with add-ons paths from the github
         server wasn't successful, rate limit exceeded. The previous
         downloaded json file will be used if exists.

so that there is a time when it will be possible to make request/response from GitHub REST API server again, something like that.

FATAL: The download of the JSON file with add-ons paths from the GitHub 
       server wasn't successful, due rate limit exceeded. The previous
       downloaded JSON file not be used because doesn't exists. Please 
       try it again on Monday, February 11, 2022 13:47:53 PM.

[petrasovaa]

@tmszi do you know what the limit is? Is there a scenario when you compile without git and don't hit the limit given 500 modules? I don't think it's likely somebody would be paying for the access. It just seems to me this whole thing is very complicated and nobody will be able to use it, or?

[neteler]

BTW:
See this (unrelated) PR to avoid the excess of github API rate_limit:
mundialis/g.extension.github#4

Not sure if it helps, just FYI.

[tmszi]

do you know what the limit is?

For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the person making requests.

tomas@gentoo-gnu-linux:/tmp$ curl -s -o /dev/null -D - https://api.github.com/repos/OSGeo/grass-addons/git/trees/grass8?recursive=1 | grep "x-ratelimit-limit"
x-ratelimit-limit: 60

Is there a scenario when you compile without git and don't hit the limit given 500 modules?

No it's not. But solution could be use authenticated user. User-to-server requests are limited to 5,000 requests per hour and per authenticated user. But I think we would get over it pretty quickly anyway. Another solution could be #2140.

I don't think it's likely somebody would be paying for the access.

I meant if GRASS GIS GitHub repo would be promoted to GitHub Enterprise account (which is paid) which has higher limit for authenticated user 15,000 API calls per hour.

It just seems to me this whole thing is very complicated and nobody will be able to use it, or?

I agree with you, but we are currently dependent on a proprietary remote GitHub REST API solution.

[lucadelu]

No it's not. But solution could be use authenticated user. User-to-server requests are limited to 5,000 requests per hour and per authenticated user. But I think we would get over it pretty quickly anyway. Another solution could be #2140.

I fully support the #2140 solution

It just seems to me this whole thing is very complicated and nobody will be able to use it, or?

I agree with you, but we are currently dependent on a proprietary remote GitHub REST API solution.

This the worst of part of all the problem. We are using to much this proprietary solution, first for code, after for discussions and now also in the code, I'm a little bit worried if Github will change it's rules...

[wenzeslaus]

We are using to much this proprietary solution, first for code, after for discussions and now also in the code, I'm a little bit worried if Github will change it's rules...

We definitively lack an exit strategy. However, there are practices we can follow to minimize the issue, for example source code in Git is easy to transfer elsewhere, or it is at least well-defined and common issue, while API calls sprinkled in the source code are tricky.

[ninsbl]

With #2895 this issue should be less likely to show up (if it shows up at all). #2895 uses git checkout and thus preserves git history that can be injected in the manual page. Therefore no API calls are needed any more.
With some adjustment of the code in #2895, g.extension would also be able to get the history for code from gitlab or bitbucket (for example)...

[ninsbl]

Is this still an issue?

[echoix]

Closing as no response from author, and outdated