g.extension: doesn't return list of available extensions in the official GRASS GIS Addons repository #2025
Comments
This is stuck at OSGeo/grass-addons#613 which we need to implement asap. |
|
Related to #1960 |
|
Sorry I closed this issue prematurely, my mistake. |
|
This report relates to grass 7.8.6 but it seems related to this issue for v8.0 The works-for-me solution was to edit g.extension.py based on a posting found here. I do not have sufficient understanding of proper usage of SSL and python networking to suggest this is a good solution, only that it worked. Details:I recently installed a completely new and updated osgeo4w bundle on my Windows 10 laptop. ERROR: Cannot open URL: I'm not experienced with python networking code so added a bunch of grass.message(...) statements to identify the failure was occurring with Running python from the grass command window and trying to retrieve the url directly using urllib in the interpreter produced an error related to an SSL failure... |
IIRC, modules/addons linking to pdal aren't available in winGRASS due to C++ compiling issues on incompatible build environment. |
|
Sorry if I distracted you... After the previous noted change for the SSL failure I am able to have g.extension return the list and all extensions I attempted to install were successful. (including r.in.pdal) |
|
I just updated the main branch and when running When I try to do the same in the GUI, i.e., go to Settings-> Addons extensions-> Install extension, I get this in the GUI console tab: Installing extensions from the terminal works just fine though. System Info: |
|
It seems, that problem is missing build GRASS GIS 8 version addons on CTU server, missing modules.xml and toolboxes.xml file, see this URL directory https://grass.osgeo.org/addons/grass8/. In this line code is missing version 8, same here and here and etc., it seems that building GRASS GIS Addons (matching addons xml files) on CTU server doesn't count with version 8. |
Could we generate these (missing) files on grass.osgeo.org itself to not be dependent on remote files?
Good catch! Maybe we need to introduce a config file which defines GRASS_VERSION_MIN and GRASS_VERSION_MAX or so, for better loops and a central place to configure it (with ´source config.txt` in the beginning of the respective shell files). |
Yes we could. We will have to complete this PR and at the same time fix this issue #1960.
Yes I agree a good idea. |
Since I didn't find a sh/bash/... compatible way to parse a |
|
I looked at the metadata of that module.xml file (require for |
|
Yes, so I wonder if we can generate that on the GRASS server itself? |
I'm going to look at it. |
On github? |
No, I thought of grass.osgeo.org where the addon manual pages are already built. |
I tried to overcome the issue too by myself two weeks ago, but, couldn't get pas the SSL error. I have difficulty understanding the root of the issue, since this is some urllib Python 2 code ported to Python 3, by mimicking the same behaviour as Python 2. What is clear with the approach in this comment is that disabling the ssl verification is not a solution and too risky as a real workaround. I took a second look, to try to understand the issue. I'm on Windows 10, using a OSGeo4W install (of QGIS 3.22.2, including GRASS 7.8.6). I found out that how the code is written, a https website should be working normally, since it works when using a different hardcoded URL than the http://wingrass.fsv.cvut.cz/ URL. On Chrome (at least), the http://wingrass.fsv.cvut.cz/ URL is redirected with a 301 response to https://wingrass.fsv.cvut.cz/. Certificates are valid. When using a different https URL, there is no SSL error (and the requests themselves are successful, but they aren't the good content). So this might mean that by default, there are some outdated certificates, but I couldn't find a way to verify that. Is there a website configuration issue? Either way, the g.extension code must be able to handle this. So I continued to try and find a solution and found a solution, but I don't think it's a valid fix, since I introduce a dependency, and I can't figure out where the python requirements are listed. The solution works on Windows from the OSGeo4W QGIS install, since some other packages are installed too. Using urllib for web requests is quite challenging to do right, and isn't recommended even by the Python docs, which refer to using the ubiquitous So my fix is in two parts: Using a default urllib opener ( Finally, adding everything together, and adding a protection so an import error doesn't fail and the extension continues to work, gives the following: At the imports, add try:
from ssl import create_default_context
import certifi
except ImportError:
# If not built with SSL or certifi not present, do not configure
# the HTTPSHandler and only use default opener.
create_default_context = None
certifi = Noneand in the main() function at the end of the file, replace the proxy section with the following (since there are some changes before and after): if create_default_context and certifi:
ssl_context = create_default_context(cafile=certifi.where())
https_handler = urlrequest.HTTPSHandler(context=ssl_context)
opener = urlrequest.build_opener(https_handler)
else:
opener = urlrequest.build_opener()
# manage proxies
global PROXIES
if options["proxy"]:
PROXIES = {}
for ptype, purl in (p.split("=") for p in options["proxy"].split(",")):
PROXIES[ptype] = purl
proxy = urlrequest.ProxyHandler(PROXIES)
# opener = urlrequest.build_opener(proxy)
# urlrequest.install_opener(opener)
# Since urllib.request.build_opener() creates many types of handlers when possible,
# including ProxyHandler, but it wouldn,t be configured correctly, we are need to replace it.
for handler in opener.handlers:
if isinstance(handler, urlrequest.ProxyHandler):
opener.handlers.remove(handler)
opener.add_handler(proxy)
urlrequest.install_opener(opener)So, the complete import section looks like: from __future__ import print_function
import fileinput
import http
import os
import codecs
import sys
import re
import atexit
import shutil
import zipfile
import tempfile
import json
import xml.etree.ElementTree as etree
from distutils.dir_util import copy_tree
try:
from ssl import create_default_context
import certifi
except ImportError:
# If not built with SSL or certifi not present, do not configure
# the HTTPSHandler and only use default opener.
create_default_context = None
certifi = None
from six.moves.urllib import request as urlrequest
from six.moves.urllib.error import HTTPError, URLError
from six.moves.urllib.parse import urlparse
# Get the XML parsing exceptions to catch. The behavior changed with Python 2.7
# and ElementTree 1.3.
from xml.parsers import expat # TODO: works for any Python?
if hasattr(etree, "ParseError"):
ETREE_EXCEPTIONS = (etree.ParseError, expat.ExpatError)
else:
ETREE_EXCEPTIONS = expat.ExpatError
import grass.script as gscript
from grass.script.utils import try_rmdir
from grass.script import core as grass
from grass.script import task as gtask
# temp dir
REMOVE_TMPDIR = True
PROXIES = {}
HEADERS = {
"User-Agent": "Mozilla/5.0",
}
HTTP_STATUS_CODES = list(http.HTTPStatus)
GIT_URL = "https://github.com/OSGeo/grass-addons"and the def main():
# check dependencies
if not flags["a"] and sys.platform != "win32":
check_progs()
original_url = options["url"]
branch = options["branch"]
if create_default_context and certifi:
ssl_context = create_default_context(cafile=certifi.where())
https_handler = urlrequest.HTTPSHandler(context=ssl_context)
opener = urlrequest.build_opener(https_handler)
else:
opener = urlrequest.build_opener()
# manage proxies
global PROXIES
if options["proxy"]:
PROXIES = {}
for ptype, purl in (p.split("=") for p in options["proxy"].split(",")):
PROXIES[ptype] = purl
proxy = urlrequest.ProxyHandler(PROXIES)
# Since urllib.request.build_opener() creates many types of handlers when possible,
# including ProxyHandler, but it wouldn,t be configured correctly, we are need to replace it.
for handler in opener.handlers:
if isinstance(handler, urlrequest.ProxyHandler):
opener.handlers.remove(handler)
opener.add_handler(proxy)
urlrequest.install_opener(opener)
# define path
options["prefix"] = resolve_install_prefix(
path=options["prefix"], to_system=flags["s"]
)
if flags["j"]:
get_addons_paths(gg_addons_base_dir=options["prefix"])
return 0
# list available extensions
if flags["l"] or flags["c"] or (flags["g"] and not flags["a"]):
# using dummy extension, we don't need any extension URL now,
# but will work only as long as the function does not check
# if the URL is actually valid or something
source, url = resolve_source_code(
name="dummy", url=original_url, branch=branch, fork=flags["o"]
)
xmlurl = resolve_xmlurl_prefix(original_url, source=source)
list_available_extensions(xmlurl)
return 0
elif flags["a"]:
list_installed_extensions(toolboxes=flags["t"])
return 0
if flags["d"] or flags["i"]:
flag = "d" if flags["d"] else "i"
if options["operation"] != "add":
grass.warning(
_(
"Flag '{}' is relevant only to"
" 'operation=add'. Ignoring this flag."
).format(flag)
)
else:
global REMOVE_TMPDIR
REMOVE_TMPDIR = False
if options["operation"] == "add":
check_dirs()
if original_url == "" or flags["o"]:
"""
Query GitHub API only if extension will be downloaded
from official GRASS GIS addon repository
"""
get_addons_paths(gg_addons_base_dir=options["prefix"])
source, url = resolve_source_code(
name=options["extension"], url=original_url, branch=branch, fork=flags["o"]
)
xmlurl = resolve_xmlurl_prefix(original_url, source=source)
install_extension(source=source, url=url, xmlurl=xmlurl, branch=branch)
else: # remove
remove_extension(force=flags["f"])
return 0 |
|
See also: OSGeo/grass-addons#656 |
Update: developed and deployed on grass.osgeo.org (may take up to 24hs to get to the next cronjob cycle), for both G7 and G8. |
|
Fixed with GRASS GIS Addons PR OSGeo/grass-addons#656. |
Describe the bug
g.extension doesn't return list of available extensions in the official GRASS GIS Addons repository.
To Reproduce
Steps to reproduce the behavior:
g.extension -lExpected behavior
g.extension should return a list of available extensions
System description (please complete the following information):
Additional context
g.extension tries find module.xml file at this URL https://grass.osgeo.org/addons/grass8/modules.xml, but grass8 directory doesn't exists on the server, only grass6 and grass7 exists. Please check them using this URL https://grass.osgeo.org/addons/.
The text was updated successfully, but these errors were encountered: