Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc building: import GDAL Python bindings early #9783

Merged
merged 1 commit into from
Apr 30, 2024

Conversation

rouault
Copy link
Member

@rouault rouault commented Apr 26, 2024

We don't strictly need to import them at the beginning of conf.py, but importing them early helps in having a better error message if they cannot be imported rather than waiting later when autodoc tries to load them

In my case, this was due to an obscure situation. My libgdal.so in my build tree links to a /path/to/install-libjpeg-turbo/lib/libjpeg.so.8, with it being libjpeg-turbo 3.0 with new symbols. That full path to libjpeg is embedded in libgdal.so, but /path/to/install-libjpeg-turbo/lib was not pointed in LD_LIBRARY_PATH.
This worked fine when doing "from osgeo import gdal" from a 'regular' Python interpreter, or running pytest. But strangely, this failed within the sphinx-build process. The autodoc error message wasn't helpful just saying it couldn't import the osgeo module, whereas if trying to import in conf.py I got an exception pointing to the fact a symbol from libjpeg.so couldn't be loaded. I suspect that something in the sphinx-build process loads the system libjpeg library, that does not have the new symbols of libjpeg-turbo 3.0 that my libgdal.so was using, hence causing later loading of libgdal to fail.

Now that I've added /path/to/install-libjpeg-turbo/lib in LD_LIBRARY_PATH, things work and this patch isn't actually needed, but it may help diagnose further similar situations

CC @dbaston

@rouault rouault force-pushed the doc_building_python_bindings branch from 86d9e7b to 2be3c80 Compare April 26, 2024 16:43
@dbaston
Copy link
Member

dbaston commented Apr 30, 2024

I would be hesitant about messing up the doc build for a casual contributor who wants to tests a change to another part of the docs and doesn't have the Python bindings set up correctly. Here is a variant that raises a warning at the start of the build instead of requiring an environment variable to be set.

Importing early also provides a good opportunity to check for version mismatches.

import traceback
from sphinx.util import logging
logger = logging.getLogger(__name__)
try:
    from osgeo import gdal
except ImportError as e:
    logger.warn("Failed to load GDAL Python bindings. The Python bindings must be accessible to build Python API documentation.")
    for line in traceback.format_exception(e):
        logger.info(line)
else:
    version_file = os.path.join(os.path.dirname(__file__), os.pardir, os.pardir, "VERSION")
    doc_version = open(version_file).read().strip()
    gdal.__version__

    if doc_version.strip() != gdal.__version__:
        logger.warn(f"Building documentation for GDAL {doc_version} but osgeo.gdal module has version {gdal.__version__}. Python API documentation may be incorrect.")

@rouault rouault force-pushed the doc_building_python_bindings branch from 2be3c80 to 4aa76b4 Compare April 30, 2024 14:17
We don't strictly need to import them at the beginning of conf.py, but
importing them early helps in having a better error message if they
cannot be imported rather than waiting later when autodoc tries to load them

In my case, this was due to an obscure situation. My libgdal.so in my
build tree links to a /path/to/install-libjpeg-turbo/lib/libjpeg.so.8, with
it being libjpeg-turbo 3.0 with new symbols. That full path to libjpeg is
embedded in libgdal.so, but /path/to/install-libjpeg-turbo/lib was not
pointed in LD_LIBRARY_PATH.
This worked fine when doing "from osgeo import gdal" from a 'regular'
Python interpreter, or running pytest. But strangely, this failed within the
sphinx-build process. The autodoc error message wasn't helpful just
saying it couldn't import the osgeo module, whereas if trying to import
in conf.py I got an exception pointing to the fact a symbol from
libjpeg.so couldn't be loaded. I suspect that something in the
sphinx-build process loads the system libjpeg library, that does not
have the new symbols of libjpeg-turbo 3.0 that my libgdal.so was using,
hence causing later loading of libgdal to fail.

Now that I've added /path/to/install-libjpeg-turbo/lib in
LD_LIBRARY_PATH, things work and this patch isn't actually needed, but
it may help diagnose further similar situations
@rouault rouault force-pushed the doc_building_python_bindings branch from 4aa76b4 to be51102 Compare April 30, 2024 14:17
@rouault
Copy link
Member Author

rouault commented Apr 30, 2024

@dbaston Thanks for the suggestion. I've slightly updated it to work with Python < 3.10, and also support gdal.__version__ with a "dev" suffix

@rouault rouault closed this Apr 30, 2024
@rouault rouault reopened this Apr 30, 2024
@rouault rouault merged commit c2cb6ce into OSGeo:master Apr 30, 2024
3 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants