Processing algorithms silently fail to load when psycopg2 is missing

[timlinux]

So this thread on the mailing list raised what I think is the same issue I encountered some time ago when setting up a new computer. All algorithms except native QGIS ones were silently failing to load. I was able to do the following to understand what was going on:

Python Console
Use iface to access QGIS API interface or Type help(iface) for more info
Security warning: typing commands from an untrusted source can harm your computer

import os
import traceback
from qgis.PyQt.QtCore import Qt, QCoreApplication
from qgis.PyQt.QtWidgets import QApplication
from qgis.PyQt.QtGui import QCursor
from qgis.utils import iface
from qgis.core import (QgsMessageLog,
                       QgsApplication,
                       QgsMapLayer,
                       QgsProcessingProvider,
                       QgsProcessingAlgorithm,
                       QgsProcessingException,
                       QgsProcessingParameterDefinition,
                       QgsProcessingOutputVectorLayer,
                       QgsProcessingOutputRasterLayer,
                       QgsProcessingOutputMapLayer,
                       QgsProcessingOutputMultipleLayers,
                       QgsProcessingFeedback,
                       QgsRuntimeProfiler)
import processing
from processing.core.ProcessingConfig import ProcessingConfig
from processing.gui.MessageBarProgress import MessageBarProgress
from processing.gui.RenderingStyles import RenderingStyles
from processing.gui.Postprocessing import handleAlgorithmResults
from processing.gui.AlgorithmExecutor import execute
from processing.script import ScriptUtils
from processing.tools import dataobjects
with QgsRuntimeProfiler.profile('Import GDAL Provider'):
    from processing.algs.gdal.GdalAlgorithmProvider import GdalAlgorithmProvider  # NOQA
QgsApplication.processingRegistry().providers()
[<qgis._core.QgsProcessingProvider object at 0x7fcce901d700>, <qgis._core.QgsProcessingProvider object at 0x7fcce901d8b0>, <qgis2web.qgis2webProvider.qgis2webProvider object at 0x7fcceb368c10>, <QuickOSM.quick_osm_processing.provider.Provider object at 0x7fcceb400940>]
from processing.algs.gdal.GdalAlgorithmProvider import GdalAlgorithmProvider
Traceback (most recent call last):
  File "/usr/lib64/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/home/timlinux/dev/cpp/QGIS-Debug-Build/output/python/qgis/utils.py", line 792, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "/home/timlinux/dev/cpp/QGIS-Debug-Build/output/python/plugins/processing/algs/gdal/GdalAlgorithmProvider.py", line 33, in <module>
    from .GdalUtils import GdalUtils
  File "/home/timlinux/dev/cpp/QGIS-Debug-Build/output/python/qgis/utils.py", line 792, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "/home/timlinux/dev/cpp/QGIS-Debug-Build/output/python/plugins/processing/algs/gdal/GdalUtils.py", line 30, in <module>
    import psycopg2
  File "/home/timlinux/dev/cpp/QGIS-Debug-Build/output/python/qgis/utils.py", line 792, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'psycopg2'

I think these 'with' profiling clauses in Processing.py:

with QgsRuntimeProfiler.profile('Import QGIS Provider'):
    from processing.algs.qgis.QgisAlgorithmProvider import QgisAlgorithmProvider  # NOQA

with QgsRuntimeProfiler.profile('Import GRASS Provider'):
    from processing.algs.grass7.Grass7AlgorithmProvider import Grass7AlgorithmProvider

with QgsRuntimeProfiler.profile('Import GDAL Provider'):
    from processing.algs.gdal.GdalAlgorithmProvider import GdalAlgorithmProvider  # NOQA

with QgsRuntimeProfiler.profile('Import OTB Provider'):
    from processing.algs.otb.OtbAlgorithmProvider import OtbAlgorithmProvider  # NOQA

with QgsRuntimeProfiler.profile('Import SAGA Provider'):
    from processing.algs.saga.SagaAlgorithmProvider import SagaAlgorithmProvider  # NOQA

with QgsRuntimeProfiler.profile('Import Script Provider'):
    from processing.script.ScriptAlgorithmProvider import ScriptAlgorithmProvider  # NOQA

Hide the python exception that gets raised if a dependency is missing....

---

Installing python-psycopg2 fixed the problem. Perhaps we can trap this error some how and surface it since it blocks lags from loading that do not even need psycopg2.

CC @nyalldawson to capture our off list thread somewhere more accessible

[nyalldawson]

@timlinux too slow #41985 😆