Could not load plugin "ModuleNotFoundError: No module named 'pandas'"

[dbrakenhoff]

I couldn't load the IMOD qgis plugin because pandas was not installed. Pandas is imported in /imodqgis/widgets/unique_color_widget.py. It is used to drop NaNs and get the uniques from self.data. Replacing this code with its numpy equivalent np.unique(self.data[~np.isnan(self.data)]) and removing the pandas import worked for me.

I'm on QGIS 3.24.2, Ubuntu 20.04.

[dbrakenhoff]

This is an Ubuntu thing. QGIS uses the system Python and does not install its own Python version like it does on Windows. I can probably install pandas in my system python and that would also solve the problem. Still seems like a fairly unnecessary dependency though, so removing it is also a solution.

[Huite]

Thanks for reporting this!

QGIS uses the system Python and does not install its own Python version like it does on Windows.

That's good to know, we've really only tested on Windows so far. I don't have a Linux machine available at the moment, though I suppose Windows Subsystem for Linux (WSL) should work okay.

We use pandas primarily to do some reading of IPF files -- we haven't found a format that's really much more suitable for time series or boreholes. A different form would likely be more like a basic CSV (or TSV), but then pandas still provides a lot of convenience. We could turn this into an optional dependency, by using the numpy unique instead as you suggest, and only raising an ImportError for the read functionality. The pandas unique is also a little more robust however in general, I believe, e.g.:

import numpy as np
import pandas as pd

a = np.arange(10.0)
a[:5] = np.nan

np.unique(a) 
# array([ 5.,  6.,  7.,  8.,  9., nan, nan, nan, nan, nan])

pd.unique(a)
# array([nan,  5.,  6.,  7.,  8.,  9.])

A different thought though: the reason that we felt comfortable including pandas is because on Windows, it's included with the Python installation that comes with QGIS. Numpy has been included for as long as I can remember.

It seems likely to me that numpy, just like pandas, is not included in the system Python by default.
How do the other Python modules required by QGIS (e.g. gdal bindings?) end up in the system Python?

[dbrakenhoff]

We use pandas primarily to do some reading of IPF files

Ah, that makes sense. I only looked at what was preventing the plugin from loading in QGIS, which was this one pandas import.

It seems likely to me that numpy, just like pandas, is not included in the system Python by default.
How do the other Python modules required by QGIS (e.g. gdal bindings?) end up in the system Python?

I can't be sure if my system Python is still clean, so I don't know if numpy was put there by something else, or already included. But in my case, numpy was installed. As for GDAL, my guess would be QGIS installed them there? But also not sure about that...

I think the best solution is just to include a note somewhere for Linux users that the plugin requires certain packages, that might not be installed on the system python. Installing packages, e.g. pandas, can be done with python3.8 -m pip install pandas. Where the python version has to match the one being used by QGIS of course.

[JoerivanEngelen]

Thanks for reporting this,

The tests run on a docker image containing QGIS which includes pandas, so I assumed that every Linux version of QGIS included it. I agree that we need to add a note on this in the documentation. Probably in the the "Known Issues" section of the documentation: https://deltares.github.io/iMOD-Documentation/viewer_known_issues.html

[visr]

It seems like QGIS no longer comes with pandas by default on Windows.

I just did a fresh install of OSGeo4W QGIS 3.30, and hit this same error, which was fixed after adding the python3-pandas package in the OSGeo4W installer. I got the same thing on the standalone installer of QGIS 3.30.

[JoerivanEngelen]

Ah that's very unfortunate for us. I experienced that the OSGeo4W installer of 3.18 did not install pandas by default, but that the standalone installer now also does not ship with it is very unfortunate.

[JoerivanEngelen]

FYI: I sent an e-mail to the qgis developer mailing list, so let's see if somebody answers.

[JoerivanEngelen]

@visr, just checked with the RC installer of yesterday (QGIS-OSGeo4W-3.30.0-2.msi, downloaded from here), pandas is in there (as well as geopandas! :-O). Imports fine within QGIS. Are you sure you are not dealing with some other python installation in your PATH causing a mess?

[visr]

Oh that is reassuring! I wonder what happened, I don't have python in my PATH. Maybe the earlier build had an issue? With the OSGeo4W network installer I did an advanced install and only added the qgis package. It could be that pandas does come along with an express install. Sorry for the noise!