BUG: UI is accessible, active modules work, but marketplace is empty and config files are not loading

[KKdev15]

Describe the Bug

Is this an issue with the CLI or GUI?
GUI

Is this something that should be put in the Wiki?
NO

A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Open MMPM GUI

Expected behavior
I expect to see modules that are available for install in Marketplace and the config files in the Config Editor

Screenshots

Operating System, Python3 Version, and Web Browser
(Please complete the following information)

• output from cat /etc/os-release

PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

• output from python3 --version
  Python 3.9.2

• Web Browser (firefox, chrome, safari, chromium, etc)
  Chrome

MMPM Environment Settings

• output from mmpm env

{
  "MMPM_MAGICMIRROR_ROOT": "/home/pi/MagicMirror",
  "MMPM_MAGICMIRROR_URI": "http://192.168.123.102:8080",
  "MMPM_MAGICMIRROR_PM2_PROCESS_NAME": "mm",
  "MMPM_MAGICMIRROR_DOCKER_COMPOSE_FILE": "",
  "MMPM_IS_DOCKER_IMAGE": false,
  "MMPM_LOG_LEVEL": "INFO"
}

MMPM Log Files

• Attach the file produced Download MMPM log files within the Control Center of the GUI or from the mmpm log --zip
  mmpm-logs-2024-1-14.zip

MagicMirror Config

• your MagicMirror config.js

 *
 * By Michael Teeuw https://michaelteeuw.nl
 * MIT Licensed.
 *
 * For more information on how you can configure this file
 * see https://docs.magicmirror.builders/configuration/introduction.html
 * and https://docs.magicmirror.builders/modules/configuration.html
 *
 * You can use environment variables using a `config.js.template` file instead of `config.js`
 * which will be converted to `config.js` while starting. For more information
 * see https://docs.magicmirror.builders/configuration/introduction.html#enviromnent-variables
 */
let config = {
	address: "localhost",	// Address to listen on, can be:
							// - "localhost", "127.0.0.1", "::1" to listen on loopback interface
							// - another specific IPv4/6 to listen on a specific interface
							// - "0.0.0.0", "::" to listen on any interface
							// Default, when address config is left out or empty, is "localhost"
	port: 8080,
	basePath: "/",	// The URL path where MagicMirror² is hosted. If you are using a Reverse proxy
									// you must set the sub path here. basePath must end with a /
	ipWhitelist: [],	// Set [] to allow all IP addresses
									// or add a specific IPv4 of 192.168.1.5 :
									// ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.1.5"],
									// or IPv4 range of 192.168.3.0 --> 192.168.3.15 use CIDR format :
									// ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.3.0/28"],

	useHttps: false,			// Support HTTPS or not, default "false" will use HTTP
	httpsPrivateKey: "",	// HTTPS private key path, only require when useHttps is true
	httpsCertificate: "",	// HTTPS Certificate path, only require when useHttps is true

	language: "de",
	locale: "de-DE",
	logLevel: ["INFO", "LOG", "WARN", "ERROR"], // Add "DEBUG" for even more logging
	timeFormat: 24,
	units: "metric",

	modules: [
/* some modules... */
{
    module: "MMM-mmpm"
},
	]
};

/*************** DO NOT EDIT THE LINE BELOW ***************/
if (typeof module !== "undefined") { module.exports = config; }

Additional Information
From the error shown in the Screenshots, I assumed, that the error war part of the mmpm.api process started by pm2. Therefore I checked the logs and found the following error repeating constantly:

[2024-01-14 18:47:27 +0100] [2439] [ERROR] Worker (pid:2440) exited with code 3
[2024-01-14 18:47:27 +0100] [2439] [ERROR] Shutting down: Master
[2024-01-14 18:47:27 +0100] [2439] [ERROR] Reason: Worker failed to boot.
[2024-01-14 18:47:28 +0100] [2441] [INFO] Starting gunicorn 21.2.0
[2024-01-14 18:47:28 +0100] [2441] [INFO] Listening at: http://0.0.0.0:7891 (2441)
[2024-01-14 18:47:28 +0100] [2441] [INFO] Using worker: gevent
[2024-01-14 18:47:28 +0100] [2442] [INFO] Booting worker with pid: 2442
[2024-01-14 18:47:32 +0100] [2442] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 147, in init_process
    super().init_process()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/pi/.local/lib/python3.9/site-packages/mmpm/wsgi.py", line 28, in <module>
    from mmpm.api.entrypoint import app
  File "/home/pi/.local/lib/python3.9/site-packages/mmpm/api/entrypoint.py", line 8, in <module>
    from flask import Flask, Response
  File "/home/pi/.local/lib/python3.9/site-packages/flask/__init__.py", line 14, in <module>
    from jinja2 import escape
ImportError: cannot import name 'escape' from 'jinja2' (/home/pi/.local/lib/python3.9/site-packages/jinja2/__init__.py)

From this Stackoverflow issue, I figured out, that maybe the used version of Flask and jinja don't match: https://stackoverflow.com/questions/71718167/importerror-cannot-import-name-escape-from-jinja2

Is this a general dependency problem in mmpm or is this just on my Raspi? If it is just on my Raspi, which version of Flask and jinja do I need and how do I install them into the correct place? Are Flask and jinja installed globally?

Thanks!

EDIT
These are the moduled installed on my Raspi (Flask = 1.1.2 and Jinja2 = 3.1.3):

pi@smartmirror:~ $ python3 -m pip list
Package            Version
------------------ ---------
arandr             0.1.10
argcomplete        3.2.1
astroid            2.5.1
asttokens          2.0.4
automationhat      0.2.0
beautifulsoup4     4.12.2
bidict             0.22.1
blinker            1.7.0
blinkt             0.1.2
buttonshim         0.0.2
Cap1xxx            0.1.3
certifi            2020.6.20
chardet            4.0.0
charset-normalizer 3.3.2
click              8.1.7
colorama           0.4.4
colorzero          1.1
cryptography       3.3.2
cupshelpers        1.0
dbus-python        1.2.16
distro             1.5.0
docutils           0.16
drumhat            0.1.0
envirophat         1.0.0
ExplorerHAT        0.4.2
Flask              1.1.2
Flask-Cors         4.0.0
Flask-SocketIO     5.3.6
fourletterphat     0.1.0
gevent             23.9.1
gevent-websocket   0.10.1
gitdb              4.0.11
GitPython          3.1.41
gpiozero           1.6.2
greenlet           3.0.3
gunicorn           21.2.0
h11                0.14.0
html5lib           1.1
idna               2.10
importlib-metadata 7.0.1
isort              5.6.4
itsdangerous       2.1.2
jedi               0.18.0
Jinja2             3.1.3
lazy-object-proxy  0.0.0
logilab-common     1.8.1
lxml               4.6.3
MarkupSafe         2.1.3
mccabe             0.6.1
microdotphat       0.2.1
mmpm               4.1.0
mote               0.0.4
motephat           0.0.3
mypy               0.812
mypy-extensions    0.4.3
numpy              1.19.5
oauthlib           3.1.0
packaging          23.2
pantilthat         0.0.7
parso              0.8.1
pexpect            4.8.0
pgzero             1.2
phatbeat           0.1.1
pianohat           0.1.0
picamera           1.13
picamera2          0.3.12
pidng              4.0.9
piexif             1.1.3
piglow             1.2.5
pigpio             1.78
Pillow             8.1.2
pip                23.3.2
prompt-toolkit     3.0.43
psutil             5.8.0
pycairo            1.16.2
pycups             2.0.1
pygame             1.9.6
Pygments           2.17.2
PyGObject          3.38.0
pyinotify          0.9.6
PyJWT              1.7.1
pylint             2.7.2
PyOpenGL           3.1.5
pyOpenSSL          20.0.1
PyQt5              5.15.2
PyQt5-sip          12.8.1
pyserial           3.5b0
pysmbc             1.0.23
python-apt         2.2.1
python-engineio    4.8.2
python-prctl       1.7
python-socketio    5.11.0
rainbowhat         0.1.0
reportlab          3.5.59
requests           2.31.0
requests-oauthlib  1.0.0
responses          0.12.1
roman              2.0.0
RPi.GPIO           0.7.0
RTIMULib           7.2.1
scrollphat         0.0.7
scrollphathd       1.2.1
Send2Trash         1.6.0b1
sense-hat          2.6.0
setuptools         52.0.0
simple-websocket   1.0.0
simplejpeg         1.6.4
simplejson         3.17.2
six                1.16.0
skywriter          0.0.7
smmap              5.0.1
sn3218             1.2.7
soupsieve          2.2.1
spidev             3.5
ssh-import-id      5.10
termcolor          2.4.0
thonny             4.0.1
toml               0.10.1
touchphat          0.0.1
twython            3.8.2
typed-ast          1.4.2
typing-extensions  3.7.4.3
unicornhathd       0.0.4
urllib3            1.26.5
v4l2-python3       0.3.2
wcwidth            0.2.13
webencodings       0.5.1
websocket-client   1.7.0
Werkzeug           1.0.1
wheel              0.34.2
wrapt              1.12.1
wsproto            1.2.0
yaspin             3.0.1
zipp               3.17.0
zope.event         5.0
zope.interface     6.1

[KKdev15]

I was able to solve the bug.

Install Jinja==3.1.0 and flask==2.2.0
As reported by pip when installing an old version of Jinja:

mmpm 4.1.0 requires flask>=2.2.0, but you have flask 1.1.2 which is incompatible.
mmpm 4.1.0 requires jinja2>=3.1.0, but you have jinja2 3.0.3 which is incompatible.

Shell commands:

python3 -m pip install Jinja2==3.1.0
python3 -m pip install flask==2.2.0

After this I got another error in the mmpm.api logs:

[2024-01-14 19:53:33 +0100] [1630] [INFO] Worker exiting (pid: 1630)
[2024-01-14 19:53:33 +0100] [1629] [ERROR] Worker (pid:1630) exited with code 3
[2024-01-14 19:53:33 +0100] [1629] [ERROR] Shutting down: Master
[2024-01-14 19:53:33 +0100] [1629] [ERROR] Reason: Worker failed to boot.
[2024-01-14 19:53:34 +0100] [1631] [INFO] Starting gunicorn 21.2.0
[2024-01-14 19:53:34 +0100] [1631] [INFO] Listening at: http://0.0.0.0:7891 (1631)
[2024-01-14 19:53:34 +0100] [1631] [INFO] Using worker: gevent
[2024-01-14 19:53:34 +0100] [1632] [INFO] Booting worker with pid: 1632
[2024-01-14 19:53:38 +0100] [1632] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 147, in init_process
    super().init_process()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/pi/.local/lib/python3.9/site-packages/mmpm/wsgi.py", line 28, in <module>
    from mmpm.api.entrypoint import app
  File "/home/pi/.local/lib/python3.9/site-packages/mmpm/api/entrypoint.py", line 8, in <module>
    from flask import Flask, Response
  File "/home/pi/.local/lib/python3.9/site-packages/flask/__init__.py", line 5, in <module>
    from .app import Flask as Flask
  File "/home/pi/.local/lib/python3.9/site-packages/flask/app.py", line 29, in <module>
    from werkzeug.urls import url_quote
ImportError: cannot import name 'url_quote' from 'werkzeug.urls' (/home/pi/.local/lib/python3.9/site-packages/werkzeug/urls.py)

I could solve this by installing an old version of Werkzeug: https://stackoverflow.com/questions/77213053/why-did-flask-start-failing-with-importerror-cannot-import-name-url-quote-fr
python3 -m pip install Werkzeug==2.2.2

[KKdev15]

Maybe it would make sense to specify the package-dependencies more strictly in the requirements.txt of the mmpm Python module. This way incompatibilities like that should not be a problem.

[Bee-Mar]

@KKdev15 what version of mmpm do you have installed? Are you on the latest 4.1.0?

[Bee-Mar]

@KKdev15 also, I'm unsure of how you have flask version 1.1.2 installed, with MMPM operating considering the dependencies within the pyproject.toml are:

dependencies = [
  "Flask-Cors>=3.0.8",
  "argcomplete>=2.0.0",
  "argparse>=1.4",
  "beautifulsoup4>=4.11.1",
  "colorama>=0.4.3",
  "flask-socketio>=5.3.2",
  "flask>=2.2.0",
  "gevent>=23.9.1",
  "gunicorn>=20.1.0",
  "jinja2>=3.1.0",
  "pygments>=2.12.0",
  "python-socketio[client]>=5.7.0",
  "requests>=2.28.0",
  "yaspin>=2.3.0",
  "pip>=23.3.1",
  "gitpython>=3.1.40",
  "gevent-websocket>=0.10.1",
  "prompt-toolkit>=3.0.43",
  "importlib_resources; python_version<'3.9'",
]

[Bee-Mar]

@KKdev15 what version of mmpm do you have installed? Are you on the latest 4.1.0?

@KKdev15 can you provide the version of mmpm you have installed?

[KKdev15]

@KKdev15 what version of mmpm do you have installed? Are you on the latest 4.1.0?

@KKdev15 can you provide the version of mmpm you have installed?

@Bee-Mar I have installed the latest version (4.1.0) of mmpm.

I guess the problem was, that I also have the plugin MMM-FastNotes installed. It's dependencies (flask == 1.1.2) are conflicting with the dependencies of mmpm.

Anyway MMM-FastNotes seems to work with the mmpm dependencies as well...

A general question from my side: Is it possible to install different versions of the same package to make sure, that each package has exactly the dependencies it requires? I'm not a Node.js expert, but maybe there is something like venvs in Python?

[Bee-Mar]

@KKdev15 ah, well that's not something I expected. Yep, python has virtual environments. You could install mmpm using a virtualenv if you'd like. You'd basically do the following to get it setup:

python3 -m venv venv # creates a virtualenv named 'venv'
source venv/bin/activate
python3 -m pip install mmpm

You'd have run source venv/bin/activate prior to using the mmpm CLI, or you could alias it to make it easier. The installation of the UI will pickup the correct python path even if you use a virtualenv.

Another option is to use pipx https://github.com/pypa/pipx. That handles the virtualenv creation for you and makes an alias so you don't need to do anything special other than run the CLI like you normally would:

python3 -m pip install pipx
pipx install mmpm

Then just run mmpm from terminal.

Something else that could be interesting from my end is detecting if the module being installed has python requirements and prompting the user if they want to setup the module in a virtualenv and having mmpm handle that so things are isolated.

[Bee-Mar]

@KKdev15 one issue with that module is they made the requirements too strict. Requiring version numbers with == hamstrings anyone else installing the same dependency. It's typically a very bad idea to specify dependencies like that unless you're running the application in a container or environment you know won't conflict with anything else.

[Bee-Mar]

@KKdev15 in short, for simplicity sake, I'd recommend the following:

python3 -m pip uninstall mmpm
python3 -m pip install pipx
pipx install mmpm

Then go about things how you normally would. The should resolve the issue.

[Bee-Mar]

@KKdev15 I'm going to close this issue since the problem was related to another MagicMirror module. If you're still having issues feel free to respond in this thread or email me.

[minkshaman]

@Bee-Mar Hi, I've had the same issue after installing FastNotes, and I've followed your code block, but it hasn't helped!

do you have a suggestion?

[minkshaman]

Additional: HTML inspector shows CORS is being blocked.

How do I fix this?