MacOS standalone not working due to pygments error

[dalthviz]

Seems like although the action building and testing the MacOS installer passes there is an error with the generated app. Looking the action log you can see a traceback related with pygments (from action execution: https://github.com/spyder-ide/spyder/runs/8079765883?check_suite_focus=true#step:11:17):

Run ./test_app.sh -t 60 -d 10 ${DISTDIR}
Traceback (most recent call last):
  File "/Users/runner/work/spyder/spyder/dist/Spyder.app/Contents/Resources/__boot__.py", line 251, in <module>
    _run()
  File "/Users/runner/work/spyder/spyder/dist/Spyder.app/Contents/Resources/__boot__.py", line 174, in _run
    exec(compile(source, path, "exec"), globals(), globals())
  File "/Users/runner/work/spyder/spyder/dist/Spyder.app/Contents/Resources/spyder", line 3, in <module>
    start.main()
  File "/Users/runner/work/spyder/spyder/dist/Spyder.app/Contents/Resources/lib/python3.9/spyder/app/start.py", line 128, in main
    from spyder.config.manager import CONF
  File "/Users/runner/work/spyder/spyder/dist/Spyder.app/Contents/Resources/lib/python3.9/spyder/config/manager.py", line 22, in <module>
    from spyder.config.main import CONF_VERSION, DEFAULTS, NAME_MAP
  File "/Users/runner/work/spyder/spyder/dist/Spyder.app/Contents/Resources/lib/python3.9/spyder/config/main.py", line 21, in <module>
    from spyder.config.appearance import APPEARANCE
  File "/Users/runner/work/spyder/spyder/dist/Spyder.app/Contents/Resources/lib/python3.9/spyder/config/appearance.py", line 15, in <module>
    from spyder.plugins.help.utils.sphinxify import CSS_PATH
  File "/Users/runner/work/spyder/spyder/dist/Spyder.app/Contents/Resources/lib/python3.9/spyder/plugins/help/utils/sphinxify.py", line 36, in <module>
    from sphinx.application import Sphinx
  File "/Users/runner/work/spyder/spyder/dist/Spyder.app/Contents/Resources/lib/python3.9/sphinx/application.py", line 32, in <module>
    from sphinx.highlighting import lexer_classes
  File "/Users/runner/work/spyder/spyder/dist/Spyder.app/Contents/Resources/lib/python3.9/sphinx/highlighting.py", line 12, in <module>
    from pygments.formatters import HtmlFormatter, LatexFormatter
  File "pygments/formatters/__init__.pyc", line 132, in __getattr__
  File "pygments/formatters/__init__.pyc", line 28, in _load_formatters
ModuleNotFoundError: No module named 'pygments.formatters.latex'
2022-08-29 21:14:56.679 Spyder[36541:97505] Launch error
2022-08-29 21:14:56.680 Spyder[36541:97505] Launch error
See the py2app website for debugging launch issues

Spyder launched successfully!
Shutting down Spyder...

./test_app.sh: line 72: 36541 Terminated: 15          $1/Spyder.app/Contents/MacOS/Spyder

Spyder shut down successfully in 0s

And when launching the installed app you see an error dialog from py2app (I checked this using MacOS 11.6.8 Big Sur):

Checking seems like pygments released version 2.13.0 a couple of weeks ago and PRs which GitHub actions checks runs are older than two weeks ago seem like where able to properly pass the MacOS installer validation test. Could then the new pygments release be the cause of the issue?

What do you think @mrclary ?

Also, I think someone reported something similar on the google group, right @ccordoba12 ?

[phjo33]

I confirm Spyder standalone don't run on Monterey either, and obviously, you can mark my own bug report #19237 as a duplicate of this one !

[ccordoba12]

Also, I think someone reported something similar on the google group, right @ccordoba12 ?

Correct. This seems caused by a an incompatibility between Sphinx and Pygments. I think for now the simplest thing we can do to solve it is to add an upper constraint to Pygments (2.13 was released last August 15th, so that should be the version that introduced this issue).

@mrclary, could you take care of this?

[dalthviz]

@ccordoba12 should we, for the moment, remove the Spyder.dmg and the link to download it from the Spyder 5.3.3 release page?

[ccordoba12]

Ok, I think that's a good idea given that this is a hard crash, so it'll give a very bad impression to new users.

Please also leave a note in the 5.3.3 release entry saying that the Mac app has a critical error but we'll try to upload it shortly (you can reference this issue if you want too).

Also, you need to check that the 5.3.2 dmg can still be downloaded from our main webpage (i.e. spyder.org).

[ccordoba12]

I think for now the simplest thing we can do to solve it is to add an upper constraint to Pygments

About this error, I think it's in our side because I couldn't find a similar one in the Sphinx repo. So maybe we're not packaging all Pygments lexers somehow?

[mrclary]

@mrclary, could you take care of this?

If there is not a problem with the Windows or conda installs, then I hesitate to restrict the pygments version because it may be that only the macOS installer (py2app) didn't capture pygments.formatters.latex.

[mrclary]

I'll take a look. If that is the issue, I can patch that and upload a new artifact (built locally) for 5.3.3. Then we can decide how best to proceed, either a patch or PR to py2app or both.

[dalthviz]

Just in case, checked on Windows and both standalone and conda version start normally 👍

[mrclary]

So it was definitely py2app not capturing all the components of the package. The quick fix is to force py2app to include the entire package. I'll upload new artifacts shortly.

The long-term fix for this would be to submit a PR for a pygments recipe for py2app. However, given the imminent conda-based installer, I'll forgo that and just submit a PR here for this patch.

[mrclary]

Another concern is why the app test erroneously declares that Spyder launched successfully...

[mrclary]

New artifacts are now uploaded.

[ccordoba12]

Thanks for your prompt help with this @mrclary!

[dalthviz]

Another concern is why the app test erroneously declares that Spyder launched successfully...

I was thiking about this too 🤔. My guess is that since py2app shows a dialog when the application is unable to launch, that was causing the faulty installer launched to properly handle the exit signal and then causing the test script to exit normally. Maybe we should add some validation to the output generated when runnning Spyder to check that no tracebacks are found @mrclary ?

[ccordoba12]

That's an interesting idea @dalthviz. When looking at the last logs of the Mac app on PRs, you can see that the error was shown there but it wasn't flagged as such:

https://github.com/spyder-ide/spyder/runs/8079765883?check_suite_focus=true#step:11:1

[mrclary]

py2app does not show the dialog. That is a system (macOS) dialog. I think the "py2app" reference comes from the info.plist in the Spyder.app bundle. The test only checks whether Spyder is still running. In Spyder, we have a flag dedicated for this test which causes spyder to quit when it raises an exception. In this instance, spyder did not raise the exception, so it did not quit, but instead it was somehow picked up by the os. I'll take a deeper look at it, but @dalthviz's suggestion may be the right thing to do.