How to use libsass with boussole through a single python process?

[kopax-polyconseil]

Describe the bug

I have written the following function in python that use python-libsass to preprocess sass on runtime in production, and in development, that watch for change using boussole.

• Boussole documentation: https://pypi.org/project/boussole/
• Python libsass: https://pypi.org/project/libsass/

This work well for production, however, with boussole it watch for change, but I keep having my python process crashing with :

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/shutil.py", line 713, in rmtree
ImportError: sys.meta_path is None, Python is likely shutting down

I already took caution not to rerun the process when I have the flask hotreload on *.py edition by clearing the previous subprocess on restart. I checked and I only have one process running all the time.

It work fine when I do not run the boussole subprocess with python.

This is my boussole.yml configuration:

---
SOURCES_PATH: "."
TARGET_PATH: "../css/compiled"
SOURCE_MAP: "true"
EXCLUDES: "../css/compiled"

Environment
Describe your environment:

• Plateform: Debian
• Python version: 3.10
• Boussole version: 2.1.0
• libsass-python version: 0.22.0

To Reproduce

import glob
import os
from pathlib import Path
import signal
from subprocess import DEVNULL
from subprocess import Popen
from subprocess import STDOUT

import sass

from pcapi import settings


def preprocess_scss(watch: bool) -> None:
    source = Path("src/pcapi/static/backofficev3/scss")
    destination = Path("src/pcapi/static/backofficev3/css/compiled")
    configuration = Path("src/pcapi/static/backofficev3/scss/boussole.yml")
    pid_file_path = Path("src/pcapi/static/backofficev3/scss/boussole.pid")

    Path(destination).mkdir(parents=True, exist_ok=True)

    if settings.IS_RUNNING_TESTS is not True:
        if watch:
            if os.environ.get("WERKZEUG_RUN_MAIN") != "true":
                has_never_compiled_css = len(glob.glob(f"{destination}/**/*.css", recursive=True)) == 0
                if has_never_compiled_css:
                    sass.compile(
                        dirname=(source, destination),
                        output_style="compressed",
                        source_map_contents=True,
                        source_map_embed=True,
                        source_map_root=destination,
                    )

                # kill previous boussole process if python previously crashed
                try:
                    if os.path.isfile(pid_file_path):
                        with open(pid_file_path, "r", encoding="utf8") as pid_file:
                            pid = int(pid_file.read())
                            pid_file.close()
                            os.kill(pid, signal.SIGTERM)
                except Exception:  # pylint: disable=broad-except
                    pass

                proc = Popen(  # pylint: disable=consider-using-with
                    ["boussole", "watch", "--config", configuration, "--backend", "yaml"],
                    stdout=DEVNULL,
                    stderr=STDOUT,
                )

                # save new process pid in case of python crash
                with open(pid_file_path, "w", encoding="utf8") as pid_file:
                    pid_file.write(str(proc.pid))
                    pid_file.close()

                print("💅 Scss compiler attached and watching, enjoy styling 💅", flush=True)
        else:
            sass.compile(
                dirname=(source, destination),
                output_style="compressed",
                source_map_contents=True,
                source_map_embed=True,
                source_map_root=destination,
            )
            print("💅 Scss compiler has compiled css 💅", flush=True)

Expected behavior

I expect boussole to watch for change without error

Additional context

NA

[sveetch]

It seems a problem with threads and popen usage. Boussole stands on watchdog for the watch mode and it did have some issues with this kind of usage.

As i know, Boussole don't keep any unclosed files and i never tried to use Boussole in the same way you did.

Maybe you could try to directly use the watchdog script "watchmedo" with "boussole compile ..." to see if it changes anything. If the issue is still the same, it probably comes from watchdog, else it the issue does not appear anymore this should be a bug from Boussole.

As for now i don't have any idea of a solution yet, let me know if you found something new.

[sveetch]

I forget to advise that with the watchmedo + boussole compile, obviously you will have to define yourself the sources to watch, which is not as efficient than the boussole watch that make some smart inspection about source inheritance.

[kopax-polyconseil]

Hi and thanks @sveetch for your fast reply, it is well appreciated !

I am not sure we use watchdog in our project and I am afraid to propose to add new library for this purpose, I already had to negotiate to get libsass and boussole :D

This seems like a common use case for webpack dev server, I was expecting to get the same experience in python development, but if I understand right, your advice is to run boussole along the python process?

This is actually the fix I was thinking of, but I hope there was a way or fix that could avoid that extreme solution.

[sveetch]

I am not sure we use watchdog in our project and I am afraid to propose to add new library for this purpose, I already had to negotiate to get libsass and boussole :D

Watchdog itself is a requirement from Boussole so if you have Boussole installed you already got watchdog. However the solution i was giving use an extra dependency from watchdog (called watchmedo) so it may not fit to your constraints.

Honestly i don't know what is going on with your script, i looked at your exceptions and the ImportError does not really seems to involve a PID problem, but you truncated the stacktrace so i can't trace from where it is happening.

I never used watch mode through a subprocess so i can ensure this is related or not to watchdog, as fas as i know Boussole is not playing with imports, process or threads.

[kopax-polyconseil]

Hello, I just opened a pull request to remove this features, but before mergin, I'd like to know stacktrace I can give you to help your further.

This is how my console look like:

/home/dka/workspace/github.com/pass-culture/pass-culture-main/api/env/bin/python /srv/pycharm-2022.3.3/plugins/python/helpers/pydev/pydevd.py --module --multiprocess --qt-support=auto --client 127.0.0.1 --port 45975 --file pcapi.backoffice_app 
warning: PYDEVD_USE_CYTHON environment variable is set to 'NO'. Frame evaluator will be also disabled because it requires Cython extensions to be enabled in order to operate correctly.
Connected to pydev debugger (build 223.8836.43)
💅 Scss compiler attached and watching, enjoy styling 💅
 * Serving Flask app 'pcapi.flask_app' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
warning: PYDEVD_USE_CYTHON environment variable is set to 'NO'. Frame evaluator will be also disabled because it requires Cython extensions to be enabled in order to operate correctly.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/shutil.py", line 713, in rmtree
ImportError: sys.meta_path is None, Python is likely shutting down
warning: PYDEVD_USE_CYTHON environment variable is set to 'NO'. Frame evaluator will be also disabled because it requires Cython extensions to be enabled in order to operate correctly.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/shutil.py", line 713, in rmtree
ImportError: sys.meta_path is None, Python is likely shutting down
warning: PYDEVD_USE_CYTHON environment variable is set to 'NO'. Frame evaluator will be also disabled because it requires Cython extensions to be enabled in order to operate correctl

How should I give you more verbose logs ?

[sveetch]

I don't know for the verbose logs, commonly a traceback is longer that yours, but it does not seems there is more to see.

Whatever, i've just seen the exception comes from usage of shutil.rmtree but there is not any from Boussole code and i've also searched through watchdog and it does not use it ever.

I think your problem is higher level in your script or else, but searching for rmtree in your code and its requirements could help you to find the real cause.

[kopax-polyconseil]

searching for rmtree in your code and its requirements could help you to find the real cause.

Our repository is public : https://github.com/pass-culture/pass-culture-main

I think your problem is higher level in your script or else

I can only disagree as this problem started when I added subprocess boussole from our python app, and goes away as soon as I remove this subprocess from boussole.

This bug also happen while we just edit html or py code (we reload our flask service on .py edition), it's not just only while editing scss, but it was working fine before, and it work fine if we do not run boussole from py, perhaps this could be a reason.

[kopax-polyconseil]

I'll close the issue as I am in the process to revert this feature. Thanks for helping !