Cannot Run Sarracenia due to Missing Magic

[gc-nrcan-michael]

Hi Peter,

I saw there were some updates related to Magic:

#698

When I run Sarracenia, it gives me an error that it cannot load Magic/libmagic:

I looked through the code and I see the loader attempts to search for the appropriate library.

What is the best way to get this module working again?

[petersilva]

simplest is to install the dependency. A couple of methods:

on ubuntu:

sudo apt install python3-magic

in a (non-ubuntu, but still normal) python environment:

pip install python-magic

(might be pip3, depending on installation.)

but you're on windows... hmm... this brings up the whole python environment morass... Are you using Anaconda? how did you set this up?

[gc-nrcan-michael]

Neither pip install python-magic or pip3 install python-magic work.

This is being set up using Python 3.10 in a venv.

[petersilva]

sanity checking: you need to be in the venv when doing the pip...

I guess you ran activate before pip?

https://stackoverflow.com/questions/21240653/how-to-install-a-package-inside-virtualenv

[gc-nrcan-michael]

Yes, I created a fresh venv for this.

activated the venv

ran the install commands and got the above error.

The same error was produced by the Docker instance.

The same commands did not produce this error, or any error, in the past.

[gc-nrcan-michael]

To clarify, the Docker instance had a different missing module error, it's Ubuntu.

Maybe running the above install commands in the GitHub Action will allow Sarracenia to work in the instance.

I will try this tomorrow.

This still doesn't cover the windows case, which would be nice to have, but not necessary if it works in Docker.

[gc-nrcan-michael]

After testing the Docker instance with either, or both installation commands, Sarracenia is still producing the libmagic module not found error.

[gc-nrcan-michael]

In addition, even if this were to work, it is probably still important to have Sarracenia working on Windows machines too since many developers are using them.

[petersilva]

ooh... It isn't complaining about the python module, but the underlying C library that the python library calls. I thought that would have been in the base OS... ok... let me see:

https://pypi.org/project/python-magic/

on windows it says you need to "pip install python-magic-bin"

on ubuntu linux it needs:

sudo apt install libmagic1

I'm puzzled that a linux environment doesn't have that built-in, I would have expected it to be there by default. I'm doubly confused that installing the python package doesn't pull in the binary... you would think it would be a dependency.
anyways the two things above should help.

We're interested in your thinking about windows support. I've been working on keeping windows compatibility for the package, and every once in while we check, as >99% of usage is on linux, and several devs want to just provide a container (which is just a slimmed down linux) for people to use on windows, and not bother with a native binary.

It would probably be easier to validate that... what do you think?

[gc-nrcan-michael]

I will try those adding those particular commands on Monday and see if the instances work.

And I don't know where your stats are from although people from NRCan-RNCan use Windows.

This should work on Windows.
It worked on Windows before.
It now doesn't work on Windows.
It would be better if it worked on Windows.

[gc-nrcan-michael]

Also, how would the container work in terms of processing messages?

Sarracenia moth is a Python library with functions that are distributed throughout the code where it's used.

This would require significant reworking of existing code for an equivalent Sarracenia moth integration.

[petersilva]

for container: one would define a volume the container would look for plugins and configs, and the container would run using the configurable containers... there would be no change in the code.

It's good to hear you were successfully using it on windows... we will put more effort on testing there.

[gc-nrcan-michael]

I don't understand, it sounds unnecessarily complicated.

Why can't it just be a module you download and use?

[gc-nrcan-michael]

And when are you going to reintroduce Windows and Ubuntu support?

[gc-nrcan-michael]

To be clear:

installing:
metpx-sr3==3.00.40
works

installing:
metpx-sr3==3.00.41
does not work

[petersilva]

On your python install, did you try: "pip install python-magic-bin" ? does that work to get a working magic library?

[gc-nrcan-michael]

I tried that and it appears to be working now.

Thank you for looking into this.

[gc-nrcan-michael]

actually wait, one second

[gc-nrcan-michael]

It works on Windows, although when I try to install it on Ubuntu, the install commands don't work!

[gc-nrcan-michael]

pip install python-magic
or
pip install python-magic-bin

do not work in the Ubuntu Dockerfile

[petersilva]

Are you using a venv on ubuntu or just system packages?

sudo apt install libmagic1
sudo apt install python3-magic

[gc-nrcan-michael]

I am not using venv inside the Docker instance, there is no purpose or need.

I tried those commands, they did not work.

[petersilva]

a little more information please, do you have an error message? or is the dockerfile something we can see?

[gc-nrcan-michael]

This Dockerfile works with metpx-sr3==3.00.40, not metpx-sr3==3.00.41.

FROM ubuntu:latest


RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    libssl-dev \
    libffi-dev \
    python3-dev \
    python3-pip \
    python3-setuptools \
    python3-wheel \
    && rm -rf /var/lib/apt/lists/*


WORKDIR /app

COPY ./app .

RUN pip3 install -r requirements.txt

RUN pip install metpx-sr3
RUN pip3 install metpx-sr3[amqp,mqtt,vip]
RUN pip3 install metpx-sr3


EXPOSE 5671
EXPOSE 5672
EXPOSE 15671
EXPOSE 15672


CMD [ "python3", "./__main__.py" ]

[petersilva]

so... in the list of packages installed via apt-get install you should be able to add the two others: python3-magic, and libmagic1 ....

If you want to do it with a separate line... then:

# after the apt update...
RUN apt install python3-magic libmagic1

an error message might be helpful.

[petersilva]

fwiw, the reason why it's broken is because I thought libmagic1 was included with every ubuntu... no need to explicitly include it. it's still weird not to be there, now that I know it's missing sometimes, I will add code to gracefully degrade...

[gc-nrcan-michael]

Just saw your message,

I tried adding the command and it produced an error:

docker.errors.BuildError: The command '/bin/sh -c apt install python3-magic libmagic1' returned a non-zero code: 100

venv\lib\site-packages\docker\models\images.py:306: BuildError

also found in the logs:

last_event = {'stream': '\x1b[91mE: Unable to locate package python3-magic\nE: Unable 
to locate package libmagic1\n\x1b[0m'}
image_id = None, result_stream = <itertools._tee object at 0x000001AF1AEF3B00>        
internal_stream = <itertools._tee object at 0x000001AF1AEF1380>
chunk = {'error': "The command '/bin/sh -c apt install python3-magic libmagic1' returned a non-zero code: 100", 'errorDetail': {'code': 100, 'message': "The command '/bin/sh -c apt install python3-magic libmagic1' returned a non-zero code: 100"}}
match = None

[petersilva]

to the docker file, try adding a line:

RUN pip3 install python-magic

[gc-nrcan-michael]

The Dockerfile worked with this command, although on run still the same missing module error:

Traceback (most recent call last):
  File "/app/./__main__.py", line 7, in <module>
    import sarracenia.moth
  File "/usr/local/lib/python3.10/dist-packages/sarracenia/__init__.py", line 34, in <module>
    import magic
  File "/usr/local/lib/python3.10/dist-packages/magic/__init__.py", line 209, in <module>
    libmagic = loader.load_lib()
  File "/usr/local/lib/python3.10/dist-packages/magic/loader.py", line 49, in load_lib
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation

[petersilva]

okaay....

above the other RUN... add this:

RUN apt install libmagic1

[gc-nrcan-michael]

Produces the same error:

last_event = {'stream': '\x1b[91mE: Unable to locate package libmagic1\n\x1b[0m'}, image_id = None
result_stream = <itertools._tee object at 0x000002178AFBE600>
internal_stream = <itertools._tee object at 0x000002178AFBE180>
chunk = {'error': "The command '/bin/sh -c apt install libmagic1' returned a non-zero code: 100", 'errorDetail': {'code': 100, 'message': "The command '/bin/sh -c apt install libmagic1' returned a non-zero code: 100"}}    
match = None

[petersilva]

perhaps apt-get in place of apt ... again... should always be there, but perhaps stripped out in the name of minimalism. it's like the repo isn't configured, but when I look it's in main... so no ... weird.

[gc-nrcan-michael]

Okay got the message, I'll run the script tomorrow.

[gc-nrcan-michael]

Using apt-get instead produces the same error.

[petersilva]

I think the issue is that you are not rebuilding your docker image from scratch. I just built my own, with a copy/paste of your Dockerfile from a few posts up, inserted python3-magic into the main apt-get list... and it worked fine.

I think you need some judicious use of docker rm and docker rmi, followed by docker build. (if you are using vanilla docker, ymmv for other environments.)

[gc-nrcan-michael]

Your assumption is incorrect, I am building with rm and nocache.

Adding python3-magic into the main apt-get list is different than your prior suggestions although does work.

We now figured out how to run Sarracenia metpx-sr3==3.00.41 in Ubuntu, it just requires an extra setup step.

[petersilva]

ok the work for #738 makes dependencies optional and obvious. So users should not have this problem. in future:

• one should be able to install without "python-magic" and it will still mostly work (degrading gracefully) nothing will fail if a given module (any module) is missing.
• to find out what features are degraded/missing, use the sr3 features command.
• if the missing modules are subsequently installed, that will be detected on restart and the feature will be enabled automatically.

There were also more tests done on the self-extracting executable on windows, as well as an editable installation using a standard python installation and pip install -e.
Found and fixed a bug there (#748), and some missing dependencies, and some added notes, and it seems to work fine when installed from a self-extracting exe now.

All of the above released in v3.00.43