Installing diambra CLI system wide cause import error when using Conda env

[discordianfish]

When diambra CLI is installed system wide (i.e. outside a conda env), it is installed in $HOME/.local/lib/python3.8/site-packages

If then a conda env is created to host diambra-arena, running the gist example with diambra run python diambra_arena_gist.py fails as python is not able to resolve the diambra.arena import.

It is probably due to the fact it is looking for the lib in the CLI path and not finding it.

Python 3.8.16 (default, Jan 17 2023, 23:13:24) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
> import sys
> >>> for path in sys.path:
> ...   print(path)
> ... 
> 
> /home/alexpalms/miniconda3/envs/diambra-arena-test/lib/python38.zip
> /home/alexpalms/miniconda3/envs/diambra-arena-test/lib/python3.8
> /home/alexpalms/miniconda3/envs/diambra-arena-test/lib/python3.8/lib-dynload
> /home/alexpalms/.local/lib/python3.8/site-packages
> /home/alexpalms/miniconda3/envs/diambra-arena-test/lib/python3.8/site-packages
> 

installing the CLI inside the virtual env solves the problem and is the current workaround.

Docs updated accordingly, removing suggestion to install diambra system wide, but instead suggesting to install it in a virtual env

[discordianfish]

I can't reproduce this. I've tried using this Dockerfile:

FROM ubuntu:22.04

RUN apt-get -qy update && apt-get -qy install curl libgl1

RUN adduser fish
USER fish
WORKDIR /home/fish

RUN curl -Lsfo anaconda.sh https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh && \
  bash anaconda.sh -b

ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/fish/anaconda3/bin
RUN pip install diambra
RUN diambra
RUN which diambra
RUN conda create --name some-agent
ENTRYPOINT ["conda", "run", "-n", "some-agent", "--no-capture-output"]

• the entrypoint should be equivalent to conda activate some-agent (which doesn't work in a Dockerfile
• diambra is available even though it was installed outside of the some-agent conda env

Now with this built using docker build -t anaconda .:

$ diambra run --agent.image anaconda -p=false bash
fish@e0a71c2c6af3:~$ diambra
...
Use "diambra [command] --help" for more information about a command.
fish@e0a71c2c6af3:~$ which diambra
/home/fish/anaconda3/bin/diambra

-> still in PATH

$ pip install diambra-arena
...
wget https://github.com/diambra/arena/raw/main/examples/diambra_arena_gist.py
python diambra_arena_gist.py

-> works fine.

@alexpalms I think this is an issue with your local anaconda setup. You installed diambra using your system's python but anaconda seems to also look into that system's python paths but it shouldn't.

[alexpalms]

@discordianfish Performed a few tests on my end using your Dockerfile (and other things).

The first thing I noted is that the entry point is not correctly activating the conda environment "some-agent", and in fact the command pip install diambra-arena is installing it in the anaconda3 base environment (more precisely here ~/anaconda3/lib/python3.9/site-packages) and not inside the some-agent one (that is inside ~/anaconda3/envs/) so I could not replicate the exact same behavior I was having locally.

Testing it in docker that has those limitations I believe introduces additional unknowns in terms of behavior (e.g. the conda activate command not being available).

In addition, there are at least two things:

1. The conda base environment auto activate "topic" (https://stackoverflow.com/questions/54429210/how-do-i-prevent-conda-from-activating-the-base-environment-by-default) that can be deactivated at installation time or later, that introduces different combinations and possibilities (for example I deactivated it at installation time, while the installation in docker is the default one that keeps it activated)
2. Other configurations, for example I am using miniconda that is way lighter than anaconda. They should behave the same, but, well, should. Will it be the case? In every condition?

And finally, even if we find out that an option that works in this specific case of mine is suggesting our users to install the CLI in the base virtual env, it might not be the solution as I clearly recall we explicitly added the instruction in the docs saying to users to deactivate the base conda env since we found wrong behavior with windows (it was last summer, I remember it very clearly).

So finally, I think the best way to move forward is to leave the instructions as we have now, meaning to suggest to install the CLI in the same environment in which they install diambra-arena. In the end is a small package, quick to download and install.

In general, I believe that being sure what you are executing lives in the very same confined and isolated environment is way clearer than leveraging packages that are buried somewhere in the lower levels. And I don't see any cons in this choice.

I believe I represent the high end of our potential users in terms of experience with python, conda envs, and similar things. So the less possibilities we give them to be confused (if it happened to me it can happen to them, even if it is a "broken" conda env) the better. Especially because I don't see any drawback in telling them to install all our packages in the same env.

If we go this way, I believe we can close this and forget it as we have more pressing things to work on ;)

[discordianfish]

Here the tl;dr: This issue is about the 'bug' that is installing the cli in the 'wrong' way causing python to not find diambra-arena anymore. What the right suggestion for users is, is out of scope.

You're right that the conda env is somehow not activated. I've updated the Dockerfile and again showed that this is working, so I'm going to close this issue:

FROM ubuntu:22.04

RUN apt-get -qy update && apt-get -qy install curl libgl1

RUN adduser fish
USER fish
WORKDIR /home/fish

RUN curl -Lsfo anaconda.sh https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh && \
  bash anaconda.sh -b

ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/fish/anaconda3/bin
RUN pip install diambra
RUN diambra
RUN which diambra
RUN conda create --name some-agent && \
    conda init && \
    wget https://github.com/diambra/arena/raw/main/examples/diambra_arena_gist.py
ENTRYPOINT [ "/bin/bash", "--login" ]

$ diambra run --agent.image anaconda -p=false
(base) fish@09a95a6e157f:~$ diambra
...
Use "diambra [command] --help" for more information about a command.
$ pip install diambra-arena
Collecting diambra-arena
....
Successfully installed dacite-1.8.0 diambra-arena-2.1.0rc5 diambra-engine-2.1.0rc8 distro-1.8.0 grpcio-1.51.1 gym-0.21.0 inputs-0.5 opencv-python-4.7.0.68 protobuf-3.20.3 screeninfo-0.8.1 tk-0.1.0
$ python ./diambra_arena_gist.py
...
🏟️ (b00d) (0)Round won
(0)Moving to next round