envt build conflicts

[lwasser]

Right now our docker image is failing because of conflicts. it seems like this is likely due to pyton 3.9 being the default version in the newer version of miniconda.

moving from - conda-forge/cartopy-feedstock#93

I have tried the following to address it

1. Just run with python 3.9 doesn't work - lots of conflicts.

I tried to create an envt file to make install simpler and it's failing at the cleanup state - saying find: paths must precede expression: RUN'`

FROM continuumio/miniconda3:4.7.12

ENV PYTHONDONTWRITEBYTECODE=true

# Only for the time being before we migrate to 3.9
#RUN conda install -y python=3.8

COPY environment.yml environment.yml

RUN conda update conda --yes \
    && conda config --add channels conda-forge \
    && conda config --set channel_priority strict \
    && conda env create -f environment.yml \
    && conda clean --all --yes --force-pkgs-dirs \
    && find /opt/conda/ -follow -type f -name '*.a' -delete \
    && find /opt/conda/ -follow -type f -name '*.pyc' -delete \
    && find /opt/conda/ -follow -type f -name '*.js.map' -delete \

RUN conda init bash \
    conda activate ea-lessons \
    && conda list


COPY import_check.py import_check.py
RUN python import_check.py

i also tried

    && conda create -n ea-lessons \
    python=3.8 \
    packages here

and it didn't like that syntax.

i'm happy to share more code / docker file examples with you but i don't want to consume your time. i just keep getting conflicts or errors.
The main issue i'm having is you have to initialize bash to activate an envt and it doesn't seem to be working for me. that is why we did use base to begin with to keep it simpler. I also did try installing python 8 at the top and ran into many conflicts.

@xylar do you have any suggestions for how to best force this envt to python 3.8 while keeping the docker layers to a minimum and avoiding conflicts?
THank you !!

[xylar]

@lwasser, I would be happy to see code. I'm not very experienced with docker and it's not clear to me how specific your difficulties are to docker itself. I think not very much.

I can help with the conda activate part but let's see if we can get things working in the base environment first.

What happens if you start with the docker image as it exists on master just install python=3.8 (removing all these packages: https://github.com/earthlab/r-python-eds-lessons-env/blob/master/Dockerfile#L10-L93)? Does that work or do you see conflicts?

[xylar]

My suggested change (which maybe you tried already) is:

...
RUN conda update conda --yes \
    && conda config --add channels conda-forge \
    && conda config --set channel_priority strict \
    && conda install --yes python=3.8 \
    && conda install --yes \
    r-base>=4.0 \
    autopep8 \
    cartopy \
...
    tweepy \
    xarray \
    && conda clean --all --yes --force-pkgs-dirs \
    && find /opt/conda/ -follow -type f -name '*.a' -delete \
    && find /opt/conda/ -follow -type f -name '*.pyc' -delete \
    && find /opt/conda/ -follow -type f -name '*.js.map' -delete \
    && conda list

That is, install python 3.8 on its own first, then all the rest.

[xylar]

I would be happy to make a fork and do a pull request to play around with this. But I don't know if I will be able to see your docker builds. I can's wee the result from the PR here: https://hub.docker.com/login/?next=/redirect/%3Fresource_uri%3D/api/audit/v1/action/708a951a-daa6-4824-b4a9-bc5197e94036/
Is that private to you and your collaborators?

[xylar]

As I'm sure you saw, I tried it out: #41. Looks like no luck but I can't see the docker output.

I'd love to have a way to debug this but I've had some bad side effects in the past from installing Docker on my laptop so I'm not sure if that's a good plan. And I can install all the contents of the package on my local linux miniconda without any conflicts. I'm going to see if I can update to python 3.9 and then see the issue...

[xylar]

Regarding the option to activate a new virtual environment, here is what has worked for me in y CI:

RUN eval "$(conda shell.bash hook)" \
    && conda activate ea-lessons \
    && conda list

Could you see if that works for you as well?

[xylar]

Weird, I really can't reproduce these conflicts locally no matter what I do. I installed Miniconda3, updated conda and python both to the latest (so python 3.9) and then ran your install command from master without a problem.

I'm kind of at a loss to help further.

[lwasser]

ok @xylar @ocefpaf here is what is interesting

forget all of the packages.
this builds for me just find using docker build -- it SEEMS that something happens with setting strict channel priority and trying to update python at a bare minimum.

FROM continuumio/miniconda3:4.7.12

ENV PYTHONDONTWRITEBYTECODE=true

RUN conda update conda --yes \
    && conda install --yes python=3.8

this has conflicts:

FROM continuumio/miniconda3:4.7.12

ENV PYTHONDONTWRITEBYTECODE=true

RUN conda update conda --yes \
    && conda config --add channels conda-forge \
    && conda config --set channel_priority strict \
    && conda install --yes python=3.8

this also works

FROM continuumio/miniconda3:4.7.12

ENV PYTHONDONTWRITEBYTECODE=true

RUN conda update conda --yes \
    && conda config --add channels conda-forge \
#    && conda config --set channel_priority strict \ # commented out
    && conda install --yes python=3.8 \
    && conda install --yes \
    r-base=4.0 \
    autopep8 \
    cartopy \

but not this - this has a lot of conflicts : (strict channel priority)

FROM continuumio/miniconda3:4.7.12

ENV PYTHONDONTWRITEBYTECODE=true

RUN conda update conda --yes \
    && conda config --add channels conda-forge \
    && conda config --set channel_priority strict \
    && conda install --yes python=3.8 \
    && conda install --yes \
    r-base=4.0 \
    autopep8 \
    cartopy \

So what is failing is installing python 3.8. When i build the image wtihout installing python 3.7.4 is the default python
I hate to ask this but is it possible that there is something wrong with python 3.8 when installed on top of the miniconda docker container ONLY using conda-forge? Again it runs fine when i don't set conda as the primary channel.

I'm going to try to install it again in a separate line prior to setting conda-forge as the default channel.

[lwasser]

thank you @xylar i really appreciate your looking at this. there's something really odd going on with strict channel priority and installing python 3.8. because i can get the same issue with NO other packages is what i'm seeing now. so you're idea about the python version is definitely spot on.

[ocefpaf]

@lwasser sorry for taking so long to get to this. I think the problem may be the miniconda on the base. Let me try a few things. BTW, since I last touched this Dockerfile a lot as happened. We now have conda-lock to "lock" the dependencies and ensure higher reproducibility of docker images. Another thing that is new is miniforge, conda-forge own "miniconda." Those two should help avoid issues like this on in the future. Hold on... I'm play with those ideas today and I should have a PR soon.

[lwasser]

@ocefpaf gosh do not apologize! i was just happy to get the image working and was thankful for @xylar helping with that as i was stumped.

I look forward to seeing how you change things in our dockerfile. i did play with different miniconda installs and i wonder if miniforge will be the way to go for us! thank you again!