Add Dockerfile

[yoshitomo-matsubara]

This PR will resolve #56 and #57

[yoshitomo-matsubara]

@lacava Can you try to build a docker image on your machine, following an instruction I added to README?

[yoshitomo-matsubara]

Hi @lacava
Did you have a chance to confirm it works on your end as well? If so, could you merge this PR?

Thank you

[lacava]

sorry for the delay and thanks for the reminder. I have to better understand how to integrate the docker container with our continuous integration setup before merging it. I'll try to do that by end of next week.

[yoshitomo-matsubara]

Hi @lacava

Any update about the integration to merge this PR? I created this PR as you requested Dockerfile in #56 and #57, but didn't see any update for almost 3 months

[lacava]

no updates but I haven't forgotten, just wrapped up. thanks for your patience

[MilesCranmer]

@yoshitomo-matsubara - what do you think about replacing the lines:

WORKDIR /opt/app/
RUN git clone https://github.com/EpistasisLab/srbench.git
WORKDIR /opt/app/srbench/

with this instead?

WORKDIR /opt/app/srbench/
COPY . .

This way it will just copy the current version of the repository (assuming you build the image inside the root directory) and any uncommitted files rather than checking out the master branch. So this would let you use the docker image for testing and CI.

[MilesCranmer]

One other suggestion: use micromamba instead of conda (mamba is a C++ rewrite of conda - it uses the same package servers but is much faster). The docker image mambaorg/micromamba which has micromamba built-in seems to work if all conda calls are replaced with micromamba. It's much faster for me!

[yoshitomo-matsubara]

@MilesCranmer

I pushed a new commit with micromamba.
Could you confirm that it meets what you requested?

[MilesCranmer]

Need to include:

USER root

here (to get micromamba to work).

[MilesCranmer]

python3.7-dev is unavailable on this image. I think micromamba should have most things already, but if not, the following packages may do it:

    zlib1g-dev \
    libedit-dev \
    libncurses5-dev \
    libssl-dev \
    bzip2 \
    libbz2-dev \
    libreadline-dev \

[MilesCranmer]

Should be apt update and apt install. Also, the --fix-missing doesn't seem to work with this image so I would delete it.

[MilesCranmer]

I don't have a GPU so I can't verify these steps work, so just double check on your end!

[MilesCranmer]

Should be deleted as the srbench env will have its own version of pip which installs in the correct location.

[MilesCranmer]

configure.sh uses conda instead of micromamba so this will not work. Maybe paste the commands of configure.sh in the dockerfile, and swap with micromamba?

Alternatively, you could set up micromamba as an alias to conda so all other commands will work? (e.g., in the various install scripts).

[yoshitomo-matsubara]

@MilesCranmer
Thank you for the comments.

I think all the new points are specifically for micromamba and I am not a micromamba user.
So, it would be better to merge it first and take a close look on your side at these micromamba-specific settings.

I confirmed on my end that the initial Dockerfile works, but cannot guarantee that the big changes with micromamba still doesn't change the behavior. Also, it's been several months since the last time I have touched the code of srbench, merging this PR first and customizing it on your side would be more efficient.

[MilesCranmer]

Sounds good to me!

[yoshitomo-matsubara]

@MilesCranmer Thank you, could you merge this PR or tell the person in charge to do so?

[lacava]

@MilesCranmer micromamba sounds interesting, let's open a separate issue and PR to discuss using it.

[yoshitomo-matsubara]

@lacava

let's open a separate issue and PR to discuss using it.

Then, what do you do with this PR ?

[lacava]

@yoshitomo-matsubara we will review this PR and merge it, don't worry. My bandwidth has been extremely limited since I've moved jobs and cities. I've just invited a number of contributors to be maintainers of this repo (@foolnotion, @marcovirgolin , @folivetti , etc), so if any of them would like to work on this PR they are welcome.

I'm hesitant to add the docker container without having an element of CI that is checking /using the docker image. Otherwise there is no one to maintain it once it is merged, and it is going to need to be maintained. I can't use docker on the compute cluster that I'm using to run the experiments, and because I won't be using it, the tests are extra important.

@yoshitomo-matsubara are you willing/able to integrate the docker build into test.yml? Some of the steps @MilesCranmer mentioned appear relevant. Otherwise you'll have to wait for me or someone else to do it.

[MilesCranmer]

You can build docker containers with GitHub actions using https://github.com/docker/build-push-action/. That should let you test whether this Dockerfile builds, and also try to run the test suite inside the container as part of another GitHub action.

I would recommend having a separate GitHub workflow (e.g., docker.yml) so they can run in parallel, since the test is long already.

[foolnotion]

my PR yoshitomo-matsubara#1 was supposed to fix the remaining issues with this dockerfile. I suppose that now it's a little outdated but the changes are small and they could even be manually picked and merged into the current branch.

[MilesCranmer]

One additional comment:

This docker container (and actually, ALL attempts to build the conda environment srbench) fails to build on some architectures. For example, I have an Apple M1 cpu, and I see this output:

Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound: 
  - shogun-cpp=6.1.4
  - armadillo=9.900.4
  - tensorflow=1.14
  - taskflow

The reason for this is because these packages are not built for aarch64 in my case. Luckily, docker helps us get around this by emulating a compatible architecture. The way you can set this up is by using the platform tag:

FROM [--platform=<platform>] <image> [AS <name>]

If I select --platform=linux/amd64, then I can successfully build this container on my M1, since it uses qemu to emulate the amd64 architecture which has the conda env set up.

[yoshitomo-matsubara]

@foolnotion Sorry, I didn't notice your PR to my forked repo for a while. I didn't have any issue on my machine with the original docker file and environment.yaml. Maybe there were some minor updates that crashed the dependencies.

@MilesCranmer I think that the original one worked on Ubuntu 18LTS with x86_64 architecture, but different environments may require you some minor changes.

[lacava]

moved to PR #75