grpcio dependency: No such file or directory: 'cc'

[nick-youngblut]

Steps to reproduce:

env.yaml file:

channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - python=3.9
  - scikit-learn=0.23.2
  - h5py=2.10.0
  - numpy=1.19.2
  - diamond=2.0.4
  - tensorflow>=2.1.0,<2.6.0
  - lightgbm=3.2.1
  - pandas<=1.4.0
  - scipy
  - prodigal>=2.6.3
  - setuptools
  - requests
  - packaging
  - tqdm

Dockerfile:

FROM mambaorg/micromamba:1.1.0

ARG MAMBA_DOCKERFILE_ACTIVATE=1
COPY --chown=$MAMBA_USER:$MAMBA_USER env.yaml /tmp/env.yaml
RUN micromamba install -y -n base -f /tmp/env.yaml git && \
    micromamba clean --all --yes

RUN git clone --recursive https://github.com/chklovski/checkm2.git && \
    cd checkm2 && \
    python setup.py install && \
    cd ../ && rm -rf checkm2

Build:

docker build --platform linux/amd64 -t checkm2:1.0.0 .

Error

Installed /opt/conda/lib/python3.9/site-packages/wheel-0.38.4-py3.9.egg
Searching for grpcio~=1.32.0
Reading https://pypi.org/simple/grpcio/
Downloading https://files.pythonhosted.org/packages/0e/5f/eeb402746a65839acdec78b7e757635f5e446138cc1d68589dfa32cba593/grpcio-1.32.0.tar.gz#sha256=01d3046fe980be25796d368f8fc5ff34b7cf5e1444f3789a017a7fe794465639
Best match: grpcio 1.32.0
Processing grpcio-1.32.0.tar.gz
Writing /tmp/easy_install-eop8d0xx/grpcio-1.32.0/setup.cfg
Running grpcio-1.32.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-eop8d0xx/grpcio-1.32.0/egg-dist-tmp-s4mfdn9f
/tmp/easy_install-eop8d0xx/grpcio-1.32.0/src/python/grpcio/commands.py:104: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if exit_code is not 0:
error: [Errno 2] No such file or directory: 'cc'

[nick-youngblut]

A dockerfile that builds:

FROM mambaorg/micromamba:1.1.0

USER root
RUN apt-get update && \
    apt-get install -y build-essential git && \
    apt-get clean && \
    apt-get purge && \
    rm -rf /var/lib/apt/lists/* /tmp/*

USER $MAMBA_USER
ARG MAMBA_DOCKERFILE_ACTIVATE=1
COPY --chown=$MAMBA_USER:$MAMBA_USER env.yaml /tmp/env.yaml
RUN micromamba install -y -n base -f /tmp/env.yaml && \
    micromamba clean --all --yes

RUN git clone --recursive https://github.com/chklovski/checkm2.git && \
    cd checkm2 && \
    python setup.py install && \
    cd .. && rm -rf checkm2

ENTRYPOINT ["/usr/local/bin/_entrypoint.sh", "checkm2", "-h"]

...in case https://github.com/chklovski/CheckM2/blob/main/docker/Dockerfile.in isn't working.

More generally, it would be good to point out the OS-level dependencies (e.g., gcc).

[nick-youngblut]

If you'd like to set up CI for the repo, this action should get you most of the way there:

name: CheckM2

on: [push, pull_request]

jobs:
  build:
    name: build (${{ matrix.python-version }}, ${{ matrix.os }})
    runs-on: ubuntu-latest
    env:
      DATADIR: data
    strategy:
      matrix:
        python-version: [3.9]
        build_type: [Release]
        compiler: [g++-9]
        include:
          - compiler: g++-9
    steps:
    - name: Checkout repository
      uses: actions/checkout@v2
    - name: Checkout submodules
      run: git submodule update --init --recursive
    - uses: conda-incubator/setup-miniconda@v2
      with:
        miniconda-version: 'latest'
        auto-update-conda: true
        mamba-version: "*"
        python-version: ${{ matrix.python-version }}
        channels: conda-forge,bioconda
        environment-file: checkm2.yml
        activate-environment: checkm2
    - name: package install
        shell: bash -l {0}
        working-directory: ${{runner.workspace}}/checkm2/
        run: |
          python setup.py install
    - name: checkm2 unit tests
        shell: bash -l {0}
        working-directory: ${{runner.workspace}}/checkm2/tests
        run: |
          pytest [checkm unit tests]
    - name: checkm2 CLI tests
        shell: bash -l {0}
        working-directory: ${{runner.workspace}}/checkm2/tests
        run: |
          checkm2 -h
          checkm2 predict -h

[nick-youngblut]

I found that the grpcio dependency issue is not a problem if python=3.8 is used.
If python=3.9 is used, then conda installs a version of grpcio much more recent than 1.32.0.
So, python setup.py install with the python=3.9 conda env results in pip trying to install grpcio=1.32.0, and that leads to the issues.

@chklovski I'm guessing that you've been developing checkm2 with python3.8?

This is an instance where CI testing using multiple versions of python can be very helpful.

[chklovski]

Thank you, that was very helpful. I've integrated some basic CI testing now. The docker file is still incomplete, but will work on that shortly.