Provide a way to replicate Ansible/Python environment of Docker image

[dalbani]

Feature Request

Describe the problem you need a feature to resolve.

In order to run the ansible-operator locally for development purposes (with make run), one has to setup an Ansible / Python (virtual) environment.
Ideally, it would have to be identical to the one found in the Docker image quay.io/operator-framework/ansible-operator which is then deployed on production.
However, due to how the Python dependencies are declared in the Dockerfile, it is difficult to do so accurately.
See https://github.com/operator-framework/operator-sdk/blob/master/images/ansible-operator/Dockerfile#L19:

RUN yum clean all && rm -rf /var/cache/yum/* \
  && yum -y update \
  && yum install -y libffi-devel openssl-devel python36-devel gcc python3-pip python3-setuptools \
  && pip3 install --no-cache-dir \
    ipaddress \
    ansible-runner==1.3.4 \
    ansible-runner-http==1.0.0 \
    openshift~=0.10.0 \
    ansible~=2.9 \
    jmespath \
  && yum remove -y gcc libffi-devel openssl-devel python36-devel \
  && yum clean all \
  && rm -rf /var/cache/yum

The first solution is of course to track down the Dockerfile on Github and copy/paste the pip3 install ... command locally.
It works, but it's not very elegant and requires manual work which would be tricky to automate reliably.

Then I thought that I could simply extract the list of installed Python packages with pip install $(docker run --rm -ti --entrypoint bash quay.io/operator-framework/ansible-operator:v1.2.0 -c 'pip3 freeze').
But I ran into the problem of system packages: the freeze list also contains Python packages installed system-wide via YUM.
Which means I ended up getting gpg==1.13.1 in the freeze list, for example, because the RPM package gpg is present on the system. And gpg==1.13.1 is not a package available on PyPI...

So the pip freeze command becomes a bit more complicated (and still depending on the content of the Dockerfile!):

$ pip3 freeze | grep -E '^ipaddress|ansible-runner|openshift|ansible|jmespath'

This solution works quite well, even though it's not perfect: there's always the possibility of locally installing a different version of some dependencies of those packages, should a new release be made since the Docker image has been built for example.

Describe the solution you'd like.

My suggestion would be to have a requirements.txt file available in the Docker image at a documented location.
And that the packages mentioned inside would have to be pinned to specific versions for reproducibility purposes.
So one can easily set up a local environment with:

pip install $(docker run --rm -ti --entrypoint cat quay.io/operator-framework/ansible-operator:v1.2.0 /path/to/requirements.txt)

Any other idea?

/language ansible

[estroz]

Steps to install these deps need to be documented.

/kind documentation
/assign @asmacdo

[reinvantveer]

👍 for the suggestion of transparent local dependency installation. A couple of thoughts. There's a lot of potential for dependency hell here, pip3 install-ing in the global site-packages, I'd much rather see a solution that uses virtualenvs and deterministic builds through the Makefile. We've been using pipenv successfully for quite some time now, and it would be a welcome addition for managing python dependencies in custom ansible modules as well.

[asmacdo]

Adding release blocker since this blocks #4202

[asmacdo]

This no longer blocks #4202, but this turned out to be a little more involved than I realized. I dont want this to block the release, so I'm bumping to 1.4

I think I agree with @reinvantveer. What I have in mind is creating a virtualenv in the image, and installing the top-level deps via a pinned requirements.txt.

TODO:

• Base image:
  • pull the top level Python dependencies out of Dockerfile and into a requirements.txt.
• Local development:
  • Add a requirements.txt to the Ansible scaffolding with the same top level dependencies
  • Add a test-requirements.txt to the Ansible scaffolding that installs molecule
  • Add a make target that creates a virtualenv and installs both prod and dev requirements
  • update make run to use the venv
  • update the installation docs
• CI:
  • Install from requirements.txt and test-requirements.txt
• Update Script should allow all references to be maintained in a single place

[reinvantveer]

Agreed that having a *requirements.txt for a virtual environment is a huge improvement. However, this leaves one thing still in limbo: the Python version itself - a requirements.txt file doesn't allow you to specify this, while pipenv can (I'm unsure whether poetry can). In a nutshell: I don't quite feel that a requirements.txt file is a substitute for proper dependency management.

On this note on dependency management: it's becoming harder to keep Python 3.6 for local development - I switched to Ubuntu Focal recently and now you need something like the deadsnakes repo to get a hold of mothballed Python versions. 3.6 now only gets security fixes: see https://www.python.org/downloads/release/python-3612/. It's time to move on to a more recent version.