Process collection python requirements metadata

[AlanCoding]

Example execution-environment.yml

---
# subset of what may eventually be the container to replace the Ansible venv in AWX
version: 1
dependencies:
  galaxy: scm_metadata.yml

where the file scm_metadata.yml contains:

collections:
   - name: https://github.com/alancoding/azure.git
     version: ee
     type: git

Just to link it, this references the tree https://github.com/AlanCoding/azure/tree/ee which importantly contains:

https://github.com/AlanCoding/azure/blob/ee/meta/execution-environment.yml

  
dependencies:
  python:
  - packaging
  - requests[security]
  - xmltodict
  - azure-cli-core==2.0.35
  - azure-cli-nspkg==3.0.2
  - azure-common==1.1.11
  - azure-mgmt-authorization==0.51.1
  - azure-mgmt-batch==5.0.1
  - azure-mgmt-cdn==3.0.0
  - azure-mgmt-compute==10.0.0
  - azure-mgmt-containerinstance==1.4.0
  - azure-mgmt-containerregistry==2.0.0
  - azure-mgmt-containerservice==4.4.0
  - azure-mgmt-dns==2.1.0
  - azure-mgmt-keyvault==1.1.0
  - azure-mgmt-marketplaceordering==0.1.0
  - azure-mgmt-monitor==0.5.2
  - azure-mgmt-network==4.0.0
  - azure-mgmt-nspkg==2.0.0
  - azure-mgmt-redis==5.0.0
  - azure-mgmt-resource==2.1.0
  - azure-mgmt-rdbms==1.4.1
  - azure-mgmt-servicebus==0.5.3
  - azure-mgmt-sql==0.10.0
  - azure-mgmt-storage==3.1.0
  - azure-mgmt-trafficmanager==0.50.0
  - azure-mgmt-web==0.41.0
  - azure-nspkg==2.0.0
  - azure-storage==0.35.1
  - msrest==0.6.10
  - msrestazure==0.6.2
  - azure-keyvault==1.0.0a1
  - azure-graphrbac==0.40.0
  - azure-mgmt-cosmosdb==0.5.2
  - azure-mgmt-hdinsight==0.1.0
  - azure-mgmt-devtestlabs==3.0.0
  - azure-mgmt-loganalytics==0.2.0
  - azure-mgmt-automation==0.1.1
  - azure-mgmt-iothub==0.7.0
  - azure >= 2.0.0
  system: []
version: 1

Building that should produce an image with those python requirements, which should run azure.azcollection modules inside of playbooks.

[pabelanger]

For metadata, at first glance could we not use the top level requirement.txt files that python projects use? Either by default or some way to reference an external file? This would avoid the need to cut down on managing 2 places for dependencies. For example, https://github.com/ansible-collections/ansible.netcommon/blob/master/requirements.txt

Additional, for system packages, we'd been using bindep for the last few years to help manage those: https://docs.openstack.org/infra/bindep/readme.html

It is a project born out of openstack that has been working well for the last 5 years on helping projects in CI work, bootstrap things needed for creating images for testing. Could we consider using it or again allow reference of that file.

We have a lot of tooling in place today, on CI side for creating custom environments, it would be awesome to maybe see if we can keep using that.

[shanemcd]

@pabelanger +1 to your idea of just using a requirements file. I had this as a hand-wavy example of what the execution environment definition side might look like:

---
version: 1
requirements:
  galaxy: requirements.yml
  python: requirements.txt
  system:
    - krb5-workstation

I hadn't considered bindep for usage here, but I will spend some time familiarizing myself with it to have an informed opinion. Thanks for all the feedback.

[AlanCoding]

In the sniffer script I was using I was looking at requirements.txt, and that's where many of those come from. In practice, I can see how it's almost never going to list in the metadata itself, unless things get complicated. Ideally, the collection-level execution-environment.yml will follow the exact same syntax rules as the user's execution environment specification.

In the near-term, I might try to re-jigger my script to make branches in the collections that simply reference requirements.txt.

[pabelanger]

In openstack, we have almost the same idea. The ability to take any python project and create a docker container from it. The goal, was to try and keep each project dockerfile small, while parenting to a common base. Almost creating an API for other Dockerfile to use.

https://opendev.org/opendev/system-config/src/branch/master/docker/python-builder/scripts/install-from-bindep

is an example for how pip / OS packages are installed. While the code is different, we are almost taking about the same idea.

https://wiki.openstack.org/wiki/Giftwrap was another idea around this, that eventually died off.

Some of the tricks done are around 2 stage builds, as not to pull in development libraries into production images, that is why we install wheels from outside the container.

[shanemcd]

@pabelanger Do you know if bindep is available for usage downstream anywhere? I just checked dist-git and I'm not seeing it there anywhere.

[pabelanger]

I'd have to look, we likely could work with rdoproject to package it if missing.

[AlanCoding]

Will be naming branches ee_req for those that defer to a requirements file. Here is one:

https://github.com/ansible-collections/azure/compare/dev...AlanCoding:ee_req?expand=1

I anticipate this to be used in the proof-of-concept. In this example, I don't think we miss anything this way.

[kdelee]

Tried this out given the azure example:

cat execution-environment.yml 
---
# subset of what may eventually be the container to replace the Ansible venv in AWX
version: 1
dependencies:
  galaxy: scm_metadata.yml

cat scm_metadata.yml 
collections:
   - name: https://github.com/alancoding/azure.git
     version: ee_req
     type: git

then I did ansible-builder build in that directory, build completed and ran container and found indeed python deps were installed correctly.

Let's get this in an integration test case when we decide where those are going to be