Proposed conda-vendor improvements #34

rigzba21 · 2022-03-31T12:18:46Z

Edit: Closed #32 in favor of tracking progress here as this is a much bigger refactor.

Background

conda-lock has some awesome improvements in 1.x 🔭 that will allow us to reduce duplicated functionality in conda-vendor's implementation.

Example conda-lock usage for 1.x:

Given an environment.yaml:

name: minimal
channels:
  - conda-forge
dependencies:
  - python
  - pip
  - micromamba

Generating a lockfile (conda-lock supports multiple solvers such as mamba and micromamba):
conda lock --file environment.yaml -p linux-64 --mamba

Produces the following conda-lock.yml:

# This lock file was generated by conda-lock (https://github.com/conda-incubator/conda-lock). DO NOT EDIT!
#
# A "lock file" contains a concrete list of package versions (with checksums) to be installed. Unlike
# e.g. `conda env create`, the resulting environment will not change as new package versions become
# available, unless you explicitly update the lock file.
#
# Install this environment as "YOURENV" with:
#     conda-lock install -n YOURENV --file conda-lock.yml
# To update a single package to the latest version compatible with the version constraints in the source:
#     conda-lock lock --lockfile conda-lock.yml --update PACKAGE
# To re-solve the entire environment, e.g. after changing a version constraint in the source file:
#     conda-lock -f environment.yaml --lockfile conda-lock.yml
metadata:
  channels:
  - url: conda-forge
    used_env_vars: []
  content_hash:
    linux-64: 0e3f55cef4b08ecec24c4dda8e5ce0617f9f55f4e457bca0d9c16c9d8fe00bc5
  platforms:
  - linux-64
  sources:
  - environment.yaml
package:
- category: main
  dependencies: {}
  hash:
    md5: d7c89558ba9fa0495403155b64376d81
    sha256: fe51de6107f9edc7aa4f786a70f4a883943bc9d39b3bb7307c04c41410990726
  manager: conda
  name: _libgcc_mutex
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
  version: '0.1'
- category: main
  dependencies: {}
  hash:
    md5: 575611b8a84f45960e87722eeb51fa26
    sha256: d13c8774129e0d8d1427f5758fba53cfa915b6a12cd4dbd2bfe612d9eab0506d
  manager: conda
  name: ca-certificates
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2021.10.8-ha878542_0.tar.bz2
  version: 2021.10.8
- category: main
  dependencies: {}
  hash:
    md5: bd4f2e711b39af170e7ff15163fe87ee
    sha256: ad7985a9ff622880cf87c42db1ffe2dfb040d8175c1bb352fc8f3705c7e0962f
  manager: conda
  name: ld_impl_linux-64
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.36.1-hea4e1c9_2.tar.bz2
  version: 2.36.1
- category: main
  dependencies: {}
  hash:
    md5: 24072cb5ef3fa80347bd35f184dfdaed
    sha256: f8d6d9ab832401f8f32e161d5043b28fd7f043d8f0829ab5388f6e4a4256524a
  manager: conda
  name: micromamba
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/micromamba-0.22.0-0.tar.bz2
  version: 0.22.0
- category: main
  dependencies: {}
  hash:
    md5: 84be5301069417a2221187d2f435e0f7
    sha256: 74d8c1fbccae1a78c9bd2b2d1cda73df425cc28717a637198c23bd1c9b53b60e
  manager: conda
  name: tzdata
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/noarch/tzdata-2022a-h191b570_0.tar.bz2
  version: 2022a
- category: main
  dependencies:
    _libgcc_mutex: 0.1 conda_forge
  hash:
    md5: a77fb1a92411cb8d979de1c2d81dd210
    sha256: 1da28d8d10c93e43c78fb5020dd9022fe24687f759acc25de699185bdfa84e9b
  manager: conda
  name: libgomp
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libgomp-11.2.0-h1d223b6_14.tar.bz2
  version: 11.2.0
- category: main
  dependencies:
    _libgcc_mutex: 0.1 conda_forge
    libgomp: '>=7.5.0'
  hash:
    md5: 561e277319a41d4f24f5c05a9ef63c04
    sha256: 81c74d38c80345e195106dc3a5b4063b61f2209402bf9f6c7e2abadef4f544a3
  manager: conda
  name: _openmp_mutex
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-1_gnu.tar.bz2
  version: '4.5'
- category: main
  dependencies:
    _libgcc_mutex: 0.1 conda_forge
    _openmp_mutex: '>=4.5'
  hash:
    md5: 47e6c01d149b26090748d9d1ac32491b
    sha256: d24e25272239827012441e3376abcd2859a29418da825e6a593fc517b0c20f61
  manager: conda
  name: libgcc-ng
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-11.2.0-h1d223b6_14.tar.bz2
  version: 11.2.0
- category: main
  dependencies:
    libgcc-ng: '>=9.3.0'
  hash:
    md5: a1fd65c7ccbf10880423d82bca54eb54
    sha256: cb521319804640ff2ad6a9f118d972ed76d86bea44e5626c09a13d38f562e1fa
  manager: conda
  name: bzip2
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-h7f98852_4.tar.bz2
  version: 1.0.8
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
  hash:
    md5: d645c6d2ac96843a2bfaccd2d62b3ac3
    sha256: ab6e9856c21709b7b517e940ae7028ae0737546122f83c2aa5d692860c3b149e
  manager: conda
  name: libffi
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.2-h7f98852_5.tar.bz2
  version: 3.4.2
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
  hash:
    md5: 39b1328babf85c7c3a61636d9cd50206
    sha256: 32f4fb94d99946b0dabfbbfd442b25852baf909637f2eed1ffe3baea15d02aad
  manager: conda
  name: libnsl
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.0-h7f98852_0.tar.bz2
  version: 2.0.0
- category: main
  dependencies:
    libgcc-ng: '>=9.3.0'
  hash:
    md5: 772d69f030955d9646d3d0eaf21d859d
    sha256: 54f118845498353c936826f8da79b5377d23032bcac8c4a02de2019e26c3f6b3
  manager: conda
  name: libuuid
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.32.1-h7f98852_1000.tar.bz2
  version: 2.32.1
- category: main
  dependencies:
    libgcc-ng: '>=10.3.0'
  hash:
    md5: 757138ba3ddc6777b82e91d9ff62e7b9
    sha256: b46b66d1cb171be2227a275e226195ca9e56c6f5b16250b85645e82a69518378
  manager: conda
  name: libzlib
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.2.11-h166bdaf_1014.tar.bz2
  version: 1.2.11
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
  hash:
    md5: fb31bcb7af058244479ca635d20f0f4a
    sha256: bcb38449634bfe58e821c28d6814795b5bbad73514f0c7a9af7a710bbffc8243
  manager: conda
  name: ncurses
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.3-h9c3ff4c_0.tar.bz2
  version: '6.3'
- category: main
  dependencies:
    ca-certificates: ''
    libgcc-ng: '>=10.3.0'
  hash:
    md5: 49bf4e64d1e86676b90a8657c1142f01
    sha256: 123f0bd67843220fb27da6b71ba126934edbe714415a630ddec0f8c8a2b88cf0
  manager: conda
  name: openssl
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/openssl-3.0.2-h166bdaf_1.tar.bz2
  version: 3.0.2
- category: main
  dependencies:
    libgcc-ng: '>=7.5.0'
  hash:
    md5: 33f601066901f3e1a85af3522a8113f9
    sha256: 1e2823cb2a526bc3a7031ad5dbfb992891f9ff9740d1c17cb6dbb8ebdfd33b27
  manager: conda
  name: xz
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.5-h516909a_1.tar.bz2
  version: 5.2.5
- category: main
  dependencies:
    libgcc-ng: '>=9.3.0'
    ncurses: '>=6.2,<7.0.0a0'
  hash:
    md5: 5788de3c8d7a7d64ac56c784c4ef48e6
    sha256: 30464670b3c81ac739e8df6b2c3c57b56d1e1408572540dec63bf4b8713163e4
  manager: conda
  name: readline
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/readline-8.1-h46c0cb4_0.tar.bz2
  version: '8.1'
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
    libzlib: '>=1.2.11,<1.3.0a0'
  hash:
    md5: 5b8c42eb62e9fc961af70bdd6a26e168
    sha256: 032fd769aad9d4cad40ba261ab222675acb7ec951a8832455fce18ef33fa8df0
  manager: conda
  name: tk
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.12-h27826a3_0.tar.bz2
  version: 8.6.12
- category: main
  dependencies:
    libgcc-ng: '>=10.3.0'
    libzlib: 1.2.11 h166bdaf_1014
  hash:
    md5: def3b82d1a03aa695bb38ac1dd072ff2
    sha256: ccfdb4dcceae8b191ddd4703e7be84eff2ba82b53788d6bb9298e531bae4eaf9
  manager: conda
  name: zlib
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/zlib-1.2.11-h166bdaf_1014.tar.bz2
  version: 1.2.11
- category: main
  dependencies:
    libgcc-ng: '>=10.3.0'
    libzlib: '>=1.2.11,<1.3.0a0'
    ncurses: '>=6.3,<7.0a0'
    readline: '>=8.1,<9.0a0'
    zlib: '>=1.2.11,<1.3.0a0'
  hash:
    md5: 8057ac02d6d10a162d7eb4b0ca7ed291
    sha256: 5b1f7e51e6f6453c295cd911b826327b7eba4785b0366cf63cf6f828ec346076
  manager: conda
  name: sqlite
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.37.1-h4ff8645_0.tar.bz2
  version: 3.37.1
- category: main
  dependencies:
    bzip2: '>=1.0.8,<2.0a0'
    ld_impl_linux-64: '>=2.36.1'
    libffi: '>=3.4.2,<3.5.0a0'
    libgcc-ng: '>=10.3.0'
    libnsl: '>=2.0.0,<2.1.0a0'
    libuuid: '>=2.32.1,<3.0a0'
    libzlib: '>=1.2.11,<1.3.0a0'
    ncurses: '>=6.3,<7.0a0'
    openssl: '>=3.0.2,<4.0a0'
    readline: '>=8.1,<9.0a0'
    sqlite: '>=3.37.1,<4.0a0'
    tk: '>=8.6.12,<8.7.0a0'
    tzdata: ''
    xz: '>=5.2.5,<5.3.0a0'
  hash:
    md5: 0f72b088a5471e97309031e1636e7b3f
    sha256: 70eb462c28c5467c6e4860d5f574d240350b6ac718990b23cb0cc144d1dbea3f
  manager: conda
  name: python
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/python-3.10.4-h2660328_0_cpython.tar.bz2
  version: 3.10.4
- category: main
  dependencies:
    python: 3.10.*
  hash:
    md5: 9e7160cd0d865e98f6803f1fe15c8b61
    sha256: e7e52aaec7cba6e17e45d731f9d38ede007aea0d72aee66670ab71016f5783ed
  manager: conda
  name: python_abi
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.10-2_cp310.tar.bz2
  version: '3.10'
- category: main
  dependencies:
    python: '!=3.0,!=3.1,!=3.2,!=3.3,!=3.4'
  hash:
    md5: 1ca02aaf78d9c70d9a81a3bed5752022
    sha256: aede66e6370f3b936164a703e48362f9080d7162234058fb2ee63cc84d528afc
  manager: conda
  name: wheel
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/noarch/wheel-0.37.1-pyhd8ed1ab_0.tar.bz2
  version: 0.37.1
- category: main
  dependencies:
    python: '>=3.10,<3.11.0a0'
    python_abi: 3.10.* *_cp310
  hash:
    md5: 2bf50027b62c5e607310c1755c27e482
    sha256: 2d5aba1f98b586b637e66bd1593424f4d5530cbd73b06883b460f2947abc244e
  manager: conda
  name: setuptools
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/setuptools-61.2.0-py310hff52083_3.tar.bz2
  version: 61.2.0
- category: main
  dependencies:
    python: '>=3.7'
    setuptools: ''
    wheel: ''
  hash:
    md5: b1239ce8ef2a1eec485c398a683c5bff
    sha256: d36bb23fa250be2d6a21cafe1760a7ae434318fb397c85223dd6a0c8e6e5562b
  manager: conda
  name: pip
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/noarch/pip-22.0.4-pyhd8ed1ab_0.tar.bz2
  version: 22.0.4
version: 1

Proposed conda-vendor changes + improvements:

Remove conda-vendor's meta-manifest generation

I propose that we remove the meta-manifest generation, as conda-lock's new lockfile format now includes all of the necessary information we use to vendor dependencies into a local channel.

Remove intermediary step of generating a meta-manifest in favor of conda-lock's 1.0.x new API
Add in vendor command as the primary
Add a subcommand to generate formatted output for IronBank's hardening_manifest.yaml resources block, using conda-lock's 1.0.x FetchAction object.

Remove conda-vendor's combined manifest functionality

conda-lock now has compound specification for lockfile generation, where you can create a conda-lock.yaml from one or more environment files.

Remove intermediary step of generating a meta-manifest in favor of using conda-lock's 1.0.x compound-specification API
NOTE: this would be best tracked as it's own issue

Signing and Verification

I propose that we introduce signing and verification of the vendored dependencies within a local channel (and/or the local channel itself), and generate a SLSA compliant in-toto spec attestation.
NOTE: this would be best tracked as it's own issue

Define attestation + SBOM format (see notes below)
sigstore digital signing

The text was updated successfully, but these errors were encountered:

rigzba21 · 2022-03-31T17:28:41Z

Possible integration point for SBOM generation? anchore/syft#932

rigzba21 · 2022-03-31T20:03:38Z

New Usage:

# use conda as the solver for linux-64
conda-vendor vendor --file environment.yaml --solver conda --platform linux-64

# use mamba as the solver for osx-64
conda-vendor vendor --file environment.yaml --solver mamba --platform osx-64

# use micromamba as the solver for the host platform
conda-vendor vendor --file environment.yaml --solver micromamba

Now supports conda, mamba, and micromamba solvers.

Attest a Vendored Channel

Generate a SLSA spec compliant attestation

# "attest" as a subcommand
conda-vendor attest --vendored-channel path/to/my/vendored-channel/

which would produce an attestation file attestation.yaml.

Attestation Format (WIP):

Components (from in-toto attestation spec):

Envelope: Handles authentication and serialization

Statement: Binds the attestation to a particular subject and unambiguously identifies the types of the predicate

Predicate: Contains arbitrary metadata about the subject, with a type-specific schema

Bundle: Defines a method of grouping multiple attestations together

(WIP) for conda-vendor:

Envelope: TBD
- DSSE?
- other?
Statement: in-toto/attestation
Predicate: TBD
- SLSA Provenance?
- SPDX Spec?
- what works best for conda channels? packages? environments?
Bundle: TBD
- vendored conda channels

References, Notes, and Links:

This was referenced Mar 31, 2022

Conda lock import changes + environment signing/verification #32

Closed

Upstream conda-lock changes are non-compatible with the IronBank workflow's update_ib_files.py #33

Open

rigzba21 mentioned this issue Mar 31, 2022

conda-vendor improvements for version 1.0.0 #35

Merged

8 tasks

rigzba21 self-assigned this Apr 11, 2022

rigzba21 added the enhancement New feature or request label Apr 11, 2022

rigzba21 pinned this issue Apr 11, 2022

rigzba21 added the question Further information is requested label Apr 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposed conda-vendor improvements #34

Proposed conda-vendor improvements #34

rigzba21 commented Mar 31, 2022 •

edited

rigzba21 commented Mar 31, 2022 •

edited

rigzba21 commented Mar 31, 2022 •

edited

Proposed conda-vendor improvements #34

Proposed conda-vendor improvements #34

Comments

rigzba21 commented Mar 31, 2022 • edited

Background

Example conda-lock usage for 1.x:

Proposed conda-vendor changes + improvements:

Remove conda-vendor's meta-manifest generation

Remove conda-vendor's combined manifest functionality

Signing and Verification

rigzba21 commented Mar 31, 2022 • edited

rigzba21 commented Mar 31, 2022 • edited

New Usage:

Attest a Vendored Channel

Attestation Format (WIP):

rigzba21 commented Mar 31, 2022 •

edited

rigzba21 commented Mar 31, 2022 •

edited

rigzba21 commented Mar 31, 2022 •

edited