Compose tree never reproducible (because of python)

[mtalexan]

Host system details
Fedora 37, podman 4.3.1, running via a coreos-assembler container image.

libostree:
 Version: '2022.1'
 Git: d3bc6a00d93850cad9c0b64088010e358ad714d6
 Features:
  - libcurl
  - libsoup
  - gpgme
  - ex-fsverity
  - libarchive
  - selinux
  - openssl
  - libmount
  - systemd
  - release
  - p2p
rpm-ostree:
 Version: '2022.3'
 Git: 03616c0a0feebd2bba2b65f031a7d2cb6c156658
 Features:
  - rust
  - compose
  - fedora-integration
Previous build: none
Previous commit: none

Expected vs actual behavior

Actual calls made to coreos-assembler

export SOURCE_DATE_EPOCH=$(date --utc "00:00:00 2000-01-01" '+%s')
coreos-assembler fetch --strict
coreos-assembler build ostree --version=TEST

Calls made to rpm-ostree on my behalf:

export SOURCE_DATE_EPOCH=$(date --utc "00:00:00 01-January-2023" '+%s')
rpm-ostree compose tree --repo=/srv/tmp/repo --touch-if-changed /srv/tmp/treecompose.changed --cachedir=/srv/cache --unified-core /srv/tmp/override/coreos-assembler-override-manifest.yaml --download-only --ex-lockfile=/srv/src/config/manifest-lock.x86_64.json --ex-lockfile=/srv/src/config/manifest-lock.overrides.yaml --ex-lockfile-strict
rpm-ostree compose tree --repo=/srv/tmp/repo --touch-if-changed /srv/tmp/treecompose.changed --cachedir=/srv/cache --unified-core /srv/tmp/override/coreos-assembler-override-manifest.yaml --cache-only --add-metadata-from-json /srv/tmp/build/tmp/commit-metadata-input.json --write-composejson-to /srv/tmp/compose.json --ex-write-lockfile-to /srv/tmp/repo/tmp/manifest-lock.generated.x86_64.json.tmp --ex-lockfile=/srv/src/config/manifest-lock.x86_64.json --ex-lockfile=/srv/src/config/manifest-lock.overrides.yaml --add-metadata-string=version=TEST --no-parent

Composing a tree using a lockfile and matched commit of the manifest.yaml from the same previously cached RPM files should generate a deterministic/reproducible ostree commit. Currently it does not because Python modules byte compile automatically on first use, and Python byte compilation isn't deterministic.

Expected:
The same ostree commit hash every time.

Steps to reproduce it

1. Pick a config with a manifest.yaml.
2. Set SOURCE_DATE_EPOCH to a fixed value
3. Precache all your RPMs in a separate OStree using --dry-run and --cachedir
4. Save the lockfile as well
5. Compose the tree, and take note of the ostree commit hash.
6. pull-local the commit into a separate archive ostree repo, and add a reference for it
7. Remove the previously tree and prune.
8. Re-compose, using the same lockfile and arguments.
9. Take note of the ostree commit hash
10. pull-local the commit into the same separate archive ostree repo as Step 6, and add a reference for it

Notice the hashes in step 5 and 9 are different. Diff the ostree references in step 6 and 10 to see why.

Additional Data
This existing issue talks about the source of the problem, which is Python modules. Python byte code still isn't reproducible even on Python 3.10 with SOURCE_DATE_EPOCH set to a value. Since SELinux uses Python tools, every rpm installation is running Python modules that get byte-compiled on first use during policy updates and file relabeling. Additionally, some RPMs will use Python scripts in their Install-time scriptlets, which causes further byte-compilation of modules in the system.

The consensus, led heavily by nixpkgs, is that until a Python version is released that can actually have reproducible byte-compiled modules, the only way to really deal with it is to wipe all byte-compiled caches. In nix's case this is before and after builds, but in the rpm-ostree case this is likely as an automatic compose postprocess step.

Would you like to work on the issue?

I can try, but I'm not experienced in Rust or the convoluted cross-calling between the C++ and Rust code.

The solution seems to be to add a new function to the tasks list in the compose_postprocess_final() here, which will find all __pycache__ folders in /usr and remove them.
Though arguably this might actually need to occur after user-specified post-process and post-process-script treefile scripts that might also make use of Python modules instead. If it's not after treefile-specified post-processes, it should at least be documented that any post-process scripts and/or post-process-script files need to remove __pycache__ directories themselves if they make use Python.

[mtalexan]

The current workaround when composing from a manifest.yaml treefile is to add this:

postprocess: 
  - |
    echo "Removing all module __pycache__ folders under /usr"
    find /usr -type d -name '__pycache__' -exec rm -r '{}' +