Turning the v80++ linker into a well-behaved Python package

[JOOpdenhoevel]

This PR converts the v80++ linker from a hand-rolled, two-mode script into a proper Python package (v80pp) with its resources shipped as package data and its v80++ entry point generated by the packaging backend. As a consequence, the source-tree and installation modes converge, the bespoke "find a Python interpreter, vendor jinja2/markupsafe, copy resources, write a shim" logic in the packaging scripts disappears, the AVED git submodule moves out of the linker's source tree, and the supported Python version drops to the system default on both Ubuntu 22.04 and Rocky 9 (3.10 and 3.9 respectively). To verify the new packaging end-to-end without an actual installation, this PR also introduces minimal Ubuntu/Rocky Dockerfiles and a single scripts/run-with-docker.sh entry point that can both build the packages inside a container and spin up a clean container with those packages installed.

Motivation

My current primary goal is adding integration tests to SLASH. These tests should exercise the software stack as close to a real installation as possible. However, I do not have access to Docker on Otus compute nodes and cannot create new Singularity containers there, so the tests have to run without an actual installation step — but they still need to look like a proper installed setup.

The biggest obstacle to that is the linker's deployment model. Today it assumes one of two scenarios:

• Source-tree mode: the resource directory is resolved relative to the
  linker's source files (Path(__file__).parent.parent.parent / "resources").
• Installation mode: the resource directory is assumed to live in a
  well-defined location like /usr/share/v80++.

On top of that, the packaging step does several things that the source-tree mode does not:

• It redistributes some of the linker's runtime dependencies, because the linker is supposed to run on Python ≥ 3.10 and jinja2 / markupsafe are not available as packages for those interpreters on Rocky 9.
• It excludes a sub-directory of the resource directory — the AVED git submodule — from the shipped artifacts.
• It generates a v80++ shell script that searches for the best Python interpreter on the system and uses it to invoke main.py.

The net effect is that a non-trivial set of behaviors and edge cases only exists in the packaged form: running from the source tree is too different from running from an installed package to be a useful proxy in tests.

It would be much better if the linker were a well-behaved Python package:

• Resources can be stored as Python modules and shipped together with the code, so Python does the path resolution for us.
• The v80++ entry-point script can be created by pip / the packaging backend, with the correct Python interpreter baked in.
• The linker can then be installed into a virtual environment and behave almost exactly like an installed system package; source-tree mode remains available, but the gap to "package mode" is minimal.

Changes

Turn the linker into a well-behaved Python package

• Source directory becomes the v80pp module
  • Renamed linker/src/ to linker/v80pp/.
  • Renamed main.py to __main__.py.
  • Prefixed the affected imports with v80pp. where necessary.
  • After this change alone, you can already cd linker && python -m v80pp to run the linker.
• Resource directory becomes the v80pp.resources module
  • With this, we no longer have to resolve the path to the resources directory ourselves — Python does it for us via importlib.resources.
  • Being well-behaved means accessing resources via importlib.resources instead of as filesystem paths.
    • That works fine for individual files and Jinja templates, but directories cannot be accessed as a single unit.
    • In some cases I copy/export the contents of a resource sub-module into the build directory before use.
    • The *_build.tcl scripts used to receive the entire resource directory as an argument and pull what they needed out of it.
      • Copying the entire resource tree just to satisfy that contract would have been overkill / pollution of the build directory.
      • I therefore refactored the build TCL scripts so they receive every argument they need directly, instead of one resource-root path.
  • Having the AVED git submodule sitting inside a Python module also did not look right.
    • Other components reference the AVED submodule and would now have to reach multiple layers deep into the linker for it.
    • The linker only needs AVED while building the base design / abstract shell files.
    • The old setup does not ship AVED in the package anyway.
    • I therefore moved the AVED submodule to submodules/AVED (top-level) and changed the linker's install subcommand to clone AVED from GitHub into its build directory.
      • If you prefer, I can swap the git clone out for a CLI argument that points at the existing local submodule path.
• Adding a pyproject.toml to turn v80pp into a real Python package
  • Nice side effect: you can declare executables there.
    • The packager creates them during installation.
    • They use the correct Python interpreter to load and run the v80pp module — no more handwritten interpreter search in a shell shim.

Integrate the v80pp Python package into the packaging system

• root-design-build.sh now just runs the linker as a Python module from the linker directory.
• pbuild.sh then builds the v80pp wheel.
• pinstall.sh then installs the v80pp wheel into the destination directory.
• The only workarounds that are still needed:
  • Creating a venv to install a modern pip for building the wheel — Rocky's default pip is too old.
  • Moving the unpacked files from /usr/local/ to /usr/ on Ubuntu — Ubuntu's pip ignores the requested /usr/ prefix and unpacks under /usr/local/ regardless.
• All the manual "find the Python interpreter, copy files into place, write the v80++ shim" logic is gone.
• Issue with this approach: we end up using the system's default Python interpreter and library packages.
  • On Rocky, python3-jinja2 is only available for the default Python interpreter, which is version 3.9.
  • Previously we assumed the linker needs Python ≥ 3.10, because that is what Ubuntu 22.04 ships.
  • However, with the Docker setup described below I was able to verify that this assumption does not actually hold — the linker runs fine on Python 3.9.
  • The previous redistribution of jinja2 and markupsafe can therefore be removed.
  • The linker package now declares the default Rocky / Ubuntu jinja2 packages as runtime dependencies instead.

Dockerfiles for packaging and running the linker

• I changed the packaging steps and needed a way to verify that the packaging still works.
• I therefore introduced Docker containers and a script that runs the full packaging procedure inside one of them, on a minimal Ubuntu 22.04 or Rocky 9 installation.
• I can then install the freshly built packages in a separate container and check that everything still works after installation.
• The whole setup is inspired by FINN's run-docker.sh and Dockerfile.finn.
  • Vivado is mounted into the container at the exact same path it has on the host.
  • Same for the current working directory.
  • The license files are mounted in too, and a few container-related workarounds are applied.
  • With those, it works just fine.

[hpc-aulmamei]

lgtm

[JOOpdenhoevel]

As agreed, I have renamed v80pp/v80++/"the linker" into "slashkit." I will run a full build to verify that this doesn't break anything and report back.