Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure packaging/ci infrastructure #2014

Merged
merged 123 commits into from May 10, 2023
Merged

Restructure packaging/ci infrastructure #2014

merged 123 commits into from May 10, 2023

Conversation

sdrave
Copy link
Member

@sdrave sdrave commented Apr 14, 2023

This PR contains a major overhaul of pyMOR's packaging an CI infrastructure.

Main packaging changes:

  • pyproject.toml-based package with hatch(ling) as build backend.
  • All (optional) dependencies are specified in pyproject.toml using extras. This includes additional dependencies for CI, etc.
  • Dependency specification is completely static. dependencies.py is removed.
  • versioneer is replaced in favor of statically setting the version in __init__.py, which is extracted by the buildsytem to determine the package version.
  • All tool configuration is now in pyproject.toml. setup.cfg has been removed.

Main CI changes:

  • We generate requirements-*-txt lock files from pyproject.toml using pip-compile.
    To update, call make ci_requirements.
  • CI images are based on a single Dockerfile located in docker/. The Dockerfile installs the current locked requirements into the image. The pymor/docker is obsolete.
    To generate, call make ci_images. To push to the registry, call make ci_images_push.
  • The tag of the CI image is the sha256sum of the used requirements file. The preflight phase in the gitlab pipeline computes this hash for the currently tested source tree. The following stages use this hash to select the CI image.
  • ATM, we have ci-current, ci-oldest and ci-fenics. ci-current will have it's requirements and Python version periodically updated to follow upstream. ci-oldest will only be updated, when a new dependency is added or a minimal required version is bumped.
  • Since Fenics is not pip-installable, we have an additional CI image, for it. In the future we might have additional images, e.g. for FenicsX. The general idea is to use some current 'official' docker image for external solver which are not available via pip and install pyMOR into that image.
  • I have removed several CI jobs, in particular all installed tests and the binder-related stuff. (The binder links should still work.)
  • Conda tests use an environment lock file generated from pyproject.toml using conda-lock, which works simliar to pip-compile.
  • Hypothesis profile for conda tests has been set to dev to avoid build timeouts.

Remarks:

  • We don't really need many features of the chosen build backend, but hatch seems to be popular these days. Also, we use its hatch-vcs plugin for version management.
  • Neither requirements nor docker images are updated automatically right now. Regarding the requirements I plan to add a github action which warns to call make ci_dependencies if necessary. I am thinking about automating the image generation in the future in the CI pipeline. I would leave this for a future PR, however.
  • Generating the requirements only uses docker to ensure that pip-comile runs with the right Python version. I have good experience with rootless podman (export DOCKER=podman).
  • pip-compile will only bump versions when explicitly instructed to or required by a new dependency. Unfortunately, conda-lock will not lock at the current lock file and always use most recent versions.

@github-actions
Copy link
Contributor

Please wait for #2016 to be merged before merging this PR.

@sdrave sdrave force-pushed the setup_py_removal branch 13 times, most recently from b8891c3 to 4cca912 Compare April 26, 2023 16:09
@sdrave sdrave force-pushed the setup_py_removal branch 9 times, most recently from a0a0893 to 2320254 Compare April 27, 2023 18:35
@sdrave sdrave added the pr:change Change in existing functionality label Apr 27, 2023
@sdrave sdrave force-pushed the setup_py_removal branch 2 times, most recently from a9bc78a to 2905155 Compare April 28, 2023 09:41
@sdrave sdrave merged commit 8dc97c7 into main May 10, 2023
18 checks passed
@sdrave sdrave deleted the setup_py_removal branch May 10, 2023 21:13
@pmli pmli added this to the 2023.1 milestone May 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Buildsystem, packages and CI pr:change Change in existing functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants