CI thoughts and worries

[SimonHeybrock]

Recent trouble with the packaging highlighted a number of shortcomings of the current setup:

• The manual coping of files in build_conda.py was very hard to debug and led to many slow iterations while attempting to fix the packaging. If I remember correctly the reason for this was to avoid rebuilds when calling conda --build?
• A sub-problem of the former was the mix of platform-specifics in this script, which meant fixing one platform broke the other platform's packages (more than once).
• We build install and all-tests before running any of the tests. One reason for splitting the scipp codebase into sublibraries was to be able to abort early, e.g., to run scipp-core-test before compiling scipp-variable. This advantage was lost at some point. While this may sometimes be irrelevant, during the past week or so I did observe that we had jobs waiting for free builders at times when we were merging/pushing in multiple projects (scipp, scippneutron, ess) at the same time.
  • Would maybe require fixing the unresolved MSVC issue of full recompilation of all object files and libraries for every target.
• Could consider building without IPO for PRs?
• Having to maintain a series of different env files is becoming a nightmare. We have:
  • dev envs (multiple platforms), used also for parts of CI
  • conda build meta.yml
  • User-facing envs (linked to from docs)
• Do we actually want to publish a package for every push to main?
• We have multiple times broken the ability to make patch releases by updating pipelines. Should probably use tags in pipelines repo, and refer to concrete tags from downstream CI? -> fixed in Use pipelines tags to avoid breaking ability for patch releases #2207.

[nvaytet]

• I agree with running the core tests before building the variable.

• "The manual coping of files in build_conda.py was very hard to debug". Yes I came across that multiple times when setting up pipelines and common build tools. This is a remnant of what Dan set up initially, and I don't really remember the original reason for doing it this way. What is the alternate way of doing it? (i don't know enough about conda build)

[SimonHeybrock]

Notes from inspecting our current setup (work in progress):

• Why is CONDA_PREFIX used in our build scripts, rather than PREFIX?
• Why can we not simply cmake install into PREFIX?
• There is no need to build the (non-pybind11) C++ components for multiple Python versions. Can we use the same conda build with a build matrix?
• We are not defining a host section? See https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#requirements-section.
• Why are we not defining run_exports? See https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#export-runtime-requirements.
• Why can we not decouple building and running C++ tests from making the conda package?
  • I believe in the past there were some issues with environment mismatches, but now that we get few (C++) dependencies via conda, is this still a concern?
  • Would need to ensure that main non-conda-build CI completes (and passes) C++ tests before starting conda build.
  • Is this related to the attempts to avoid re-compilation and make ccache work?
• Why are we running conda build on PRs in addition to the main pipeline?
• Can we use conda debug to create environments rather than manually maintaining scipp-developer.yml and friends?

[SimonHeybrock]

Proposal

This is an attempt to outline a possibly simpler solution. For now this is incomplete and probably has some holes:

• Improve meta.yml as described in Review conda setup  #2085 (comment).
• Disable IPO in non-package builds to speedup CI.
• Use conda debug to create env from meta.yml. Use this instead of scipp-developer.yml for local builds and non-package CI builds. TODO: Check if this really works.
• Get rid of tools/build_conda.py and manual file copying. Rely on files installed by cmake from install target (by tools/build_cpp.py).
• Do not use conda build in PRs.
  • How to handle docs builds?
• Figure out if we actually want to run conda build for every merge/push to main.
  • Should CI for main be independent of packaging?
  • Should we use a cron job instead?
  • Are the scipp/label/dev packages actually useful?
  • How would docs build/publish work? Part of conda build.
• tools/build_cpp.py:
  • Run tests after test target has been built, don't build everything before running first tests.
  • Consider option to disable benchmark and test builds for conda build, provided that we either
    • have an independently tested package of the scipp non-python libraries or
    • have a reliable CI setup ensuring that a specific commit has been tested before running conda build.

[jl-wynen]

How to handle docs builds?

cmake --build . --target docs should work. But need to make sure it is run in the same directory where _scipp was built to prevent recompilation.

Are the scipp/label/dev packages actually useful?

Yes, for updating scippneutron to a new scipp version, see past weeks. But we could enable building those packages only when needed.

This sounds like we potentially miss bugs in the day-to-day CI since we would not build the final package. Can we ensure that all dependencies are set up correctly without calling conda build?

[OwenArnold]

Do not use conda build in PRs

This is the way we originally had things, but @nvaytet changed this at some effort for consistency. I believe the driver for this was to eliminate potential bugs that would get through between PR pipelines but only be realised atmain/release pipelines. However I prefer the idea of getting back to a state were we could use ccache without having to do the cpp builds outside of conda build.

[SimonHeybrock]

Do not use conda build in PRs

This is the way we originally had things, but @nvaytet changed this at some effort for consistency. I believe the driver for this was to eliminate potential bugs that would get through between PR pipelines but only be realised atmain/release pipelines. However I prefer the idea of getting back to a state were we could use ccache without having to do the cpp builds outside of conda build.

I feel that this may have been an instance of fixing the symptoms, rather than the underlying issues? With the above suggestion we would (hopefully) have a consistent way of creating envs (which would eliminate once source of bugs).

[SimonHeybrock]

How to handle docs builds?

cmake --build . --target docs should work. But need to make sure it is run in the same directory where _scipp was built to prevent recompilation.

Well, but there is also the question of publishing, and avoiding publishing unless packages were built/published.

Are the scipp/label/dev packages actually useful?

Yes, for updating scippneutron to a new scipp version, see past weeks. But we could enable building those packages only when needed.

This sounds like we potentially miss bugs in the day-to-day CI since we would not build the final package.

Why would we miss bugs? I am not suggesting to not run CI, just to not build/publish packages. The only bugs this would miss are bugs in the packaging env and build scripts, which is unrelated.

Can we ensure that all dependencies are set up correctly without calling conda build?

You mean dependencies specified in meta.yml? If the suggestion to use conda debug for creating dev/CI envs is viable, I think this would be ensured?

[OwenArnold]

Do not use conda build in PRs

This is the way we originally had things, but @nvaytet changed this at some effort for consistency. I believe the driver for this was to eliminate potential bugs that would get through between PR pipelines but only be realised atmain/release pipelines. However I prefer the idea of getting back to a state were we could use ccache without having to do the cpp builds outside of conda build.

I feel that this may have been an instance of fixing the symptoms, rather than the underlying issues? With the above suggestion we would (hopefully) have a consistent way of creating envs (which would eliminate once source of bugs).

Yes environment is one source of bugs. cmake settings and flags is another, and possibly the bigger issue if I recall.

[SimonHeybrock]

Do not use conda build in PRs

This is the way we originally had things, but @nvaytet changed this at some effort for consistency. I believe the driver for this was to eliminate potential bugs that would get through between PR pipelines but only be realised atmain/release pipelines. However I prefer the idea of getting back to a state were we could use ccache without having to do the cpp builds outside of conda build.

I feel that this may have been an instance of fixing the symptoms, rather than the underlying issues? With the above suggestion we would (hopefully) have a consistent way of creating envs (which would eliminate once source of bugs).

Yes environment is one source of bugs. cmake settings and flags is another, and possibly the bigger issue if I recall.

Shouldn't this be solved by the recent introduction of build_cpp.py?

[OwenArnold]

Shouldn't this be solved by the recent introduction of build_cpp.py?

Yes that would work. Though I think we would want to adapt it to support a more conda build like approach. This needs more thought.

[SimonHeybrock]

Shouldn't this be solved by the recent introduction of build_cpp.py?

Yes that would work. Though I think we would want to adapt it to support a more conda build like approach. This needs more thought.

Can you elaborate? I thought conda build would call this script? It is basically a unified version of build.sh and build.bat.

[OwenArnold]

Can you elaborate? I thought conda build would call this script? It is basically a unified version of build.sh and build.bat.

I had thought that build_cpp.py was internally activating environments, but after checking, that's not where it's done, so indeed this wouldn't need to change.