build and test conda packages in CI by jameslamb · Pull Request #99 · rapidsai/rapids-cli

jameslamb · 2025-05-28T02:08:34Z

Closes #89

adds conda CI (for PRs and branch builds)
uses rattler-build for builds, matching the direction RAPIDS is moving (Replace conda-build with rattler-build build-planning#47)
move version into a root-level VERSION file (so it can be referenced in both pyproject.toml and the rattler-build recipe)
bumps version to 0.1.1 (so we'll see packages get published)
introduces rapids-dependency-file-generator
- to start, just to avoid duplicating testing dependencies into test_python.sh
- and for consistency with other RAPIDS projects
- if we agree on this, I'll go further in a follow-up PR and use it to automatically manage pyproject.toml

…port noarch packages yet

jameslamb · 2025-05-28T15:19:56Z

    rev: v0.10.0.1
    hooks:
      - id: shellcheck
+        args: ["--severity=warning"]


This suppresses warnings like this that come from source-ing scripts from https://github.com/rapidsai/gha-tools in CI scripts:

In ci/build_python.sh line 6: source rapids-date-string ^----------------^ SC1091 (info): Not following: rapids-date-string was not specified as input (see shellcheck -x). In ci/build_python.sh line 14: source rapids-rattler-channel-string ^---------------------------^ SC1091 (info): Not following: rapids-rattler-channel-string was not specified as input (see shellcheck -x). In ci/test_python.sh line 6: . /opt/conda/etc/profile.d/conda.sh ^-- SC1091 (info): Not following: /opt/conda/etc/profile.d/conda.sh was not specified as input (see shellcheck -x).

I think saying "don't treat info-level findings as errors" is preferable to individually skipping those points with # shellcheck disable comments, just for readability and reduced development friction.

Not sure if apply but maybe a comment in the yaml might help for future reference.

Sure!

Did that here: f224cf3

jameslamb · 2025-05-28T18:15:27Z

@@ -0,0 +1 @@
+0.1.1


As of this PR, proposing we just use a version in a file.

Updating this file will be required to do a new release. In the near future, it'd probably be better to read this version from git tags, as most other RAPIDS projects do.

jameslamb · 2025-05-28T18:17:02Z

 [build-system]
 requires = [
-    "setuptools>=64.0.0",
+    "setuptools>=77.0.0",


To get the new behavior for project.license-files from setuptools.

ref: https://github.com/pypa/setuptools/blob/f9bf42241cc345988e6acdb99f0cbaf634e8c15f/setuptools/config/_apply_pyprojecttoml.py#L214-L215

jameslamb · 2025-05-28T18:18:46Z

@@ -0,0 +1,46 @@
+# Dependency list for https://github.com/rapidsai/dependency-file-generator
+files:


For this PR, adding just enough in dependencies.yaml to set up the environment for conda-based tests in CI.

If we agree on using rapids-dependency-file-generator here, I'll do a follow-up PR using this for pyproject.toml too.

Is there any advantage to use rapids-dependency-file-generator at this stage of the project? why should we?

Yes I think so. I think that will become clearer in the follow-up PR I'll add that also manages pyproject.toml dependencies... less stuff to keep in sync across multiple places.

ncclementi

Just left couple of comments and question mostly for understanding, but no blockers.

ncclementi · 2025-05-28T22:36:23Z

      - name: Run pre-commit
        uses: pre-commit/action@v3.0.1
-  build-wheel:
+  conda-python-build:


Do we want to build the package on every PR? or are we doing this juts to test?

Yes I think we should build it on every PR. We do that across much of RAPIDS.

Building is cheap, and doing it often spreads the development work like "some upstream dependency broke and our stuff is broken" more evenly over time than, say, only building at release time.

If we wanted to avoid that in the future, we could use something like the changed-files workflow to implement behavior like "only run certain types of CI workflows based on which files were changed".

Example of that: https://github.com/rapidsai/cuml/blob/9f829c3e87fd3ee59297dfafd60a475620635342/.github/workflows/pr.yaml#L61-L78

ncclementi · 2025-05-28T22:56:16Z

    rev: v0.10.0.1
    hooks:
      - id: shellcheck
+        args: ["--severity=warning"]


Not sure if apply but maybe a comment in the yaml might help for future reference.

ncclementi · 2025-05-28T22:59:40Z

@@ -0,0 +1,46 @@
+# Dependency list for https://github.com/rapidsai/dependency-file-generator
+files:


Is there any advantage to use rapids-dependency-file-generator at this stage of the project? why should we?

jameslamb · 2025-05-29T14:33:11Z

/merge

Follow-up to #99 Fixes the failing conda builds: > The workflow is not valid. .github/workflows/build.yaml (Line: 18, Col: 13): Job 'conda-python-build' depends on unknown job 'checks'. .github/workflows/build.yaml (Line: 25, Col: 13): Job 'upload-conda' depends on job 'conda-python-build' which creates a cycle in the dependency graph. ([build link](https://github.com/rapidsai/rapids-cli/actions/runs/15326288414)) Adds more use of `rapids-dependency-file-generator` for managing dependencies. Also bumps the version to 0.1.2, to pick up the slight change to wheel metadata (`importlib-metadata`, not `importlib_metadata`: https://pypi.org/project/importlib-metadata/) Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Naty Clementi (https://github.com/ncclementi) URL: #101

build and test conda packages in CI

2c2d153

jameslamb added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels May 28, 2025

jameslamb and others added 5 commits May 27, 2025 21:19

reduce build matrix size

0f72926

fix metadat

3b2975c

more setuptools fiddling

6a54b23

Merge branch 'main' into add-conda-packages

5d079f3

just build the whole matrix, because RAPIDS workflows don't fully sup…

ef86853

…port noarch packages yet

jameslamb commented May 28, 2025

View reviewed changes

jameslamb added 2 commits May 28, 2025 10:58

actually install rapids-cli in tests

7f638e5

use strict channel priority

8f13733

jameslamb commented May 28, 2025

View reviewed changes

jameslamb changed the title ~~WIP: build and test conda packages in CI~~ build and test conda packages in CI May 28, 2025

jameslamb marked this pull request as ready for review May 28, 2025 18:18

jameslamb requested a review from a team May 28, 2025 18:19

ncclementi approved these changes May 28, 2025

View reviewed changes

jameslamb and others added 2 commits May 28, 2025 18:33

Update .github/workflows/build.yaml

58e503a

add comment

f224cf3

rapids-bot Bot merged commit 4a846b9 into rapidsai:main May 29, 2025
22 checks passed

jameslamb deleted the add-conda-packages branch May 29, 2025 14:33

jameslamb mentioned this pull request May 29, 2025

fix conda uploads, use more rapids-dependency-file-generator #101

Merged

		@@ -0,0 +1,46 @@
		# Dependency list for https://github.com/rapidsai/dependency-file-generator
		files:

		@@ -0,0 +1 @@
		0.1.1

Conversation

jameslamb commented May 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ncclementi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jameslamb commented May 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants