Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate proper, consistent nightly versions for pip and conda packages #1347

Merged
merged 32 commits into from
Oct 31, 2023
Merged

Generate proper, consistent nightly versions for pip and conda packages #1347

merged 32 commits into from
Oct 31, 2023

Conversation

vyasr
Copy link
Contributor

@vyasr vyasr commented Sep 22, 2023 •
edited
Loading

Description

This PR changes conda C++/Python packages and wheels to all generate a consistent version for nightlies. The nightly version is of the form YY.MM.DDaN, where N is the number of commits from the last tag. The version is embedded in both the package metadata and in the rmm.__version__ attribute. In addition the commit hash itself is embedded into the package as rmm.__git_commit__. These changes ensure that 1) the conda Python package for a given nightly will reliably choose the correct C++ package (previously we relied on build strings and build times, which is more fragile w.r.t. the conda solver); 2) wheels are properly considered nightlies and are treated accordingly by pip (e.g. requiring --pre for installation, not conflicting with normal releases, etc); and 3) wheels and conda packages are aligned on versions so that they can be easily compared if necessary.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@vyasr vyasr added non-breaking Non-breaking change improvement Improvement / enhancement to an existing function labels Sep 22, 2023
@vyasr vyasr self-assigned this Sep 22, 2023
@github-actions github-actions bot added Python Related to RMM Python API conda ci labels Sep 22, 2023
vyasr
vyasr commented Sep 22, 2023
View reviewed changes
dependencies.yaml Outdated Show resolved Hide resolved
vyasr
vyasr commented Sep 22, 2023
View reviewed changes
ci/get_version.sh Outdated Show resolved Hide resolved
@github-actions github-actions bot added CMake cpp Pertains to C++ code labels Sep 29, 2023
@vyasr vyasr changed the base branch from branch-23.10 to branch-23.12 September 29, 2023 16:36
@github-actions github-actions bot removed CMake cpp Pertains to C++ code labels Sep 29, 2023
@leofang
Copy link
Member

leofang commented Sep 30, 2023

0dd834b is genius! Is there a minimal version requirement for setuptools to support it?

@vyasr
Copy link
Contributor Author

vyasr commented Oct 2, 2023 •
edited
Loading

Should work for anything >=61.0.0, which is what added support for PEP 621-style storage of project metadata in pyproject.toml
https://setuptools.pypa.io/en/latest/history.html#id337

#3068: [EXPERIMENTAL] Add support for pyproject.toml configuration (as introduced by PEP 621). Configuration parameters not covered by standards are handled in the [tool.setuptools] sub-table.

Unfortunately it only gets you so far since we still have to patch in the version for conda packages.

@vyasr vyasr mentioned this pull request Oct 4, 2023
Merged
@vyasr vyasr removed the 5 - DO NOT MERGE Hold off on merging; see PR for details label Oct 27, 2023
This was referenced Oct 27, 2023
Merged
Merged
Merged
bdice
bdice reviewed Oct 30, 2023
View reviewed changes
python/rmm/_version.py
import importlib.resources

__version__ = (
importlib.resources.files("rmm").joinpath("VERSION").read_text().strip()
Copy link
Contributor

@bdice bdice Oct 30, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really like that importing the package now requires reading another file on disk, just to populate a version string. Is "single-source" the only benefit of the VERSION approach?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming that you include the dynamic version metadata support in pyproject.toml as part of single-sourcing (and not just sharing across multiple packages), yes that is the primary benefit. It is the only approach to single-sourcing that is compatible with the range of different build processes that we're currently supporting. I agree that it's not perfect, but given the number of iterations that we've gone through with these PRs across all of RAPIDS I would prefer to leave this as is for now. Every alternative has tradeoffs, so there isn't a clear winner, and at least in the specific case of how we minimize duplication of the version we're going to have to revisit this again in the near future once we switch over to scikit-build-core.

bdice
bdice reviewed Oct 30, 2023
View reviewed changes
ci/build_python.sh Outdated Show resolved Hide resolved
@divyegala divyegala mentioned this pull request Oct 30, 2023
Merged
grlee77 added a commit to grlee77/cucim that referenced this pull request Oct 31, 2023
@grlee77
refactor VERSION management following example in rapidsai/rmm#1347
dff57f8
rapids-bot bot pushed a commit to rapidsai/cuml that referenced this pull request Oct 31, 2023
@vyasr
Allow nightly dependencies and set up consistent nightly versions for…
4c36b19 
… conda and pip packages (#5607)

This PR applies analogous changes to rapidsai/rmm#1335 and rapidsai/rmm#1347.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Divye Gala (https://github.com/divyegala)
  - Simon Adorf (https://github.com/csadorf)
  - AJ Schmidt (https://github.com/ajschmidt8)
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #5607
rapids-bot bot pushed a commit to rapidsai/cudf that referenced this pull request Oct 31, 2023
@vyasr
Update versioning strategy (#14285)
7358ecd 
See rapidsai/rmm#1347

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: #14285
rapids-bot bot pushed a commit to rapidsai/ucx-py that referenced this pull request Oct 31, 2023
@vyasr
Update versioning strategy (#997)
c6ab353 
See rapidsai/rmm#1347

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Peter Andreas Entschev (https://github.com/pentschev)
  - https://github.com/jakirkham

Approvers:
  - Peter Andreas Entschev (https://github.com/pentschev)
  - Lawrence Mitchell (https://github.com/wence-)
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: #997
rapids-bot bot pushed a commit to rapidsai/cuspatial that referenced this pull request Oct 31, 2023
@vyasr
Update versioning strategy (#1285)
8731137 
See rapidsai/rmm#1347

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Mark Harris (https://github.com/harrism)
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: #1285
rapids-bot bot pushed a commit to rapidsai/cuxfilter that referenced this pull request Oct 31, 2023
@vyasr
Update versioning strategy (#548)
24d3465 
See rapidsai/rmm#1347

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)
  - Ajay Thorve (https://github.com/AjayThorve)

URL: #548
@vyasr
Copy link
Contributor Author

vyasr commented Oct 31, 2023

/merge

@rapids-bot rapids-bot bot merged commit 207b898 into rapidsai:branch-23.12 Oct 31, 2023
55 checks passed
@vyasr vyasr deleted the feat/nightly_versions branch October 31, 2023 16:59
rapids-bot bot pushed a commit to rapidsai/wholegraph that referenced this pull request Oct 31, 2023
@divyegala
Setup Consistent Nightly Versions for Pip and Conda (#82)
c3bd5ef 
See rapidsai/rmm#1347

Authors:
  - Divye Gala (https://github.com/divyegala)

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: #82
rapids-bot bot pushed a commit to rapidsai/raft that referenced this pull request Oct 31, 2023
@divyegala
Setup Consistent Nightly Versions for Pip and Conda (#1880)
f720605 
See rapidsai/rmm#1347

Authors:
  - Divye Gala (https://github.com/divyegala)

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #1880
This was referenced Nov 1, 2023
Merged
Merged
Merged
Merged
rapids-bot bot pushed a commit to rapidsai/cugraph that referenced this pull request Nov 1, 2023
@divyegala
Setup Consistent Nightly Versions for Pip and Conda (#3933)
f4bcdc2 
See rapidsai/rmm#1347

Authors:
  - Divye Gala (https://github.com/divyegala)

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)
  - Chuck Hastings (https://github.com/ChuckHastings)
  - Rick Ratzel (https://github.com/rlratzel)

URL: #3933
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bdice bdice bdice approved these changes

@ajschmidt8 ajschmidt8 ajschmidt8 approved these changes

@wence- wence- wence- approved these changes

Assignees

@vyasr vyasr

Labels
ci conda improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Python Related to RMM Python API
Projects
RMM Project Board
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@vyasr @leofang @wence- @bdice @ajschmidt8