Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start building and publishing conda nightlies #986

Merged
merged 6 commits into from
Mar 25, 2024

Conversation

charlesbluca
Copy link
Member

Should unblock the issues around pinning described in dask/distributed#8575 (comment)

@fjetter
Copy link
Member

fjetter commented Mar 22, 2024

dask-expr 1.0.4+6.gd7238e9 has requirement dask==2024.3.1, but you have dask 2024.3.0+3.gf9310c440. Is this a chicken egg problem where we have to merge on failing build to let the other pass or what is the expected behavior?

@bdice
Copy link

bdice commented Mar 22, 2024

@fjetter I discussed this a bit with @charlesbluca. It looks like dask-expr’s pip check cannot pass because dask is versioned incorrectly. The conda package is versioned as an alpha of 2024.3.1, but the internal versioneer version is a post-release of 2024.3.0. This would have to be aligned in dask to make this check pass.

The options are probably:

  • don’t add 1 to the patch release in the conda recipe, match the versioneer post-release versioning (make conda match versioneer)
  • hack the versioneer versioning to add 1 to the patch and make it an alpha (make versioneer match conda)
  • turn off pip check

Copy link
Member

@jrbourbeau jrbourbeau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @charlesbluca. Btw, a commit just got pushed to dask/dask, so there should be a new nightly build uploaded shortly which should fix CI here (based off our offline conversation earlier today)

@charlesbluca
Copy link
Member Author

Thanks @jrbourbeau! Just pulled the latest changes to retrigger CI, looks like that resolved the conda issues 🎉

The conda package is versioned as an alpha of 2024.3.1, but the internal versioneer version is a post-release of 2024.3.0.

This is correct, and I filed dask/community#369 to track efforts to make these versions consistent.

This would have to be aligned in dask to make this check pass.

I don't think this is the case, I think the main reason the check failed before (as opposed to now) is that before the latest dask-core nightlies we were picking up were 2024.3.1a that preceded the last release (hence the internal 2024.3.0+... version). Now that a commit has been pushed to dask/dask since the last release (dask/dask@8c2199e) to trigger a nightly build, we now have2024.3.2a nightlies available with the internal 2024.3.1+... version that dask-expr is looking for ("dask-expr 1.0.4+6.gd7238e9 has requirement dask==2024.3.1...").

That being said, we shouldn't have had to wait for a random commit to land in the repo post-release to unblock things here, which is why I opened dask/dask#11014 so that, moving forward, the normal release process of pushing a tag to dask/dask will trigger a nightly build/publish and resolve this blocker immediately.

@rjzamora
Copy link
Member

@jrbourbeau - Any objections to getting this in?

@bdice
Copy link

bdice commented Mar 22, 2024

Yes, @charlesbluca is right. Thanks for the correction!

@jrbourbeau
Copy link
Member

jrbourbeau commented Mar 22, 2024

Any objections to getting this in?

No objection from me generally.

That said, I think we still need to add the conda token as a repo secret for uploading, right?

EDIT: Or is that handled at a higher level, like the dask org?

@charlesbluca
Copy link
Member Author

EDIT: Or is that handled at a higher level, like the dask org?

My understanding from this comment when we first set up nightlies is that it was set at the org level?

dask/dask#8469 (comment)

Copy link
Member

@rjzamora rjzamora left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @charlesbluca!

@rjzamora rjzamora merged commit b0f610b into dask:main Mar 25, 2024
9 checks passed
@rjzamora
Copy link
Member

@rjzamora
Copy link
Member

Okay yeah, @charlesbluca shared this link with me offline: https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#creating-secrets-for-an-organization

From the Repository access dropdown list, choose an access policy.

Seems like dask-expr needs to be granted access from a dropdown list at the org level.

@quasiben
Copy link
Member

I updated the DASK_CONDA_TOKEN to be shared with the dask-expr repo

@charlesbluca
Copy link
Member Author

Thanks @quasiben! Looks like that unblocked publishing nightlies:

https://github.com/dask/dask-expr/actions/runs/8437562794/job/23107649428
https://anaconda.org/dask/dask-expr/files

I've opened dask/distributed#8601 to apply the necessary changes to the dask nightly recipe, which supersedes dask/distributed#8575 and should unblock Distributed's conda CI.

@jrbourbeau
Copy link
Member

Great, thanks @charlesbluca @rjzamora @quasiben for handling this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants