Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-42111: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock #42114

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

amoeba
Copy link
Member

@amoeba amoeba commented Jun 11, 2024

Rationale for this change

See #42111.

What changes are included in this PR?

  • Sync'd config files from arrow-cpp-feedstock
  • Sync'd auxiliary files
  • Updated tasks.yml to match above changes

Are these changes tested?

No, we need to test in CI.

Are there any user-facing changes?

No.

@amoeba
Copy link
Member Author

amoeba commented Jun 11, 2024

@github-actions crossbow submit -g conda

@github-actions github-actions bot added the awaiting review Awaiting review label Jun 11, 2024
@amoeba amoeba changed the title GH-42111: [Python][CI] Re-synchronize conda_recipes with arrow-cpp-feedstock GH-42111: [CI][Packaging] Re-synchronize conda_recipes with arrow-cpp-feedstock Jun 11, 2024
Copy link

Revision: 6ce51d5

Submitted crossbow builds: ursacomputing/crossbow @ actions-1622970cbf

Task Status
conda-clean Azure
conda-linux-aarch64-cpu-py3 Azure
conda-linux-aarch64-cuda-py3 Azure
conda-linux-ppc64le-cpu-py3 Azure
conda-linux-ppc64le-cuda-py3 Azure
conda-linux-x64-cpu-py3 Azure
conda-linux-x64-cuda-py3 Azure
conda-osx-arm64-cpu-py3 Azure
conda-osx-x64-cpu-py3 Azure
conda-win-x64-cpu-py3 Azure
conda-win-x64-cuda-py3 Azure

@amoeba
Copy link
Member Author

amoeba commented Jun 11, 2024

I initially thought this might be straightforward but after looking at past PRs, such as #37624, I see there's a bit more to it. @h-vetinari would you have any time to look at this? I've done a first pass and there's almost definitely more to do here so another set of eyes would be appreciated.

@@ -245,7 +245,7 @@ tasks:
ci: azure
template: conda-recipes/azure.linux.yml
params:
config: linux_64_cuda_compiler_versionNone
config: linux_64_c_compiler_version12c_stdlib_version2.12cuda_compilerNonecuda_compiler_versionNonecxx_compiler_version12
artifacts:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My guess is that the number and names of artifacts have changed so I need to figure that out for all of these tasks.

@github-actions github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jun 11, 2024
@h-vetinari
Copy link
Contributor

So in general, what you'd need to do is:

In the case of this update, the split of the python outputs into a separate feedstock means you'll have to integrate a full third recipe here, and build the pyarrow bits on top of libarrow (like currently done for r-arrow already). On the plus side, this will automatically resolve #15045.

Some more inspiration: #35637 #34652 #14832 #14102

@amoeba
Copy link
Member Author

amoeba commented Jun 12, 2024

Thanks @h-vetinari, I did at least some of those steps after studying past PRs but will go over your list more thoroughly tomorrow. I did find I had to edit meta.yml in a few places to make it work here. Re: @kou's comment about scripting, it seems like we could probably do all of the above with a script plus maybe a few patchfiles for the arrow repo specifics.

@amoeba amoeba marked this pull request as ready for review June 12, 2024 03:44
@amoeba amoeba force-pushed the sync-arrow-cpp-feedstocks branch from 6ce51d5 to 353c3fe Compare June 12, 2024 04:05
@raulcd
Copy link
Member

raulcd commented Jun 19, 2024

@github-actions crossbow submit -g conda

Copy link

Revision: 353c3fe

Submitted crossbow builds: ursacomputing/crossbow @ actions-ba6a36d7ee

Task Status
conda-clean Azure
conda-linux-aarch64-cpu-py3 Azure
conda-linux-aarch64-cuda-py3 Azure
conda-linux-ppc64le-cpu-py3 Azure
conda-linux-ppc64le-cuda-py3 Azure
conda-linux-x64-cpu-py3 Azure
conda-linux-x64-cuda-py3 Azure
conda-osx-arm64-cpu-py3 Azure
conda-osx-x64-cpu-py3 Azure
conda-win-x64-cpu-py3 Azure
conda-win-x64-cuda-py3 Azure

@raulcd
Copy link
Member

raulcd commented Jun 19, 2024

The current problem I see is that now that we did split arrow-cpp and pyarrow to different recipes if we still want to provide nightly conda packages for pyarrow we will require to build also the recipes for:
https://github.com/conda-forge/pyarrow-feedstock
Increasing the maintenance cost too.
This is required for our development cookbooks at the moment and I am unsure if there are other places where we are using those.

@jonkeane
Copy link
Member

I was doing some triaging + cleaning up of various jobs in our crossbow tests that are failing frequently. I've opened #43451 to temporarily disable running these every night so we don't continue getting alarmed while we work out what we need to do to synchronize these. I appreciate the work y'all are putting into getting these back up, and would happily close the temporary disabling PR if we can get these back running soon.

raulcd pushed a commit that referenced this pull request Jul 29, 2024
There's some work ongoing to update the conda jobs #42114, but many of them have been failing for many days (some have no history of success in our [crossbow report](http://crossbow.voltrondata.com)).

Let's turn those off temporarily so that we stop ignoring other failures alongside it, and we can re-enable them once we get them back running. Alternatively, we could merge those fixes and close this PR.
* GitHub Issue: #43450

Authored-by: Jonathan Keane <jkeane@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting committer review Awaiting committer review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants