Downgrade to Arrow 12.0.0 for aws-sdk-cpp and fix cudf_kafka builds for new CI containers #14296
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bdice
reviewed
Oct 18, 2023
jakirkham
reviewed
Oct 18, 2023
jakirkham
reviewed
Oct 18, 2023
jakirkham
reviewed
Oct 18, 2023
bdice
reviewed
Oct 18, 2023
bdice
reviewed
Oct 18, 2023
|
@vyasr I left a review but I would accept doing the changes that I requested in a follow-up PR if this PR can unblock CI as it is currently. |
github-actions
bot
added
libcudf
vyasr
changed the title
raydouglass
approved these changes
Oct 19, 2023
| @@ -25,8 +25,8 @@ gtest_version: | |||
| aws_sdk_cpp_version: | |||
| - "<1.11" | |||
|
|
|||
| libarrow_version: | |||
| - "=12" | |||
| libarrow: | |||
There was a problem hiding this comment.
Was it intentional to drop the _version suffix?
There was a problem hiding this comment.
No 🙁 good catch. But now that it's done and we have CI passing, I'm inclined to merge and fix in a follow-up to unblock CI.
|
/merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The aws-sdk-cpp pinning introduced in #14173 causes problems because newer builds of libarrow require a newer version of aws-sdk-cpp. Even though we restrict to libarrow 12.0.1, this restriction is insufficient to create solvable environments because the conda (mamba) solver doesn't seem to consistently reach far back enough into the history of builds to pull the last build that was compatible with the aws-sdk-cpp version that we need. For now, the safest way for us to avoid this problem is to downgrade to arrow 12.0.0, for which all conda package builds are pinned to the older version of aws-sdk-cpp that does not have the bug in question.
Separately, while the above issue was encountered we also got new builds of our CI images that removed system installs of CTK packages from CUDA 12 images. This changes was made because for CUDA 12 we can get all the necessary pieces of the CTK from conda-forge. However, it turns out that the cudf_kafka builds were implicitly relying on system CTK packages, and the cudf_kafka build is in fact not fully compatible with conda-forge CTK packages because it is not using CMake via scikit-build (nor any other more sophisticated library discovery mechanism like pkg-config) and therefore does not know how to find conda-forge CTK headers/libraries. This PR introduces a set of temporary patches to get around this limitation. These patches are not a long-term fix, and are only put in place assuming that #14292 is merged in the near future before we cut a 23.12 release.
Checklist