Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-14932: [CI] Prefer mamba over conda #11816

Closed
wants to merge 6 commits into from
Closed

Conversation

kszucs
Copy link
Member

@kszucs kszucs commented Nov 30, 2021

Mamba should provide quicker image builds compared to conda.

@kszucs kszucs requested review from pitrou and xhochy November 30, 2021 17:51
@kszucs kszucs changed the title [CI][Python] Prefer mamba over conda ARROW-14932: [CI] Prefer mamba over conda Nov 30, 2021
@github-actions
Copy link

@github-actions
Copy link

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@apache apache deleted a comment from github-actions bot Nov 30, 2021
@kszucs
Copy link
Member Author

kszucs commented Nov 30, 2021

@github-actions crossbow submit test-conda-*

@github-actions
Copy link

Revision: ad41c19

Submitted crossbow builds: ursacomputing/crossbow @ actions-1220

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou
Copy link
Member

pitrou commented Nov 30, 2021

Perhaps you can also make the same changes for AppVeyor?

@@ -29,8 +29,7 @@ dask=$1
if [ "${dask}" = "master" ]; then
pip install https://github.com/dask/dask/archive/main.tar.gz#egg=dask[dataframe]
elif [ "${dask}" = "latest" ]; then
conda install -q dask
pip install dask
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason to switch to pip here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These scripts might be used from non-conda images so rather prefer pip here.

Copy link
Member

@jorisvandenbossche jorisvandenbossche Dec 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This broke the nightly integration build, because the pip version not installing pandas automatically (now this is easily solved with pip install dask[dataframe] like is already done above for the main branch)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, creating a PR for it.

@kszucs
Copy link
Member Author

kszucs commented Dec 1, 2021

@github-actions crossbow submit test-conda-*

@github-actions
Copy link

github-actions bot commented Dec 1, 2021

Revision: b709bd7

Submitted crossbow builds: ursacomputing/crossbow @ actions-1225

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@kszucs
Copy link
Member Author

kszucs commented Dec 1, 2021

@github-actions crossbow submit test-conda-*

@github-actions
Copy link

github-actions bot commented Dec 1, 2021

Revision: bff9147

Submitted crossbow builds: ursacomputing/crossbow @ actions-1226

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

Copy link
Member Author

@kszucs kszucs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, merging on green

@kszucs kszucs closed this in f416111 Dec 1, 2021
@ursabot
Copy link

ursabot commented Dec 1, 2021

Benchmark runs are scheduled for baseline = bca0681 and contender = f416111. f416111 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️3.23% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.27% ⬆️0.13%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants