Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-35946: [CI][Packaging] Free up more disk space for Linux packages #35947

Merged
merged 8 commits into from
Jun 7, 2023

Conversation

raulcd
Copy link
Member

@raulcd raulcd commented Jun 6, 2023

Rationale for this change

Fix some Linux packaging jobs that fail due to out of space

What changes are included in this PR?

Cleaning up some more cache

Are these changes tested?

With crossbow jobs

Are there any user-facing changes?

No

@github-actions
Copy link

github-actions bot commented Jun 6, 2023

⚠️ GitHub issue #35946 has been automatically assigned in GitHub to PR creator.

@github-actions github-actions bot added the awaiting committer review Awaiting committer review label Jun 6, 2023
@raulcd
Copy link
Member Author

raulcd commented Jun 6, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 6, 2023

Revision: 021ec59

Submitted crossbow builds: ursacomputing/crossbow @ actions-02374678ab

Task Status
ubuntu-jammy-amd64 Github Actions

@raulcd
Copy link
Member Author

raulcd commented Jun 6, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 6, 2023

Revision: 9ffd9f0

Submitted crossbow builds: ursacomputing/crossbow @ actions-e7cc538dd3

Task Status
ubuntu-jammy-amd64 Github Actions

@raulcd
Copy link
Member Author

raulcd commented Jun 6, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 6, 2023

Revision: e209617

Submitted crossbow builds: ursacomputing/crossbow @ actions-c4c9731ed5

Task Status
ubuntu-jammy-amd64 Github Actions

@raulcd
Copy link
Member Author

raulcd commented Jun 6, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 6, 2023

Revision: d6e3008

Submitted crossbow builds: ursacomputing/crossbow @ actions-f977e71a38

Task Status
ubuntu-jammy-amd64 Github Actions

@raulcd
Copy link
Member Author

raulcd commented Jun 6, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 6, 2023

Revision: 4acdd01

Submitted crossbow builds: ursacomputing/crossbow @ actions-92d7715b46

Task Status
ubuntu-jammy-amd64 Github Actions

@raulcd
Copy link
Member Author

raulcd commented Jun 7, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 7, 2023

Revision: 3b1a84d

Submitted crossbow builds: ursacomputing/crossbow @ actions-106e5e0c2a

Task Status
ubuntu-jammy-amd64 Github Actions

@raulcd
Copy link
Member Author

raulcd commented Jun 7, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 7, 2023

Revision: fab055d

Submitted crossbow builds: ursacomputing/crossbow @ actions-f783d66906

Task Status
ubuntu-jammy-amd64 Github Actions

@raulcd
Copy link
Member Author

raulcd commented Jun 7, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 7, 2023

Revision: 98ee28e

Submitted crossbow builds: ursacomputing/crossbow @ actions-1e729f7314

Task Status
ubuntu-jammy-amd64 Github Actions

@raulcd
Copy link
Member Author

raulcd commented Jun 7, 2023

@kou @assignUser at this point I'm not sure what else to clean up, I've removed more than 2GB and I still get a No space left on device.
I'm thinking on adding a bigger self-hosted AMD 64 on crossbow just for the Linux packaging jobs at least temporarily for the release.
What are your thoughts?

@raulcd
Copy link
Member Author

raulcd commented Jun 7, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 7, 2023

Revision: 26ac7c7

Submitted crossbow builds: ursacomputing/crossbow @ actions-12c86c6b16

Task Status
ubuntu-jammy-amd64 Github Actions

@raulcd
Copy link
Member Author

raulcd commented Jun 7, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 7, 2023

Revision: 56341df

Submitted crossbow builds: ursacomputing/crossbow @ actions-bf96eec3a5

Task Status
ubuntu-jammy-amd64 Github Actions

@raulcd
Copy link
Member Author

raulcd commented Jun 7, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 7, 2023

Revision: 27be074

Submitted crossbow builds: ursacomputing/crossbow @ actions-5e44b7a2f2

Task Status
ubuntu-jammy-amd64 Github Actions

@raulcd
Copy link
Member Author

raulcd commented Jun 7, 2023

@github-actions crossbow submit ubuntu-jammy-amd64

@github-actions
Copy link

github-actions bot commented Jun 7, 2023

Revision: 5dcf34d

Submitted crossbow builds: ursacomputing/crossbow @ actions-4baff69314

Task Status
ubuntu-jammy-amd64 Github Actions

@kou
Copy link
Member

kou commented Jun 7, 2023

Short term workarounds:

  1. Use self-hosted runner like arm64
  2. Disable some large components such as S3 and GCP

Long term solution:

  1. Build deb packages for large dependencies such as aws-sdk-cpp and google-cloud-cpp separately and use them to build our deb packages

    FYI: https://github.com/datafusion-contrib/datafusion-c/ provides deb packages at https://apache.jfrog.io/ui/native/arrow/ubuntu/pool/jammy/main/d/datafusion-c/ . Apache Arrow C++ dependencies can use the same approach.

@kou
Copy link
Member

kou commented Jun 7, 2023

One more idea: #35961

@raulcd
Copy link
Member Author

raulcd commented Jun 7, 2023

@kou It seems that after removing 3.5GB it is able to build again, we probably can merge this as a temporary fix and open a new issue to tackle some of the long term improvements you proposed?

@kou
Copy link
Member

kou commented Jun 7, 2023

Yes!

@raulcd
Copy link
Member Author

raulcd commented Jun 7, 2023

I've created the issue to tackle the longer term solution: #35964 and will merge this.

Copy link
Member Author

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jun 7, 2023
@raulcd raulcd merged commit 9630258 into apache:main Jun 7, 2023
7 checks passed
raulcd added a commit that referenced this pull request Jun 7, 2023
…35947)

### Rationale for this change
Fix some Linux packaging jobs that fail due to out of space

### What changes are included in this PR?
Cleaning up some more cache

### Are these changes tested?
With crossbow jobs

### Are there any user-facing changes?

No
* Closes: #35946

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
@ursabot
Copy link

ursabot commented Jun 8, 2023

Benchmark runs are scheduled for baseline = 1d75816 and contender = 9630258. 9630258 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed] ec2-t3-xlarge-us-east-2
[Finished ⬇️2.69% ⬆️0.18%] test-mac-arm
[Finished ⬇️0.33% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.48% ⬆️0.06%] ursa-thinkcentre-m75q
Buildkite builds:
[Failed] 96302584 ec2-t3-xlarge-us-east-2
[Finished] 96302584 test-mac-arm
[Finished] 96302584 ursa-i9-9960x
[Finished] 96302584 ursa-thinkcentre-m75q
[Finished] 1d758162 ec2-t3-xlarge-us-east-2
[Finished] 1d758162 test-mac-arm
[Finished] 1d758162 ursa-i9-9960x
[Finished] 1d758162 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting changes Awaiting changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CI][Packaging] ubuntu-jammy packaging job fails due to out of space
3 participants