-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Open
Labels
development-processRelated to development process of DataFusionRelated to development process of DataFusionenhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
We've faced issues with storage space on our CI runners, so we introduced steps to free up space for them across various actions:
datafusion/.github/workflows/extended.yml
Lines 89 to 113 in 302dbc8
| # Run extended tests (with feature 'extended_tests') | |
| linux-test-extended: | |
| name: cargo test 'extended_tests' (amd64) | |
| needs: [linux-build-lib] | |
| runs-on: ubuntu-latest | |
| # note: do not use amd/rust container to preserve disk space | |
| steps: | |
| - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0 | |
| with: | |
| ref: ${{ github.event.inputs.pr_head_sha }} # will be empty if triggered by push | |
| submodules: true | |
| fetch-depth: 1 | |
| - name: Free Disk Space (Ubuntu) | |
| uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1 | |
| - name: Install Rust | |
| run: | | |
| curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y | |
| source $HOME/.cargo/env | |
| rustup toolchain install | |
| - name: Install Protobuf Compiler | |
| run: sudo apt-get install -y protobuf-compiler | |
| # For debugging, test binaries can be large. | |
| - name: Show available disk space | |
| run: | | |
| df -h |
datafusion/.github/workflows/rust.yml
Lines 267 to 288 in 302dbc8
| # Library and integration tests | |
| linux-test: | |
| name: cargo test (amd64) | |
| needs: linux-build-lib | |
| runs-on: ubuntu-latest | |
| container: | |
| image: amd64/rust | |
| volumes: | |
| - /usr/local:/host/usr/local | |
| steps: | |
| - name: Remove unnecessary preinstalled software | |
| run: | | |
| echo "Disk space before cleanup:" | |
| df -h | |
| # remove tool cache: about 8.5GB (github has host /opt/hostedtoolcache mounted as /__t) | |
| rm -rf /__t/* || true | |
| # remove Haskell runtime: about 6.3GB (host /usr/local/.ghcup) | |
| rm -rf /host/usr/local/.ghcup || true | |
| # remove Android library: about 7.8GB (host /usr/local/lib/android) | |
| rm -rf /host/usr/local/lib/android || true | |
| echo "Disk space after cleanup:" | |
| df -h |
datafusion/.github/workflows/rust.yml
Lines 359 to 391 in 302dbc8
| linux-test-example: | |
| name: cargo examples (amd64) | |
| needs: linux-build-lib | |
| runs-on: ubuntu-latest | |
| container: | |
| image: amd64/rust | |
| steps: | |
| - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0 | |
| with: | |
| submodules: true | |
| fetch-depth: 1 | |
| - name: Setup Rust toolchain | |
| uses: ./.github/actions/setup-builder | |
| with: | |
| rust-version: stable | |
| - name: Rust Dependency Cache | |
| uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1 | |
| with: | |
| save-if: ${{ github.ref_name == 'main' }} | |
| shared-key: "amd-ci-linux-test-example" | |
| - name: Remove unnecessary preinstalled software | |
| run: | | |
| echo "Disk space before cleanup:" | |
| df -h | |
| apt-get clean | |
| rm -rf /__t/CodeQL | |
| rm -rf /__t/PyPy | |
| rm -rf /__t/Java_Temurin-Hotspot_jdk | |
| rm -rf /__t/Python | |
| rm -rf /__t/go | |
| rm -rf /__t/Ruby | |
| echo "Disk space after cleanup:" | |
| df -h |
Describe the solution you'd like
We should look into consolidating them together if possible so can apply to all our runners/actions.
Need to consider that some actions are on bare machine runners and others are on containers.
Describe alternatives you've considered
No response
Additional context
See #18709
alamb
Metadata
Metadata
Assignees
Labels
development-processRelated to development process of DataFusionRelated to development process of DataFusionenhancementNew feature or requestNew feature or request