CI test flake: failed to parse manifest while downloading crates

[ispivey]

Run cargo test --features "strict"
  cargo test --features "strict"
  shell: /bin/bash -e {0}
  env:
    RUST_BACKTRACE: 1
    Updating crates.io index
 Downloading crates ...
  Downloaded byte-tools v0.3.1
  Downloaded config v0.10.1
  Downloaded crossbeam-utils v0.7.2
  Downloaded crc32fast v1.2.0
  Downloaded proc-macro-hack v0.5.16
  Downloaded semver-parser v0.7.0
  Downloaded siphasher v0.3.3
  Downloaded sloggers v0.3.6
  Downloaded take_mut v0.2.2
  Downloaded tokio-tls v0.3.1
  Downloaded treeline v0.1.0
  Downloaded zip v0.5.5
  Downloaded which v3.1.1
  Downloaded ws v0.9.1
  Downloaded rand v0.7.3
  Downloaded filetime v0.2.10
  Downloaded url v2.1.1
  Downloaded linked-hash-map v0.3.0
  Downloaded futures-task v0.3.5
  Downloaded mio v0.6.22
  Downloaded futures-macro v0.3.5
  Downloaded nom v5.1.2
  Downloaded reqwest v0.10.6
  Downloaded mime_guess v2.0.3
  Downloaded serde_json v1.0.55
  Downloaded hyper v0.13.6
error: failed to parse manifest at `/Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.13.6/Cargo.toml`

Caused by:
  readme file with name 'README.md' was not found
##[error]Process completed with exit code 101.

See build log at: https://github.com/cloudflare/wrangler/pull/1379/checks?check_run_id=765543689

I haven't been able to reproduce it, but want to start documenting flaky CI failures (in case there are multiple root causes) so we can knock 'em down one by one.

[ispivey]

Re-ran and hit a different error that feels like the same issue – being unable to fetch stuff when downloading crates: https://github.com/cloudflare/wrangler/pull/1379/checks?check_run_id=765614864

2s
    RUST_BACKTRACE: 1
Run cargo test --features "strict"
  cargo test --features "strict"
  shell: /bin/bash -e {0}
  env:
    RUST_BACKTRACE: 1
    Updating crates.io index
 Downloading crates ...
error: failed to download `futures v0.3.5`

Caused by:
  unable to get packages from source

Caused by:
  failed to parse manifest at `/home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.3.5/Cargo.toml`

Caused by:
  readme file with name '../README.md' was not found
##[error]Process completed with exit code 101.

[ispivey]

One of the things I've got so far is, we're likely caching bad or partially downloaded artifacts. The 1.10.0 PR failed a few times in a row:

https://github.com/cloudflare/wrangler/pull/1379/checks?check_run_id=765784577

The failure happens when downloading and parsing crates:

Run cargo test --features "strict"
  cargo test --features "strict"
  shell: /bin/bash -e {0}
  env:
    RUST_BACKTRACE: 1
    Updating crates.io index
 Downloading crates ...
error: failed to download `futures v0.3.5`

Caused by:
  unable to get packages from source

Caused by:
  failed to parse manifest at `/home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.3.5/Cargo.toml`

Caused by:
  readme file with name '../README.md' was not found
##[error]Process completed with exit code 101.

The entire .cargo/registry directory is being cached – I notice in the "Cache Cargo Registry" step, there's always a cache hit:

5s
  
Run actions/cache@v1
  with:
    path: ~/.cargo/registry
    key: linux-nightly-cargo-registry-a760e20aa0e2940b31f09359b8dd574b861f607b7f869da88e477c2f8f4b6419
    restore-keys: linux-nightly-cargo-registry-
  
Cache Size: ~96 MB (100211002 B)
/bin/tar -xz -f /home/runner/work/_temp/5f6fc1e7-6967-474e-a965-b725340792c8/cache.tgz -C /home/runner/.cargo/registry
Cache restored from key: linux-nightly-cargo-registry-5cdeb281705989d9d9fc0751e1c6f80741a5507d8d320fe92f62d328bbefb001

This makes sense – we specify:

          restore-keys: |
            ${{ matrix.build }}-cargo-registry-

which means we'll "match" on any cache key that matches that prefix, so pretty much any cached registry in the past whatever amount of time. That might explain why once the failures start, they recur a bunch of times in a row, until our cache is expired.

[ispivey]

I tried some old-fashioned cache-busting (changing the key name) on another PR that was failing:

https://github.com/cloudflare/wrangler/pull/1381/checks?check_run_id=765809720

Run actions/cache@v2
  with:
    path: target
    key: -cargo-target-dir-blorp-8d69840ab92ea7f4d323420088dd8c9775f180cd-5cdeb281705989d9d9fc0751e1c6f80741a5507d8d320fe92f62d328bbefb001
  env:
    TARGET: 
    SKIP_TESTS: 
Cache not found for input keys: -cargo-target-dir-blorp-8d69840ab92ea7f4d323420088dd8c9775f180cd-5cdeb281705989d9d9fc0751e1c6f80741a5507d8d320fe92f62d328bbefb001

Neat, no cache hit.

Run cargo test --features "strict"
  cargo test --features "strict"
  shell: /bin/bash -e {0}
  env:
    TARGET: 
    SKIP_TESTS: 
    RUST_BACKTRACE: 1
    Updating crates.io index
 Downloading crates ...
error: failed to download `nom v5.1.1`

Caused by:
  unable to get packages from source

Caused by:
  failed to parse manifest at `/home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/nom-5.1.1/Cargo.toml`

Caused by:
  readme file with name 'README.md' was not found
##[error]Process completed with exit code 101.

Still failing to download/parse.

[ashleymichal]

any chance we're being rate limited by crates.io?

[ispivey]

Actually, it's not supposed to cache if the build fails; should probably test that.

[ispivey]

hrm rust-lang/cargo#8354

[ispivey]

hrm rust-lang/cargo#8351