Receiving "stream did not contain valid UTF-8" errors in recent builds when unpacking

[milesj]

In a recent release, I think this one https://github.com/astral-sh/python-build-standalone/releases/tag/20251007, since it started triggering with Python 3.14.0, the prebuilts now error with "stream did not contain valid UTF-8" when attempting to unpack.

More info: moonrepo/plugins#87

For unpacking, I'm using the tar and zstd crates: https://github.com/moonrepo/starbase/blob/master/crates/archive/src/tar.rs#L176 I've tried different versions of these crates with no luck.

Did something change recently in the build process that could have possibly introduced this?

[zanieb]

Does it occur every time or is it spurious?

[milesj]

Every time. Also seems to happen for many (maybe all) users.

[zanieb]

Specifically, you're using the -full archives? Have you tried with the -install-only archives?

If helpful, here's our archive extraction code https://github.com/astral-sh/uv/blob/2a14edf75c56fc7372a9ab9aa346dd217cab3e3f/crates/uv-extract/src/stream.rs#L747-L779

[zanieb]

I confirmed uv is also not failing on the -full archives, e.g., https://github.com/astral-sh/python-build-standalone/releases/download/20251014/cpython-3.15.0a1%2B20251014-aarch64-apple-darwin-freethreaded%2Bpgo%2Blto-full.tar.zst

[zanieb]

I can't see what would have changed on our end?

[milesj]

Yeah it's weird because nothing has changed on our end in a while either. Let me test with the install only builds.

[geofft]

Testing starbase_archive in isolation on Linux seems happy:

[package]
name = "starb"
version = "0.1.0"
edition = "2024"

[dependencies]
starbase_archive = { version = "0.11.8", features = ["tar-zstd"] }

use starbase_archive::{tar::*, *};
use std::path::Path;

fn main() {
    let mut t = TarUnpacker::new_zstd(
        Path::new("/tmp/starboard"),
        Path::new("/tmp/cpython-3.14.0+20251010-aarch64-apple-darwin-pgo+lto-full.tar.zst"),
    )
    .unwrap();
    t.unpack(
        "",
        &mut TreeDiffer {
            files: Default::default(),
        },
    )
    .unwrap();
}

works fine. This is with binstall-tar 0.4.42 and zstd 0.13.3.

Could you get us a more detailed Rust traceback from that error message? What exactly is the stream that it's unhappy about?

[indygreg]

This smells like a corrupted file on a CDN or caching proxy.

Or a client somewhere not following the HTTP 3xx redirects issued by GitHub release assets.

We should compare SHA-256 digests of specific URLs.

[milesj]

So the install_only builds resolved this problem for the time being. I haven't had a chance to debug further into the failing ones.