Fix tar upload content-length mismatch (#1086)

[noamzbr]

Fixes #1086

Use portable mode for deterministic gzip output

[changeset-bot]

⚠️ No Changeset found

Latest commit: e8db0f6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2be4177c0c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[mishushakov]

I don't think we can do it this due to how hashes are calculated. Is there another way to ensure these are same?
Maybe pipe the stream to two readers at the same time?

[noamzbr]

Note I've fixed it from tar portable: true to gzip: { portable: true }. With gzip: { portable: true } - we expect only gzip wrapper headers to be tweaked, so the fileHash should be identical.

But sure, if you prefer avoiding changing the archive mechanism entirely - can you tell me more about what you had in mind? If the presigned PUT requires Content-Length (SDK currently assumes it does), we can’t really stream once to upload + counter because headers must be sent before the body. Did you mean switching the upload to chunked (no Content-Length), making the two-pass tar generation stable (e.g. normalize/strip volatile pax atime/ctime via onWriteEntry) so both passes produce identical bytes. or something else?

[mishushakov]

Okay, I see.
What we require is that the files inside the tar keep same permissions as the source.

As far as I understand with gzip: { portable: true } the contents of the tar should be unchanged?

For the upload of the files we are using pre-signed URLs that require Content-Length to be specified before the upload. My initial idea was to have 2 streams - one for counting the archive length and another for actual upload.

[noamzbr]

Okay, I see. What we require is that the files inside the tar keep same permissions as the source.

As far as I understand with gzip: { portable: true } the contents of the tar should be unchanged?

For the upload of the files we are using pre-signed URLs that require Content-Length to be specified before the upload. My initial idea was to have 2 streams - one for counting the archive length and another for actual upload.

yes, gzip: { portable: true } will not change file modes (permissions)

[mishushakov]

sounds good, did you verify this fixes #1086 for you?

[noamzbr]

sounds good, did you verify this fixes #1086 for you?

yes, being using it for around a couple of days now

[mishushakov]

Thanks for your contribution, we merged it here #1095