fix(sdk): DSPX-4590 zip64 conformance

[dmihalcik-virtru]

Jira: https://virtru.atlassian.net/browse/DSPX-4590

Part 3 of a 3-PR stack: #3933 (base) <- #3979 <- this PR.

Based on #3979 (default per-segment sizes when a writer omits them), which
is itself based on #3933. This PR carries findings 1-6 of the DSPX-4590
investigation (go-sdk's ZIP64/APPNOTE conformance issues); finding 7
(per-segment size defaults) is fixed in #3979 below it and is unrelated to
these changes -- the two were originally one PR (#3967) and are split here
so each can be reviewed independently.

go-sdk's zip layer disagrees with the other SDKs in a handful of places,
and this addresses all six.

Finding 1 (interop): writer switched to ZIP64 at 4 GiB instead of 2 GiB

Finalize compared against ^uint32(0), so a payload between 2 GiB and 4 GiB
was written as a zip32 archive with a value in the top half of the unsigned
32-bit range. java-sdk widens those central-directory fields signed, so
deployed Java clients read the size/offset back as a negative number and cannot
open the container. web-sdk always writes ZIP64, java-sdk (since java-sdk#393)
switches at Integer.MAX_VALUE.

• New maxNonZip64Value = math.MaxInt32 in zip_primitives.go, mirroring
  java-sdk's MAX_NON_ZIP64_VALUE.
• The rule is applied to the uncompressed size, the compressed size and
  entry.Offset (the local-header offset), which was previously not checked
  at all -- an archive under 2 GiB of payload could still place a later
  entry's header past the boundary.
• The threshold is injectable: Config.MaxNonZip64Value plus a
  WithMaxNonZip64Value option (clamped to (0, maxNonZip64Value]), so the
  tests can drive the ZIP64 path with a 1 KiB threshold instead of
  allocating gigabytes.

Finding 2: ZIP64 extra field parsed positionally

The reader assumed the ZIP64 extended-information field was first in the
extra-field area and that all three values were always present. Per APPNOTE
4.5.3 the values appear in the order original size, compressed size, local
header offset, and each is present only when its central-directory
counterpart holds the 0xFFFFFFFF sentinel. parseZip64ExtraField now walks
the whole extra area, skips foreign tags, reads values in spec order gated on
the sentinels, and rejects a field that claims to run past the end of the
area.

Finding 3: ZIP64 detected from the CD offset alone

NewReader only looked at CentralDirectoryOffset == 0xFFFFFFFF. An archive
that overflows the entry count (0xFFFF) or the central-directory size but
not the offset was read as zip32. eocdNeedsZip64 now checks all three EOCD
sentinel fields.

Finding 4: per-entry ZIP64 extra field ignored without a ZIP64 EOCD

A writer may put a ZIP64 extra field on an individual entry while leaving
the EOCD in zip32 form. The reader now consults the extra field whenever the
central-directory header carries a sentinel, independent of the EOCD form.

Finding 5: central-directory cursor could wrap at uint16

nextCD was advanced with uint16 arithmetic and did not include the file
comment. A 65000-byte filename plus a 600-byte extra field wraps to 110 and
the reader walks into the middle of a header. The advance is now done in
uint64 and includes FileCommentLength.

Finding 6: silent truncation when a value does not fit 32 bits

The zip32 paths narrowed with a bare cast. checkFitsInCentralDirectory now
returns a new ErrFieldOverflow for the compressed size, uncompressed size,
local-header offset, central-directory size/offset and entry count instead
of writing a corrupt archive. (With finding 1 in place this is unreachable
in normal operation; it is a backstop against future callers.)

Tests

sdk/internal/zipstream/zip64_conformance_test.go -- hand-assembles raw zip
archives (buildRawZip) so the reader can be pointed at containers no Go
writer would produce: extra field not first, differing compressed and
uncompressed sizes so the APPNOTE 4.5.3 ordering is actually asserted, per-
entry ZIP64 under a zip32 EOCD, a central-directory file comment, the
65646-byte name+extra case that wraps to 110 at uint16, ZIP64 implied by the
entry count, and a malformed extra field. Writer side:
TestWriterSwitchesToZip64AtInjectedThreshold uses the injected 1 KiB
threshold, TestEntryNeedsZip64AtTwoGiB pins maxNonZip64Value == math.MaxInt32 and covers size / compressed size / offset,
TestCentralDirectoryNarrowingGuard covers finding 6.
sdk/internal/zipstream/fuzz_test.go -- two new seeds for the nextCD
overflow and the file-comment case.

Follow-up required in opentdf/tests (NOT covered by this PR)

The xtest cell test_tdfs.py::test_chunky_roundtrip currently SKIPS
for go, because xtest/sdk/go/cli.sh answers no to supports chunky.
That shim lives in the opentdf/tests repo, so merging this stack does
not flip it -- the go column will stay skipped and the interop
regression will stay invisible in CI. When this fix ships in a release,
someone needs to version-gate the chunky) case in xtest/sdk/go/cli.sh
so it reports support at or above that version.

Testing

• cd sdk && go test ./... -race
• make fmt, make lint (0 new issues)

Supersedes the zip64 portion of #3967, which is left open, unmodified, for
reference.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	153.769214ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	78.340304ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	269.885813ms
Throughput	370.53 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	35.476608296s
Average Latency	354.023502ms
Throughput	140.94 requests/second

[github-actions]

X-Test Failure Report

opentdfplatformUW0E3Q.dockerbuild
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	147.005046ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	87.962554ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	283.454448ms
Throughput	352.79 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	39.771002646s
Average Latency	396.829653ms
Throughput	125.72 requests/second

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	221.215411ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	120.793391ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	437.020703ms
Throughput	228.82 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	56.487553578s
Average Latency	563.647924ms
Throughput	88.52 requests/second

[github-actions]

X-Test Results

✅ go@pull-3981-main
✅ go@pull-3981-pull-3981
✅ go@main-pull-3981
✅ go@main-main
✅ go@pull-3981-v0.9.0
bats-test-results
✅ go@main-v0.9.0
✅ java@main-main
✅ js@main-pull-3981
✅ java@main-v0.9.0
✅ java@main-pull-3981
cukes-report
✅ js@main-v0.9.0
✅ js@main-main
opentdfplatformK8ECVP.dockerbuild
govulncheck-failure-8
govulncheck-failure-1
govulncheck-failure-3

[github-actions]

X-Test Failure Report

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	226.380952ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	124.051175ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	457.231154ms
Throughput	218.71 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	59.360582491s
Average Latency	592.476556ms
Throughput	84.23 requests/second

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	195.744576ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	103.907686ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	367.41732ms
Throughput	272.17 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	51.529894041s
Average Latency	514.155542ms
Throughput	97.03 requests/second

[github-actions]

X-Test Failure Report

bats-test-results
govulncheck-failure-8
opentdfplatformQD354O.dockerbuild
govulncheck-failure-1
govulncheck-failure-3

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	219.251682ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	121.475667ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	446.480946ms
Throughput	223.97 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	55.794171102s
Average Latency	556.648353ms
Throughput	89.62 requests/second

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	253.937424ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	138.927697ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	442.519697ms
Throughput	225.98 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	1m1.935929654s
Average Latency	617.897325ms
Throughput	80.73 requests/second

[github-actions]

⚠️ Govulncheck found vulnerabilities ⚠️

The following modules have known vulnerabilities:

• otdfctl
• service
• tests-bdd

See the workflow run for details.

[github-actions]

X-Test Results

✅ go@pull-3981-main
✅ go@main-pull-3981
✅ go@pull-3981-pull-3981
✅ go@main-main
✅ go@pull-3981-v0.9.0
✅ go@main-v0.9.0
✅ java@main-pull-3981
bats-test-results
✅ java@main-v0.9.0
✅ js@main-pull-3981
✅ js@main-main
✅ java@main-main
✅ js@main-v0.9.0
cukes-report
opentdfplatformM6Y0FD.dockerbuild
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1