Add forward-only ZIP stream reader (ZipStreamReader) reusing ZipArchiveEntry#130400
Draft
alinpahontu2912 wants to merge 6 commits into
Draft
Add forward-only ZIP stream reader (ZipStreamReader) reusing ZipArchiveEntry#130400alinpahontu2912 wants to merge 6 commits into
alinpahontu2912 wants to merge 6 commits into
Conversation
Add ZipStreamReader, a forward-only reader for ZIP archives that walks local file headers sequentially and decompresses on the fly, suitable for non-seekable sources (network streams, pipes). Rather than introducing a separate entry type, GetNextEntry(Async) return ZipArchiveEntry itself. A new internal constructor builds an entry from a local file header with no owning archive (Archive is null), and Open() returns a single-use forward-only data stream. Data-descriptor entries have their CRC and sizes populated after the data is drained. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…/zip-forward-read-archiveentry
Contributor
|
Tagging subscribers to this area: @karelz, @dotnet/area-system-io-compression |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new forward-only ZIP reader (ZipStreamReader) intended for sequential entry consumption from non-seekable sources, implemented by reusing ZipArchiveEntry as the returned entry type and extending internal ZIP stream helpers. The PR also adds a dedicated test suite and updates public reference surface to include the new reader.
Changes:
- Introduces
ZipStreamReader(public API) to enumerate ZIP local file headers sequentially and provide entry data streams. - Extends
ZipArchiveEntrywith an internal “forward-read” construction path and single-use stream behavior inOpen(). - Adds forward-read focused tests and wires them into the compression test project.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/libraries/System.IO.Compression/tests/ZipArchive/zip_StreamEntryReadTests.cs | New tests for forward-only enumeration, data descriptor behavior, copyData, and basic error paths. |
| src/libraries/System.IO.Compression/tests/System.IO.Compression.Tests.csproj | Includes the new zip_StreamEntryReadTests.cs in the test build. |
| src/libraries/System.IO.Compression/src/System/IO/Compression/ZipStreamReader.cs | Implements the forward-only reader that parses local headers and creates entry data streams. |
| src/libraries/System.IO.Compression/src/System/IO/Compression/ZipCustomStreams.cs | Adds BoundedReadOnlyStream and ReadAheadStream; exposes CRC tracking needed by ZipStreamReader. |
| src/libraries/System.IO.Compression/src/System/IO/Compression/ZipArchiveEntry.cs | Adds internal constructor + state for forward-only entries; routes Open()/Open(FileAccess) to a forward-read stream. |
| src/libraries/System.IO.Compression/src/System.IO.Compression.csproj | Adds ZipStreamReader.cs to compilation; also introduces a UTF-8 BOM on the first line. |
| src/libraries/System.IO.Compression/src/Resources/Strings.resx | Adds new error strings for forward-only reader scenarios. |
| src/libraries/System.IO.Compression/ref/System.IO.Compression.cs | Adds ZipStreamReader to public ref surface. |
Comment on lines
464
to
+470
| public Stream Open() | ||
| { | ||
| if (_isForwardReadEntry) | ||
| { | ||
| return OpenForwardReadMode(); | ||
| } | ||
|
|
Comment on lines
+115
to
+124
| if (!headerBytes.AsSpan().StartsWith(ZipLocalFileHeader.SignatureConstantBytes)) | ||
| { | ||
| if (IsKnownEndOfEntriesSignature(headerBytes)) | ||
| { | ||
| _reachedEnd = true; | ||
| return null; | ||
| } | ||
|
|
||
| throw new InvalidDataException(SR.ZipStreamInvalidLocalFileHeader); | ||
| } |
Comment on lines
+211
to
+220
| if (!headerBytes.AsSpan().StartsWith(ZipLocalFileHeader.SignatureConstantBytes)) | ||
| { | ||
| if (IsKnownEndOfEntriesSignature(headerBytes)) | ||
| { | ||
| _reachedEnd = true; | ||
| return null; | ||
| } | ||
|
|
||
| throw new InvalidDataException(SR.ZipStreamInvalidLocalFileHeader); | ||
| } |
Comment on lines
+347
to
+351
| ZipArchiveEntry entry = _previousEntry; | ||
| _previousEntry = null; | ||
|
|
||
| DrainStream(entry.ForwardDataStream); | ||
|
|
Comment on lines
+40
to
+44
| Assert.Equal(ZipCompressionMethod.Deflate, entry.CompressionMethod); | ||
|
|
||
| Stream dataStream = entry.Open(); | ||
| byte[] decompressed = await ReadStreamFully(dataStream, async); | ||
| Assert.Equal(expectedContents[i], decompressed); |
| @@ -1,4 +1,4 @@ | |||
| <Project Sdk="Microsoft.NET.Sdk"> | |||
| <Project Sdk="Microsoft.NET.Sdk"> | |||
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
+109
to
+113
| if (bytesRead < ZipLocalFileHeader.SizeOfLocalHeader) | ||
| { | ||
| _reachedEnd = true; | ||
| return null; | ||
| } |
Comment on lines
+205
to
+209
| if (bytesRead < ZipLocalFileHeader.SizeOfLocalHeader) | ||
| { | ||
| _reachedEnd = true; | ||
| return null; | ||
| } |
Comment on lines
+491
to
+495
| // Probe: if 4 bytes after 32-bit sizes form a known ZIP signature, | ||
| // the descriptor uses 32-bit sizes; otherwise assume 64-bit. | ||
| bool isZip64 = !IsKnownZipSignature(buffer.AsSpan(totalRead + 8, 4)); | ||
| int sizesBytes = isZip64 ? 16 : 8; | ||
|
|
Comment on lines
+522
to
+526
| // Probe: if 4 bytes after 32-bit sizes form a known ZIP signature, | ||
| // the descriptor uses 32-bit sizes; otherwise assume 64-bit. | ||
| bool isZip64 = !IsKnownZipSignature(buffer.AsSpan(totalRead + 8, 4)); | ||
| int sizesBytes = isZip64 ? 16 : 8; | ||
|
|
Comment on lines
+359
to
+365
| DrainStream(entry.ForwardDataStream); | ||
|
|
||
| if (entry.HasDataDescriptor && entry.ForwardDataStream is CrcValidatingReadStream crcStream) | ||
| { | ||
| ReadDataDescriptor(entry, crcStream); | ||
| } | ||
| } |
| @@ -1,4 +1,4 @@ | |||
| <Project Sdk="Microsoft.NET.Sdk"> | |||
| <Project Sdk="Microsoft.NET.Sdk"> | |||
Comment on lines
+977
to
+979
| _forwardStreamOpened = true; | ||
| return _forwardDataStream; | ||
| } |
Comment on lines
+377
to
+382
| await DrainStreamAsync(entry.ForwardDataStream, cancellationToken).ConfigureAwait(false); | ||
|
|
||
| if (entry.HasDataDescriptor && entry.ForwardDataStream is CrcValidatingReadStream crcStream) | ||
| { | ||
| await ReadDataDescriptorAsync(entry, crcStream, cancellationToken).ConfigureAwait(false); | ||
| } |
| @@ -1,4 +1,4 @@ | |||
| <Project Sdk="Microsoft.NET.Sdk"> | |||
| <Project Sdk="Microsoft.NET.Sdk"> | |||
Comment on lines
+11
to
+14
| /// <summary> | ||
| /// Provides a forward-only reader for ZIP archives that reads entries sequentially | ||
| /// from a stream without requiring the stream to be seekable. | ||
| /// </summary> |
Comment on lines
+107
to
+113
| int bytesRead = _archiveStream.ReadAtLeast(headerBytes, headerBytes.Length, throwOnEndOfStream: false); | ||
|
|
||
| if (bytesRead < ZipLocalFileHeader.SizeOfLocalHeader) | ||
| { | ||
| _reachedEnd = true; | ||
| return null; | ||
| } |
Comment on lines
550
to
+555
| public Stream Open() | ||
| { | ||
| if (_isForwardReadEntry) | ||
| { | ||
| return OpenForwardReadMode(); | ||
| } |
3 tasks
Comment on lines
+588
to
+595
| if (isZip64) | ||
| { | ||
| entry.UpdateDataDescriptor( | ||
| crc32, | ||
| compressedLength: BinaryPrimitives.ReadInt64LittleEndian(buffer[sizesOffset..]), | ||
| length: BinaryPrimitives.ReadInt64LittleEndian(buffer[(sizesOffset + 8)..]), | ||
| crcStream.RunningCrc, crcStream.TotalBytesRead); | ||
| } |
Comment on lines
+145
to
+147
| int dynamicLength = GetDynamicHeaderLength(headerBytes); | ||
| byte[] dynamicBuffer = new byte[dynamicLength]; | ||
| _archiveStream.ReadExactly(dynamicBuffer); |
Comment on lines
+573
to
+576
| if (_isForwardReadEntry) | ||
| { | ||
| return OpenForwardReadMode(); | ||
| } |
Comment on lines
+135
to
+138
| if (_isForwardReadEntry) | ||
| { | ||
| return Task.FromResult<Stream>(OpenForwardReadMode()); | ||
| } |
Comment on lines
+235
to
+245
| _archive = null!; | ||
|
|
||
| _isForwardReadEntry = true; | ||
| _originallyInArchive = true; | ||
| _forwardDataStream = dataStream; | ||
|
|
||
| _diskNumberStart = 0; | ||
| _versionMadeByPlatform = CurrentZipPlatform; | ||
| _versionMadeBySpecification = ZipVersionNeededValues.Default; | ||
| _versionToExtract = (ZipVersionNeededValues)versionNeeded; | ||
| _generalPurposeBitFlag = (BitFlagValues)generalPurposeBitFlag; |
Comment on lines
22
to
+29
| public Task<Stream> OpenAsync(CancellationToken cancellationToken = default) | ||
| { | ||
| cancellationToken.ThrowIfCancellationRequested(); | ||
|
|
||
| if (_isForwardReadEntry) | ||
| { | ||
| return Task.FromResult<Stream>(OpenForwardReadMode()); | ||
| } |
| @@ -1,4 +1,4 @@ | |||
| <Project Sdk="Microsoft.NET.Sdk"> | |||
| <Project Sdk="Microsoft.NET.Sdk"> | |||
Comment on lines
+207
to
+215
| public sealed partial class ZipStreamReader : System.IAsyncDisposable, System.IDisposable | ||
| { | ||
| public ZipStreamReader(System.IO.Stream stream, bool leaveOpen = false) { } | ||
| public ZipStreamReader(System.IO.Stream stream, System.Text.Encoding? entryNameEncoding, bool leaveOpen = false) { } | ||
| public void Dispose() { } | ||
| public System.Threading.Tasks.ValueTask DisposeAsync() { throw null; } | ||
| public System.IO.Compression.ZipArchiveEntry? GetNextEntry(bool copyData = false) { throw null; } | ||
| public System.Threading.Tasks.ValueTask<System.IO.Compression.ZipArchiveEntry?> GetNextEntryAsync(bool copyData = false, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } | ||
| } |
Comment on lines
+492
to
+496
| private static async ValueTask<Stream> OpenEntry(ZipArchiveEntry entry, bool async) => | ||
| async ? await entry.OpenAsync() : entry.Open(); | ||
|
|
||
| private static async ValueTask<Stream> OpenEntry(ZipArchiveEntry entry, FileAccess access, bool async) => | ||
| async ? await entry.OpenAsync(access) : entry.Open(access); |
This was referenced Jul 14, 2026
Comment on lines
+207
to
+215
| public sealed partial class ZipStreamReader : System.IAsyncDisposable, System.IDisposable | ||
| { | ||
| public ZipStreamReader(System.IO.Stream stream, bool leaveOpen = false) { } | ||
| public ZipStreamReader(System.IO.Stream stream, System.Text.Encoding? entryNameEncoding, bool leaveOpen = false) { } | ||
| public void Dispose() { } | ||
| public System.Threading.Tasks.ValueTask DisposeAsync() { throw null; } | ||
| public System.IO.Compression.ZipArchiveEntry? GetNextEntry(bool copyData = false) { throw null; } | ||
| public System.Threading.Tasks.ValueTask<System.IO.Compression.ZipArchiveEntry?> GetNextEntryAsync(bool copyData = false, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) { throw null; } | ||
| } |
Comment on lines
+373
to
+381
| // Bounds forward reads to exactly compressedSize bytes so the decompressor (or the | ||
| // encrypted-drain path) cannot read past the entry into the next local file header. | ||
| // SubReadStream supports non-seekable super streams; anchoring the window at the | ||
| // current position keeps its internal position in lockstep with _archiveStream so it | ||
| // never issues a SeekOrigin.Begin (which ReadAheadStream does not support) on a | ||
| // non-seekable source. | ||
| private SubReadStream CreateBoundedStream(long compressedSize) | ||
| => new SubReadStream(_archiveStream, _archiveStream.Position, compressedSize); | ||
|
|
Comment on lines
+744
to
+769
| public void Dispose() | ||
| { | ||
| if (!_isDisposed) | ||
| { | ||
| _isDisposed = true; | ||
|
|
||
| if (!_leaveOpen) | ||
| { | ||
| _archiveStream.Dispose(); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| public async ValueTask DisposeAsync() | ||
| { | ||
| if (!_isDisposed) | ||
| { | ||
| _isDisposed = true; | ||
|
|
||
| if (!_leaveOpen) | ||
| { | ||
| await _archiveStream.DisposeAsync().ConfigureAwait(false); | ||
| } | ||
| } | ||
| } | ||
| } |
| stream.Write(contents); | ||
| } | ||
|
|
||
| private const string EncryptionPassword = "forward-read-secret"; |
| @@ -1,4 +1,4 @@ | |||
| <Project Sdk="Microsoft.NET.Sdk"> | |||
| <Project Sdk="Microsoft.NET.Sdk"> | |||
Comment on lines
+1298
to
+1301
| if (password.IsEmpty) | ||
| { | ||
| throw new InvalidDataException(SR.PasswordRequired); | ||
| } |
Comment on lines
+209
to
+212
| if (password.IsEmpty) | ||
| { | ||
| throw new InvalidDataException(SR.PasswordRequired); | ||
| } |
Comment on lines
+93
to
+97
| /// <param name="copyData"> | ||
| /// <see langword="true"/> to copy the entry's decompressed data into a <see cref="MemoryStream"/> | ||
| /// that remains valid after the reader advances; <see langword="false"/> to read directly | ||
| /// from the archive stream (invalidated on the next <see cref="GetNextEntry"/> call). | ||
| /// </param> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
ZipStreamReader, a forward-only reader for ZIP archives that walks local file headers sequentially and decompresses on the fly, suitable for non-seekable sources (network streams, pipes). Mirrors theTarReaderpattern.Rather than introducing a separate entry type,
GetNextEntry/GetNextEntryAsyncreturnZipArchiveEntryitself:Archiveisnull), reusing the already-legal post-Deletenull-archive state.Open()is extended (no signature change) to return a single-use, forward-only data stream for such entries.Notes for reviewers
upstream/mainat the time of authoring and may need a rebase.ZipStreamReader; the previously-prototypedZipForwardReadEntrytype was removed in favor of reusingZipArchiveEntry(smaller surface).System.IO.Compressionbuilds clean; full test suite passes (2048 cases, 0 failures), including 41 new forward-read cases.