perf: parse headers in blocks and scan for magic numbers with memchr

[cosmicexplorer]

Motivation

Parsing in Blocks

Aesthetics

We currently parse the zip binary format with a series of e.g. .read_u32_le() calls. While this does work, I believe this approach obscures some of the control flow and makes it more difficult to correlate to the zip specification.

Performance

Additionally, when reading a zip file from a non-buffered stream, I believe a series of individual reads may take longer than a single bulk read (I have not verified this).

Scanning with memchr

We currently perform an extremely inefficient byte-by-byte loop to locate central directory records, e.g.:

zip2/src/spec.rs

Lines 66 to 80 in 103f1ec

	while pos >= search_upper_bound {
	reader.seek(io::SeekFrom::Start(pos))?;
	if reader.read_u32_le()? == CENTRAL_DIRECTORY_END_SIGNATURE {
	reader.seek(io::SeekFrom::Current(
	BYTES_BETWEEN_MAGIC_AND_COMMENT_SIZE as i64,
	))?;
	let cde_start_pos = reader.seek(io::SeekFrom::Start(pos))?;
	if let Ok(end_header) = CentralDirectoryEnd::parse(reader) {
	return Ok((end_header, cde_start_pos));
	}
	}
	pos = match pos.checked_sub(1) {
	Some(p) => p,
	None => break,
	};

We can lean on string scanning utilities like memchr in order to speed up this loop that occurs whenever we read a zip file!

Implementation

• Create *Block structs declared with #[repr(packed)] so we can extract metadata using bulk reads.
• Create a Block trait with several utility methods for the different types of "block" objects introduced in this PR.
• Use memchr to scan for central directory magic numbers.
• Add benchmarks to demonstrate the speedup when reading zip files with large comment sections.

Result

We get a very noticeable speedup in the new benchmarks!

> git checkout HEAD~1
> cargo bench -- parse
...
running 2 tests
test parse_comment       ... bench:      42,446 ns/iter (+/- 704)
test parse_zip64_comment ... bench:     400,349 ns/iter (+/- 16,882)
> git checkout -
> cargo bench -- parse
...
running 2 tests
test parse_comment       ... bench:      15,307 ns/iter (+/- 682)
test parse_zip64_comment ... bench:     153,766 ns/iter (+/- 9,240)

Note that the benchmarks have been renamed upon review, so the output looks more like this:

test parse_archive_with_comment       ... bench:      16,403 ns/iter (+/- 1,177) = 3055 MB/s
test parse_zip64_archive_with_comment ... bench:     167,784 ns/iter (+/- 7,847) = 2980 MB/s

[Pr0methean]

Thanks for the update; but I see a number of my comments from earlier are still unresolved. Any clue how soon you'll be able to address them?

[cosmicexplorer]

I've since refactored this to cover your comments. The current implementation is not quite complete, but please let me know if this is easier to follow.

[Pr0methean]

Please feel free to revert cc4fe7f if it helps, as long as if you delete changes then you add them back in a merge PR.

[cosmicexplorer]

@Pr0methean:

as long as if you delete changes then you add them back in a merge PR.

I just rebased against master and force pushed, which deleted your merge history. I believe the merge commit was just a bump against master, and not any additional work you added on top of my changes, so I'm not missing anything at fa2c8e1? Just making sure!