Skip to content

fix(shasum): version 1.1.5 - #1559

Merged
tcely merged 1 commit into
mainfrom
tcely-shasum-unbuffered
Aug 22, 2026
Merged

fix(shasum): version 1.1.5#1559
tcely merged 1 commit into
mainfrom
tcely-shasum-unbuffered

Conversation

@tcely

@tcely tcely commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator
  • use unbuffered reads into a memory view
  • cast the memory view for faster operations
  • process buffered files first
  • switch the file_buffer_pool to queue.SimpleQueue
  • reduce the number of tasks sent to ThreadPoolExecutor
  • translate Windows path separators

Previous: #1558

@tcely tcely self-assigned this Aug 20, 2026
@tcely
tcely marked this pull request as ready for review August 20, 2026 13:43
@tcely tcely moved this to Todo in Status Aug 20, 2026
@tcely
tcely force-pushed the tcely-shasum-unbuffered branch 8 times, most recently from 8a1d846 to 602e72e Compare August 20, 2026 22:43
- use unbuffered reads into a memory view
- cast the memory view for faster operations
- process buffered files first
- switch the file_buffer_pool to `queue.SimpleQueue`
- reduce the number of tasks sent to `ThreadPoolExecutor`
- translate Windows path separators
@tcely
tcely force-pushed the tcely-shasum-unbuffered branch from 602e72e to fbb4fb0 Compare August 20, 2026 23:08
@tcely
tcely requested a review from meeb August 22, 2026 16:18
@tcely tcely moved this from Todo to In Progress in Status Aug 22, 2026
@github-project-automation github-project-automation Bot moved this from In Progress to Ready in Status Aug 22, 2026
@meeb

meeb commented Aug 22, 2026

Copy link
Copy Markdown
Owner

You just fully re-implementing shasum in Python?

@tcely

tcely commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator Author

I gave up on the variations between busybox and coreutils and whatever windows/mac are using in various versions a while ago. Since I had decided to implement a verification tool in Python, I then gave it the default behaviors that I wanted with some minimal speed / consistency improvements.

These latest changes are about speeding things up and accepting manifest files containing Windows paths.

The performance across multiple files has become quite good. I'd consider switching to this tool to be a win for anyone who has to verify files.

v1.1.3

Ran 26 tests in 1.065s

OK

--- Stress Testing shasum v1.1.3 ---

   [Perf] Generating 100.0 MiB of test data...
   [Perf] Verified 100 files (100.0 MiB)
   [Perf] Total Time: 0.0769 seconds
   [Perf] Throughput: 1301.23 MiB/s

SUCCESS: All 26 nightmare cases passed.

v1.1.4

Ran 26 tests in 0.729s

OK

--- Stress Testing shasum v1.1.4 ---

   [Perf] Generating 100.0 MiB of test data...
   [Perf] Verified 100 files (100.0 MiB)
   [Perf] Total Time: 0.0930 seconds
   [Perf] Throughput: 1075.38 MiB/s

SUCCESS: All 26 nightmare cases passed.

v1.1.5

Ran 26 tests in 1.001s

OK

--- Stress Testing shasum v1.1.5 ---

   [Perf] Generating 100.0 MiB of test data...
   [Perf] Verified 100 files (100.0 MiB)
   [Perf] Total Time: 0.0756 seconds
   [Perf] Throughput: 1322.99 MiB/s

SUCCESS: All 26 nightmare cases passed.

@tcely
tcely merged commit bec4106 into main Aug 22, 2026
7 checks passed
@tcely
tcely deleted the tcely-shasum-unbuffered branch August 22, 2026 16:56
@github-project-automation github-project-automation Bot moved this from Ready to Done in Status Aug 22, 2026
@meeb

meeb commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Thanks! The implementation looks solid and the reasoning is sound. I was mostly just curious.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants