Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compress the languages data #468

Merged
merged 1 commit into from
May 27, 2024
Merged

Conversation

apocelipes
Copy link
Contributor

There is a lot of repeated content in languages.json, so using gzip compression can
significantly reduce the size of both the generated Go file and the compiled binary file.
A small binary file can use less momory and have a better cache hit rate.

On my windows machine with go1.22, it shows:

  • "processor/constants.go": 170k -> 20k, 150k reduced.
  • scc binary file: 8909k -> 8836.5k, 72.5k reduced.

Gzip is very fast, as hyperfine reports there's nearly no harm on the performance:

$ cd scc
$ hyperfine --warmup 5 'scc.exe .' 'scc-nocompress.exe .'
Benchmark 1: scc.exe .
  Time (mean ± σ):      59.0 ms ±   2.6 ms    [User: 9.0 ms, System: 18.2 ms]
  Range (min … max):    55.6 ms …  65.3 ms    45 runs

Benchmark 2: scc-nocompress.exe .
  Time (mean ± σ):      59.2 ms ±   2.9 ms    [User: 8.8 ms, System: 17.8 ms]
  Range (min … max):    55.5 ms …  66.5 ms    46 runs

There is a lot of repeated content in languages.json, so using
gzip compression can significantly reduce the size of the compiled
binary file.

On my windows machine with go1.22, it shows:

- "processor/constants.go": 170k -> 20k, 150k reduced.
- scc binary file: 8909k -> 8836.5k, 72.5k reduced.

Gzip is very fast, as hyperfine reports there's nearly no harm on
the performance.
@boyter
Copy link
Owner

boyter commented May 27, 2024

This is very interesting. Not the biggest saving, but having scc on a diet isnt a bad idea.

Going to merge but possibly revert if I notice any issues in the future or slowdown.

@boyter boyter merged commit 4144d52 into boyter:master May 27, 2024
4 of 5 checks passed
@apocelipes apocelipes deleted the feat-compress branch July 23, 2024 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants