Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[contrib] largeNbDicts bugfix + improvements #3161

Merged
merged 4 commits into from
Jun 15, 2022

Conversation

embg
Copy link
Contributor

@embg embg commented Jun 14, 2022

These are local changes I've been cherry-picking on builds for a while. Figured it's time to put them on GitHub.

  • Fixes a bug in the memory estimation found by @terrelln
  • Fixes a bug which prevented forceAttach/forceCopy from working properly (ZSTD_compress_usingCDict -> ZSTD_compress2)
  • Prints results to a csv file with the executable name
    • The idea is you bake the commit hash and compiler into the executable name. (This is my setup).
    • largeNbDicts will append the best speed to the CSV file, so you can run multiple times and then get statistics. I like to interleave runs between different versions to correct for background load changes.
    • I have a benchmark runner script which creates a directory tree of CSV files for different scenarios, and then a python script which aggregates all the statistics. Will put them on GitHub when I have time :)

@embg embg force-pushed the largeNbDictsImprovements branch from a4ca296 to e0c4863 Compare June 14, 2022 00:27
@embg embg changed the title largeNbDicts bugfix + improvements [contrib] largeNbDicts bugfix + improvements Jun 14, 2022
@embg embg requested a review from terrelln June 14, 2022 21:53
@embg
Copy link
Contributor Author

embg commented Jun 14, 2022

  • Fixed FILE handle leak
  • Fixed a bug which prevented forceAttach/forceCopy from working properly (ZSTD_compress_usingCDict -> ZSTD_compress2).

Co-authored-by: Nick Terrell <nickrterrell@gmail.com>
@embg embg merged commit b7b7edb into facebook:dev Jun 15, 2022
@embg embg deleted the largeNbDictsImprovements branch June 15, 2022 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants