Skip to content

[hf-xet v1.7.1]: Improved dedupe in long upload session

Choose a tag to compare

@github-actions github-actions released this 28 Oct 15:06
f466508

Improvements and bug fixes

  • Fix a bug that dedup silently stops once the in-memory chunk index reaches size limit, this bug previously lead a long upload session by MicroAGI to slowdown dramatically.
  • Fix a bug that commit and download-group construction errors, such as a token-refresh 403 (bad CAS token refresh URL or providing a HF_TOKEN with nonmatching access), cause all subsequent group constructions to fail.
  • Respect HF_XET_MIN_SPACING_BETWEEN_GLOBAL_DEDUP_QUERIES (aliased as HF_XET_DATA_MIN_SPACING_BETWEEN_GLOBAL_DEDUP_QUERIES) that controls minimum number of chunks to wait for between generating global dedup queries.
  • Simulation and comment fixes, dependency bumps.

Full Changelog: v1.7.0...v1.7.1