Repository navigation
[hf-xet v1.7.1]: Improved dedupe in long upload session
Improvements and bug fixes
- Fix a bug that dedup silently stops once the in-memory chunk index reaches size limit, this bug previously lead a long upload session by MicroAGI to slowdown dramatically.
- Fix a bug that commit and download-group construction errors, such as a token-refresh 403 (bad CAS token refresh URL or providing a HF_TOKEN with nonmatching access), cause all subsequent group constructions to fail.
- Respect
HF_XET_MIN_SPACING_BETWEEN_GLOBAL_DEDUP_QUERIES(aliased asHF_XET_DATA_MIN_SPACING_BETWEEN_GLOBAL_DEDUP_QUERIES) that controls minimum number of chunks to wait for between generating global dedup queries. - Simulation and comment fixes, dependency bumps.
Full Changelog: v1.7.0...v1.7.1