v0.8.1 is a general clean up of the code base, following the stabilization of
There are a few welcomed evolutions and improvements, but for the most part, this release consists of fixes for multiple corner cases and scenarios, that shall improve usability of
xxhsum across a wide range of platforms.
API entry points have not changed, all entry points labelled "stable" will continue to work as intended in this release and future ones.
While the "big picture" is unchanged, there are a few notable improvements.
XXH128 feature a large speed improvement in streaming mode, which is particularly sensible for
clang was already in good shape), by as much as +40%, making streaming speed essentially on par with single-shot mode when ingesting large quantities of data.
XXH64 and even
XXH32 feature improved latency performance for small inputs of random sizes. Perhaps as importantly, their binary size is smaller.
There is a new experimental
XXH3 variant, named
_withSecretandSeed(). In a nutshell, it combines
seed for small inputs, with
secret for large inputs.
The main driver for this variant is a wish to skip the delay from
secret's transparent generation when using
_withSeed() variant with large inputs, resulting in measurable performance drop for "not so large" sizes (< 1 KB) (note: this delay is insensible for "large" inputs, such as > 256 KB). Coupled with new function
XXH3_generateSecret_fromSeed(), which generates the same secret as the one generated internally when using the
_withSeed() variant, it results in exactly the same return values, while skipping the
secret generation stage, thus improving speed.
XXH3_generateSecret() has been extended to allow generation of
secret of any size (though respecting the specification's minimum size). It's generally recommended to use this generator to ensure a source of "high entropy" for the
On the CLI front, a highly demanded
xxhsum feature was an ability to generate
XXH3 checksum values. This is achieved in
v0.8.1, using the
--tag format, which ensures that
XXH3 results cannot be confused with (default)
XXH64 ones, even though they feature the same 64-bit width.
- perf : much improved performance for
XXH3streaming variants, notably on
- perf : improved
XXH64speed and latency on small inputs
- perf : small
XXH32speed and latency improvement on small inputs of random size
- perf : minor stack usage improvement for
- api : new experimental variants
- api : updated
XXH3_generateSecret(), can now generate secret of any size (
- cli :
xxhsumcan now generate and check
XXH3checksums, using command
- build: can build xxhash without
XXH3, with new build macro
- build: fix
xxh_x86dispatchbuild with MSVC, by @apankrat
XXH_INLINE_ALLcan always be used safely, even after
XXH_NAMESPACEor a previous
- build: improved PPC64LE vector support, by @mpe
- install: fix pkgconfig, by @ellert
- install: compatibility with Haiku, by @Begasus
- doc : code comments made compatible with doxygen, by @easyaspi314
- misc :
XXH_ACCEPT_NULL_INPUT_POINTERis no longer necessary, all functions can accept
NULLinput pointers, as long as
size == 0
- misc : complete refactor of CI tests on Github Actions, offering much larger coverage, by @t-mat
- misc :
xxhsumcode base split into multiple specialized units, within directory
cli/, by @easyaspi314
xxHash v0.8.0 - Stable XXH3
After more than a year in the making,
XXH3 has finally reached
stable status, for both its 64-bit and 128-bit variants.
While the code itself was in good enough shape for production use, the generated values could still change between versions. This limited
XXH3 to local sessions only.
From now on, output values produced by
XXH3 for a given input and parameter set will remain identical across systems and across future versions. It makes it possible to store these values for later comparison, or to exchange them across network connections.
Official stabilization being the main goal of this release, there are only minimal additional changes.
A notable one though is the ability for
xxhsum CLI to produce and check BSD-style checksum lines, using command
One advantage of
--tag format is that it explicitly specifies the algorithm and format used to represent the checksum. For example, it explicitly mentions if a checksum value follows the canonical format (
XXH32) or the alternative little-endian format (
Generating BSD-style checksum lines was actually already possible, but as the CLI was unable to
--check them, it remained a hidden option.
This situation changes with
v0.8.0, thanks to a patch by @WayneD which makes it possible to
--check BSD-style checksum lines.
- api : stabilize XXH3
- cli :
xxhsumcan produce BSD-style lines, with command
- cli :
xxhsumcan parse and check BSD-style lines, using command
--check, by @WayneD
- cli :
xxhsum -accepts console input, requested by @jaki
- cli :
--separator, by @jaki
- cli : fix : print correct default algo for symlinked helpers, by @martinetd
- install: improved pkgconfig script, allowing custom install locations, requested by @ellert
xxHash v0.7.4 - Finalizing XXH3 and XXH128
v0.7.4 is the last evolution of
xxh128, primarily designed to finalize the algorithm.
It is considered release candidate for
v0.8.0, which means that if all goes right, this version will rebranded
v0.8.0, almost "as is", within the next few weeks, after receiving sufficient feedback.
v0.8.0 is the official version after which
XXH128 are considered "stabilized", meaning that return values will never change given the same
seed, making the hash suitable for long-term storage and transmission.
Beyond these "final touches", the new version also brings a few notable improvements.
Automatic vector detection
x64 systems can enjoy a new unit,
xxh_x86dispatch, which can detect at runtime the best vector instruction set present on host system (none,
avx512), thanks to a cpu feature detector designed by @easyaspi314. It then automatically runs the appropriate vector code.
This makes it safer to deploy a single binary with advanced vector instruction sets, such as
AVX2, since there is no hard requirement for all target systems to actually support it : the binary can automatically switch to
As a proof of concept, the windows builds provided alongside this release are compiled with this new capability.
A new vector instruction set is supported, thanks to @gzm55 :
AVX512. It can be applied on
XXH128, using some of the most recent Intel cpus, such as IceLake on laptop. It typically offers +50% more performance compared to
Advanced users can be interested in the highly customizable variant
_withSecret(), which makes it possible to run
XXH128 algorithms using one's own
However, the quality of the hash depends on the high entropy (randomness) of the
secret. And sometimes, it can be difficult to ensure that the candidate
secret is "random enough".
In order to produce a
secret of high quality, a new function
XXH3_generateSecret() is proposed in the advanced API section. It will convert any blob of bytes, named
customSeed, into a high quality
secret which respects all conditions expected by
XXH128. This is true even if
customSeed itself is of poor quality, such as a bunch of
\0 bytes or some short or repeated common sequence.
No API modification
The existing API present in 0.7.3 has remained unchanged in 0.7.4. Any programs linking with 0.7.3 should continue to work as-is.
Note however that
xxh128 return values are not comparable across these versions.
0.7.x are labelled development versions, and should only be used for ephemeral data (hash produced and consume in the same local session).
(note : this limitation does not extend to
XXH64, which are considered fully stable and specified).
There are multiple smaller bug fixes and minor improvements that have been brought to this repository by great contributors. Here is a summarized list:
- perf: automatic vector detection and selection at runtime (
xxh_x86dispatch.h), initiated by @easyaspi314
- perf: added
AVX512support, by @gzm55
- api : new: secret generator
XXH_generateSecret(), suggested by @koraa
- api : fix:
XXH3_state_tis movable, identified by @koraa
- api : fix: state is correctly aligned in AVX mode (unlike
malloc()), by @easyaspi314
- api : fix: streaming generated wrong values in some combination of random ingestion lengths, reported by @WayneD
- cli : fix unicode print on Windows, by @easyaspi314
- cli : can
-ccheck file generated by
libxxhashwith runtime vector detection (x86/x64 only)
- install: cygwin installation support
- doc : Cryptol specification of
XXH64, by @weaversa
xxHash v0.7.3 is major evolution for
xxh128, with a focus on speed and dispersion performance.
v0.7.3 pays a lot of attention to small data, by delivering generally faster latency metrics (about +10%).
Inlining is now a first class citizen, as it is generally key to best performance on small inputs.
Among the visible changes:
XXH_INLINE_ALLcan always be set before including
xxhash.h, even if
xxhash.hwas previously included (for example transitively, as part of a prior
- The algorithm implementation has been transferred into
xxhash.h. It's no longer necessary to keep a copy of
/includedirectory for inlining to work correctly.
xxhash.cstill exists, as it's useful to instantiate xxhash functions as public symbols accessible from a library or a
*.oobject file. It also remains compatible with existing projects.
Large data has also received a boost, which can go up to +20% for very large samples (> many MB).
Let's underline the remarkable optimization work of @easyaspi314, who hand optimized several hot loops and instructions, and even added a new Z-vector target for
No API modification
The API has remained completely stable between 0.7.2 and 0.7.3. Any programs linking with 0.7.2 should work as-is.
xxh128 results are not comparable across these versions.
New test tool
Testing a 64-bit hash algorithm for its collision rate has remained elusive for most. The sheer volume of data required to assess quality at this scale is too large for traditional test tools like
SMHasher. As a general guide, it requires 4 billion hashes to reach a 50% probability of getting a single collision. Accurate collision ratio evaluation requires many more hashes to actually measure something meaningful.
A new open-source tool in
tests/collisions offers this capability. It requires a lot of memory to run, with a minimum of 32 GB to measure anything significant. But provided that one has a system with enough capacity, it can accurately measure the collision ratio of any 64-bit hash algorithm.
Several algorithms were measured thanks to this tool, the result of which is currently consolidated on this wiki page. More can be added in the future.
This new development round also introduced several improvements to the
SMHasher test suite, uncovering new requirements for new scenarios. This proved beneficial to improve the general dispersion qualities of
Here is a summarized list of changes for this version:
- perf: improved speed for large inputs (~+20%)
- perf: improved latency for small inputs (~10%)
- perf: s390x Vectorial code, by @easyaspi314
- cli: Improved support for Unicode filenames on Windows, thanks to @easyaspi314 and @t-mat
xxhash.hcan now be included in any order, multiple times, with and without
- build: xxHash's implementation has been transferred into
xxhash.h. There is no more need to have
- install: created pkg-config file, by @bket
- install: VCpkg installation instructions, by @LilyWangL
- doc: Highly improved code documentation, by @easyaspi314
- misc: New test tool in
/tests/collisions: brute force collision tester for 64-bit hashes
This a maintenance release, focused on the newer 128-bit variant.
XXH3 is still labelled experimental : return values from this version are not comparable with other versions.
- Fixed collision ratio of
XXH128for some specific input lengths, reported by @svpv
NEONvariants, by @easyaspi314
- Improved performance of scalar code path (
XXH_VECTOR=0), by @easyaspi314
xxhsum: can generate 128-bit hash with command
-H2(note : for experimental purposes only !
XXH128is not yet frozen)
-qremoves status notifications
The main feature of this release is an update of
XXH3, building upon many user feedbacks during this test period. The main points are :
- Secret first : the algorithm computation can be altered by providing a "secret", which is any blob of bytes, of size >=
seedis still available, and acts as a secret generator
- As a consequence of these changes, note that new return values of
XXH3are not compatible with v0.7.0
ARM NEONvariant by @easyaspi314
- Streaming implementation is available
- Improve compatibility and performance with Visual Studio, with help from @aras-p
- Better integration when using
XXH_INLINE_ALL: do not pollute host namespace, use its own macros, such as
- 128-bits variant provide helper function, for comparison of hashes.
XXH3 is still considered experimental at this stage. It will have to remain stable for at least 2 releases before being branded "stable". After which stage, the algorithm and produced results will no longer evolve.
Several general improvements are also present in this release :
The main highlight of this release is the introduction of
XXH3, a new hash algorithm offering much improved speed, for both large and small inputs.
XXH3 is still labelled experimental, and must be unlocked with macro
XXH_STATIC_LINKING_ONLY. The source code is located into its own
xxh3.h file, which is automatically included (and therefore required) by
xxhash.c. It's also possible to include
xxh3.h directly, which will have a similar effect as triggering
At this stage,
XXH3 is suitable for ephemeral data and tests, but avoid storing long term hash values yet.
XXH3 will be transferred into stable in a future release, after a period dedicated to gather users' feedback.
For more details on
XXH3 performance, see this article.
note : there are known compilation issues under Visual Studio, which have been later fixed in
- Improved performance on small keys, thanks to suggestions from Jens Bauer
- New build macro,
XXH_INLINE_ALL, extremely effective for small keys of fixed length (see this article for details)
XXH32(): better performance on OS-X
clangby disabling auto-vectorization
- Improved benchmark measurements accuracy on small keys
- Included xxHash specification document