Embeddings 0.4.0 precompilednormalizer gethashcode

Lodestar.Embeddings 0.4.0. This page is frozen at that release. Read the current documentation for what main says now. A link to a decision or a migration page follows main, and leaves the archive.

PrecompiledNormalizer.GetHashCode

A hash consistent with that equality.

public int GetHashCode()

Returns — int, derived from the charsmap bytes.

Example — equal normalizers hash alike.

using Lodestar.Embeddings.Tokenization;

// charsMap is the precompiled_charsmap blob from the model's own spiece.model.
byte[] charsMap = File.ReadAllBytes("spiece.model");

PrecompiledNormalizer first = PrecompiledNormalizer.FromCharsMap(charsMap);
PrecompiledNormalizer second = PrecompiledNormalizer.FromCharsMap(charsMap);

bool consistent = first.Equals(second) && first.GetHashCode() == second.GetHashCode();

Remarks — the same bytes that Equals compares feed the hash, which is the contract. A charsmap is hundreds of kilobytes, so this is not free — cache the normalizer rather than rebuilding it, which is what a vocabulary already does by holding one.

Applies to — net10.0, netstandard2.0.

See also — PrecompiledNormalizer.Equals.