Embeddings npyfile

Development build. This page describes main, not a released package. The latest published Lodestar.Embeddings is 0.4.0 — read its documentation.

NpyFile

Reads and writes a block of float in numpy's .npy format.

public static class NpyFile

Example — a block out and the same block back, without touching a file.

using Lodestar.Embeddings.Persistence;

float[] vectors = [1f, -2f, 3.5f, 0f, 0.25f, -0.5f];

using var buffer = new MemoryStream();
NpyFile.Write(buffer, vectors, 3, 2);
buffer.Position = 0;

NpyBlock block = NpyFile.Read(buffer);
string read = $"{block.Shape[0]}x{block.Shape[1]} {block.Values.Length}";   // => 3x2 6

Remarks — this is an interop format for a float matrix, not a second artifact format. EmbeddingIndex.Save is unchanged and still writes the versioned JSON decision 0011 chose; a .npy carries no ids, no normalization flag and no schema header, so it cannot stand in for one.

What it is for is the other direction: vectors that came from numpy, or vectors going to it.

The header is a Python dict literal and is never evaluated. numpy writes {'descr': '<f4', 'fortran_order': False, 'shape': (3, 4), } — executable source, which is the hazard decision 0011 refused pickle over. Only a fixed grammar is accepted here: three known keys, each with one of a closed set of values, and anything else refused rather than interpreted.

descr: '|O' — numpy's object dtype, whose payload is a pickle — is refused by name, before a byte of that payload is read.

Only float32, C order, little-endian, one or two dimensions. A file outside that is refused with a message naming what it held, rather than read approximately.

Applies to — net10.0, netstandard2.0.

See also — NpyBlock, EmbeddingIndex, the persistence index.

Members

Member	What it does
NpyFile.Read	Reads a .npy into a block and its shape.
NpyFile.Write	Writes a block as a .npy.