Gamut 1.2.0

Security: this release fixes GHSA-7pj7-847p-f2c9. Upgrade if you open, validate or append to .gamut files from sources you do not control.

Install: pip install --upgrade gamut-format; MATLAB R2023b or later: open Gamut.mltbx below; Julia, C and C++: the gamut-core-1.2.0-<platform> archive below.

A file can no longer make a reader or a writer touch another path. A
dataset whose values HDF5 keeps outside the file, through external storage
or as a virtual dataset, passed validation and was read from that other
file (a constant as soon as the file opened), and appending wrote to it,
creating it if need be. Such a dataset is now an error (E22): the
validator reports it, opening refuses the file, and appending refuses it
before the file is opened to write. Appending also refuses a run-table
column that cannot grow (E20), before writing anything.

A read is refused (E23) before any buffer is sized when it would return
more than 2³⁰ values (8 GiB at 8 bytes a value); a file declaring billions
of rows made the Python, Julia and MATLAB layers allocate for them first.
The threaded reader no longer reads past its arrays when a field has fewer
axes than its selections.

The core reads compressed datasets itself, a chunk at a time, with one
path for one thread or many, and HDF5 is never handed a chunk it has not
checked. A chunk that is not stored whole, or that gzip does not inflate
to its full size, is refused (E23) on reading and before appending
rewrites it; HDF5 trusts these sizes, and a crafted chunk made it read past
its memory and return what lay there, or write past it when appending. A
filter other than gzip and shuffle is not read (E20). Reading a few
scattered rows or nodes is several times faster, up to twenty times on a
million-row table, and other reads and writes take the time they did.
Building the core needs HDF5 1.14 or later (every package carries 1.14.6).

MATLAB installs from a toolbox, Gamut.mltbx, attached to each release:
MATLAB R2023b or later on Linux, Windows or an Apple silicon Mac, and no
compiler. Each release also carries the core on its own for each platform
(gamut-core-<version>-<platform>, with its headers and CMake package) for
Julia, C and C++. Every package now includes the licences of HDF5 and zlib,
which the core links in.

Evaluating a damaged model file is refused in every language when the file
lacks what evaluation reads: a nearest model's numbers (columns, lower,
upper, x, outputs; E17), or a declaration such as a parameter's
bounds, a result's model or output, or a model's type or inputs. Python
and Julia raised a key error and MATLAB a field error instead. In Python
and Julia, related likewise refuses a quantile without its q.

Calls from several threads no longer crash. HDF5 keeps state for the whole
process and is built without its thread-safety option, so two threads each
reading its own file could bring it down; every call that reaches a file
now takes its turn. threads= remains the way to read one large field
faster.

Appending from C, C++ or MATLAB refuses a file of another major format
version, as reading already did, and does so before opening the file to
write, which leaves it byte for byte as it was.

Messages say more about what to fix. A number left as text ("4.0" from
a spreadsheet) is named as such, rather than reported as a missing value;
an illegal name is refused with what a legal one is; W2 counts the rows
of each status and says when no status is named converged, the one
name that marks a run as usable; W1 names the group a split divides.

The languages agree in more places. gamut_create from C, or
gamut::create from C++, with a mesh and no per is refused (E6), as
Python, MATLAB and Julia already refused it, where it took fixed. Cell
offsets of the wrong length are refused (E5) in Python, MATLAB and Julia
before the core reads them, as the C++ layer already did, and so are a
nearest model's inputs of unequal length. A parameter given only one of
lower and upper is refused (E16) in Python, MATLAB and Julia, where the
other was taken as 0 or the one given was dropped. An evaluation that
fails leaves no file behind, in every language. In Julia, a true note is stored as the integer 1,
and related keys a derivative along each coordinate axis separately, as
in Python.

A run table from a spreadsheet goes straight in: Python's add_runs
takes a pandas DataFrame and add_run one of its rows, and MATLAB's
addRuns takes a table, as readtable gives, with labels as text or
categorical.

The README and the guide start from the simplest study, a table of runs,
shown in Python, MATLAB, Julia and C++, and step up to a mesh from there;
the guide opens on the pattern every file follows and lists its pages by
what a study has. It shows how to publish a train and test split, says
that only converged marks a run as usable, that a new category name
extends a label's categories, and that unit names are checked only when a
read converts them. MATLAB can read a file with ncread and nothing
installed. Installing covers xarray, Xcode for MATLAB on macOS, and adds
the Julia package at the release tag that matches the wheel.

help(gamut) opens on the whole pattern, help(gamut.create) documents
every declaration, and llms.txt gives coding agents what Gamut is for,
the pattern, the mistakes that are easy to make, and every page.

A model in a file has its own step in the README and leads the models
page: a quadratic, of a model type defined in a few lines, is fitted in
Python to the README's sweep, saved, and evaluated in MATLAB by a MATLAB
class of the same type, with no Python involved; the built-in nearest
model is evaluated from Julia and C++. help(gamut.create_model) and
llms.txt show the whole model pattern and its easy mistakes, and
help gamut.registerModelType in MATLAB says what call receives and
returns. A model's save that returns an object which is not model data
is refused with the object named, where Python reported a
zero-dimensional array; so are a list mixing text with other values and
text with more than one axis, which Python flattened into strings (E18).
gamut info names the studies a model was trained on, and a model's
one-line description. bench/scaling.py times writes and reads as run
tables, one mesh and a mesh per run grow.

An answer from evaluate says, row by row, which inputs lie outside the
range the model was fitted on, and how far. The answer already kept the
model's bounds on its parameters; outside(name) measures each row's
value against them, in the parameter's units: 0 inside or on a bound,
negative below, positive above, NaN for a missing value. A parameter with
no bounds is refused rather than reported as inside. The core computes it
once (gamut_outside in C) for any data file, and it is study.outside
in Python, MATLAB and C++ and Gamut.outside in Julia. evaluate still
warns as well (W4). The models page and the MATLAB, Julia and C++ pages
show the call.

The validator holds a mesh's computed coordinates to the writer's rules: a
computing model only in a model file whose mesh is per run, which must
name one (E16), and that model existing with its output named (E17). The
writer now also refuses a computing model without an output. An array
holding more values than the validator reads, 2²⁶, is now reported as
unchecked (W11) rather than passed over in silence.

Opening a file takes about a third less time (0.81 to 0.51 ms on the
benchmarks' capsule study): the reader opens each object once, for its
attributes, type and shape together, and leaves the storage settings to the
validator.