srbf 0.21.0

Added

• Operon as a subprocess worker (worker: operon): pyoperon 0.6.1's SymbolicRegressor in the configuration
  its first author published for benchmarking (NSGA-II on R² and length, one Levenberg–Marquardt step per individual,
  population 1,000, length at most 50, depth at most 10, MDL selection from the Pareto front), without its
  hyperparameter search. srbf sets the operators and a budget in evaluations (rung files evals_<n>.pkl); the whole
  Pareto front is stored.
• DSO as a subprocess worker (worker: dso; DSO v3.0.0 in its own Python 3.7 environment,
  scripts/envs/build_dso_env.sh): options.arm: dsr runs DSO's shipped regression defaults, udsr the uDSR paper's
  benchmark configuration (uDSR*: priority-queue training, GP-meld, the LINEAR token). Three patches, documented in
  docs/models.md: GP-meld evaluates the trees it breeds, LINEAR's inverse table gains neg and n4, and a GP-meld
  clone shares the primitive set. On the NGGP Nguyen anchor the uDSR* path recovers 14 of 15 runs.
• GP-GOMEA as a subprocess worker (worker: gpgomea): the original GP-GOMEA at commit 6a92cb6, the one SRBench 2021
  ran, in its first author's benchmark configuration. Its protected operators are written as the functions it
  computes, and the linear-scaling coefficients are recomputed in float64. On SRBench 2021's Feynman protocol it
  recovers 71.7 % (published: 70.9 %).
• RILS-ROLS as a subprocess worker (worker: rilsrols): rils-rols 1.6.7 in its first author's SRBench
  configuration, with an output-only patch that prints constants at full precision; the budget is fit calls
  (evals_<n>.pkl). On Feynman it recovers 81.7 % (published: 82.59 %).
• The Flash-ANSR oracle: with generation_config.method: oracle (flash-ansr's OracleConfig), the Flash-ANSR adapter
  hands the model each problem's ground truth as its only candidate, which shows how well the fitting does when the
  form is given. A ladder over the refiner's restarts (restarts_<n>.pkl,
  configs/evaluation/scaling/flash-ansr-v25.0-T8-oracle_srbf.yaml).
• srbf table judges result trees (every method, draw, catalog, rung and shard) into one CSV with every metric, in
  parallel and with a per-file cache. With scripts/site_timing.py (the time axis from reference-machine result files;
  --pattern names a method's own rung files) and scripts/catalog_mu.py, the results site is rebuilt from the result
  files with srbf alone.
• QLattice as a subprocess worker (worker: qlattice): feyn 3.5.0 through its authors' SRBench loop (200 epochs,
  at most 10 edges, wide_parsimony, squared error). srbf sets the operators, the budget (n_epochs, rung files
  epochs_<n>.pkl) and the seed; the answer is the first model the loop returns, written at full precision, with
  feyn's protected functions written out where they clip. scripts/envs/build_qlattice_env.sh builds a pinned
  Python 3.12 environment; feyn runs offline without a licence key (CC BY-NC-ND 4.0: research use). On SRBench 2025's
  first-principles data it reproduces the published QLattice accuracy: mean mid-rank 0.41 among the published runs
  (2 epochs: 0.07).
• The flash_ansr_hybrid adapter type is built in again. The hybrid method moved into flash-ansr 0.19
  (flash_ansr.hybrid); srbf's adapter hands it each problem through its evaluation path (HybridRegressor.solve)
  and records the answer like every adapter. Configs that named the private plugin (hybrid_adapter:build) use
  type: flash_ansr_hybrid. pip install srbf[hybrid] brings flash-ansr with PySR; flash-ansr 0.19 is allowed.
• srbf table stores the expressions it judged: predicted_expression (in the ground truth's variable names and
  the engine's spelling) and ground_truth_expression, as prefix tokens at full precision.
• The results site's Predictions view: one problem at a time, the true formula and every method's formula, typeset,
  with whether each recovered it. scripts/site_export_v2.py writes one file per method, problem set, budget,
  finished run and block of 500 problems; a finished file never changes.
• srbf status -c CONFIG: how far every run of a config is (done, started, not started, with the
  row counts), without loading a model; the exit code is 0 when every run is done.
  Benchmark.runs_from_config(..., build_adapter=False) is the same from Python.
• An adapter from your own package: model_adapter.type: mypackage.module:function names a builder
  function; nothing in srbf has to be edited for an in-process adapter.
• What a worker reports about itself is stored with the results: __meta__["worker"] holds the return
  value of the worker's info() (its interpreter, package versions, checkpoint).
• Symbolic recovery at three levels of masking. symbolic_recovery masks every number (the structure is
  the ground truth's); symbolic_recovery_mask_fittable masks only the fittable constants, so exponents and root
  indices have to be the ground truth's as well; symbolic_recovery_mask_none also asks for the constants, measured as
  numeric recovery is. Each level implies the one before it, and agreement at a stricter level is a witness
  at the looser one. The two new columns need an engine.
• precision_score, recall_score and edit_distance_norm among the derived metrics: the two parts of
  the token F1, and the edit distance over the length of the longer sequence.
• The documentation is tested against the package (tests/test_docs.py): every command, flag, adapter
  type, suite catalog and derived metric is documented; every documented flag, import and repository path
  exists; every example parses; and the site builds with broken links as errors.

Changed

• srbf no longer depends on flash-ansr, and its root is SRBF_ROOT. {{ROOT}} in a config stands for the
  SRBF_ROOT directory (the current directory when it is unset); FLASH_ANSR_ROOT means nothing to srbf any more,
  so a script that set only it must set SRBF_ROOT. srbf reads its configs and resolves {{ROOT}} itself
  (srbf.paths). flash-ansr is an optional install, pip install "srbf[flash-ansr]", needed only by the adapters
  built on it (flash_ansr, and flash_ansr_hybrid through srbf[hybrid]) and by the two baselines that fit their
  constants with its refiner (lample_charton, brute_force); they import it when they are built and say how to
  install it when it is missing. A run of any other method never imports it: the FVU the NeSymReS and hybrid adapters
  record is srbf's own srbf.metrics.fvu, and the run's provenance reads flash-ansr's version without importing it.
  Flash-ANSR checkpoints are downloaded with hf download <checkpoint> --local-dir "$SRBF_ROOT/models/<checkpoint>".
  CI runs the suite with and without flash-ansr, and a test imports every srbf module where flash-ansr cannot be
  imported.
• The results site averages over problem sets, with the problem as the unit. A problem's value is the mean of
  its runs, and the catalogs are combined by a random-effects average (Paule–Mandel between-catalog variance,
  intervals over these problem sets, rates on the logit scale, ratios as geometric means), so erbench-syneq no longer decides
  a pooled number on its own; tables and tooltips add where one more catalog would fall. Paired contrasts compare
  every run of one method with every run of the other and test over catalogs (rates on the difference, other metrics
  on the per-problem superiority); ranks are built from the pairwise chances to beat, with Holm-corrected pairwise
  tests in place of Friedman and the Nemenyi critical difference. Two predictions that both meet Numeric Recovery tie
  in comparisons, and a failed run counts as worst in the medians, distributions, ranks and comparisons of log10 FVU
  and R². The time axis averages the catalogs the same way. scripts/site_random_effects.py is the reference; the
  site suite runs the page's own code against it.
• log10_fvu is floored at the float64 epsilon (srbf.metrics.numeric.LOG10_FVU_FLOOR, (\log_{10} 2^{-52}
  \approx -15.65)). An exact fit used to be (-\infty), which every mean left out, while the same formula written
  another way landed at finite rounding noise anywhere down to about (-320), which every mean took in: on the
  2026-09 board, 22 % of T8-120M's usable predictions sat below the float64 epsilon and at budget 128 values below
  (-20) made 45 % of its mean's sum. Below the epsilon the unexplained variance is smaller than about one rounding
  unit of the variance it is divided by, so all of these are one value now, and an exact fit counts in a mean.
  Blow-ups stay (+\infty). The judge's fingerprint changes, so srbf table re-judges its cache.
• The results site is written for a first-time reader. Every hint, popover, metric definition, method note and
  protocol text, and the prose below the explorer, say what they mean in plain words ("problem set", "budget",
  "run"; no pipeline vocabulary), with the longer explanations moved to a from-scratch "How to read the results"
  section. The Numeric Recovery threshold is described as what it is (FVU at most 2^-23: a typical error of at most
  0.035 % of the values' spread). A method missing at a budget is said to be not run there, or not finished yet;
  the export carries each method's planned budgets for that. copy_lint.py and a rendered-text test in the site
  suite keep the pipeline's words out.
• A worker is handed the problem and nothing of the ground truth. meta carries the problem's identifiers and
  sampling parameters (benchmark_eq_id, eval_row_index, n_support, noise_level, the variable
  names); the ground truth's skeleton, expression, constants and complexity stay on srbf's side.
• Every method sees every column. drop_unused_variables (and the PySR block's padding, and
  remove_padding of the NeSymReS and E2E adapters) selected the columns the ground truth uses before handing a problem
  over. The keys are still accepted, and ignored.
• The documentation is rewritten: a quickstart that needs no GPU and no model, a concepts page, a command
  reference, a metric reference with the definition of every derived column, the 29 catalogs with their
  sources and licenses, and one page each for running, results, paired comparisons, models, adapters and
  fairness.
• The results explorer lists a metric that repeats another in every published cell only once: recovery
  relative to the reference ground truth equals numeric recovery wherever the targets are computed from the ground truth.
• The tables and figures of srbf.analysis take a metric by the name of its column as well as a Metric.
  The report's table shows its Scaling column only when a run has a ladder.
• scripts/run_timing_ladder.py measures on any machine; --host NAME restricts it to one.
• Rung-file names live in srbf/rungs.py, and they and the out-of-process workers are left out of the judge's
  fingerprint: a new budget unit or a new worker no longer re-judges every cached file.

Fixed

• E2E's and NeSymReS's square roots and absolute values are read. Both baselines print through SymPy, which
  spells sqrt(u) and Abs(u); the engine spells rootn(u, 2) and abs(u) and kept the unknown names as bare
  tokens, so such a prediction had no description length and could never match its ground truth: 29,289 of E2E's
  139,348 usable predictions (21 %) and 917 of NeSymReS's. The in-process adapters now store the engine's spelling
  (srbf.spelling.engine_spelling, as the subprocess adapter already did), and srbf table rewrites the files
  written before. The judge's fingerprint changes, so the table re-judges its cache. A prediction that also carries a
  function outside the vocabulary (SymPy's conjugate) keeps its square roots as written instead of failing the whole
  file: the first re-judge lost three E2E files that way.
• The results site's paired wins and losses count better and worse. scripts/site_export_v2.py counted a
  problem as a win when the method's value was higher than the baseline's; for an error (log10 FVU) that is a loss,
  and for a ratio whose ideal is 1 the sign says nothing. They are now oriented by the metric's own terms (lower
  error, ratio closer to 1), the same rule the ranks use, and the sign test reads them.
• The shards of one unit no longer race on the candidate store's manifest. Every shard of a unit writes the
  manifest of the unit's store directory; the temp file they renamed had one shared name, so two shards renaming
  at once ended one of them with FileNotFoundError after hours of work. The temp name now carries the writer's
  identity, and a manifest that cannot be written is a warning: the manifest is advisory, the reader globs the
  directory.
• Symbolic recovery judges ground truth and prediction by one function. Both are brought into canonical form with
  their numbers, then every number is masked (pi and e included) and the skeleton is simplified and masked
  until it stands still. Before, the ground truth was judged in the form its catalog wrote and a number that
  simplification wrote itself stayed unmasked, so x1 * x1 and x1 ** 2, pow(u, 0.5) and rootn(u, 2), or
  x2 * x4 / (c * x4 ** 3) and c * x2 / x4 ** 2 were different skeletons: a prediction in the other spelling
  could not be recovered symbolically. A match of the skeletons as written counts as well, since no simplifier
  is complete. skeleton_simplified holds the ground truth's judged form.
• What a failed problem counts in an analysis metric follows from the metric's range. derive_metrics
  wrote the worst fit into a problem without a prediction (FVU infinite, R² 0) and left the structural columns
  empty, so a mean R² mixed how often a method returns a prediction into how well it fits, and a mean token overlap rose
  when a method failed on the hard problems. A failed problem counts 0 in the overlaps, whose range ends there:
  f1_score, precision_score, recall_score and the three variable-set columns
  (srbf.result_processing.WORST_VALUE). No other analysis metric is filled in: most can be arbitrarily
  bad, and a failed problem has no value there (None, NaN in the fit columns). A problem is failed when prediction_success is false, whatever text the method left behind.
  derive_metrics(..., impute_failed=False) leaves the failed problems out of every analysis metric.
• A failed prediction has recovered nothing. numeric_recovery_* and numeric_recovery_relative_* are
  false for a problem whose prediction_success is false, whatever values the method left behind.
• R² has no floor. r2_* is 1 - FVU: negative for a prediction worse than the mean predictor, -inf for a
  non-finite one. It was clipped to [0, 1]. One diverging prediction decides a mean of it, so srbf analyze
  reports its median (Metric(..., statistic="median")).
• The precision of an empty prediction is 0, not NaN (srbf.metrics.precision).
• A method that raises has failed the problem. An exception escaping an in-process adapter was recorded as a
  placeholder row, which every summary leaves out; it is a failed prediction, a miss on every rate, like a
  failure the adapter reports itself. A placeholder marks only a problem the catalog could not pose.
  derive_metrics reads such a row in a result file of an earlier version as the failed prediction it is.
• Symbolic recovery of a failed prediction is a miss. derive_metrics wrote None for a problem
  without a prediction, so bootstrap_report, srbf analyze and the paired reports computed symbolic
  recovery over the successful predictions only, while numeric recovery counted every problem. Both are
  rates: a method that returns nothing has missed.
• A prediction may use SymPy's spellings. sqrt(u), Abs(u) and the constant E are read; a prediction that
  uses a function the judge does not know fails with a message that names it.
• A shard without problems leaves its file. A one-expression catalog split eight ways has seven empty shards;
  they wrote nothing, and srbf merge then reported them missing. They write an empty shard file, and
  srbf merge takes the columns from the shards that hold rows.
• {{ROOT}} means the same directory everywhere a config names a file. It was left as it is in
  data_source.catalog, in holdouts and in a worker's options and env; it is replaced there too, and a
  catalog path may start with ~.
• A run started from Python documents itself like srbf run. Benchmark.run() on a benchmark built from
  a config file stored only the provenance label, and resuming a file written by the command line replaced
  its record of what ran. It now stores the config, its hash, the versions, the git state and the inputs.
• scripts/audit_pysr_maxsize.py reads the expressions with the engine the catalogs are judged with and audits any
  catalog (--catalogs, --suite, --maxsize).
• srbf new printed the {{ROOT}} token of the config language inside a shell command, and srbf run -v
  inside the path it reported; both print the real path. srbf analyze wrote nan as the budget of a
  method without a ladder when it stood next to one with a ladder.

Removed

• The results site's 2026-07 (paper) release: its explorer, data and prose. Its links open the current page.
• The scaling configs of the v25.0-T7 checkpoints; the v25.0-T8 configs are the reference.