Skip to content

v3.13.1 – Native Gemma & Mistral, Faster Hash Joins, CSV & Statistical Fixes

Choose a tag to compare

@Nodibell Nodibell released this 14 Nov 21:43
· 10 commits to main since this release

Native Model Support (SwiftLLM, SwiftNLP)

  • Mistral uses grouped-query attention with a configurable sliding window, RMSNorm, and a SiLU feed-forward network. Tests compare cached decoding against a full forward pass, including beyond the sliding window.
  • Gemma 2 uses offset RMSNorm, GELU, alternating local and global attention, and soft caps for both attention scores and output logits. Tests cover the attention window and check that cached decoding matches a full forward pass.
  • Gemma 3 text decoding reads nested text_config and explicit attention head dimensions. It applies query and key normalization, uses separate rotary position settings for local and global layers, and loads a separate output head when the checkpoint provides one.
  • Gemma 4 text decoding supports proportional rotary embeddings, shared key/value states, per-layer token inputs, and sparse mixture-of-experts routing. Tests cover the E2B configuration and mixed-bit MLX weights with packed 4-bit experts and an 8-bit router.
  • Checkpoint loading selects the decoder and tokenizer from the local Hugging Face configuration. The loader normalizes checkpoint tensor names across shards and can keep supported MLX weights packed during inference.
  • SentencePiece BPE and chat templates handle whitespace markers, UTF-8 byte fallback, and added tokens for supported Mistral and Gemma tokenizers. Gemma 2 and 3 fold the system message into the first user turn. Gemma 4 uses its own turn markers and follows the tokenizer's beginning-of-sequence rules to avoid inserting a duplicate token.
  • Verification scope covers text inference and architecture behavior with deterministic fixtures. Tests that compare against external Hugging Face checkpoints are optional.

Scientific Correctness & Data Integrity (SwiftStats, SwiftPreprocessing, SwiftDataFrame)

  • SwiftStats Hypothesis Testing & Normality Calibration (Stats+HypothesisTesting.swift):
    • Corrected upper-tail Chi-square survival/CDF evaluation via regularized incomplete gamma precision fixes so chiSquareTest / goodness-of-fit $p$-values match SciPy (stat=156.274131, p=2.147307e-04, 65.67× faster at $N=100$ on clean idle hardware).
    • Implemented Royston AS R94 Shapiro-Wilk W and calibrated $p$-value transformation across $n \in [3, 5000]$ (verified exact W and $p$-value match at $n = 100, 1000, 5000$).
    • Calibrated Kolmogorov-Smirnov Lilliefors normal test (ksTest / normal parameters estimated from data) against empirical/regression-table Lilliefors critical-value curves (d=0.044160 at $N=200$, d=0.010317 at $N=2000$).
  • SwiftPreprocessing Transformers (FeatureSelection.swift, PolynomialFeatures.swift, TargetEncoder.swift):
    • Replaced naive sum-of-squares computation in VarianceThreshold with numerically stable mean-shifted variance, avoiding catastrophic cancellation on high-offset features (1e12).
    • Added exact Bessel degrees-of-freedom handling and stable group variance accumulation in SelectKBest ANOVA $F$-test (f_classif).
    • Corrected RFE surviving feature index remapping back to original feature coordinates.
    • Enforced canonical non-decreasing multi-index ordering and bias-column semantics in PolynomialFeatures.
    • Enforced genuine strictly out-of-fold (OOF) K-fold cross-fitting inside TargetEncoder.fitTransform (DataFrame and numeric matrix paths) to eliminate target leakage, with smooth prior blending and minimum sample thresholds.
  • SwiftDataFrame Deduplication, Variance & CSV Quote/Null Round-Trip (DataFrame+Operations.swift, TypedColumn.swift, CSVWriter.swift, CSVNullMatcher.swift, CSVReader.swift, CSVEagerReader.swift, CSVColumnBuilder.swift, SystemsCSVParser.swift):
    • Updated DataFrame.unique to distinguish heterogeneous typed keys canonicalized across column storages instead of lossy string coercion.
    • Replaced naive TypedColumn.variance() / stdDev() paths with shifted/two-pass Welford formulation immune to $10^{12}$-offset cancellation.
    • Fixed CSV quoting and null-token round-trip contract across writer, eager parser, and streaming parser: unquoted empty/null tokens ("", NA, null, NaN, None) decode as nil, whereas quoted empty strings and quoted literal null tokens preserve their string values (wasQuoted propagated distinctly from escapedQuotesPresent; CSVWriter automatically quotes literals colliding with active null tokens).

Performance Optimizations (SwiftDataFrame)

  • Dedicated Compact Inner Hash Join (DataFrame+SIMDJoin.swift): Added a specialized .inner fast path without unnecessary BitSet / unmatched right-side tracking, promoted compact mixed-storage key handling, and consolidated output gathering:
    • medium-inner-join (100k rows): 11.155 ms → 2.185 ms (-80.4% vs pre-opt; -79.5% vs 3.12.0 10.635 ms).
    • mixed-medium-inner-join (100k rows): 13.856 ms → 2.209 ms (-84.1% vs pre-opt; -84.0% vs 3.12.0 13.808 ms).
  • Compact Mixed-Storage Gather (TypedColumn.swift): Routed gatheredScalar through gatheredNumeric where contiguous numeric storage applies, improving mixed-medium-filter from 0.429 ms to 0.298 ms (-30.5% vs pre-opt; -27.9% vs 3.12.0).
  • Fast Pre-Reserved CSV Serialization (CSVWriter.swift): Replaced per-cell boxing with direct ColumnAccessor extraction and pre-reserved single String output buffer (9.88 ms numeric = 4.94× faster than Pandas; 15.71 ms null tokens = 3.64×; 17.53 ms plain strings = 3.45× on clean idle runs; quoted null-token eager/streaming CSV reads 3.51 / 3.33 ms = 3.02–3.92× vs Pandas).

Verification, Conformance & Documentation

  • Audited standardized runs: certification (27/27 passed, max abs err 5.588e-09), dataframe-conformance (24/24 passed, max err 0.000e+00), scientific-workflow (4/4 passed, max err 1.421e-14), standard (14/14 passed), and migration (36/36 passed).
  • Restructured PERFORMANCE.md and ACCURACY.md into concise index landing pages backed by versioned archives (PERFORMANCE/{3.10.3..3.13.1}.md and ACCURACY/{3.10.3..3.13.1}.md), updated DocC articles, SwiftSciCLI (3.13.1), BenchmarkMain (v3.13.1), Engineering Handbook (70/70 CookbookVerification recipes passed), 1,833 unit tests passing in Debug & Release (0 failures, 3 optional skipped), and 100.00% (2,551/2,551) heuristic public API documentation coverage.

Reports and downloads