Skip to content

0.5.9

Choose a tag to compare

@rootcoder007 rootcoder007 released this 07 Oct 09:36

Standards vectors, at scale

  • The pinned C2SP Wycheproof post-quantum sets (ML-KEM-512/768/1024 key generation, encapsulation, decapsulation and key validation; ML-DSA-44/65/87 signing with and without seed, and verification) and the NIST ACVP sets (ML-KEM keyGen/encapDecap, ML-DSA keyGen/sigGen/sigVer, SLH-DSA keyGen/sigGen/sigVer for all twelve parameter sets, SHA-3, SHAKE, HMAC-SHA-256/512, PBKDF2) run in CI on every change to the kernels (.github/workflows/vectors.yml, tools/vectors/): 6,488 tests checked, 0 failures. The 3,822 not applicable are named with their reason: bit-length SHA-3 messages (the API hashes bytes), gigabyte messages, hedged ML-DSA signatures whose randomness the vectors do not carry, and PBKDF2 over SHA-224. A subset of 1.5 MB ships in tests/testthat/vectors and runs offline through the same runner.
  • Found by the vectors: C_rmbl_mlkem_decaps accepted a decapsulation key whose stored H(ek) did not match its own ek; FIPS 203 section 7.3 requires rejecting it, and it is now refused.
  • Pre-hash: HashML-DSA and HashSLH-DSA take every hash function FIPS 204 and FIPS 205 approve: sha224, sha384, sha512_224, sha512_256, sha3_224, sha3_256, sha3_384 and sha3_512 join sha256, sha512, shake128 and shake256. SHA-224, SHA-512/224, SHA-512/256, SHA3-224 and SHA3-384 are new native digests, checked against OpenSSL at every block boundary; each OID is bound into the signature, so two digests of the same length (SHA-512/224, SHA3-224) never verify for each other.
  • SLH-DSA gains its FIPS 205 internal interface (slh_sign_internal / slh_verify_internal, not exported) so the ACVP internal vectors check the core the public interface wraps.
    Side channels: timing, power, masking
  • First-order masking, on by default. kem_decapsulate() (ML-KEM-512/768/1024) and every ML-DSA signature (capsule_sign() with an ML-DSA key, fips_sign_mu()) run with each value that depends on the secret key held as two shares, refreshed with fresh randomness from the operating system for every call. The output is byte-identical to the unmasked computation: the full vector suite compares the two on every ML-KEM decapsulation and ML-DSA signing vector. kem_decapsulate(masked = FALSE) keeps the plain path. Cost: about 5 times the plain decapsulation and 16 times the plain signature (13 ms for ML-DSA-65 on a laptop).
  • The gadgets (src/rmbl_masked.h): Trichina's AND, a bitsliced Boolean adder, arithmetic-to-Boolean and Boolean-to-arithmetic conversion modulo 3329 and 8380417, masked Compress_1 and the decompressed ciphertext comparison (Bhasin et al. 2021), masked CBD sampling, a masked Keccak, and for ML-DSA a masked SHAKE for rho'' and y, HighBits revealed one bit at a time (w1 is public: it is hashed into the challenge), and the three rejection tests revealing one bit each.
  • Simulated power analysis (inst/tvla, CI on every change): the gadgets and the ML-KEM arithmetic, compiled for a Cortex-M4 and run in an emulator, are assessed with TVLA (fixed-versus-random Welch t, two experiments) under two leakage models, the value (Hamming-weight) model and the transition (Hamming-distance) model. Every masked kernel shows no first-order leakage under either, and every unmasked control leaks (|t| up to 163). CI fails on a masked leak under either model.
  • The masking gadgets are assembly for Cortex-M (Thumb-2), x86-64 and aarch64: every operation that touches both shares of a value (AND, XOR, the public-constant operations, the copies) loads, computes and stores in a fixed register order, zeroes each temporary before it takes a new operand and each destination before it is written, and stores a zero between the share-0 and share-1 stores; code that handles one share at a time runs each share in its own pass with the general-purpose and SIMD registers zeroed in between. From C, the compiler had moved one share into the register that held the other, which the transition model showed as leakage (|t| up to 135) and the value model could not.
  • Timing measured on hardware (inst/dudect, CI on x86-64 and arm64 Linux and on macOS): the dudect fixed-versus-random test on ML-KEM decapsulation (plain, masked, and each phase on its own), encapsulation, HQC decapsulation, the digest comparison and PBKDF2, with an early-exit comparison as the control that must be flagged. Decapsulation is also measured valid-against-invalid with both classes varying (fresh valid ciphertexts against random ones), so the only systematic difference is the rejection bit: an arm64 runner had flagged the fixed-ciphertext test at under one tick in 65,000 on one pass and not on the next, while every phase measured on its own was clean.
  • Differential fuzzing against OpenSSL 3.5 (inst/fuzz/fuzz_ossl.cpp, libFuzzer): ML-KEM, ML-DSA (valid and mutated signatures), SLH-DSA-SHA2-128f, the SHA-2, SHA-3 and SHAKE digests, BLAKE2b-512, HMAC and PBKDF2 must agree with OpenSSL byte for byte.
  • Found and fixed along the way: compression divided by q with a hardware divide at -O0 (KyberSlash) and now multiplies; three "add q when negative" corrections compiled to Thumb-2 conditional execution on a share's sign and are branch-free; the masked share split drew by rejection and now draws in constant time.
  • A hex logo (man/figures/logo.png, source data-raw/hex_logo.svg), on the README and the package website.
    The SIU parser, third pass
    Two further reviews of 0.5.8 (the diff review and a dedicated SIU review) showed the
    C-stack-overflow class still live from three sites the 0.5.8 rewrite did not reach, and
    a correctness cost of the fix it did make. All closed here, each at every site of its
    kind, pinned by test-review5.R and by fuzz seeds for every shape the reviewers used.
  • 25,000 form feeds (or vertical tabs, or carriage returns) through the plain-text path killed R. normalize_text() collapsed space and tab only; flatten_ws() had the full six-character set 180 lines away. The first pass now collapses every whitespace byte \s matches except the newline the second pass bounds.
  • strip_boilerplate() ran two unbounded, newline-crossing regexes over the whole document ([\s\S]*? to the privacy paragraph's end, [^.]* to the glossary sentence's end); the line cap never bounded them and 44 KB of period-free text after "this information may include", standard boilerplate in every real report, overflowed the stack, from bricklayer_parse_siu(), bricklayer_siu_resolve_so() and the network path bricklayer_fetch_parse_siu(). Both passes are find() scans now; no regex in either SIU file crosses a line without a bound, and the hygiene test asserts it over the sources.
  • The 2000-character line split is said, and placed better. A line longer than the cap is split at the last sentence end inside the window when there is one, else at the last space; every split is counted and bricklayer_parse_siu(), bricklayer_siu_text() and bricklayer_siu_resolve_so() warn how many lines were split, because a field spanning a split can come back incomplete. 0.5.8 split at any space, silently.
  • Two of the new regex-free passes were quadratic (strip_elements() rescanned to the end from every unclosed <script, 40 minutes at the 2 MiB cap; strip_tags() rescanned for > from every bare <). Both are linear, and the whole SIU path now polls for a user interrupt through a hook the host installs (rmbl::check_interrupt() here; rmorie's and morie's bindings install their own).
  • bricklayer_siu_iso_date() errors past 4096 bytes instead of returning ""; small_int() reads up to nine digits (SO #1234567 is 1234567 again, not 1); zero-padded numeric entities (&#0000000233;) decode; named entities are case-sensitive, as in HTML and in 0.5.7.
    From the 0.5.8 diff review
  • rmbl_pbkdf2_sha256() and rmbl_mann_kendall(), the two LinkingTo kernels that poll for an interrupt, raise it from a frame that holds none of their buffers; 0.5.8 raised from inside the poll, over live std::vectors. rmbl_interrupt_pending() never raises now; rmbl_interrupt_raise_unbarriered() does, after the kernel returned. The header documents rmbl_mann_kendall()'s NA-on-interrupt contract.
  • capsule_drift()'s identifier heuristic gates on distinct values (20 or more), not on a hundred rows a side: an 80-row id column is an identifier again, a 5-level factor on ten rows still gets its verdict.
  • The exact Mann-Kendall memo evicts its oldest table instead of flushing all 32.
  • mcar_test()'s log_p_value is documented; download_data(mode =) says what it accepts; bricklayer_json_to_json() names keep_vec_names and quiet_vec_names; MORIE_SERVICES_URL is documented and a mirror without a .json suffix finds its .sig; bricklayer_download(timeout = 1e18) is an error, not an undefined cast; a single-label host name is reported as a local address; bricklayer_llm_ask()'s help says an own endpoint is taken when configured, not probed, and bricklayer_llm_status()'s example switches the local route off.
  • An endpoint of your own (MORIE_LLM_BASE_URL) may be a literal private LAN address over plain http (10/8, 172.16/12, 192.168/16, fc00::/7: an LM Studio or vLLM box on your own network, the documented use), through the same per-call relaxation the loopback host has; the diff review (D4) found it refused by two independent rules with no way through. Link-local addresses and host names over plain http stay refused, and a redirect off such an address meets the full policy. bricklayer_services() documents MORIE_SERVICES_URL; bricklayer_parse_siu() and bricklayer_siu_text() document the 2 MiB cap and the line split; capsule_drift() documents the identifier gate; bricklayer_download() documents its headers and timeout ranges.
  • The services cache serves only when it is newer than the document bundled with the package: a cache earns its place by being newer, the same date falls back to the bundled copy, and an older one is deleted. morie's resolver gets the same floor.
  • Fuzzing: a fuzz_url target over url_check() and redirect_policy(), seeds for every shape the reviewers used, and the SIU target runs under an 8 MB stack like R's.
    The fourth review (network and service discovery, 2026-10-06) confirmed the 0.5.8
    transport closes every bypass it was built for, then found eight things at its edges.
    Each is fixed at every site of its kind and pinned by test-review4.R.
  • The loopback relaxation is an argument, not an option. options(rmoriebricklayer.allow_loopback = TRUE) relaxed every fetch in the process; it is gone. The local-model routes pass allow_loopback down one call, only when the address is the loopback host, and a redirect from a remote origin into the loopback interface is refused. An endpoint of your own is either the loopback host or a public https address.
  • Request headers do not follow a redirect to another origin unless they are content negotiation. The old rule dropped three named credential headers and let everything else travel; X-Api-Key, api-key, X-Auth-Token and the rest now stay with the first origin. "Another origin" compares scheme, host and effective port, so a redirect to another port on the same host drops them too.
  • A signed document older than the copy shipped with the package is refused. A validly signed but retired document in the cache (or served live) can no longer pin the hosted endpoint, which is where the user's key goes; the bundled document is the floor, and a stale cache below it is deleted.
  • Bytes a server sent are parsed as text, never as a second address. One helper, .rmbl_json_text(), parses every network body, cached manifest and signature file; the URL and file branches of bricklayer_json_from_json() are for a user's argument only. A body or a .sig that is a bare URL is "not JSON", not a fetch.
  • Keys are redacted by value. A gateway's error text has the key that was sent, any Bearer token and any sk- key removed before it reaches an R condition.
  • MORIE_HOSTED_AUTH_URL is validated like every other endpoint, and the sign-in service's verification_uri is opened in a browser only when it is a public https address.
  • Resource limits. A transfer under 64 bytes/s for 30 s is ended (it used to hold the connection for the whole timeout); an in-memory reply past 16 MiB is dropped, not handed back as a partial body; bricklayer_download() has max_bytes (default 2 GiB) and the services document and JSON fetches cap at 1 MiB and 16 MiB; the scratch file is private to the call and the body is moved into place on a 2xx only.

Documentation

  • Sixteen new vignettes, each runnable offline: post-quantum signatures (ML-DSA, SLH-DSA, XMSS); key encapsulation (ML-KEM, HQC); hashing, keyed digests, key derivation and Merkle trees; attestations, signed bundles and tamper-evident chains; side-channel assurance (what is verified, how, and what is outside the checks); JSON parsing and exact serialisation; the language-model routes and the signed services document; trends and change in short count series; small-area rates (expected counts, empirical Bayes, funnel limits, Moran's I); Hawkes processes; banded tables and concentration; stock and flow (average daily population, length of stay); falsification controls, power curves and the E-value; validation rules, inferred schemas and synthetic stand-ins; missingness, duplicates, outliers and distinct counts; and the SIU director's-report parser. The package website groups them by topic.
  • DESCRIPTION now also names the Hawkes-process fitting and the stock-and-flow measures; the standards-vector total reads 6,848 (it was misprinted as 6,488).

Air pollution, from source to burden, and the footprint of the computation

  • Atmospheric dispersion: pg_sigmas(), briggs_plume_rise(), gaussian_plume(), gaussian_puff(), advection_diffusion_2d(), lagrangian_particles() (Pasquill-Gifford coefficients with Briggs's fits, plume rise with its buoyant and stable branches, Gaussian plume with ground reflection and mixing-lid images, an instantaneous puff, an explicit grid solver and a Lagrangian random walk with a counting grid). Formula for formula the same as rmorie's AirDispersion; the particle walk accepts any normal generator through normals=.
  • Concentration-response and burden: crf_pm25(), crf_no2(), attributable_fraction(), mortality_displaced(), pollution_burden(), pollution_burden_by_area() (WHO 2021 review pooled estimates: Chen & Hoek 2020 RR 1.08 per 10 ug/m3 PM2.5, Huangfu & Atkinson 2020 RR 1.02 per 10 ug/m3 NO2; Burnett et al. 2014 integrated exposure-response curves for IHD and stroke; Levin's attributable fraction; the BenMAP displaced-mortality function; the GBD burden chain). Every coefficient is cited and asserted in the tests.
  • Exposure-response estimation: plr_crossfit() (Chernozhukov et al. 2018 partially linear model, DML2 cross-fitting, OLS nuisances, sandwich standard error, recomputed step by step in the tests) and exposure_response_plr() (percentile bootstrap; any estimator with the same interface can be passed).
  • Equity: exposure_concentration_index(), the Wagstaff, Paci & van Doorslaer (1991) concentration index of exposure by income.
  • The pipeline: verify_pollution() and pollution_report_text() (demo data, a CSV, a NAPS pull with NO2 ppb converted at 1.88, or scalars; an assumption log; one reference concentration throughout; exit statuses 0/1/2).
  • The footprint of the computation: compute_footprint(), carbon_intensity(), carbon_intensity_table(), detect_location(), rapl_available(), footprint_equivalents(). Electricity and CO2-equivalent of an expression, measured from RAPL energy counters where readable (codecarbon's Linux method; wrap-around handled, subdomains not double-counted), otherwise modelled with the Green Algorithms formula (Lannelongue, Grealey & Inouye 2021). Utilisation is measured, per process (codecarbon's process mode) or per whole machine (/proc/stat, codecarbon's machine mode) or fixed by number, with Green Algorithms' linear or codecarbon's load curves. The grid's carbon intensity is bundled for offline use anywhere: every country at the latest year Our World in Data publishes from Ember's yearly data (2024 or 2025), sub-national zones from Electricity Maps' 2024 yearly data via Green Algorithms data v3.1, and the world average, each row with its year and source; locations match by code, alpha-3 code or name, an unknown zone falls back to its country, and location = NULL detects the location offline from the environment, time zone or locale and records how. Every default that replaced a measurement is listed in the result. Cross-checked against codecarbon 3.3.1.
  • A seventeenth vignette, vignette("air-pollution-and-health"), walks all of it on offline examples.

Coverage to the last line

  • Every line codecov reported unreached at 75a0529 (204 lines, 53 files) now has a test reaching it with a real input; guards that no input could reach were removed from expected_counts(), sir(), drift_homogeneity(), mahalanobis_outliers(), clean_column_names(), hill_tail_index() and the gzip writer rather than excluded. Two defects found on the way and fixed: capsule_attest() with a raw context could never be verified (signed raw, recorded hex), and the dispersion models read a plain-vector receptors column-wise. A two-part hashing self-test now exercises the SHA-256/512 partial-block buffering. The handful of lines that remain unreached by design (OS entropy failures, the barrier's post-interrupt error, prime-order-curve impossibilities, ML-DSA rejection refills) are listed in NEWS.