Skip to content

Releases: rootcoder007/rmorie-bricklayer

0.5.14

Choose a tag to compare

@rootcoder007 rootcoder007 released this 16 Oct 15:26
v0.5.14
1c19b00
  • A new data key is issued only after the data access terms are accepted. rmbl login and bricklayer_llm_login() show the terms summary and their address, ask Do you accept ...? [y/N], and then ask for your name, organisation (optional) and why you need the data; for the restricted tables (Schedule B: ICPSR, NCVS 38604, A2AJ case law) they also ask for a supervisor, department chair, director or coordinator and their email. In a script, give them as arguments (accept_terms = TRUE, name, affiliation, purpose, restricted, supervisor_name, supervisor_role, supervisor_email) or flags (--accept-terms, --name, --affiliation, --reason, --restricted, --supervisor-name, --supervisor-role, --supervisor-email). Nothing is sent until every required answer is there; the data service checks the same things. login --token with a key you already hold is unchanged.

0.5.13

Choose a tag to compare

@rootcoder007 rootcoder007 released this 15 Oct 21:03
v0.5.13
98601b2
  • The survey-estimation vignette loads the CSUS 2023 public-use file and its 1,000 mean bootstrap weights with bricklayer_data_load("csus_2023/pumf") and ("csus_2023/bootstrap"), now hosted on data.rmorie.com, or downloads them from Health Canada with bricklayer_download(); it no longer calls rmorie, which this package never depends on.
  • A new vignette, Causal effects with cross-fitted double machine learning: plr_crossfit() recovers a known effect on simulated data and, on the NSW job-training experiment (LaLonde 1986; Dehejia and Wahba 1999, hosted as nsw_dw/* on data.rmorie.com), moves the observational estimate against the PSID controls from -15,205 to 618 dollars, against the experimental 1,794.
    Fixes from the CRAN stress test of b617a3f
  • A key next to punctuation never reaches a model. A question with no verb is refused as one when a key-shaped run appears anywhere in it (KEY?, "my key is KEY, thanks?", a key after a no-break or zero-width space), not only as a whole word: those went to the first provider, the wrong one for the key. Function names (bricklayer_json_base64url_dec) are not taken for keys. A verb typo with a ? apart (rmbl "doctr ?"), a leading ideographic space or quotes is a near miss, named in "did you mean"; a CJK question ending in a full-width question mark is asked.
  • survey_design(): genuine replicates passed by name are no longer refused when two of them are identical (216 of 2,400 test designs); JKn replicates of other strata, which equal the weight on a stratum subset, are accepted (every replicate equal to the weight is refused); a stratified bootstrap whose replicates equal the weight by chance, and post-stratified or calibrated jackknife subsets, are no longer warned about; a replicate a tenth of the weight's scale (a normalised weight, a multiplier column) is said, as one ten times it was. Rows reordered and then a column dropped or one cell edited are refused (the SE was silently wrong); a row-name reset is accepted. The Rd says how to read the survey package's own bootstrap replicates.
  • Interrupts and time limits: every POST polls for Ctrl-C and setTimeLimit(), not only when the wait label shows (Wayback /save, OCSP, the data-key check, ask with stderr redirected); a Ctrl-C in bricklayer_fetch() or wayback_snapshot_url_native() is raised at once instead of running the Wayback lookups on to their timeouts; the Wayback back-off without a terminal sleeps a second at a time, so a time limit ends it.
  • bricklayer_data_load() applies the option rmoriebricklayer.max_csv_bytes (1 GiB by default) on both of its read paths.
  • CSV reader parity with read.csv(): a line of NUL bytes is skipped; integers with leading zeros past ten characters are integers; a header of "" alone is a table of no columns; blanks outside a quoted header name are dropped; quotes after a closing quote are read as quotes; a header past a million fields is refused before it is held.
  • Language models: the last turn after the help-page lookups keeps the tools declared with calls turned off and rides on the last tool result (it was a second user turn with no tools, which Anthropic refuses); an SSE error event, or a 200 that carries an error, is said with its message; the saved-answer key includes a digest of the API key in use.
  • C and C++: argument errors and large allocations inside the entry points unwind through the barrier, so their buffers are freed (sen_slopes() leaked 858 MB at a failed allocation; small leaks in the HTTP header, gini, merkle, XMSS and BLAKE2b paths); a NaN or huge timeout is no longer cast to an integer (undefined behaviour).
  • Statistics: hurwitz_zeta(s, 1/2) is (2^s - 1) zeta(s), exact beside the trivial zeros; a point at -1e300 no longer collapses the Knox and hot-spot grids.
  • Arguments: a seed that is not one whole number is refused in words wherever a function seeds the RNG, and seeds now run from -2147483647 to 2^53 - 1: past set.seed()'s 32 bits the Mersenne-Twister state is built from the seed (each seed its own stream; seeds in the integer range give the stream they always did); lagrangian_particles(normals =) must be a function returning n finite numbers; a plot method given something its class's function did not make says so.
  • SIU parsing of a long run of letters is linear (a 1 MB <p>AAAA... took four minutes; now 2.4 s).
  • The courts vignette's case-mix paragraph is computed from its table; a test no longer writes Rplots.pdf into the tests directory; a complex-NA test follows R's own coercion (it failed on R 4.3); the heaviest signature-vector blocks (about 100 s) are skipped on CRAN and run in CI.
    Live progress for long work
  • bricklayer_progress(label, total), exported: the animated bar of bricklayer_download() for any loop, as an object with $update(done, info), $tick(by, info), $wait(seconds) and $done(summary). It follows the download bar's rule (drawn while a person watches a terminal: an interactive session, or a command line that sets options(morie.progress = TRUE); a line at each tenth when stderr is not a terminal; nothing with options(morie.quiet = TRUE) or quiet = TRUE). Nothing is drawn for the first 0.75 s, so a fast call never flashes a bar; a bar inside another function's running bar stays silent, so nested work shows the outer loop; an update is a stored value when quiet and a throttled frame when drawing. With total = 0 it is a spinner with the time so far and the current stage. on.exit(p$done(), add = TRUE) clears the line when the work fails or is interrupted.
  • Wired into the long-running functions: the bootstrap of exposure_response_plr(), the folds of plr_crossfit(), the permutations of knox_test() and step_change(), the controls of capsule_falsify(), the repetitions of capsule_power(), the splits of drift_calibrate(), the columns of capsule_drift(), profile_columns(), correlation_table() and top_correlations(), the stages of mahalanobis_outliers(), the EM iterations of mcar_test() and of core_hawkes_fit(method = "em"), the replicate checks of survey_design(), the steps of lagrangian_particles() and advection_diffusion_2d(), the receptors of gaussian_plume(), gaussian_puff() and plume_field(), the files of capsule_bundle() and capsule_bundle_verify(), the columns and the writing of make_synthetic_csv(), the packages of capture_dependencies(), the groups and exact p-values of analyse_table() and yoy_pvalues(), and long vectors in benford_test() and distinct_sketch() (read in blocks of a million; the counts and registers are those of one pass).
  • A spinner for long native calls: XMSS key generation and signing (pqc_keygen(), a few seconds at the default height, about a minute at 14), derive_key(), the native routes of core_hawkes_fit() (with the stage: first start, second start) and every network request (bricklayer_fetch(), the portal resolvers, wayback_snapshot_url_native(), revocation and service lookups, model lists) show a turning spinner with the label and the seconds so far, drawn by the compiled layer each time the call polls for a Ctrl-C, from 0.75 s on, on a terminal only. The XMSS tree now polls every 32 leaves (it was every 1,024, once in a whole default key).
  • Live progress in the IDE consoles, and one look for it. Whether a console can redraw a line is now decided as the cli package decides it, no longer by isatty(stderr()) alone, which is FALSE in the RStudio and Positron consoles, so most R users saw only the line at each tenth. Bars, spinners and the native spinner now animate in a terminal, the RStudio console (and its build and render panes), the Positron console, R.app and RKWard. They draw on stdout in the RStudio and Positron consoles, which print stderr in their error colour, and on stderr elsewhere. Colours follow the console (24-bit, 256, 8, or none; NO_COLOR, R_CLI_NUM_COLORS, RSTUDIO_CONSOLE_COLOR), and so does the width. They never animate inside knitr, an R Markdown notebook chunk or a Jupyter kernel, which get the plain lines. options(rmoriebricklayer.dynamic, rmoriebricklayer.num_colors, rmoriebricklayer.unicode, rmoriebricklayer.width), cli's options and R_CLI_DYNAMIC override the detection. The bar shows the percent, the count (or the bytes), the rate, the time so far and, once the rate has settled, the time left; it fits the console's width and cuts the label or stage with an ellipsis. When the work ends, a green check, the label, the result and the time replace the bar; a red cross replaces it when the function that made the bar failed or was interrupted. bricklayer_download() shows the same cross. The native spinner of long compiled calls draws the same line, turns only on R's main thread, ends with a check or a cross, and in a log writes a line at 0.75 s, one every ten seconds after it and one at the end.
    Integrity scheme 2
    New Merkle trees and chain entries are hashed under scheme 2 and record it; anything sealed by
    0.5.12 or earlier (no scheme recorded) still verifies under its old rules, with a warning that it
    was sealed with the weaker scheme.
  • Merkle trees separate leaves from interior nodes (RFC 6962): a leaf is SHA-256 of the byte 0x00 and the chunk, a node SHA-256 of 0x01 and its two children. Under scheme 1 the 64 bytes of two sibling digests verified as a "leaf" with the shorter proof from their parent. merkle_root(), merkle_leaves() and merkle_proof() take scheme (2 by default; 1 recomputes a root pinned by an earlier version), merkle_proof() records it in the proof, and merkle_verify() reads it from there or takes scheme = 2 to refuse a proof without one. Roots therefore differ from those of 0.5.12 for the same chunks. capsule_report() records merkle_scheme beside merkle_root.
  • Chain entries cover their label. chain_append() records `scheme ...
Read more

0.5.12

Choose a tag to compare

@rootcoder007 rootcoder007 released this 15 Oct 06:24
v0.5.12
27d03d1

rmoriebricklayer 0.5.12

The 0.5.11 stress round (commit a5a4d10) and the review of this release's own branch (head 6e80794, 2026-10-12): the findings of both fixed, each fix applied to its siblings, and the claims the earlier fixes overstated corrected.

Language models: bring your own key; the hosted tier is retired

  • Any major provider, or your own model. bricklayer_llm_ask() takes, in order, an endpoint of your own, then the first provider whose key is set, then Ollama. Sixteen providers are built in (bricklayer_llm_providers()): OpenAI, Anthropic, Google Gemini, OpenRouter, Groq, Together AI, Fireworks AI, DeepSeek, Mistral, xAI, Cerebras, Perplexity, Hugging Face, OpenCode Zen, Cohere and NVIDIA, each from its usual key variable or a key saved privately with rmbl config set <provider>.key. Anthropic's Messages API and Gemini's generateContent are spoken natively, tool calls included; the others are OpenAI-compatible. route = "anthropic" (or any provider id) takes one route only; bricklayer_llm_models(route) and rmbl models --route list a route's models; rmbl config setup asks for a provider, its key and its model.

  • The model knows the packages. With docs = TRUE (the default) the model is given the name, arguments and title of every function of the installed MORIE packages (rmoriebricklayer, and rmoriedata and rmorie when present), read from their installed help, and five read-only tools: a search over every help page, a help page, its examples, the dataset list and a vignette. A model that does not take tools gets the matching pages in its prompt. How much goes into the prompt follows the model's context: everything for a provider's model or an Ollama cloud model, the functions nearest the question for a model Ollama runs on this machine; MORIE_LLM_CONTEXT overrides. agent_bundle() asks the same way (its backend argument is now route).

  • bricklayer_llm_route_info() says which route would answer, without sending anything; bricklayer_llm_chat() sends a whole conversation as it is (no help, no tools), for chat loops in the other MORIE packages. rmorie's help, thousands of pages, is searched by the tools, and only its functions nearest the question are listed for the model.

  • The hosted MORIE tier is retired. No route, setting or model list for it remains; route = "hosted" says so, and a saved hosted route falls back to the automatic order with a one-time message. Signing in (rmbl login) is for the curated tables at data.rmorie.com only, and rmbl doctor reports the data key.

  • A model the route does not have is named as such, with the number it has; a refused key is said as such, and the key never appears in an error.

  • The data key is checked and issued by the data service. login --token asks data.rmorie.com (/v1/whoami) whether it knows the key before storing it; the sign-in address is the services document's data.auth_url, then its llm.auth_url for documents signed earlier. A document may now leave out the llm block. A restricted table (the ICPSR studies, NCVS 38604, the a2aj case law) refused with 403 says to accept the terms at https://data.rmorie.com/terms.

  • API calls no longer end a reply that is slow to start: a model that thinks before answering is not cut off by the 30-second low-speed limit.

  • It shows that it is working. While a model thinks, the terminal shows a turning spinner, " is thinking" and the seconds so far, then how many help pages it has read; the line is wiped when the answer comes. Only for a person at a terminal (never in a script, a log or options(morie.quiet = TRUE)); Ctrl-C now ends a long wait for a model.

  • A repeated question is answered at once. The answer to the same question (spacing aside; case counts), on the same route, model, packages and versions, is kept on this machine for 30 days and comes back without asking the model: rmbl ask says so, --fresh (R: fresh = TRUE) asks again. Owner-only files under tools::R_user_dir(), at most 200, outdated ones removed on each save, the first save in a session says where; only for a person at a terminal (the command line with the terminal on stdin and stdout) or an interactive session, never from a script, a pipeline or a check; MORIE_LLM_CACHE=off turns it off. Nothing is hosted.

  • Servers can reuse the instructions. The instructions a model gets are the same for every question (the functions picked for a question travel with the question), so the prompt caches of Ollama, OpenAI, DeepSeek and Gemini apply, and long instructions are marked for Anthropic's cache: a faster, cheaper first answer on every route.

  • Better retrieval. The help search matches inflected words ("emissions" finds "emission", "testing" finds "test") and leaves out question words; the closest help pages go with every question, to large models too; and the model is told never to say the packages lack something before it has searched under other words.

  • bricklayer_explain_error() and rmbl explain-error "MESSAGE": where an error or warning of rmoriebricklayer, rmoriedata or rmorie comes from, without a model or the network. Every message is read from the packages' own code (stop(), warning(), usage errors; sprintf() and paste() parts as wildcards), matched exactly or by its words, and traced to the exported functions that reach it, whose help pages say what they expect. With no argument, the last error of the session. rmorie's 18,000 messages are read once per installed version and kept beside the saved answers.

  • Ollama has to be running. rmbl doctor and the help now say to start it (ollama serve, the Ollama app, or its system service) when nothing answers at its address.

Network and statistics stress tests of 5e1dce5 (2026-10-14)

  • A typo, a file name or a key is never sent to a model. A line with no verb is asked only when it plainly is a question: the first argument is the question itself (two words or more, or ending in ?), it is not a verb or one or two letters from one, it holds no path, line break or key-shaped word, and anything after it is an option ask takes (rmbl "how?" --fresh works). rmbl explain-eror "object 'x' not found", rmbl "My Documents/urls.txt" and rmbl conifg set anthropic.key "$(cat key.txt)" with a CRLF key file went to the model before, the key with them. An unknown verb now names the nearest one (did you mean ...); one that looks like a key is not printed back.
  • A Ctrl-C during a GET is raised (rmbl models exited 0 with "did not answer", a Wayback run polled on past its time limit): the transfer's callback records the interrupt it takes. The plain GET behind the Wayback lookups takes Ctrl-C too.
  • The key prompt finds stty and sh on PATH without a child process (only /bin and /usr/bin were looked in, so on a NixOS-style layout the key was echoed), runs the sh it found, and takes a Ctrl-C already pending before the read starts. config setup prints its key explanation as part of the prompt the shell shows once echo is off.
  • A settings or cache folder left open by an earlier version is made owner-only, not only a new one. config set writes only the settings it changes into llm.json (a full rewrite cut floats to 4 digits, unboxed arrays, renamed a "" key and rounded integers past 2^53). Answers are kept only with a person at a terminal or in a GUI session (R --interactive < file kept them). The note after the tool-call cap goes with the last tool result (Anthropic and Gemini got two user turns in a row).
  • The native CSV reader: a table that would decompress past 512 MiB is read the streamed way (read.csv(gzfile()); a 2.6 MB file of blank lines got the process killed for memory), and a .csv.gz that is plain text is read as such. As read.csv() reads them: unquoted header names are trimmed (" Operational_Capacity " came out X.Operational_Capacity.), CR and CRLF inside quotes are newlines, a line holding only "" is blank, blanks before a quote and text after one are kept. A quote inside an unquoted field (ab"c, where read.csv() swallows the rest of the file) is refused in words. Deliberate differences from read.csv(): a UTF-8 byte-order mark is not part of the first name, and extra fields and an unclosed quote are refused. Short rows are filled, as read.csv() fills them, unless filling makes more cells than the text has bytes plus ten million (a 2,000-column header over short lines made 4e8 cells from a few megabytes). A Ctrl-C while a table is inflated or split is looked for every million fields or blank lines (by rows it waited 53 s on a wide table), the inflating goes in 16 MiB steps on its second pass too, and either comes back as R's interrupt, not an ordinary error.
  • Survey replicates: what is refused, what is said and what cannot be told apart is in ?survey_design. New: a copy of the weight within 1 percent on every row is refused under JK1, JKn and BRR; warnings for more replicates equal to the weight than a bootstrap draws (judged against the m!/m^m chance for the PSUs read from the replicates), a near copy under the bootstrap or Fay, a JK or BRR replicate that leaves no row out and is no rescaled weight, and replicates totalling on average under a tenth or over ten times the weight ($repweights multipliers), and a replicate over ten times the weight on a typical row (api99 among the bootstrap replicates raised the SE 21% in silence). 360 generated genuine sets: 0 refused, 0 warned; survey's JK1 and bootstrap subsets of apiclus1: no warning. A second column with the id's name and other values is refused (it was kept); the weight beside the replicates under its own name is recognised within 1e-12.
  • Reordered survey data is caught by a key per row, computed every time: rows reordered within strata or PSUs are refused now, a permuted outcome (a permutation test) is not a reorder, and a million rows of ten columns t...
Read more

0.5.11

Choose a tag to compare

@rootcoder007 rootcoder007 released this 09 Oct 22:20
a5a4d10

rmoriebricklayer 0.5.11

Language models: set everything from the shell or R

  • A stored hosted key is used even when a local Ollama server is running with no model pulled. The automatic route used to stop at that empty server ("local Ollama has no model to use"), so a user who had logged in, and whose doctor listed every hosted model, could not reach any of them. An empty Ollama is now passed over unless OLLAMA_MODEL is set or the Ollama route is asked for.
  • bricklayer_llm_config() and rmbl config show and save every language-model setting: the route ask takes (auto, own, ollama, hosted) and the address, key and model of your own server, of Ollama and of the hosted tier. rmbl config setup walks through them. Settings are saved in ~/.config/morie/llm.json (mode 0600); an environment variable that is set still wins.
  • rmbl ask --route ROUTE picks the route for one call.
  • rmbl doctor ends with the route and model ask will use, and how to change it.
  • rmbl help is a guide: the verbs by area, and help start, help llm, help config and help r pages with examples. Error messages name the config command that fixes them.

The table list reads left to right

  • rmbl data list and printing bricklayer_data_tables() in R show one line per table: the key, the row count and the name, left-aligned and cut to the terminal or console width. They went through print.data.frame, which right-aligned the names and, once the descriptions made a line wider than the terminal, printed every key, then every name, then every row count and source as separate blocks. bricklayer_data_tables() still returns the full data frame (with source); only its printing changed.
  • A table whose manifest entry has a blank description and source (the otis tables) is named by its key instead of by an empty string.
  • The smoke suite (inst/smoke/smoke.R) prints a failing case's message in full. It cut them at 400 characters, which shortened "https://rmorie.com/access" in the no-route message of ask to "https://rmorie.co".

0.5.10

Choose a tag to compare

@rootcoder007 rootcoder007 released this 08 Oct 06:22

A version of its own. The build tagged 0.5.9 on 2026-10-07 (29ff122) and the one that followed it the same day (dcb3cb9) shared a version number while differing in the masking defaults, the FIPS 203 section 7.3 key check, capsule_attest(context = raw) and the whole air-pollution feature. The stress test of dcb3cb9 (2026-10-08) asked for a number that tells them apart, and for the fixes below; this is both.

Every result draws itself

  • plot() methods for every result the package returns, on base graphics only: stock and flow (stock_flow(), one panel per measure), period-over-period change with intervals (yoy()), rates, shares and rate change, band sensitivity, drift by column (capsule_drift(), drift_calibrate()), Benford digits, power curves, frequencies, missingness patterns, region coverage, Mahalanobis outliers, correlations, permutation nulls, rounding envelopes, concentration-response curves, burdens, concentration curves, the whole verify_pollution() report on one page, the footprint with its equivalents, concentration fields and particle clouds, count trends, and analyse_table() one panel at a time. The same customisation everywhere (main, col or palette, grid, ... to the base call), bricklayer_plot_options() for the session, bricklayer_palette() with four colour-blind-safe palettes, and every method returns the data it drew. plume_field() evaluates a plume on a grid; plot_lorenz() and plot_funnel() serve the results that are plain data frames. vignette("plotting-results"). Titles fit their panel: a title too wide is drawn a step smaller, in a panel set one that still does not fit becomes a letter with the key printed under the panels (returned as the panel_key attribute), long axis titles shrink or wrap, and the left margin is sized from the widest tick label so an axis title never overlaps the numbers.
  • To carry what the plots need, exposure_concentration_index() records its concentration curve, count_trend() its data, verify_pollution() returns class rmbl_pollution_report, advection_diffusion_2d() class rmbl_field and lagrangian_particles() class rmbl_particles (all still plain lists underneath).

Air pollution and dispersion, after the stress test

  • verify_pollution() applied the PM2.5 counterfactual (5.8) to NO2, overstating NO2 deaths by 27%; reference now defaults per pollutant (NO2 10, PM2.5 5.8), as pollution_burden() always did.
  • Levin's formula was diluted twice when prevalence was below 1: the relative risk was taken at the population mean, which already includes the unexposed, and then multiplied by the prevalence again. With a CSV or the demo data the exposed are the rows above the counterfactual floor: exposure_mean is their mean and exposure_prevalence their share, and the scalar arguments are documented the same way.
  • advection_diffusion_2d() documented a stability criterion that was false (cfl below 1 and diffusion number below 1); the explicit upwind-plus-FTCS scheme needs |u dt/dx| + |v dt/dy| + 2 (kx dt/dx^2 + ky dt/dy^2) <= 1, and a run outside it reached 10^10 from a unit pulse while reporting itself stable. The bound is returned as stability_number and stable; a call outside it is taken in enough sub-steps of a smaller dt to satisfy it (reported, with a warning), so the field is a solution over the same physical time rather than an explosion.
  • Physical guards throughout the dispersion code, on the package's own input helpers: emission rates, masses, heights and distances non-negative, wind speed, travel time and grid steps positive, n_images a whole number (2.5 was truncated silently), the stack hotter than the air for Briggs' rise (a cooler stack returned NaN), x0 and y0 single numbers (a vector doubled mean_x), and a warning when the release is above the mixing height. gaussian_plume(q = -100) returned a negative concentration and u = 0 returned Inf; gaussian_puff(t = -100) a negative one; pg_sigmas(-500) a negative sigma.
  • attributable_fraction() returned 2 for RR = -1, 0 for RR = 0 and NaN for RR = Inf, and coerced text; it now requires a single finite positive RR and a prevalence in [0, 1]. pollution_burden(), mortality_displaced() and the CRFs check their inputs the same way; population is kept as a number (8e9 is no longer NA); "PM2.5", "PM25" and "pm2.5" are accepted everywhere.
  • A NAPS pull is converted row by row: a file mixing ppb and micrograms per cubic metre no longer has every row multiplied by 1.88, a value column without a unit column is an error rather than assumed, and any other unit is an error.
  • The concentration index uses mid-ranks, so tied incomes no longer make the index depend on the row order (all incomes equal gives exactly 0).
  • Citations: 5.8 is the lower bound of the GBD 2019 theoretical minimum risk exposure level, not "the WHO 2021 guideline" (that is 5); the NO2 respiratory coefficient is Huangfu and Atkinson's RR 1.03 per 10 micrograms per cubic metre; childhood asthma is an incidence coefficient (Khreis et al. 2017, OR 1.05 per 4 micrograms per cubic metre, scaled to per 10) and its burden is reported as incident cases, not deaths. Units are stated on every page: pollution_burden() takes a rate per person-year, verify_pollution() per 100,000.
  • Exposure at or below the counterfactual floor is a result (PAF and cases 0), not an assumption failure; an infinite or missing input is one (the status contract holds).
  • The carbon-intensity table has one Kosovo row (XK), documents its provenance and licences, and says which values Ember reports as exactly 1000 or above it. The time-zone table is rebuilt from the IANA zone.tab (one country per zone) plus every alias in backward, 549 zones: Europe/Stockholm, US/Eastern, Asia/Calcutta, America/Montreal and about 130 other common names were missing and fell through to the locale, so an English (US) locale in Stockholm got the US grid. Every geographic zone in OlsonNames() now resolves, and the posix/ and right/ prefixes are stripped. compute_footprint() checks every argument before it runs expr (a negative memory_gb no longer gives negative CO2e after the computation has been paid for).

Cryptography

  • Masked ML-KEM decapsulation left both shares of the decrypted message, the hash input and K-bar on the dead stack after returning (the plain path guarded them); every share pair is now wiped.
  • Masked ML-DSA signing can be switched off: capsule_sign(masked = FALSE), fips_sign_mu(masked = FALSE), and the option rmoriebricklayer.masked sets the default for all three masked operations in a session (a chain or bundle signs many times). The documented cost was understated: masking takes about 3 to 12 times the plain time on decapsulation and 13 to 24 times on ML-DSA signing, depending on the CPU (4 ms and 26 ms on this package's test machine).
  • The TVLA workflow also runs on changes to rmbl_mlkem_body.h and rmbl_mldsa_body.h, where the composed masked algorithms live; the dudect runner re-measures every target the first pass puts above the no-evidence line (|t| > 4.5) with three times the samples before judging it, because a shared CI runner moves a target by several units of t between passes (the arm64 runner put the unmasked ML-KEM decapsulation at 6, 9.7 and 10.6 within an hour, for seven ticks in eighty-four thousand; the masked default sits at 1.6). The re-measurement decides: above 10 fails, the inconclusive band is reported with its numbers. On arm64 the run with PSTATE.DIT set is the gate, as the workflow always said; when the runner's CPU has no FEAT_DIT (the GitHub arm64 runners of 2026-10), Arm gives no data-independent-timing guarantee for the instructions, so that run is reported with its numbers rather than gating: there the unmasked ML-KEM decapsulation shows a steady eight ticks in eighty-four thousand between valid and random ciphertexts (|t| 13 at 900k samples) that x86-64, Apple silicon (DIT) and the masked default do not show. The ACVP PBKDF sample exercises SHA-224 only; PBKDF2-SHA256 is checked against OpenSSL and RFC 6070, and the README now says so.
  • C_rmbl_hash_two_part (an internal self-test entry) refuses a non-integer or logical split instead of coercing it.

Documentation

  • The statement that no third-party security audit has been commissioned had been removed from DESCRIPTION, SECURITY.md, the README and seven help pages without one having taken place; it is back in all of them, and the masking sentence no longer appears on the pages of HQC, XMSS and the DRBG, which are not masked.
  • Vignette corrections: no ChipWhisperer firmware ships (the power figures are from the emulator); ML-KEM takes a 64-byte seed and HQC v5 a 32-byte one; attestations take a signing key, not a shared secret; a bundle's note is $attestation$note; the JSON codec is R, not C, and is not a fuzz target; Windows randomness is RtlGenRandom; capsule_bundle() pins whole files, the Merkle tree pins parts of one file; there is no separate cloud-keys route; bricklayer_llm_ask() has no context argument; the SIU schema column is name and an oversize tag is ignored, not an error; the synthetic marker is the .synthetic sidecar; XMSS is not compared with OpenSSL; verification errors on an object that is not a signature; ctgrind is a dynamic check, not a proof; the share recombination sentence in the README names what signing publishes.
  • yoy_pdf() drew rows off the page when height was not 8.5: the row height follows the page height, as the pagination already did. rmbl_barrier.cpp is regenerated from its generator. The www.rmorie.com move reaches data_hub.R and the services fallback, with the trailing slashes the site expects. The offline vector subset is 3 MB, not 1.5.

Pre-existing, found by the stress test

  • The French SIU resolver was about 100 times slower than the English...
Read more

0.5.9

Choose a tag to compare

@rootcoder007 rootcoder007 released this 07 Oct 09:36

Standards vectors, at scale

  • The pinned C2SP Wycheproof post-quantum sets (ML-KEM-512/768/1024 key generation, encapsulation, decapsulation and key validation; ML-DSA-44/65/87 signing with and without seed, and verification) and the NIST ACVP sets (ML-KEM keyGen/encapDecap, ML-DSA keyGen/sigGen/sigVer, SLH-DSA keyGen/sigGen/sigVer for all twelve parameter sets, SHA-3, SHAKE, HMAC-SHA-256/512, PBKDF2) run in CI on every change to the kernels (.github/workflows/vectors.yml, tools/vectors/): 6,488 tests checked, 0 failures. The 3,822 not applicable are named with their reason: bit-length SHA-3 messages (the API hashes bytes), gigabyte messages, hedged ML-DSA signatures whose randomness the vectors do not carry, and PBKDF2 over SHA-224. A subset of 1.5 MB ships in tests/testthat/vectors and runs offline through the same runner.
  • Found by the vectors: C_rmbl_mlkem_decaps accepted a decapsulation key whose stored H(ek) did not match its own ek; FIPS 203 section 7.3 requires rejecting it, and it is now refused.
  • Pre-hash: HashML-DSA and HashSLH-DSA take every hash function FIPS 204 and FIPS 205 approve: sha224, sha384, sha512_224, sha512_256, sha3_224, sha3_256, sha3_384 and sha3_512 join sha256, sha512, shake128 and shake256. SHA-224, SHA-512/224, SHA-512/256, SHA3-224 and SHA3-384 are new native digests, checked against OpenSSL at every block boundary; each OID is bound into the signature, so two digests of the same length (SHA-512/224, SHA3-224) never verify for each other.
  • SLH-DSA gains its FIPS 205 internal interface (slh_sign_internal / slh_verify_internal, not exported) so the ACVP internal vectors check the core the public interface wraps.
    Side channels: timing, power, masking
  • First-order masking, on by default. kem_decapsulate() (ML-KEM-512/768/1024) and every ML-DSA signature (capsule_sign() with an ML-DSA key, fips_sign_mu()) run with each value that depends on the secret key held as two shares, refreshed with fresh randomness from the operating system for every call. The output is byte-identical to the unmasked computation: the full vector suite compares the two on every ML-KEM decapsulation and ML-DSA signing vector. kem_decapsulate(masked = FALSE) keeps the plain path. Cost: about 5 times the plain decapsulation and 16 times the plain signature (13 ms for ML-DSA-65 on a laptop).
  • The gadgets (src/rmbl_masked.h): Trichina's AND, a bitsliced Boolean adder, arithmetic-to-Boolean and Boolean-to-arithmetic conversion modulo 3329 and 8380417, masked Compress_1 and the decompressed ciphertext comparison (Bhasin et al. 2021), masked CBD sampling, a masked Keccak, and for ML-DSA a masked SHAKE for rho'' and y, HighBits revealed one bit at a time (w1 is public: it is hashed into the challenge), and the three rejection tests revealing one bit each.
  • Simulated power analysis (inst/tvla, CI on every change): the gadgets and the ML-KEM arithmetic, compiled for a Cortex-M4 and run in an emulator, are assessed with TVLA (fixed-versus-random Welch t, two experiments) under two leakage models, the value (Hamming-weight) model and the transition (Hamming-distance) model. Every masked kernel shows no first-order leakage under either, and every unmasked control leaks (|t| up to 163). CI fails on a masked leak under either model.
  • The masking gadgets are assembly for Cortex-M (Thumb-2), x86-64 and aarch64: every operation that touches both shares of a value (AND, XOR, the public-constant operations, the copies) loads, computes and stores in a fixed register order, zeroes each temporary before it takes a new operand and each destination before it is written, and stores a zero between the share-0 and share-1 stores; code that handles one share at a time runs each share in its own pass with the general-purpose and SIMD registers zeroed in between. From C, the compiler had moved one share into the register that held the other, which the transition model showed as leakage (|t| up to 135) and the value model could not.
  • Timing measured on hardware (inst/dudect, CI on x86-64 and arm64 Linux and on macOS): the dudect fixed-versus-random test on ML-KEM decapsulation (plain, masked, and each phase on its own), encapsulation, HQC decapsulation, the digest comparison and PBKDF2, with an early-exit comparison as the control that must be flagged. Decapsulation is also measured valid-against-invalid with both classes varying (fresh valid ciphertexts against random ones), so the only systematic difference is the rejection bit: an arm64 runner had flagged the fixed-ciphertext test at under one tick in 65,000 on one pass and not on the next, while every phase measured on its own was clean.
  • Differential fuzzing against OpenSSL 3.5 (inst/fuzz/fuzz_ossl.cpp, libFuzzer): ML-KEM, ML-DSA (valid and mutated signatures), SLH-DSA-SHA2-128f, the SHA-2, SHA-3 and SHAKE digests, BLAKE2b-512, HMAC and PBKDF2 must agree with OpenSSL byte for byte.
  • Found and fixed along the way: compression divided by q with a hardware divide at -O0 (KyberSlash) and now multiplies; three "add q when negative" corrections compiled to Thumb-2 conditional execution on a share's sign and are branch-free; the masked share split drew by rejection and now draws in constant time.
  • A hex logo (man/figures/logo.png, source data-raw/hex_logo.svg), on the README and the package website.
    The SIU parser, third pass
    Two further reviews of 0.5.8 (the diff review and a dedicated SIU review) showed the
    C-stack-overflow class still live from three sites the 0.5.8 rewrite did not reach, and
    a correctness cost of the fix it did make. All closed here, each at every site of its
    kind, pinned by test-review5.R and by fuzz seeds for every shape the reviewers used.
  • 25,000 form feeds (or vertical tabs, or carriage returns) through the plain-text path killed R. normalize_text() collapsed space and tab only; flatten_ws() had the full six-character set 180 lines away. The first pass now collapses every whitespace byte \s matches except the newline the second pass bounds.
  • strip_boilerplate() ran two unbounded, newline-crossing regexes over the whole document ([\s\S]*? to the privacy paragraph's end, [^.]* to the glossary sentence's end); the line cap never bounded them and 44 KB of period-free text after "this information may include", standard boilerplate in every real report, overflowed the stack, from bricklayer_parse_siu(), bricklayer_siu_resolve_so() and the network path bricklayer_fetch_parse_siu(). Both passes are find() scans now; no regex in either SIU file crosses a line without a bound, and the hygiene test asserts it over the sources.
  • The 2000-character line split is said, and placed better. A line longer than the cap is split at the last sentence end inside the window when there is one, else at the last space; every split is counted and bricklayer_parse_siu(), bricklayer_siu_text() and bricklayer_siu_resolve_so() warn how many lines were split, because a field spanning a split can come back incomplete. 0.5.8 split at any space, silently.
  • Two of the new regex-free passes were quadratic (strip_elements() rescanned to the end from every unclosed <script, 40 minutes at the 2 MiB cap; strip_tags() rescanned for > from every bare <). Both are linear, and the whole SIU path now polls for a user interrupt through a hook the host installs (rmbl::check_interrupt() here; rmorie's and morie's bindings install their own).
  • bricklayer_siu_iso_date() errors past 4096 bytes instead of returning ""; small_int() reads up to nine digits (SO #1234567 is 1234567 again, not 1); zero-padded numeric entities (&#0000000233;) decode; named entities are case-sensitive, as in HTML and in 0.5.7.
    From the 0.5.8 diff review
  • rmbl_pbkdf2_sha256() and rmbl_mann_kendall(), the two LinkingTo kernels that poll for an interrupt, raise it from a frame that holds none of their buffers; 0.5.8 raised from inside the poll, over live std::vectors. rmbl_interrupt_pending() never raises now; rmbl_interrupt_raise_unbarriered() does, after the kernel returned. The header documents rmbl_mann_kendall()'s NA-on-interrupt contract.
  • capsule_drift()'s identifier heuristic gates on distinct values (20 or more), not on a hundred rows a side: an 80-row id column is an identifier again, a 5-level factor on ten rows still gets its verdict.
  • The exact Mann-Kendall memo evicts its oldest table instead of flushing all 32.
  • mcar_test()'s log_p_value is documented; download_data(mode =) says what it accepts; bricklayer_json_to_json() names keep_vec_names and quiet_vec_names; MORIE_SERVICES_URL is documented and a mirror without a .json suffix finds its .sig; bricklayer_download(timeout = 1e18) is an error, not an undefined cast; a single-label host name is reported as a local address; bricklayer_llm_ask()'s help says an own endpoint is taken when configured, not probed, and bricklayer_llm_status()'s example switches the local route off.
  • An endpoint of your own (MORIE_LLM_BASE_URL) may be a literal private LAN address over plain http (10/8, 172.16/12, 192.168/16, fc00::/7: an LM Studio or vLLM box on your own network, the documented use), through the same per-call relaxation the loopback host has; the diff review (D4) found it refused by two independent rules with no way through. Link-local addresses and host names over plain http stay refused, and a redirect off such an address meets the full policy. bricklayer_services() documents MORIE_SERVICES_URL; bricklayer_parse_siu() and bricklayer_siu_text() document the 2 MiB cap and the line split; capsule_drift() documents the identifier gate; bricklayer_download() documents its headers and timeout ranges.
  • The services cache serves only when it is newer than the document bundled with the package: a cache earns its place by being newer, the sam...
Read more

0.5.8

Choose a tag to compare

@rootcoder007 rootcoder007 released this 06 Oct 12:35
a033f46

The third review of the hardening work (2026-10-06) found seven things,
every one of them a fix that had landed at the site a reproducer named and
not at its siblings. This release fixes each at every site of its kind and,
where the siblings can be listed, puts the list in the tests
(test-review3.R, test-src-hygiene.R), so a new one cannot appear
unnoticed.
Language-model routes and the signed services document

  • bricklayer_llm_ask() tries an endpoint of your own, then a local Ollama server, then the hosted MORIE tier. MORIE_LLM_BASE_URL (with MORIE_LLM_API_KEY, MORIE_LLM_MODEL) names any OpenAI-compatible server; OLLAMA_HOST / OLLAMA_BASE_URL (default http://localhost:11434, off to skip) a local Ollama, whose first listed model is the default unless OLLAMA_MODEL says otherwise. route = "own" | "ollama" | "hosted" insists on one. bricklayer_llm_status() and rmoriebricklayer doctor report all three, in order; agent_bundle() follows the same order.
  • The hosted tier's addresses come from a signed document, not from the package. bricklayer_services() reads https://rmorie.com/.well-known/morie-services.json and its detached ML-DSA-44 signature, verifies it against the public key pinned in the package, caches it for a day under tools::R_user_dir(), refuses a document older than the one it holds, and falls back to the cached copy, then the copy bundled in inst/services/, then a document with every service switched off. The endpoint, the model list, the sign-in address and whether the tier is open can now change without a package release; a document that does not verify is ignored. MORIE_HOSTED_BASE_URL, MORIE_HOSTED_AUTH_URL, MORIE_HOSTED_MODEL and MORIE_DATA_URL still override. The hosted tier is a last resort; keys are personal and issued on request at https://rmorie.com/access, and every hint in the package says so. The GitHub and emailed-code sign-ins keep working.
  • Plain http on the loopback host is admitted for the local routes only. The URL policy that guards every other address refuses http:// and loopback twice over; the own-endpoint and Ollama calls set options(rmoriebricklayer.allow_loopback = TRUE) for their own duration, which admits exactly the loopback host (localhost, 127.0.0.0/8, ::1), nothing else private, and a redirect off it meets the ordinary rules (test-llm-routes.R).
  • SIU core: every string-to-integer conversion on a regex capture goes through one bounded helper (small_int()); std::stoi on a tag such as SO #4444444444444444 threw std::out_of_range and aborted the process (found by the libFuzzer target; the input is in the fuzz corpus). Ported to the rmorie and morie copies.
  • bricklayer_siu_text() and bricklayer_parse_siu() killed R on 25 KB of whitespace. libstdc++'s regex executor recurses once per character a repeated atom consumes, so \s+ over a long run overflowed the C stack, which no tryCatch() and no exception barrier can see. Every pass that runs over a whole document (newline normalisation, script and style removal, tags, entities, whitespace) is a plain loop now, byte-identical to the regexes it replaced (the test keeps the old pipeline as an R reference and compares); the text every field extractor sees has its spaces collapsed and no line longer than 2,000 characters, so no remaining regex can recurse further than that (and MSVC's regex, which has a step budget instead of a stack, stays inside it -- the one time it did not, the public entry points now answer "no match" instead of throwing); the input cap of every SIU entry point is 2 MiB (a report page is a few hundred KB). The guard is a test that enumerates every one-string SIU entry point from the namespace and runs ten hostile inputs through each in a subprocess. XMSS key generation polls for Ctrl-C with the throwing check like the other ten sites.
  • One attestation still verified two documents. 0.5.7 wrote a level whose names were all empty as an array, so {"k":{"":"X"}} and {"k":["X"]} shared a digest. The names attribute, present or absent, is now the one thing that tells an object from an array at every level, empty names included, and a named atomic vector is an object too ({"a":1} is not [1]). Attestations of documents whose fields are named vectors change digest; capsules verified against a stored digest should be re-attested.
  • Every network byte goes through the compiled transport. bricklayer_download() gated with the validator and then transported with base R's url(): no DNS check, no pin, no redirect re-validation, no size cap on the package's own documented pipeline; bricklayer_json_from_json() fetched a bare URL with no gate at all; download_data() wrapped download.file(). All three use src/rmbl_fetch.cpp, with the live progress bar driven by a callback from libcurl. The transport itself now drops Authorization, Cookie and Proxy-Authorization on a cross-host redirect, refuses a redirect from https to plain http even when the http option is set, checks a URL before opening the destination and writes beside it, moving the body into place only on a 2xx (a refused URL or a 404 no longer deletes a pre-existing file). The redirect rule is one pure function with its own .Call entry and tests. A single-label hostname (metadata, instance-data) is refused by shape, without DNS. A test lists every network open in R/ and allows exactly the two base-R fallbacks a standalone capsule bundle needs.
  • The interrupt flag. The barrier cleared rmbl_kernel_interrupted only on the normal path; it is cleared on entry and on every exit now, and a kernel called with no barrier active (a sibling package's own .Call) re-raises the pending interrupt instead of eating it and returning NA. rmbl_pbkdf2_sha256() in the C API returns an int (0, or -1 for an argument it refuses); an interrupt inside it is raised, never a zero key.
  • capsule_drift()'s identifier heuristic applies its ratio test from a hundred rows a side: a five-level factor on ten rows gets a verdict again.
  • mahalanobis_outliers() carries log_p_value, the fifth chi-square site; log_p_value is documented at every site. drift_psi() accepts any eps in (0, 1) again (0.5.7 refused values above 0.01, a new hard error on a previously accepted argument) and the help explains what the floor does to the index.
  • The Mann-Kendall exact-distribution memo is bounded; hostnames may carry underscores; Makevars.win's libcurl fallback link line is wired in; rmbl_digest_equal() gets the same value barrier as the ML-KEM select.

0.5.7

Choose a tag to compare

@rootcoder007 rootcoder007 released this 06 Oct 07:32

A release about measuring what the 0.5.6 README could only describe. Every claim in the new "Security posture" table is a check that runs in CI.

  • Constant-time verification, every primitive. inst/ctcheck/ runs ML-KEM, ML-DSA, SLH-DSA, HQC (v5 and round 4), XMSS, the CTR_DRBG, HMAC, PBKDF2, keyed BLAKE2b, SHA-2, SHA-3 and the digest comparison under valgrind memcheck with their secrets marked undefined (ctgrind), with GCC and Clang; a branch or memory address that depends on a secret fails the constant-time workflow. The run found and this release fixes: ML-KEM's ByteEncode branched on every bit it packed (the secret key included); the hex encoder used a table indexed by the secret nibble (every HMAC, PBKDF2 and key output); ML-DSA's norm check returned at the first offending coefficient; PBKDF2 round-tripped each MAC through hex and branched on the characters; XMSS measured its hex-encoded seeds with strlen(). The values the specifications publish (rejected samples, the challenge, the hints, R) are declassified at one named line each.
  • Secrets are wiped. The same harness scans the dead stack after each operation for copies of the seed, signing key or password. Every scheme left at least one; all temporaries now clear on every return path (rmbl_ct::Guard), and the scan is part of the gate.
  • Wycheproof. 1,247 Project Wycheproof vectors (ECDSA P-256/SHA-256, P-384/SHA-384, RSASSA-PKCS1-v1_5 2048/SHA-256) run in the test suite. They found that the DER parser accepted BER (long-form lengths below 128, lengths with leading zeros, non-minimal tags, end-of-contents octets), that ECDSA-Sig-Value was accepted with extra elements, padded or negative integers, and that a PKCS#1 block was checked field by field rather than against RFC 8017's exact DigestInfo bytes. All three are fixed; every valid vector verifies, every invalid and every legacy "acceptable" one is refused.
  • Fuzzing. inst/fuzz/ holds libFuzzer targets (ASan + UBSan) for the DER parser and RSA arithmetic, the decimal-to-double conversion (compared bit for bit with the C library), and the SIU report parsers, with seed corpora; the fuzz workflow runs them on every change and for longer weekly.
  • SECURITY.md: threat model, what each check catches and does not, reporting. The README's "Security posture" is now that table.
  • rmbl_hmac_sha256_raw() joins the C API (the hex variant is a wrapper).
    The second review of 0.5.6 (2026-10-06) asked for each fix's neighbours. Every item below has a test in tests/testthat/test-review2.R.
  • Every .Call entry point has the exception barrier, not two of 110: src/rmbl_barrier.cpp, generated from init.c by inst/scripts/gen_barrier.R, wraps each entry so a C++ exception is an R error raised after every destructor has run. Seven NULLs to hawkes_rescaled, a 2 GB shake, mismatched theil_sen lengths (a 40 MB out-of-bounds read), a 4 TB sen_slopes, a 2^31 reservoir and an INT_MAX PBKDF2 are all plain errors now; the quadratic loops and the RSA arithmetic check for Ctrl-C; the RSA public exponent is capped at 64 bytes (1 KB took 1.2 s, 1 MB would have taken 20 minutes, from a certificate); shake output and PBKDF2 iterations are capped; every string argument goes through one rmbl_str0(); the DER tree is converted under R_UnwindProtect, so an allocation failure cannot skip its destructor either.
  • SSRF gate rewritten, in the transport. src/rmbl_fetch.cpp parses the authority as a URL parser does (#@ and ?@ no longer hide a host, any userinfo is refused), reads IPv4 literals with inet_aton's grammar (127.1, 0177.0.0.1, 2130706433, 0x7f000001), IPv6 with inet_pton (mapped, NAT64, link-local, unique-local), refuses local names by suffix (localhost, .local, .internal, .localdomain, .home.arpa), resolves every hostname and tests every address, pins the connection to those addresses (CURLOPT_RESOLVE, so DNS rebinding cannot answer differently), follows redirects itself one checked hop at a time (an https fetch cannot be bounced to http or to a private host), and caps a chunked download. bricklayer_fetch(), .rmbl_read_json(), the CRL/OCSP fetch, the Wayback resolver and the hosted-model client all go through it. .rmbl_check_public_url() calls the same C check.
  • verify_capsule(): provenance_file is contained like every other path (a symlinked data_provenance.json verified tampered data as intact); data_not_synthetic is a required row that always exists and reads the sidecar, the manifest and the provenance; a manifest.json in the capsule is checked by default and manifest_consistent / script_sha256 are required whenever they apply; a sha256 or filename that is not a single string, an unreadable file or a directory are failed rows, not R errors.
  • manifest_digest() is injective again: the canonical encoder writes an empty key as "" (0.5.6 renamed it to its index, so {"":"X"} and {"1":"X"} shared a digest and one attestation verified both). A list with a names attribute is an object and is sorted, empty names included: a manifest built with unnamed elements in a named list gets a different digest from 0.5.5's, which never canonicalised such a level. make_manifest() writes meta$synthetic only when it is TRUE, so a real-data manifest keeps its pre-0.5.6 digest, and make_synthetic_csv() sets the flag in-process.
  • JSON: string assembly is linear (250,000 escapes took 16 minutes); the reformatter refuses NUL and lone surrogates as the parser does; parser and encoder share one 100-level cap (the encoder used to hit the C stack first); bricklayer_json_base64_dec() refuses characters outside the alphabet instead of deleting them; whole numbers beyond 2^53 warn on the decimal branch too; bigint_warn = FALSE and duplicate_keys = "keep" are what the package's own API readers use; the gzip inflate cap is 64 MiB.
  • capsule_drift(): a character column of dates is compared as dates (two disjoint years drift); a column with more than identifier_levels distinct values, or one per five rows, is an identifier -- its unseen_share is reported and drifted is NA (the 0.5.6 PSI over the union of categories fired on every id and date column); PSI for the rest is over the reference's categories with unseen mass pooled. The result gains type, unseen_share and note columns.
  • drift_homogeneity() returns p_value = NA with method = "inapplicable: ..." when the categories outnumber half the observations (every table with those margins has the same statistic), draws its Monte Carlo p-value under a local seed and restores the caller's RNG, and like drift_chisq() refuses negative, NA and infinite counts. All chi-square sites report log_p_value next to p_value: a double has no number for exp(-2536), and the log says how far below zero the tail is. cramers_v() drops an unused level (it returned NaN) and takes a seed too.
  • parse_bands(): trailing units with punctuation ("18 to 24 (years)", "5 to 9 years, inclusive") parse again; cue words are matched as words ("overnight" is not "over") in the leading slot as well as the tail ("under 18 to 24" is NA, not 18-24); a label with two numbers that is not a range is NA ("more than 10 - 20"); thousands separators are removed only from correctly grouped numbers ("12,34,567+" is NA); a label with a decimal point is on a decimal scale; the dash family is normalised byte-wise, so an en dash parses in a C locale. band_values() guards the open-lower floor as it guards the cap, and its default floor is min(0, upper), so "under 0" has its midpoint inside the band.
  • trend_test() reports slope_ci_clamped = TRUE exactly when a rank is at an end, warns when exact = TRUE cannot be honoured (n > 8), and memoises the tied exact null; count_trend() refuses an all-zero series; step_change() and count_trend() refuse infinite values as trend_test() does; morans_i() refuses negative weights and a constant variable; gini(), top_share(), lorenz() and hill_tail_index() refuse empty input; hurwitz_zeta() returns the Bernoulli values at non-positive integers (zeta(0, 1) = -1/2); the Benford first digit of a subnormal is read from the shortest round-tripping decimal (1e-310 is 1, not 9).
  • Result changes from 0.5.6 that NEWS did not state: morans_i() excludes self-neighbours from a weight matrix; make_manifest() no longer changes the digest of a real-data manifest (0.5.6 did); the OTIS full-verification workflow runs on demand only.
    The coverage pass that followed (every branch below was untested at 96.2%) found five defects that no test had reached; the CRAN checks and the entry-point sweep found two more.
  • Revocation checking worked only on paper. revocation_fetch() ran every certificate through the single-file-path guard, so a parsed path was refused before anything was fetched; the CRL/OCSP fetch refused plain http, which is what RFC 5280 distribution points and RFC 6960 responders use (the answers are signed, the transport adds nothing); and a delegated OCSP responder's certificate was never found, because the search for the [0] certificates element landed on nextUpdate inside the SingleResponse first and the signed answer was reported as unverifiable. All three are fixed; the client is now tested end to end against recorded OpenSSL responses (good, revoked, unknown, SHA-384 without an embedded certificate, a delegated responder, an ECDSA responder, tryLater, and every malformed shape), with the two network calls behind one-line seams. The fixtures are documented in tests/testthat/x509-fixtures.txt.
  • top_share() summed from the start for every fraction: a million fractions over a million values was 10^12 additions with no interrupt check (the entry-point sweep found it as the one hang). It is one prefix sum now, interruptible.
  • manifest_restore_seed() padded a record naming only the main generator kind with NA...
Read more

0.5.6

Choose a tag to compare

@rootcoder007 rootcoder007 released this 06 Oct 01:52

A security and correctness release from a full external review of 0.5.5 (every item below was reproduced against the installed package before it was fixed, and each has a test that fails on 0.5.5).

  • verify_capsule() fails when it cannot check. A provenance file that lacked resource$sha256, resource$filename or the whole resource block -- or was exactly {} -- verified a capsule with ALTERED data as ok = TRUE, because an absent field appended no row and all() of the remaining rows was still true. Every check that cannot be made is now a FAILED row with the reason, ok requires the three required rows (provenance_readable, data_present, data_sha256) to exist and pass, a pinned script digest is checked whenever one exists, a non-CSV file with a recorded row count or schema is a failed check, a <data>.synthetic sidecar is a failed check, and the result has a class and a print method that shows what was skipped.
  • The DER parser no longer aborts R. A 22-byte file made cert_parse() (and every OCSP, timestamp and X.509 path that parses network bytes) throw std::bad_alloc through std::terminate, killing the session past any tryCatch(): a length field could be read as 2^64-1 and the bound checks wrapped. Lengths are bounded without wrapping, the node count is capped, a child must advance the cursor, and no C++ exception can cross the R boundary (rmbl_asn1.cpp). The same discipline for Rf_error() over live C++ objects in the RSA, BLAKE2b, SHAKE and digest entry points.
  • ML-DSA signing cannot loop forever. A length-correct but corrupt secret key (s1/s2 outside +-eta) is refused in words, and the rejection loop is capped. The seven unguarded FIPS 204 arithmetic probes that read past a 1-byte allocation are removed (nothing called them); the two other unreachable .Call registrations are gone too. pack_sig() uses memmove() for its aliased copy; the poly_chknorm() comment says what the code does.
  • Every .Call entry that reads a string checks it is one (rmbl_str0()); random_bytes() prefers getrandom(2) where glibc has it; XMSS key generation can be interrupted.
  • Paths and URLs are checked before use. resource$filename, bundle paths and the pipeline's download target must be plain relative paths inside the capsule (.., absolute paths, drive letters and symlinks out are refused). Every URL the resolvers return and every URL bricklayer_download(), friendly_download() and download_data() fetch must be https (plain http only with options(rmoriebricklayer.allow_http = TRUE)), never file:// (an offline test passes allow_file = TRUE), and never a loopback, link-local, private or metadata address. The Wayback availability API is queried over https; the snapshot URL it returns passes the same gate. libcurl transfers are bounded: 5 redirects pinned to http(s), 2 GiB to a file, 64 MiB into memory, and the "available": true check reads the literal value.
  • JSON. A repeated key is an error (duplicate_keys = "keep" restores jsonlite's behaviour); nesting is capped at 200 levels; a lone surrogate escape is an error rather than the text "NA"; an integer above 2^53 warns once unless bigint_as_char = TRUE. bricklayer_json_unserialize() rebuilds data types only by default and refuses functions, calls, expressions, namespaces and S4 objects (trusted = TRUE for your own session's JSON). manifest_canonical() sorts a level with an empty-string key, so a re-serialised manifest keeps its digest. json_gzip_decode() refuses a member whose ISIZE trailer declares more than 512 MiB and checks the trailer after inflating.
  • Synthetic data say so. make_synthetic_csv() writes a <path>.synthetic sidecar and returns a classed object; make_manifest() and write_manifest_json() set meta$synthetic from BRICKLAYER_SYNTHETIC themselves, and record() marks every comparison INFO on a synthetic manifest, whatever the analysis script did.
  • The reference pipeline stops on a digest mismatch (exit 7) unless --allow-digest-drift is given, and then records the override in the manifest. Its subprocess command line is fully quoted.
  • capsule_bundle() hashes dotfiles and capsule_bundle_verify() lists them; only the bundle's own file is exempt from "unlisted".
  • The hosted-tier endpoints taken from MORIE_DATA_URL / MORIE_HOSTED_BASE_URL must be https and public before a bearer key is sent; the credentials file is created 0600 before the key is written; a DRBG reseeds itself from the operating system in a forked or restored process; the launcher install_cli() writes quotes its library path.
  • Statistics. Chi-square p-values are the upper tail (they underflowed to exactly 0 in drift_chisq(), drift_homogeneity(), benford_test(), mcar_test() and the Mahalanobis screen); the KS p-value uses the theta-function series for small t (two million-row columns differing in one value were flagged as drifted); drift_psi() no longer collapses to one bin on a constant or mostly-constant reference (a 1000x rescale reported PSI = 0); drift_homogeneity() gives a Monte Carlo p-value below five expected counts, and capsule_drift() reports a PSI and the unseen-category share for categorical columns, so two disjoint date columns are flagged; a one-category table reports df 0 and NA. morans_i() honours the magnitudes of a weight matrix. The exact Mann-Kendall p-value enumerates the observed values, ties kept. eb_rates() refuses negative, missing and infinite counts. parse_bands() strips thousands separators, reads "ages 18-24", and leaves a bare "-5", inverted bounds, "100-200-300" and "15 to 19 and over" as NA; expand_bands() errors (or warns, with drop_unparsed = TRUE) instead of dropping units; band_values() refuses a cap below a band's lower bound. count_trend() uses t quantiles in its quasi-Poisson branch; trend_test() reports slope_ci_clamped when the pairwise slopes do not reach the requested confidence and refuses infinite values. Benford digits are read from the decimal rendering (2e-300 was digit 1).
  • Packaging. graphics, grDevices and tools are declared; R CMD check fails CI on a WARNING; a configure script probes pkg-config / curl-config for libcurl on Unix; an ASAN + UBSAN job runs the suite; DESCRIPTION names every scheme the package implements and states that none has been audited; the key and signature help pages carry a Security section; a FIPS 202 known-answer test covers SHA-3 and SHAKE directly; the LinkingTo consumer test calls all 45 published kernels; the stale CHANGELOG.md is gone (NEWS.md is the record).

0.5.5

Choose a tag to compare

@rootcoder007 rootcoder007 released this 05 Oct 18:09
  • XMSS: an index never signs twice. capsule_sign() signed two messages at index 0 when called twice with the same key object (the key is an R value, so it cannot advance the caller's copy), and the help said it refused. It now remembers, for the session, the highest index each key has signed with and refuses that index or an earlier one (one key object signed twice, or a stale copy), retiring the index before the signature exists (RFC 8391; NIST SP 800-208). A damaged next_index (NA, negative, fractional) is refused in words. Across sessions, save the latest key_state after every signature.
  • Hawkes processes: events at the same instant no longer excite each other. The intensity sums over strictly earlier events, as the definition of a Hawkes process (a simple point process) has it; the exponential recursion, the sum-of-exponentials recursion and the exponential and Lomax densities counted a tie as an excitation at lag 0. core_hawkes_fit() warns when times are tied, and core_hawkes_jitter() spreads times recorded to a resolution (daily dates) uniformly across their interval, as Filimonov & Sornette (2015) recommend, with the splitmix64 uniforms rmorie and morie use for the TPS fits (the same numbers on daily data). The goodness-of-fit KS test uses the exact distribution only without tied residuals (the exact computation with ties ran for minutes and exhausted memory).
  • core_hawkes_fit() is fast for the Lomax kernel: the projected BFGS now restarts its curvature estimate when a parameter reaches or leaves its bound (as L-BFGS-B does). A Lomax fit to exponential-like data drifts to the bound on alpha (its exponential limit), and the matrix learned with alpha free mis-scaled the remaining steps: 1,164 iterations and 196 s on 9,839 events, now 64 iterations and 3 s to the same optimum. The same restart ends the gamma kernel's stall on its sum-of- exponentials and truncated likelihoods (2,000 iterations to a worse optimum on 2,118 tied events; now 2.8 s to the exact route's maximum), so "auto" keeps "soe" for gamma. core_hawkes_fit() also returns at_bound, the parameters whose estimate lies on the box: converged alone did not show that day-dated times had driven a shape to its wall. A fit whose event rate n / horizon lies outside what the bounded baseline can reach (e^-15 to 100 e^15 per unit time; horizon = 1e308) is refused with a message to express the times in another unit, instead of returning its start.
  • core_hawkes_nll() is the likelihood core_hawkes_fit() maximises, on the same parameters: its Lomax was a power law with exponent alpha (a Lomax of shape alpha - 1), so its values and a fit of it disagreed with the fit and with the documented density; it now uses the shared core for every kernel, keeping its documented sentinel outside the feasible region. A non-finite horizon is refused by every Hawkes function (Inf returned a list that looked like a fit).
  • SIU reports: counts in every layout. Investigators, forensic investigators, subject and witness officials and civilian witnesses are read from 2020-on English pages ("Witness Officials ( WO )", the singular "Civilian Witness ( CW )"), French pages ("Agents témoins ( AT )", numbered "AT no 1" or "AT n o 1", "Nombre d'enquêteurs de l'UES assignés : 3") and the legacy (2005-2011) layouts ("Witness Officer #2", "Three SIU investigators and two forensic investigators", "l'agent témoin n o 1", "Cinq enquêteurs"); "No civilian witnesses were identified" is 0, and with no subject-official section the resolver's rules decide (a bare "SO" no longer counts as one). Checked against the reviewed corpus on the 29 pages of the round-8 test: witness officials 27 of 29 (16 before), civilian witnesses 24 of 25 (14), investigators 29 of 29 (20), subject officials 20 of 20 (17).
  • SIU police_service: a service named as the one that notified the SIU ("the OPS contacted the SIU to report that police officers with the VPD ...") is passed over for the subject officials' service (drid 2010: Vancouver Police Department, not Ottawa); the OPP has one French name ("la Police provinciale", a legacy header's detachment) and one English name ("OPP Sioux Lookout").
  • SIU dates: a notification that dates the incident relative to itself ("during his arrest two hours prior", "the day before", "deux heures plus tôt", "la veille") gives the incident date (drid 648: 2019-09-12, not an earlier break-in); legacy headers ("Incident date:", "Date de l'incident :") are read; the French narrative's "du 9 février 2022" is read and interview and team dates are passed over. Legacy pages are recognised by language. French abbreviated months ("5 janv. 2023"), upper-case accented months ("3 AOÛT 2017") and "March 3 , 2020" are read.
  • SIU dates: the incident date is the first dated sentence of the narrative that is not the notification or the SIU's own work (drid 670: "... contacted the SIU to report a serious injury. On July 8, 2019 at about 3:50 p.m., CKLPS were called" is 2019-07-08), "of " counts ("Just before 4:00 a.m. of January 7, 2020"), and a narrative without a date gives way to the director's analysis ("On December 4, 2020, the Complainant rolled his SUV"); drids 670, 820, 1200 and 4600 were empty. "The SIU was notified of the incident by ... on October 27, 2017" dates the notification, and French "1 er septembre 2016" and "L'UES a été avisée ... le" are read.
  • SIU fields: specific_injuries comes from the investigation, never from the mandate's definition of a serious injury, with whole words ("stab" is not "constables", "arm" not "firearm"), more body parts, and French injuries ("fracture de l'épaule droite"); French charges_recommended and relevant_legislation ("Code criminel") are read. &Agrave;, &laquo;, &raquo;, &thinsp; and the other capital and French entities decode.
  • bricklayer_siu_resolve_so(): the spelled-out plural ("Two subject officials were designated"), "no official was a subject official" and witness officials with no subject official named resolve, as the help describes.
  • bricklayer_siu_resolve_so() reads French reports: the "Agent(s) impliqué(s)" roster ("AI no 1", a lone "AI"), legacy ordinals ("agent impliqué n o 3", "l'agent(e) impliqué(e) n o 1"), "aucun agent impliqué" and witness officials (AT) alone; the privacy, definition and legal paragraphs ("des agents impliqués et témoins", "les agents impliqués sont invités ...") are not read as cues. It returned NA on every French page. bricklayer_parse_siu() falls back to these rules for a French report with no roster, as it does for an English one (drid 3658: 9, as its English twin).
  • Inputs: kem_keygen(512.5), kem_keygen(c(512, 768)), pqc_keygen(2.5) and every other whole-number argument refuse a fractional value instead of truncating it to another setting; bricklayer_fetch_siu(1.5) no longer fetches drid 1; bricklayer_parse_siu() refuses a path that does not exist (it parsed the string as an empty report); bricklayer_fetch_parse_siu(lang = "xx") is an argument error.
  • Argument errors in words where R's internals spoke: bricklayer_json_base64url_dec(character(0)) (now empty, as bricklayer_json_base64_dec()), bricklayer_json_unserialize() of a number, capsule_bundle(character(0)), yoy() of an infinite value, HyperLogLog registers beyond 64 (distinct_count(1e300)), and bricklayer_fetch_parse_siu() with an invalid drid, which is now an argument error before any network instead of a message and NULL.
  • core_hawkes_jitter(horizon = ) spreads each event over [t, min(t + resolution, horizon)], so a time dated on the horizon stays inside the window (it was pushed past it, and the fit then refused the times); the fit's refusal names the span and the remedy.
  • The ?mahalanobis_outliers example contradicted itself: its "ordinary on each variable" row had the largest weight, and its classical-versus-robust comparison printed FALSE. The row is now short and heavy within both ranges, and the comparison uses a cluster of bad rows.
  • A count argument refuses TRUE (it was read as 1: drbg_generate(d, TRUE) gave one byte), Inf and values beyond the integer range in words, with no coercion warning; hqc_sizes(Inf) no longer warns before refusing.
  • bricklayer_fetch() gives the HTTP status of a server that answered ("answered HTTP 404"; it said "could not reach"); friendly_download(attempt_wayback = FALSE) (or "") says the fallback is switched off instead of claiming the Wayback Machine has no snapshot.
  • data pull refuses an --out path it cannot write before downloading the table.
  • CLI exit codes: a bundle with no language-model route exits 1; an empty prompt or request, a malformed --email and an unknown option to help exit 2.
  • bricklayer_json_to_json() is byte-identical to jsonlite's toJSON() where jsonlite does not pass an option on: raw = "js" is written with jsonlite's defaults (new Uint8Array([35]), an array, even under auto_unbox, where it wrote the number 35), raw = "hex" is unpadded ("f"), a Date epoch never gets a decimal point, the mongo $date and $binary objects and keep_vec_names objects stay on one line under pretty, and keep_vec_names does not reach the columns of a data frame. 18,000 random objects now encode identically (30 in 3,000 differed).
  • bricklayer_json_serialize() is no longer called lossless: like jsonlite's serializeJSON() it rounds doubles to digits (8); digits = I(17) round-trips them exactly. ?bricklayer_llm_login says a token is checked with the gateway before it is stored.
  • CI checks on Intel macOS (macos-15-intel), where r-universe's check had failed.
  • hqc_compress_key() stores an HQC secret key as its seed alone: for v5 the specification's compressed format dk = seed_KEM (32 bytes), for round 4 the 96 to 112 bytes its key generation draws (round 4 defines no compressed format). hqc_decapsulate() takes such a key, re-der...
Read more