29.0.0

Highlights

~104 commits since 28.0.0, and the theme is correctness. We develop qsv with the latest models from Anthropic and review every change adversarially with OpenAI's. Codacy adds one more layer of correctness review on top of that, and it gives us free access to its platform. With all three, we swept several core commands for correctness: stats and moarstats, luau, profile, scoresql, sqlp --format avro, and every command that reads piped stdin. Thank you, Codacy.

If you run stats on data written by pandas or numpy, upgrade:

• stats no longer crashes on NaN cells (#5053). In qsv 28.0.0 and 27.0.1, stats --median, --quartiles or --everything panics on a float column of a few hundred rows or more that holds a NaN, nan or NAN cell, which is how pandas and numpy write missing values. So does moarstats, which runs stats. On smaller columns, the sum came out empty and the quartiles, MAD, IQR, fences and percentiles were empty or shifted. Every numeric statistic now skips NaN cells, as the mean and --quantile-method approx already did. A NaN cell still counts as a value for nullcount, sparsity and cardinality.

Also in this release:

• validate --sarif writes JSON Schema errors as a SARIF 2.1.0 log for GitHub code scanning and other SARIF tools, with an example GitHub Actions workflow (#5031).
• Piped stdin works in eleven more places. sample, cat, applydp, geocode --batch 0, input --skip-lastlines, partition, split --chunks, searchset, frequency, sniff and extsort dropped rows, wrote only the header, or rejected - when reading from a pipe, several of them with exit 0 (#5003, #5005, #5009, #5012).
• Maps render in sandboxed iframes such as CKAN's Webpage view. Before, every MapLibre map there stayed blank (#5033).
• The stats cache is no longer served stale when QSV_NO_HEADERS, QSV_STATS_SEPARATOR or a sniffed or default delimiter changes, or after an upgrade with --infer-dates (#5049, #5051, #5052).
• stats --cardinality, --mode and --everything are 12–27% faster and use up to 36% less memory, with and without --weight (#5011, #5013).
• sqlp runs on a newer Polars that adds correlated subqueries, LATERAL, PERCENTILE_CONT/PERCENTILE_DISC, CREATE VIEW, BOOL_AND, COUNT_IF, MAX_BY, REGEXP_EXTRACT and more. It also fixes two wrong results (#5055 and earlier bumps).
• Dependency bumps for security fixes, and bundled-library updates.
  • Security:
    • The SQLite that to sqlite bundles is now 3.54.1, which fixes RUSTSEC-2026-0459 (#5022).
    • The Polars bump moves to zstd 0.14, which clears 13 "unsound" advisories (#5021).
    • The MCP server takes hono and @hono/node-server security releases (#5042, #5043).
  • Bundled libraries:
    • Luau is now 0.745, with native-code (luau-jit) crash and code-generation fixes (#5026).
    • driver.js, the engine behind the Data Schematic Tour, is now 1.14.0 (from 1.10.2). Enter now advances every step, and the spotlight follows elements that scroll or re-render (#5038, #5057).

This is a major release because some sqlp, validate, profile --strict, luau and stats results and exit codes change. See Breaking Changes. The MSRV is now Rust 1.100.

🙏 Contributors

Thanks to everyone who made 29.0.0 possible:

• @acdha for reporting that sample behaved differently on stdin (#4999). That report led to the stdin fixes across eleven commands.
• @mitsuhiko for adding a bundled time zone database fallback to MiniJinja's datetime filters at our request (mitsuhiko/minijinja#1008). With it, template handles legacy zone names such as EST.
• @ik1ne for merging the zip 9 bump in webdriver-downloader and releasing it as 0.18.0 (ik1ne/webdriver-downloader#67), which let qsv drop its fork patch

---

Breaking Changes

• sqlp: EXPLAIN takes queries only (#5055, pola-rs/polars#30798). EXPLAIN DROP TABLE, EXPLAIN CREATE VIEW, EXPLAIN CREATE TABLE, EXPLAIN TRUNCATE TABLE and EXPLAIN SHOW TABLES now fail with EXPLAIN supports queries only. Before, EXPLAIN ran such a statement to get its plan, so EXPLAIN DROP TABLE t dropped the table.
• sqlp: some results change with the Polars bump. See the Polars entry under Changed for the full list:
  • IS DISTINCT FROM binds tighter than AND/OR (pola-rs/polars#30367).
  • POSITION/STRPOS of a NULL string returns NULL, not 0. String length, position and date-part functions return Int64 (pola-rs/polars#30493).
  • A window function query with no outer ORDER BY may return its rows in a different order (pola-rs/polars#30494).
• validate: rows that passed can now fail with jsonschema 0.61.2. The date-time, date and time formats require digits where RFC 3339 does. . in pattern no longer matches \r, U+2028 or U+2029. Whole floats past 2^53 are compared exactly against numeric bounds. See Fixed for the details.
• profile --strict fails where it used to pass (#5025). With --validate, it fails on a file that fails qsv validate, or when that pass cannot run. With or without --validate, it fails on an --initial-context override that cannot be applied. Before, all of these exited 0.
• luau: an END script error exits with 1 (#5036). Before, it printed the error and exited 0. BEGIN and MAIN script errors already exited with 1.
• stats output changes. See Fixed for the details:
  • Every numeric statistic of a float column with NaN cells is now computed without them, and a column of only NaN cells has no sum (#5053, #5056).
  • With QSV_NO_HEADERS set, fields are numbered 0, 1, …, as with --no-headers. Before, they were named after the first row while that row was also counted as data (#5049).
  • --mode without --cardinality reports *ALL for an all-unique column, instead of listing its values (#5050).
  • max_precision changes for values below 0.00001 or from 1e16 up, and no longer counts NaN or inf cells (#5050).
  • With --weight, a column whose rows all have a zero or negative weight has no median, quartiles, MAD or percentiles, and the *ALL antimode counts only rows with a positive weight (#5050).

Added

• validate --sarif <file>: write JSON Schema validation errors as a SARIF 2.1.0 log (#5031), for code scanning tools such as GitHub code scanning. Each invalid field becomes a result. The result has a rule per JSON Schema keyword (plus dynamicEnum, uniqueCombinedWith and ragged-row with --split-ragged) and points at the line its row starts on. That line accounts for multi-line quoted fields, CRLF line endings, blank and QSV_COMMENT_CHAR comment lines, and QSV_SNIFF_PREAMBLE preambles. A compressed or converted input (e.g. .csv.gz) has no such line, so its results name only the file. A run that stops early (on a ragged row without --split-ragged, or a read error) writes no SARIF file, and removes one left by an earlier run so it can't be uploaded as current. Invalid UTF-8 doesn't stop a JSON Schema run; it is reported like any other error. The file is written even when every row is valid (with no results), so a CI run clears earlier findings. Output stops at 25,000 results by default, GitHub code scanning's limit per run, with a notification when results are cut. --sarif-max-results <n> sets another cap for tools that accept more, and 0 means no limit. The file URI is relative to the root of the git checkout the CSV is in, so results point at the right file wherever qsv runs; outside a checkout, it is relative to the current directory. A --sarif path that is the input CSV or the schema (including through a link) is refused. The option works only in JSON Schema mode with a file input; RFC 4180 mode and stdin reject it. The validation-errors.tsv report is unchanged. An example is in resources/examples/validate-sarif: a sample CSV with one error of each kind, its schema, the SARIF log qsv writes for them (a test keeps it current), and a GitHub Actions workflow that uploads it to code scanning.
• docs: "Statistics at a Glance" in docs/STATS_DEFINITIONS.md: qsv compiles 159 statistic columns across stats (48), moarstats (76 + 14 bivariate), pragmastat (18) and frequency (3). A new test checks the table against real output, so the counts can't go stale silently (#4996).
• docs: the viz USAGE documents stdin input (#5018). Pass - as <input> to chart piped CSV. Every chart type accepts it except smart, which needs a file for its cached statistics. With --geojson auto or census, stdin is first copied to a temp file. There's also a new piped-input example.
• profile --initial-context: a final - in a dataset_info pointer appends to an array (#5025, #4991). For example, /projection/hasQualityMeasurement/- adds an agency's own quality measurement after the computed ones, so it no longer takes a second run to seed the whole array. - appends only to an array that exists; a numeric index still replaces the item in place.
• docs: the profile --locate census USAGE gives the state-column threshold (#5025): a state column lists states only when at least 90% of its values are US states.

Changed

• MSRV is now Rust 1.100 (was 1.99; #5039).
• geocode: the default index is rebuilt from current Geonames data, so some place labels differ from 28.0.0. For example, Toronto now has an admin2 (county-level) name of "Toronto", so %city-admin2-admin1 gives "Toronto, Toronto, Ontario" where it gave "Toronto, , Ontario". Antwerp's admin2 is now "Antwerp", not "Antwerp Province". To keep 28.0.0's labels, load that release's index with qsv geocode index-load.
• build: qsvlite and qsvdp no longer compile crates they don't use. Rust 1.100's new unused_dependencies Cargo lint found eight dependencies that only the full qsv binary needs. They are now optional and turned on by the features that use them: const_format and minijinja-contrib (feature_capable, and profile for minijinja-contrib), semver (self_update), and dunce, ext-sort, grex, hostname-validator and jaq-json (feature_capable and lite). Every binary is built from the same code as before, so the lite and dp builds just compile a little faster (#5039).
• viz smart: the guided Tour's vendored driver.js is now 1.14.0 (was 1.10.2; 1.12.1 in #5038, then 1.13.0 and 1.14.0 in #5057):
  • Enter moves to the next step on every step. Each popover focuses Next instead of the close button, and the Tour no longer moves focus off Next on the first step. Before, Enter did nothing on the first step and ended the tour on every later one.
  • Escape pressed in a text field inside the spotlighted element, such as the data viewer's search box, goes to that field, and the drawer stays open. Before, it ended the tour. While a tour runs, the dictionary and data drawers leave Escape to the tour, so Escape anywhere else ends the tour and closes the open drawer, as before.
  • Elements inside the spotlighted area keep their own pointer-events: none; before, driver.js forced every one of them to auto.
  • From 1.13.0 and 1.14.0: the spotlight and popover cover only the part of an element that is in view, so an element wider than its scrolling container is highlighted inside it. The highlight follows a panel that scrolls inside the page, and a step follows its element when the page re-renders it. The popover scrolls into view only as far as it needs to.
  • The bundle is larger: each Tour page embeds about 12.5 KB more (gzip + base64). The embedded payloads, the QSV_VIZ_CDN jsDelivr URLs and their SRI digests are updated together, and every gallery Data Schematic is regenerated with them.
• deps: the webdriver-downloader fork patch is gone (#5027). ik1ne/webdriver-downloader#67 (zip 9) was merged and released as 0.18.0. qsv's plotly pin moves to dathere/plotly rev d43be37, whose plotly_static build dependency requires 0.18, so webdriver-downloader now comes from crates.io. The fork was added in #4828 because every zip 8.x was yanked.
• deps: other bumps: calamine 0.38 (used by excel and describegpt's spreadsheet context files), rfd 0.18 (used by prompt), toml 1.2. csvlens (used by lens) now comes from a fork of YS-L/csvlens with updated dependencies.
• luau: Luau bumped to 0.745 (from 0.740, #5026). luau0-src 0.23.0+luau745 needs an unreleased mlua-sys, so mlua is patched to upstream rev a0ad453 until an mlua release includes it. The crate versions are unchanged (mlua 0.12.2, mlua-sys 0.13.0). The yank that reverted the 0.744 bump (#4923) is resolved: 0.23.0 replaces the yanked 0.22.1+luau744.
  • Native code (luau-jit) fixes (0.744): an Arm64 crash in functions with more than 256 constants, a dead-store-elimination bug, a FORGLOOP register reservation bug, and bit32/integer correctness fixes (lrotate, extract, replace).
  • Faster: string.gsub with patterns that have no special characters (0.742), table.find (0.743), and table.create/table.clear as fast calls (0.745).
• profile --validate --strict fails on a file that fails qsv validate (#5025). Before, an input with inconsistent field counts or quoting, or invalid UTF-8, passed --strict: it was recorded only as a syntacticValidity measurement of "false". It now also gets a Required syntacticValidity warning in projection_warnings, and --strict fails on it. The same happens when the qsv validate pass cannot run, since profile can then not vouch for the file.
• profile --strict fails when an --initial-context override cannot be applied, with or without --validate (#5025, #4991). See Fixed.
• stats: --cardinality, --mode and --everything are faster and use less memory (#5011). qsv-stats 0.57 stores short values inline when counting them (ByteKey). On a 1M-row file, --cardinality --mode is 16–27% faster, and --everything on an indexed file is 27% faster with 34% less peak memory. Output is unchanged; default stats is unaffected. On big-endian targets (s390x), values of 128 bytes or more were misread until a compact_bytes fix (#5015), so this was never released with the bug.
• stats --weight: mode, antimode and cardinality tracking is faster and uses less memory (#5013). The weighted tracker now stores short values inline too, as #5011 did for unweighted stats. On a 1M-row file with --weight, --cardinality --mode is 12–20% faster, and --everything on an indexed file is 21–23% faster with 36% less peak memory. Output is unchanged.
• scripts/benchmarks.sh 10.1.1: the joinp benchmarks check their exact row counts (#4997). Before, three of them only checked for at least one output row and joinp_left_anti had no check, so a partial join could be published as a speedup. Now inner and --streaming must return 883,470 rows, --left all 1,000,000, and --left-anti 116,530 (cross-checked with qsv join and DuckDB).
• scripts/benchmarks.sh 10.1.2: the sqlp and frequency benchmarks check their results (#5019). An empty or partial result that still exits 0 is now quarantined, not published as a speedup.
  • The 25 frequency benchmarks require every field's counts to add up to the row count, and every selected column to be present.
  • 13 sqlp benchmarks must return the exact expected rows or groups (cross-checked with DuckDB), in CSV, JSON, JSONL, Parquet and Arrow output.
  • Also fixed: an output check on a benchmark that has a --conclude or --cleanup hook checked the hook's output instead, because hyperfine writes it to the same file.
• deps: Polars bumped to upstream rev 2d9938d (from py-2.1.0, 283 commits: 0b47007 in #4993, a598c51 in #5021, a9bce97 in #5030, ff1b37d in #5034, 8ecd1ca in #5041, 61fdb30 in #5045, 36ea0f9 in #5048, then 2d9938d in #5055). The 2d9938d step is 8 commits. Its one sqlp change: EXPLAIN now takes queries only (#30798). Before, EXPLAIN over another statement ran it to get a plan, so EXPLAIN DROP TABLE t in a script dropped the table and EXPLAIN CREATE VIEW ... created the view; these now fail with EXPLAIN supports queries only, as do EXPLAIN CREATE TABLE, EXPLAIN TRUNCATE TABLE and EXPLAIN SHOW TABLES. It also has two speedups: group-bys skip hashing and table lookups for runs of equal string keys (#30794), and dictionary casts and sorted searches copy less (#30790). No sqlp or pivotp result changed in side-by-side tests of GROUP BYs, joins and window functions over sorted and clustered string keys (with and without --streaming), and of every pivotp aggregation. A pivot fix for pl.element() inside eval aggregations (#30781) doesn't apply: pivotp builds no eval aggregations. The other commits are Python- or Iceberg-only, or need a polars feature qsv doesn't enable (rolling_*_by, #30795). The 36ea0f9 step is 17 commits and changes no sqlp result that already worked. It adds SQL that used to fail: comma-separated LATERAL derived tables, and aggregates without GROUP BY inside them (FROM t, LATERAL (SELECT MAX(v) FROM t2 WHERE t2.k = t.k) a, LATERAL (...) b; #30768); CREATE VIEW, DROP VIEW and DESCRIBE <table> (#30778); and a leading ~ in read_csv/read_parquet/read_ipc/read_ndjson paths now means the home directory (#30774). Date parts are now computed with integer arithmetic (#30777); every EXTRACT field gave the same result on both revs for 21,260 dates and timestamps from year 1 to 9999. Arithmetic with a literal on the left now plans with the type it runs with (#30765), and temporal total_* overflows give NULL (#30743); neither changed a sqlp result in side-by-side tests. The 61fdb30 step is 11 commits. Its one sqlp result change is a fix: a signed integer literal combined with an unsigned value no longer gives NULL for the negative rows (#30742). For example, CASE WHEN side = 'buy' THEN 1 ELSE -1 END * CAST(qty AS UINTEGER) returned NULL for the -1 rows, so a SUM over it skipped them. It also adds BOOL_AND, BOOL_OR, COUNT_IF, MAX_BY, MIN_BY, ANY_VALUE, APPROX_COUNT_DISTINCT and REGEXP_EXTRACT to sqlp (#30763); qsv now enables polars' approx_unique feature for APPROX_COUNT_DISTINCT. REGEXP_EXTRACT(s, pattern[, group]) returns the whole match or the given capture group, and an empty string when nothing matches. The other fixes left sqlp results unchanged in side-by-side tests: calendar-part offsets applied in one step (#30744, checked with month-end INTERVAL arithmetic), and population variance of a single null in a sorted group-by (#30747; sqlp's VARIANCE/STDDEV are the sample forms, which this doesn't touch). The 8ecd1ca step is 14 commits of performance work: quantiles and medians computed by selection instead of sorting, in parallel (#30569, #30570), multi-column join and group-by keys packed into fixed-width integers (#30708), and joins that skip the probe side when the build side has no non-null keys (#30692). No sqlp result changed in side-by-side tests of quantiles, medians, NTILE/PERCENT_RANK, multi-key GROUP BYs and joins, and joins on all-NULL keys. The ff1b37d step is 10 commits. Its one sqlp result change is a fix: a CASE whose ELSE applies a strict CAST no longer fails on values from rows that take an earlier branch (#30682). For example, CASE WHEN a THEN 0 ELSE CAST(COALESCE(v, -1) AS UTINYINT) END used to fail when v was NULL only in rows where a is true. The rest is Decimal SUM/AVG and Parquet string decoding speedups (#30704, #30632), and slice pushdown and unnest fixes that left sqlp results unchanged in side-by-side tests. The a9bce97 step is 11 commits of join planning and performance: out-of-core joins (#30679), join reordering (#30680, #30694), a streaming build-side choice from estimated row counts (#30681), and a fix for order relaxation in sliced maintain-order joins (#30684). No sqlp result changed in side-by-side tests. Dividing a number by a date or timestamp still fails, now with a clearer message when the schema is resolved (#30163). The a598c51 step is mostly performance, including a faster CSV reader (#30639) and spillable streaming joins; no sqlp result changed in side-by-side tests. Crates stay 0.55.1; it moves to zstd 0.14 (#30316), so the 13 zstd "unsound" advisory ignores (#4930–#4942) are dropped from .cargo/audit.toml. Some sqlp results that already worked change:
  • IS DISTINCT FROM binds tighter than AND/OR (#30367). a IS DISTINCT FROM b AND c OR d is now ((a IS DISTINCT FROM b) AND c) OR d; before, the AND/OR became part of the right operand.
  • POSITION/STRPOS of a NULL string is NULL, not 0 (#30493). String length, position and date-part functions also return Int64.
  • Known issue: COUNT(*) over a UNION ALL ... LIMIT panics again (not implemented for dtype UInt128). The #30150 fix regressed in #30363; reported as pola-rs/polars#30523. Workaround: count a column (COUNT(g)) instead of COUNT(*), or add an ORDER BY inside the limited union. (COUNT(*) over a UNION ALL without LIMIT panicked on py-2.1.0 too.)
  • A window function query with no outer ORDER BY may return rows in a different order (#30494). Add an ORDER BY if you depend on row order.
  • New SQL: join keys of different numeric types (#30401); correlated subqueries with any outer condition, GROUP BY, HAVING, LIMIT, ANY/ALL, and LATERAL derived tables (#30378, #30453, #30466, #30459, #30490, #30467); PERCENTILE_CONT/PERCENTILE_DISC ... WITHIN GROUP (#30442); non-literal LIKE patterns (#30364, #30372); typed numeric/string literals (#30486); LIMIT/OFFSET expressions such as scalar subqueries (#30425); repeated unnamed constants (#30488); RANDOM(), REPEAT, CARDINALITY, constant GROUP BY keys and row values in IN (subquery).

Fixed

• stats: a NaN cell in a float column is skipped by every numeric statistic (#5053). A NaN, nan or NAN cell is read as a float (pandas and numpy write missing values this way). The mean already skipped it, but the sum came out empty, and the exact median, quartiles, IQR, MAD, fences, skewness and percentiles were computed with it. On a column of a few hundred rows or more that crashed stats (qsv 28.0.0 exits with a panic on stats --quartiles, as does moarstats, which runs it); on a smaller one they came out empty or shifted. With --weight, one such cell emptied the weighted mean and standard deviation of the whole column. They now all skip it, as --quantile-method approx already did, so the statistics are those of the column's other values. The cell still counts as a value: nullcount, sparsity and cardinality are unchanged, and the column stays Float. That holds when the other values are all integers too: such a column is summarized as floats, so its minimum and maximum read 1.0 and not 1, and its sum is no longer exact past 2^53. A column whose cells are all NaN has no sum, as it has no mean (#5056). inf cells are unchanged.
• stats: the cache follows the delimiter set by QSV_SNIFF_DELIMITER and QSV_DEFAULT_DELIMITER (#5052). The stats cache recorded only the --delimiter flag. A semicolon-separated data.csv read as one comma-delimited column kept that one-column cache after QSV_SNIFF_DELIMITER was set, so stats and the commands that read its cache went on describing the wrong columns. The cache now records the delimiter the file was read with, and stats and its readers compare against the delimiter they resolve the same way. Checking the cache still does not decompress a compressed or Parquet input. A cache written before this change did not record the delimiter it was read with, so it is recomputed on its next use.
• moarstats: two QSV_STATS_SEPARATOR fixes (#5051):
  • mode_zscore is computed for a column with several modes under any separator. The first mode was looked up by splitting on |, so with another separator a multi-modal numeric column had an empty mode_zscore.
  • A stats CSV written with another separator is recomputed. Before, its percentile cells could not be split, so kelly_skewness and percentile_ratio_90_10 were left out without a warning. moarstats now reads the separator recorded in the stats cache metadata (#5049) and reruns stats when it differs, with a warning. A stats CSV that has no metadata file is used as it is.
• stats: three stale-cache fixes (#5049):
  • QSV_NO_HEADERS now counts for the stats cache and the field names. Before, the cache only recorded the --no-headers flag, so stats computed with the header row as names were served to a run that set the environment variable (and the reverse). A run with the variable also named the fields after the first row while counting that row as data; it now numbers them 0, 1, ... as --no-headers does. The commands that read the stats cache check it against the same resolved setting, so sortcheck and extsort still use a cache built under the variable.
  • stats recomputes when QSV_STATS_SEPARATOR changes. Before, it served its cache written with the previous separator, so modes, antimodes and percentiles kept the old one. This covers stats itself; moarstats still reads an existing <FILESTEM>.stats.csv as it is.
  • --infer-dates sniffs the date columns again after a qsv upgrade. With the default --dates-whitelist sniff, the column list sniffed by the previous version was reused for the recompute that the upgrade triggers, and saved into the new cache. It stayed until the file changed or --force was used.
• stats: wrong max_precision, weighted quantiles and *ALL antimodes (#5050):
  • max_precision is right for very small and very large floats. A value below 0.00001 or from 1e16 up was miscounted: 0.000001 reported 4 decimal places, and 1e21 reported 5. They now report 6 and 1. NaN and inf cells no longer count toward it.
  • --weight: a column whose rows all have a zero or negative weight has no median, quartiles, MAD or percentiles. Before, these fell back to the unweighted values, next to a weighted mean of 0. Weighted runs also use less memory: they no longer keep a second, unweighted copy of each numeric column (8 bytes per row).
  • --weight: the *ALL antimode counts only rows with a positive weight. Before, one zero-weight row made an all-unique, equally weighted column list every value as a mode.
  • --mode without --cardinality reports *ALL for an all-unique column. Before, it listed the values (*PREVIEW: ... past 10) and gave their number as antimode_count; *ALL appeared only with --cardinality or --everything. A column with no rows no longer reports *ALL.
• viz smart: category-axis labels no longer land under the wrong bar when the categories look like numbers. (#5047) Frequency bars, measure-by-dimension bars, the Top Relationships lollipop and funnels set their axis labels with tickvals holding each category's position (0, 1, 2…). Plotly.js matches a category-axis tickval against category names first, so on an axis of codes like day_of_week (6, 7, 5, …, 0) the label for position 0 went under category "0", and so on. Hovering a bar showed one value while the axis said another, and labels that collided were dropped. tickvals now hold the category values themselves. This needed Axis::tick_values to accept strings, so qsv's plotly pin moves to dathere/plotly rev b652677. The gallery Data Schematics are regenerated.
• docs: sqlp, joinp and schema USAGE corrections (#5046). sqlp's <=> is NULL-safe equality (the same as IS NOT DISTINCT FROM: true when both sides are equal or both are NULL, never NULL), not a -1/0/1 "spaceship" comparison. A Polars schema file is named after the full input file name, so data.csv's is data.csv.pschema.json, not data.pschema.json.
• luau: fixes from a review of luau.rs (#5036):
  • An existing ./date.lua is no longer overwritten and deleted. A script using require "./date" made qsv write the bundled LuaDate over any date.lua in the current directory, then delete it at exit. An existing file is now used as-is and left alone. The bundled copy is also removed when the run fails; before, a failed run left it behind.
  • luau filter no longer keeps every row after the first error. The keep check looked at the running error count, so after one row errored, every later row was kept whatever it returned. An errored row is still kept.
  • qsv_insertrecord() rows have the output's width. It padded and truncated to the input's column count (and let one extra value through), so the inserted row was right only for map with exactly one new column. With two new columns, under filter, or with too few values, the run failed with found record with N fields. With --no-headers and unnamed columns, an insert from the MAIN script takes the mapped row's width; one from BEGIN still assumes a single new column, since no row has been mapped yet.
  • After a MAIN-script qsv_insertrecord(), later rows no longer each get a blank inserted row.
  • qsv_insertrecord() keeps numbers and booleans. They were written as empty fields; qsv_insertrecord("p", 42, true) now writes p,42,true.
  • luau map --remap works with a single return value for one new column. Only table returns dropped the input columns, so map c --remap 'return a * 2' failed with a field-count error. (A single value for two or more new columns still fails, with or without --remap; return a table.)
  • An END script error now exits with 1, like BEGIN and MAIN errors. Before, it printed the error and exited 0.
  • The END script's output is printed with --progressbar. It was dropped whenever the progress bar was on.
  • Script keywords are matched as whole words. A column whose name contains return (e.g. returns) no longer stops qsv adding the implicit return to a one-expression script. A column whose name contains _INDEX or _LASTROW (e.g. PRICE_INDEX) no longer switches on random-access mode, which needs an index.
  • The random-access progress bar's length is now the row count; it was the starting row.
• viz: map panels now load in a sandboxed frame without allow-same-origin (#5033, #4946), such as the CKAN Webpage view the wiki recommends (sandbox="allow-scripts allow-popups allow-downloads"). Before, every MapLibre map (viz map, viz choropleth --map, and the point and region maps in a Data Schematic) stayed blank there with no error. The MapLibre in plotly.js 3.x drops messages between the page and its web worker when their origins differ, and in such a frame the page reports its URL's origin while the worker reports "null". Each page now carries a small Worker wrapper that gives those messages the origin MapLibre already accepts for file:// pages. It only acts in a sandboxed frame that isn't a file:// page, so maps elsewhere behave as before. Checked in Chrome, Firefox and Safari, for region and clustered point maps. The note added in #4947 above each map in such a frame is gone, since the maps now render. The wrapper can go once plotly.js bundles MapLibre 6.17 or later (maplibre-gl-js#8912).
• validate is stricter and more correct with jsonschema 0.61.2 (from 0.60.2). Rows that passed before can now fail:
  • format: "date-time", "date" and "time" require a digit wherever RFC 3339 does (jsonschema#1829). Before, +, ,, -, . or / passed in many digit positions, so 2026-11-0/T10:00:00Z and -----11-06T10:00:00Z were valid date-times.
  • Numeric keywords judge whole floats past 2^53 correctly (jsonschema#1825). A float equal to an integer bound past 2^53 was converted to the wrong integer, so exclusiveMinimum admitted the bound itself and const rejected it.
  • . in pattern and patternProperties no longer matches \r, U+2028 or U+2029, so a value containing one of these line terminators can now fail a pattern it passed before.
  • Schema fixes: oneOf/anyOf no longer reject valid rows when a branch's $ref sits under an $id, and $ref pointers through a numeric key (#/responses/200) now resolve the references under their target.
  • Validation is slightly faster. On the 1M-row NYC 311 sample (release build; a 41-property schema with type, length, enum and range checks), validate executes 3.7–4.0% fewer instructions when every row is valid, and 2.6–2.7% fewer with one error per row. That makes it 2–3% faster with --jobs 1, with no measurable wall-clock change at the default thread count. Upstream's larger figures (up to 40% with preserve_order) are best cases for the validation call itself; these are end-to-end results on this file.
  • Large bounded counts over one character class compile and work (0.61.1). Patterns such as ^.{0,262144}$ or [^\p{C}]{1,2048} are no longer unrolled, so they fit the default --size-limit. One such as ^.{0,404600}$ no longer fails with a regex engine error on an empty value once --size-limit is raised to compile it. Anchored character-class patterns such as ^[a-z0-9-]+$ are also matched without a regex engine. This needs no change to the schema. These speedups weren't measured on qsv's benchmark file, whose schema has no patterns.
• scoresql no longer marks down well-formed queries (#5028):
  • Join keys are found when ON starts a line or the keys are unqualified (JOIN b\nON a.id = b.id, ON id = id2). Before, such joins were reported as "possible cartesian join" and lost 25 points.
    A join inside a subquery ends at the subquery's closing parenthesis, so outer-query columns aren't scored as join keys.
  • WHERE and ORDER BY are matched as whole words when collecting their columns. In SELECT nowhere FROM w WHERE ..., the WHERE inside nowhere was taken for the clause, so no WHERE columns were found and the data-distribution and skew checks skipped them.
  • Filter selectivity says when it scored only one of a column's values. Only a column's first literal is looked up (status = 'a' OR status = 'b' checks 'a'); the detail now notes "has 2 filter values — selectivity estimated from the first value only".
  • A CTE body is not a nested subquery. WITH c AS (SELECT ...) lost 2 points with the advice to use a CTE. A subquery inside a CTE still counts.
  • A String column is penalized only when the query treats it as a number. Every String column in a comparison lost 3 points, so WHERE name = 'Bob' was told to cast name. Now only a String column compared with a numeric literal is penalized.
  • --duckdb handles column references qualified by a table name (SELECT data.id FROM data JOIN ...). Table names were rewritten to read_csv_auto(...) in the SQL text, so DuckDB rejected the query. Each input is now registered as a view under its table name and _t_N alias. A bare table name that isn't a plain identifier (FROM nyc-311, as describegpt writes it when scoring) is quoted for DuckDB, so it still works.
  • An index is found for any file extension. Only <stem>.csv.idx was checked, so data.tsv was reported as unindexed even after qsv index data.tsv. The suggestion to run qsv index now says what the index is for: it speeds up building the stats and frequency caches, but Polars and DuckDB don't use it for the query. It is no longer shown for stdin input, which is a temp copy deleted on exit.
  • The "full table scan in query plan" check is removed. It looked for FULL SCAN or TABLE SCAN, which neither Polars (Csv SCAN) nor DuckDB (READ_CSV_AUTO) prints, so it never applied. The plan is still shown in the report, and the USAGE text no longer lists it as a scoring factor.
• profile --validate --no-projection warns that --validate is ignored. --validate checks the projection block, so with --no-projection it ran nothing (neither the schema check nor the qsv validate pass) and exited 0 without a word, even with --strict. To check the file alone, run qsv validate.
• profile: an --initial-context override that cannot be applied is reported, not silently dropped (#5025, fixes #4991). Before, the run exited 0, passed --strict, and left no trace of the missing value. Each such override now gets a Required projection_warnings entry (on stderr with --no-projection) naming the key and the reason, and fails --strict:
  • a key that is not a JSON Pointer (doesn't start with /);
  • an array index out of range, e.g. /projection/dataset/1/... on a one-Dataset catalog;
  • a non-numeric token inside an array, e.g. /projection/dataset/title with --catalog;
  • a - with no array to append to. Before, a - under a missing or object parent wrote a "-" key where an array was meant.
• sqlp --format avro writes every result type (#5024, fixes #5020). Polars' Avro writer rejected a lot of result types, and any one of them failed the whole write with not yet implemented: write ... to avro:
  • Rejected types: TINYINT/SMALLINT, every unsigned integer, TIME, intervals, TIMESTAMPTZ and TIMESTAMP(9).
  • After --cache-schema: the cached schema makes every non-negative integer column unsigned, so a single --cache-schema run broke every later Avro export of that file.
  • Now: small and unsigned integers become int or long (a value above 2⁶³−1 is an error), zoned timestamps become UTC timestamps (the same instant), nanosecond timestamps become microseconds, and TIME and durations become strings (HH:MM:SS[.fraction] and ISO 8601), including inside lists.
  • Still not converted: struct columns from nested Parquet/Arrow input.
• security: the bundled SQLite is updated to 3.54.1 (#5022, RUSTSEC-2026-0459, CVE-2026-119375). It's used by to sqlite. Older SQLite could return uninitialized memory when opening a database next to a crafted -wal file written by someone else. The fix comes from rusqlite 0.41 / libsqlite3-sys 0.39, through the csvs_convert fork.
• sample: --cluster, --weighted, --systematic and fractional sample sizes work from stdin (#5003, fixes #4999). They returned only the header row, because they read the input twice (a row count or a second pass) and stdin can only be read once. For those methods, stdin is now copied to a temporary file first, which needs temp-directory space for the whole input. --bernoulli, --stratified, --timeseries, and integer sizes for the default method, --varopt and --mergeable-reservoir still stream stdin.
• cat rows/rowskey/columns read piped stdin when no input is given (#5005). Before, ... | qsv cat rows failed with "No data on stdin" unless you passed -. As with headers, cat with no input in an interactive terminal now waits for input. With an empty or closed stdin (cron, CI, xargs), it still fails with "No data on stdin", as in 28.0.0 (#5007), so a script whose file list comes up empty still gets an error.
• cat rowskey writes nothing for empty input (#5008). It wrote a lone "" header line for an empty file or an empty - stdin, where rows and columns write nothing. With --group, the grouping column's header is still written.
• applydp --batch 0 works from stdin (#5005). It counted rows first, which consumed stdin and left only the header; it now reads stdin in one batch, as py does.
• geocode --batch 0 no longer drops rows from piped stdin (#5009). Counting rows to size the batch consumed the stream, so rows past the first 128 KiB were silently lost, with exit 0 (also in 28.0.0). It now reads stdin in one batch without counting, as the census and geoclient back ends already did.
• template: legacy time zone aliases such as "EST" and "US/Eastern" work on every system (#5010, via minijinja#1008). datetimeformat(tz=...) failed with "unknown timezone" where the OS time zone database lacks them, e.g. Ubuntu 26.04 without tzdata-legacy. qsv now bundles the time zone database as a fallback (about 200 KiB); OS data still takes precedence.
• input --skip-lastlines works from stdin (#5012). It counted the rows first, which consumed stdin, so it wrote no rows (exit 0, also in 28.0.0). Stdin is now saved to a temp file when the flag is set.
• partition and split --chunks work with - for stdin (#5012). Only stdin with no input argument was saved to a temp file, so with - partition wrote no files and split --chunks wrote a single chunk, both exit 0, or failed with a CSV error on larger input (also in 28.0.0).
• searchset no longer fails on large piped input (#5012). It always counted rows with a second reader, which consumed stdin past the first ~64 KiB (also in 28.0.0). It now counts only for the progress bar, and --json reports the rows it read.
• frequency shows <ALL_UNIQUE> for all-unique columns from stdin (#5012). Only --json output saved stdin to a temp file, so CSV output listed the column's values instead (also in 28.0.0). Stdin is now always saved to a temp file.
• sniff - and extsort - read stdin (#5012). - was treated as a file path ("No such file or directory"); omitting the input already worked.
• to: clearer error when no input is given (#5005). It said "No data on stdin" even when data was piped. It now says no input CSV was given and to use - to read stdin.
• tests: template_contrib_functions no longer depends on the legacy EST zone alias (#5002). ubuntu-latest is now Ubuntu 26.04, which ships aliases such as EST and US/Eastern only in the optional tzdata-legacy package, so the test failed on every Linux run. It now uses Etc/GMT+5 (the same fixed UTC−5 offset). If you use datetimeformat(tz=...) in template, prefer canonical IANA names for the same reason.

CI/Actions

• The scoresql --duckdb tests run in CI (#5029). They look for duckdb on PATH and were skipped in every job, since no workflow installed it. The macOS Polars workflow now does.
• joinp_ignore_case_maintain_order_left no longer asserts the order of tied matches (#5023). --maintain-order left orders only by the left row. After the Polars a598c51 bump, the test failed about a third of the time. It now checks the left order exactly and the matched rows as a set.
• qsvlite and qsvdp build without dead-code warnings (#5040). Items those binaries never use are now gated like their callers, and the full qsv and qsvmcp builds compile exactly as before.

Full Changelog: 28.0.0...29.0.0