-
Notifications
You must be signed in to change notification settings - Fork 104
Glossary
Tier: Beginner
Terms used across qsv's docs and this wiki, alphabetical.
The least frequently occurring value(s) in a column — the opposite of mode. Useful for spotting anomalies. Configurable max length via QSV_ANTIMODES_LEN (default 100 chars). Computed by stats --everything or stats --mode. See Aggregation & Statistics → stats.
A library of approximate algorithms with bounded memory and bounded error. qsv uses two:
-
t-digest — approximate quantiles (
--quantile-method approx) -
HyperLogLog — approximate cardinality (
--cardinality-method approx) -
Frequent Items (Misra-Gries) — approximate top-K (
frequency --sketch-method frequent_items)
Trade ~1.5% error for O(1) memory per column. Critical for stats and frequency on huge files.
A time-series join that matches each left-side row to the nearest right-side row by a key — typically the nearest prior timestamp. Implemented by joinp --strategy backward. Use case: enrich an events log with the weather observation that was in effect at the time of each event. See Recipe: Multi-Table Joins.
qsv's term for commands that use the stats / frequency cache to work smarter without user intervention. frequency short-circuits all-unique columns; pivotp auto-picks aggregations by data type; validate derives schema rules from inferred stats. See Stats Cache & Caching.
A modern cryptographic hash function — faster than SHA-2, mmap-friendly, multithreaded. qsv's blake3 command computes BLAKE3 hashes for cache keys, pipeline gating, and integrity checks. See Indexing, Compression & Diff → blake3.
The number of distinct values in a column. A column with all-unique values has cardinality equal to row count. Use stats --cardinality to compute. The cardinality value drives a lot of qsv's automagic — frequency short-circuits high-cardinality columns; validate decides whether to use an enum constraint.
An open-source data portal platform (ckan.org). qsv has first-class CKAN integration: safenames for CKAN Datastore column-name rules, applydp for DataPusher+ pipelines, ckan:// URL scheme in validate dynamicEnum and Luau qsv_register_lookup. See Recipe: CKAN Integration.
Comma-, tab-, and semicolon-separated values respectively. qsv auto-detects the delimiter based on file extension; override with --delimiter or QSV_DEFAULT_DELIMITER.
A dc:<name> input handle that reads a resource previously cached by qsv get. Any qsv command can use it (e.g. qsv stats dc:data.csv); the blob is decompressed on the fly and stale entries are auto-refreshed per the entry's TTL/refresh policy. See Get & Disk Cache.
%dyncols: {col1:fmt1},{col2:fmt2} is qsv's syntax (used by geocode and apply) for producing multiple new columns from a single command in one pass. The left side of each : is the new column name; the right is the format spec.
Commands marked 🗄️ accept directories, .infile-list files, snappy-compressed inputs, and stdin uniformly. Examples: cat, headers, sqlp, to, validate. See README → Extended Input Support.
See Apache DataSketches.
A table with field, value, count, percentage, rank. Built by qsv frequency. The companion to stats for categorical columns.
A <file>.idx sidecar that maps row numbers to byte offsets. Built by qsv index. Lets count, sample, slice work in O(1) and enables multithreading for stats, frequency, split, schema. See Indexing, Compression & Diff → index.
A jq-like JSON query language. Used by qsv fetch / fetchpost --jaq flag to extract values from JSON responses. github.com/01mf02/jaq.
Polars's deferred-evaluation API. A LazyFrame builds up a plan without reading data until you .collect(). qsv's sqlp, joinp, and pivotp use lazy frames internally to enable larger-than-RAM and multithreaded execution.
A reference CSV indexed by its first column, used at runtime by Luau / template / validate. URL schemes: file://, http(s)://, dathere://, ckan://. See Lookup Tables.
A robust measure of variability — median of |x - median(x)|. Tolerates outliers far better than stddev. Computed by qsv stats --mad.
An open protocol for exposing tools to LLMs. qsv ships an MCP server (qsvmcp binary + .mcpb bundle) that turns ~60 qsv commands into tools any MCP-aware client can call. See MCP Server.
A Rust port of Jinja2, the Python templating engine. Used by qsv template, qsv fetchpost --payload-tpl, and qsv's describegpt prompts. See Scripting (Luau / Python) → template.
A hybrid AI approach that combines symbolic computation (deterministic algorithms — qsv's stats and frequency caches) with neural computation (LLM-generated narrative). qsv describegpt is qsv's flagship neuro-symbolic command. See AI & Documentation → describegpt.
A join where the matching predicate is not equality — typically a range comparison. joinp --non-equi 'salary_left >= min_right AND salary_left <= max_right' matches employees to salary bands. See Joins & Set Ops → joinp.
A vectorized DataFrame engine in Rust (pola.rs). Powers qsv's sqlp, joinp, pivotp, scoresql, plus reading capabilities in count, lens, color, prompt, schema. Polars supports larger-than-RAM and is highly multithreaded.
A <file>.pschema.json sidecar that tells Polars the column dtypes upfront. Skips the inference scan; speeds up sqlp / joinp / pivotp. Generated by qsv schema --polars. See Validation & Schema → schema.
A robust-statistics library by Andrey Akinshin (pragmastat.dev). qsv's pragmastat command computes:
- Hodges-Lehmann center — median of pairwise averages (tolerates 29% data corruption)
- Shamos spread — median of pairwise absolute differences
Better than mean/stddev for skewed or outlier-heavy data. See Aggregation & Statistics → pragmastat.
Replacing identifying values with stable, repeatable identifiers. Same input always maps to same output. qsv pseudo does this. For irreversible hashing, use qsv blake3 --keyed. See Transform & Reshape → pseudo.
The default sampling method in qsv sample when there's no index. Visits each row exactly once, keeps memory proportional to sample size, produces a uniform-random sample. Wikipedia.
The CSV format spec: comma delimiter, double-quote field quoting, CRLF line endings. qsv validate <file> (no schema) checks RFC 4180 conformance plus UTF-8 encoding.
Human-readable markdown enriched with light, agent-parseable conventions that a companion converter turns into structured JSON (semanticmd.org). Available as the semanticmd output format in qsv describegpt (20.2.0+), which emits the Data Dictionary this way. See AI & Documentation → Semantic Markdown output.
A metric in qsv stats measuring how close a column is to being sorted, on a scale from -1 (perfectly reverse-sorted) to +1 (perfectly sorted), with 0 being completely random. Useful for picking whether to use dedup vs dedup --sorted (the latter is constant-memory streaming).
A <file>.stats.csv.data.jsonl sidecar produced by qsv stats --stats-jsonl. Many smart commands consume it. The single biggest performance win for qsv users. See Stats Cache & Caching.
A command that processes one row at a time without loading the full file. Most qsv commands are streaming. Non-streaming commands are marked 🤯 in the README legend.
A compact JSON encoding designed for LLM prompt efficiency (toonformat.dev). Available as an output format in qsv describegpt and qsv frequency.
See Apache DataSketches.
A standard text encoding for vector geometries (points, lines, polygons). qsv's apply operations wkt_point builds WKT POINT(lat lon) values; geoconvert --geometry <col> reads WKT-formatted geometry columns. See Geospatial.
A formal schema language for XML. qsv moarstats maps inferred column types to their most-specific XSD datatype — useful for data publishing.
- README — full legend symbols
-
docs/STATS_DEFINITIONS.md— every stats metric, defined - FAQ — conceptual questions
- Command Reference
qsv — GitHub · Releases · Discussions · qsv pro · Try it online · Benchmarks · datHere · DeepWiki · Dual-licensed MIT / Unlicense
Edit this page: Contributing to the Wiki
Home · Why qsv? · Tier legend
- All Commands (index)
- Selection & Inspection
- Transform & Reshape
- Aggregation & Statistics
- Joins & Set Ops
- SQL & Polars
- Validation & Schema
- Metadata Profiling (profile)
- Conversion & I/O
- Geospatial
- HTTP & Web
- Get & Disk Cache
- Scripting (Luau / Python)
- Indexing, Compression & Diff
- AI & Documentation