Skip to content

25.0.0

Latest

Choose a tag to compare

@jqnatividad jqnatividad released this 14 Oct 17:15
· 22 commits to master since this release

[25.0.0] - 2026-10-14 🌐 The "Spatiotemporal" Release ⏳

~95 commits since 24.0.0, and the theme is SPATIOTEMPORAL: where and when. Our company's name, datHere, is short for "data here", because we've always known that location and time are the universal join keys (and future clients can demand "I want that here!" πŸ˜„). A 311 complaint, a property tax roll, a building permit and a Census table share no column, yet each record happened somewhere, at some time. Pin down the where and the when, and datasets that seemed unrelated line up.

WHERE. Two new geocoders, both built on authoritative government sources, turn addresses into the geographies public data is keyed by:

  • geocode census uses the free US Census Geocoder, no API key needed. Beyond coordinates, it returns the Census geographies each address falls in, with their GEOIDs (state, county, tract, block group, block, places, congressional and legislative districts, ZCTAs and more): the keys for joining Census statistics. Large files go through Census's batch endpoints, 2,000 rows per upload.

    Be kind to the Census Geocoder. It is a free public service with no API key, so it has no per-user quota to protect it either: Census publishes no rate limit, caps batch uploads at 10,000 rows, and can throttle or block heavy or shared-IP users. We'd like it to stay free, so geocode census is deliberately polite. It keeps a persistent on-disk result cache, de-duplicates lookups, sends only uncached rows in a batch, paces single lookups at 5 requests/second, and backs off and gives up rather than hammering a struggling server. For teams and pipelines, we also built ucgeocoder, a caching, rate-limited proxy that speaks the Census API unchanged. Set QSV_CENSUS_GEOCODER_URL=http://<host>:8080/geocoder and every qsv user, script or R/Python client behind it shares one cache, so an address is fetched from Census once rather than once per person and per run. Upstream, a deployment sends at most 5 single-record requests/second and one batch upload at a time, with a circuit breaker. It keeps answering from its cache when Census is slow or down, and it can warm the cache with resumable background jobs. You get faster repeat runs, and Census gets less load.
  • geocode geoclient uses NYC's Geoclient API over the Department of City Planning's Geosupport, returning BBL, BIN, community and council districts, police precincts, school districts, NTAs and census tracts. A search that matches in more than one borough is not silently given the first one, and when Geosupport rejects a location, its reason is available in _reject.* columns (or geoclientnow's JSON). Like census, it keeps a persistent on-disk result cache (geocode-geoclient_v3.redb in the qsv cache directory) and looks up each distinct location only once. Matches, rejects with their reason, and no-match results are cached for 14 days by default (--cache-ttl; 0 keeps them indefinitely, --no-cache turns the cache off). Failed requests are not cached, so they are retried on the next run. Lookups are case-insensitive, so 314 w 100 st and 314 W 100 St share an entry. A self-hosted Geoclient (QSV_GEOCLIENT_API_URL) gets its own cache entries. Re-runs therefore cost no quota for locations already seen, which matters against the public API's 2,500 requests/minute and 500,000/day limits. geocode cache-info, cache-prune and cache-clear manage it alongside the OpenCage and Census caches. We started our entrepreneurial journey 15 years ago after winning NYCBigApps 3.0 - that's why NYC is near and dear to our hearts. We ❀️ NYC!

WHEN. A geography is a join key only for a point in time. Tract boundaries are redrawn after every decennial census, and Connecticut's nine planning regions have replaced its eight counties in Census geography, so the same address can carry different GEOIDs in a 2019 table and a 2024 one. geocode census lets you choose the moment: --vintage picks the geography release (Current_Current by default, Census2020_Current for the 2020 Census's, ACS2026_Current for the American Community Survey's) and --benchmark the address-range release. Match a dataset's vintage and its GEOIDs join; miss it and they quietly don't. qsv even looks Connecticut batch matches up again one by one, because batch results carry only the 2020 county.

Time gets its due elsewhere too. schema --polars and sqlp --cache-schema no longer write schemas in which Polars reads 04/05/2019 as 4 May while qsv reads it as April 5; writestat recognises dates and times the way Polars does, without minutes of type inference; and viz smart --smarter notes when a measure drifts in file order.

The Geonames geocoder got a review. A county-FIPS bug that dropped the first digit of codes 100 and up is fixed, missing translations no longer print blank names, index-update rebuilds in-process and keeps the index's city set and languages, concurrent first runs no longer collide downloading the index, old releases' indexes are pruned from the cache, and the command now scales across cores (up to 2.8Γ— faster with 12 jobs). geosuggest 0.9.0 also shrinks the default index by about a third.

And the rest of the geo pipeline. geoconvert now streams CSV input (857 MB β†’ 28 MB peak on a 500k-row file), skips rows it can't place instead of writing invalid GeoJSON, and no longer panics on shapefiles. viz --geocode no longer maps a place name to a bigger city whose alternate name starts with it.

Elsewhere:

  • Polars 2.0 with a major performance bump AND much-improved SQL support. It now leads DataFusion and DuckDB in TPC-H and TPC-DS benchmarks!
  • MiniJinja 3.0 hews much closer to Jinja2 and faster to boot withtemplate now ~1.6Γ— faster!

Embedding qsv? Use qsv-client. qsv is our Neuro-symbolic FAIRification engine, powering several of our projects, and other projects embed it too. From Python, qsv-client (pip install qsv-client) is now a more robust way to do that than calling a subprocess yourself. Failures raise typed exceptions (QsvCsvError, QsvTimeout, ...) built from qsv's new JSON error output, so there's no stderr to scrape. capabilities uses the new --capabilities flag to report which binary you have (qsv, qsvdp, qsvlite or qsvmcp), its version, features and commands. A timeout, cancellation or Ctrl-C stops qsv and every process it started, so an orphaned child can't hang your program, and describegpt's API keys travel in environment variables, never on the command line. Sync and asyncio clients share one API, with no runtime dependencies.

Data here, data now: with the where and the when resolved, the join is the easy part.

Note

qsv MCP Server and Cowork plugin 25.0.0:
the qsv MCP Server is a standard MCP server, so it works with MCP-capable AI assistants beyond Claude. For example, Gemini CLI has its own setup guide. Both it and the Claude Cowork plugin are updated to 25.0.0 and need qsv 25.0.0. The qsv-geocode skill now covers the new census and geoclient geocoders, and qsv-sqlp adds examples for read_csv() options and generate_series()/range(). For Claude Desktop, double-click the attached qsv-mcp-server-25.0.0.mcpb. For Cowork, install the plugin from the dathere/qsv marketplace, or use the attached qsv-data-wrangling-25.0.0.plugin. Other clients, including Claude Code and Gemini CLI, run the server from source. See the Getting Started guide and the MCP changelog.

πŸ™ Contributors

Thanks to everyone who made 25.0.0 possible:

  • @AMDmi3 for fixing the README's Repology URLs (and for creating and maintaining Repology!) (#4761)

Detailed Changelog: https://github.com/dathere/qsv/blob/master/CHANGELOG.md#v25-changes

Full Commit log: 24.0.0...25.0.0