Skip to content

OpenPlans v0.4.8

Choose a tag to compare

@openhra openhra released this 14 Nov 17:36
· 7 commits to main since this release

This release makes nearby-provider searches answer within the request limit on a database with
providers from every state. A new worker command, worker backfill provider-search, builds search
keys from the NPPES identities and their geocoded points. A radius search then reads only the
area around its origin instead of providers spread across every loaded state.

It has three migrations, 1007 to 1009, which add new empty tables and their indexes, and one
backfill to run after migrating. See Upgrading below. Search results, totals and counts don't
change. One answer does: when the count of addresses without a located point can't finish in
time, the search answers 200 and marks that count unknown. It used to answer 504. The bundled
plugins keep their v0.4.7 versions and package hashes.

What users notice

  • Provider searches in busy areas answer in time. A nearby-provider search for a common name
    such as SMITH, for a taxonomy code, or with no filter used to read identities and points in
    every loaded state. With a nationwide NPPES load it ran past the 5-second limit and answered 504
    query_timeout. After the backfill, the search reads the keys for the 0.1-degree grid cells
    around its origin, applies the same filters and distance test, and reads full records only for
    the providers on its page. The unlocated count comes from the same keys. Results, totals and
    counts match the original path.
  • The keys are used only while they match the provider tables. After an NPPES or geocode load,
    searches take the original path until the backfill runs again.

Changed

  • An unlocated count that can't finish answers 200, not 504. Without current search keys, a
    radius search counts the practice addresses that have no located point, and that count has a
    two-minute limit of its own. When the search can't wait for it, the search now answers with its
    results, meta.unlocated: null and meta.unlocated_status: "unknown". In v0.4.6 and v0.4.7 it
    answered 504 query_timeout. The response cache doesn't keep such an answer. A count that runs
    past its limit isn't started again for ten minutes, and searches in that time answer it as
    unknown. The compatibility API adds provider_unlocated_count_unknown to its gaps header, and
    the MCP search_providers tool returns unlocated: null with unlocated_status: "unknown".
  • MCP search_providers says what its unlocated count means. It now returns
    unlocated_status next to unlocated: counted when unlocated is the exact count, unknown
    when the count didn't finish in time and some candidate addresses may have no location, and
    not_applicable for an NPI lookup. Before, a null count could mean either of the last two.
  • worker migrate and worker migrate --check refuse while a provider-search backfill runs, and
    the backfill refuses while a migration runs.

Fixes

  • Background unlocated counts no longer crowd out requests. When the provider search keys
    are stale, at most one exact unlocated count runs at a time, on its own database connection
    outside the request pool. Counts and the ten-minute pause after a timed-out count follow the
    provider tables' revisions, so plan catalog writes no longer restart them.
  • scripts/smoke.sh passes on a default setup again. Household quotes are open without a token
    by default, but the smoke script still expected a 503 and failed unless you set
    OPENPLANS_PUBLIC_HOUSEHOLD=off yourself. It now checks the open default, then restarts the API
    with OPENPLANS_PUBLIC_HOUSEHOLD=off and checks the refusal. It builds its images under tags
    unique to the run and removes them afterwards, so two runs on one Docker host no longer replace
    each other's images. The new OPENPLANS_DB_IMAGE setting picks the Compose database image. Its
    default, openplans-db:local, is unchanged.
  • The backup commands in the docs no longer start the API. They now run the worker with
    --no-deps, so a restore doesn't start the API early.
  • The docs match the code. The API guide lists every route, including /v1/cache-stats,
    /healthz and /readyz, and has an error table. The configuration guide lists every config
    file key.

Development

  • Code under src/ and plugins/*/src/ no longer uses any. ESLint checks the no-unsafe-*
    rules there with type information.
  • A test checks the docs against the routes, MCP tools, environment variables, config file keys,
    the Node.js version and every relative link.
  • A test runs the smoke script's household checks against the app on an empty database, so CI
    notices when they drift from the server's defaults.
  • Release validation runs its Linux checks in a container image pinned by digest.
  • scripts/postgis-tests.sh runs every suite that needs PostGIS, including the comparison of
    keyed and original provider searches, in a disposable database container. It fails when one of
    those tests is skipped. A new CI job runs it.

Upgrading

  • Pull the new image and run the migrations. Migration 1007 adds three empty tables:
    provider_search_key_a, provider_search_key_b and provider_search_build. Migrations 1008
    and 1009 each build one index with CREATE INDEX CONCURRENTLY. The tables are still empty, so
    the builds take well under a second, and the old version keeps serving. Older versions never
    read the new tables. If a build fails, it leaves an invalid index. Drop it with
    DROP INDEX CONCURRENTLY provider_search_key_a_cell; or provider_search_key_b_cell, then
    run the migrations again.
  • Once the migrations have finished, run the backfill once:
    docker compose run --rm worker backfill provider-search --free-gb N, where N is the free space
    in GB on the database's disk, or --disk PATH with a directory on that disk. It refuses below
    8 GB, and it refuses while worker migrate runs. It reads every practice address and point once
    and fills a second copy of the keys while searches keep reading the first, then switches. With a
    nationwide provider load it takes minutes. Until it finishes, searches take the original path,
    as in v0.4.7, and answer the same.
  • Run the backfill again after every NPPES or geocode load.
  • If you stop a backfill, check that its INSERT has stopped on the database server, and cancel
    it with pg_cancel_backend if not, before you start the backfill again.
  • The bundled plugins keep their v0.4.7 versions and package hashes, so an approved plugin list
    needs no change.