Query optimization (SCHEMA CHANGE REQUIRED)

[ian-ross]

Query optimization using Claude Code. The following description is lightly edited from Claude's own description of the changes.

---

Motivation

The flight query API is going to be used in a long-lived server process to load trajectory data for projecting onto ground-based camera video. A representative query — 1-hour time window, 50 km bounding box, spatially_crosses + filter_waypoints — was taking ~1.9 s, well above the ~200 ms target needed to keep below video-stream initialization time.

What was changed

1. Vectorized point parsing with numpy (feder-common)

Root cause: Point.unpack deserialized trajectory points one at a time in a Python loop, calling struct.unpack and round(..., 3) (milli()) per point. With ~890 K points unpacked per query call, this generated 22 M Python function calls and dominated the profile at ~76 % of wall time.

Fix:

• Added a _POINT_DTYPE numpy structured dtype matching the on-disk binary layout ('>Lddddd?').
• Added Point._unpack_blob(data) — parses a full decompressed blob in one numpy.frombuffer call.
• Added Point._array_to_points(arr) — materialises Point objects only for trajectories that survive all filters.
• Replaced the Python-lambda points_checker (spatial crossing check) and _make_point_filter (waypoint filter) with numpy boolean-mask operations, so Point objects are never constructed for the ~59 % of R-tree candidates that are ultimately rejected.
• Removed milli() from the hot path. It was originally needed for round-tripping against an old CSV format; it is preserved in utils.py and remains in tests/conftest.py where it still makes comparisons robust.

Result: 1.90 s → 0.56 s (3.4×)

2. Parallel bz2 decompression with a thread pool (feder-common)

Root cause: After the numpy change, profiling showed bz2 decompression of ~1950 trajectory blobs was 92 % of remaining time. Decompressions were serial, and bz2 is a C extension that releases the GIL — straightforward to parallelise with threads.

Fix:

• Added a module-level ThreadPoolExecutor (default 8 workers — empirically the sweet spot on the test machine; more threads increase scheduling and memory-bandwidth contention without adding throughput for this workload).
• Added _process_blob(blob, points_check, pt_filter) as the per-trajectory unit of work submitted to the pool: decompress → numpy parse → spatial check → waypoint filter, all running in parallel.
• Removed the batched(ids, 50) loop in query_flights: all R-tree IDs are now passed to _retrieve in a single SQL query (SQLite handles thousands of IN-clause parameters without issue), giving the pool the maximum available parallelism in one shot.

Result: 0.56 s → ~0.18 s median (further 3.1×)

Overall results

	Time (per call)	Speedup vs baseline
Baseline	1.90 s	—
+ numpy vectorisation	0.56 s	3.4×
+ thread pool	~0.18 s median	~10.5×

Measured on a warm server process (OS page cache hot, 10 repeated calls after one warm-up). Min observed: 128 ms; p90: ~275 ms (tail variance is from shared-host load, not the query path itself).

Dependencies added

• numpy >= 1.24 added to feder-common.

What's next

The remaining ~150 ms is almost entirely bz2 decompression. Replacing bz2 with lz4 (decompresses at ~6 GB/s vs bz2's ~10 MB/s) would reduce decompression to near-zero and push all percentiles comfortably under 100 ms. That requires a data migration and ingestion-pipeline change, and is tracked separately.

---

Addendum: lz4 storage compression (Feder 1.0.0)

Motivation

After the numpy and thread-pool changes, profiling showed bz2 decompression of ~1 950 trajectory blobs still accounted for the bulk of remaining latency. lz4 decompresses at ~6 GB/s vs bz2's ~10 MB/s — roughly 600× faster — reducing decompression to near-zero.

This required converting 380 GB of historical data (1 307 daily SQLite files) and is a breaking change: old library versions will raise OSError: Invalid data stream against the new files. It is released as Feder 1.0.0, using the cumulative ~11× query speedup as justification for the compatibility break.

What was changed

3. lz4 storage format with version byte (feder-common, feder-ingest)

Blob format: each points blob is now stored as:

b'\x01' + lz4.frame.compress(Point.pack(traj.points))

The leading version byte (0x01) future-proofs subsequent compression changes without requiring another flag day.

Reader (db.py): validates the version byte and calls lz4.frame.decompress(blob[1:]). An unrecognised version byte raises ValueError with a message directing the user to upgrade.

Writer (writeable_db.py): prepends 0x01 and compresses with lz4.frame.

Thread-pool fix: switching from bz2 to lz4 exposed a granularity problem — lz4 decompressions take ~0.4 µs each (vs ~260 µs for bz2), making 1 950 individual thread-pool tasks slower than single-threaded due to per-task Python overhead. Fixed by splitting the row list into N_WORKERS chunks and submitting one future per chunk. _array_to_points is also moved inside the chunked work so Point construction is parallelised too.

4. Migration tooling (scripts/)

Nine standalone scripts for the cluster-side data migration:

Script	Purpose
convert_file.py	Per-file conversion: SQLite backup API + UPDATE + VACUUM + integrity check + row-count check
bulk_convert.sh	Slurm job array (one task per file, 20 concurrent)
bulk_convert_parallel.sh	GNU Parallel equivalent for local use
tail_convert.sh	Daily cron to keep the lz4 copy in sync until cutover
verify_counts.py	Row-count sweep across all file pairs
verify_sample.py	Decompress-and-compare spot check on a random sample
verify_queries.py	Canned FlightQuery calls against both directories, exact equality
hypothesis_test.py	Hypothesis property test with random queries, max_examples=2000
soak_test.py	while True loop for 48-hour pre-cutover confidence building

Overall results (cumulative)

	Time (per call)	Speedup vs baseline
Baseline	1.90 s	—
+ numpy vectorisation	0.56 s	3.4×
+ thread pool (bz2)	~0.18 s median	~10.5×
+ lz4 + chunked threads	~68 ms min / ~178 ms median	~11×

Measured on a warm server process against the converted live dataset. p90 is ~264 ms; tail variance is shared-host scheduling noise, not query-path cost.

Dependencies added

• lz4 >= 4.4.5 added to feder-common and feder-ingest.

Breaking change

Users on old library versions querying converted data will see OSError: Invalid data stream. The fix is pip install --upgrade feder. This will be called out prominently in the 1.0.0 release notes and announced to users before the data cutover.

[ian-ross]

A couple of change requests to @claude.

[claude]

Claude finished @ian-ross's task in 3m 57s —— View job

---

Tasks

• Gather context (read changed files)
• Simplify code at db.py:416 (too "clever clever")
• Add comment at models.py:14 explaining what it is and how it's used
• Commit and push changes

[ian-ross]

@ian-ross There is a slightly silly thing that needs fixing here: all of the scripts that compare "old" and "new" schema databases import the current API version, which doesn't work with old schema databases... We need to think of a better way to handle that.