HEALPix integration for FASTDB: pg_healpix submodule, npix_order29 column, LIGO‑skymap helpers & tests

[trivialTZ]

Introduces HEALPix support so that FASTDB can pre‑index every diaobject and cross‑match efficiently against incoming LIGO/Bayestar multi‑order sky‑maps. Key pieces are:

Area	Change
Build / deps	Adds extern/pg_healpix as a git‑submodule and compiles it inside the Postgres image
Schema migration	New script db/2025‑07‑19_npix_order29.sql • ALTER TABLE diaobject ADD COLUMN npix_order29 BIGINT • Bulk‑populate via healpix_ang2ipix_nest(8192, ra, dec) (order ≈ 13 compromise) • Create idx_diaobject_npix29 for fast look‑ups.
Helper SQL	Adds db/2025‑07‑19_skymap_helpers.sql with: decode_uniq(uniq) → (order, ipix) and is_within(npix, order, ipix) for set‑based joins.
Docker images	Postgres image now builds and installs pg_healpix; run_postgres.sh auto‑CREATE EXTENSION pg_healpix at first start.
ORM / code	Adds npix_order29 attribute to src/db.py’s DiaObject model.
Documentation	docs/developers.rst & docs/index.rst updated with notes for pg_healpix.
Tests & data	• New pytest test_skymap_crossmatch.py + Bayestar FITS sample to exercise cross‑matching end‑to‑end. • Upstream test fix for non‑det flux‑error plotting merged.

---

* pg_healpix gives us a C‑level implementation of healpix_ang2ipix_nest that scales to millions of rows.

* Helper functions keep all logic in‑database—no Python.

---

Implementation notes

• Order choice: order 29 is maximun 64‑bit integers in Postgres; but current compromise as pg_healpix uses order 13 (NSIDE=8192), which give lower resolution and still nests cleanly into LVK maps. The shift math in is_within remains the same.

---

Test

# apply migrations

psql fastdb < db/2025-07-19_npix_order29.sql

psql fastdb < db/2025-07-19_skymap_helpers.sql



# run the new test

pytest tests/test_skymap_crossmatch.py -q
#You should see the test pass with a non‑zero match count, confirming the join logic.

---

Future work

• Switch back to order 29 once we decide how to do it.
• Implement staged back-fill for npix_order29: since bulk updating tens of millions of rows may lock the table, I will build the scheme leaving the column initially empty and populating it later in batches using UPDATE … WHERE npix_order29 IS NULL.
• Add Kafka → PostgreSQL ingestion pipeline for incoming LIGO skymaps.
• Finalize and register the is_within(npix, order, ipix) SQL function to support set-based cross-matching between diaobject and decoded UNIQ pixels.

[trivialTZ]

@hernandezc1 – If this looks good to you, feel free to go ahead and merge into the branch.
@rknop – You're very welcome to review and share any feedback!

[hombit]

Wow, looks cool. It could also help exporting to HATS and cross-match using LSDB!