Add new catalogs

boom-catalogs is a companion repo. Its job is to take catalogs (e.g. LSDR10, PanSTARRS, AllWISE) and load them into the MongoDB database, so alerts can be crossmatched against them. Getting a catalog in place always happens in two separate steps: first a Python script downloads the raw catalog files (FITS, CSV, or Parquet) to disk, then a Rust binary ingests those files into MongoDB. This page walks through both, plus how to extend the pipeline to support a catalog it doesn't already handle.

Before you start: environment variables

The scripts and binaries in this repo are driven by three environment variables, normally set in a .env file (copy .env.default to .env to start):

• OUTPUT_DIR — where downloaded catalog files get saved on disk. Each downloader script writes into its own subfolder here, e.g. $OUTPUT_DIR/ls_dr10/ for Legacy Survey DR10.
• MONGODB_URI — the full connection string (host, port, and credentials) of the MongoDB instance to write into.
• MONGODB_DB — the name of the database within that instance to create collections in.

The last two matter more than they might look: boom has its own MongoDB connection configured separately (in boom's config.yaml), and if MONGODB_URI/MONGODB_DB here don't point at that same database, you'll successfully ingest a catalog that boom will never actually see or crossmatch against. On MongoDB Compass, you can find the MONGODB_URI and confirm that the MONGODB_DB actually exists.

Building the ingestion binaries

The download scripts are plain Python and need no build step, but the ingestion side is a Rust crate that needs compiling once:

cargo build --release

This produces one binary per file format under target/release/: add_fits_catalog, add_csv_catalog, add_parquet_catalog, add_ascii_catalog. Note that all of them require the system library cfitsio (and pkg-config) to already be installed, even the ones that don't ingest FITS files since they're all built from one shared crate that includes FITS support, so the dependency applies to the whole build:

# Debian/Ubuntu
sudo apt-get install -y libcfitsio-dev pkg-config

# RHEL/Fedora (what we have in production)
sudo dnf install -y cfitsio-devel pkg-config

Known issue on some RHEL/pkgconf setups: cargo build can fail inside fitsio-sys with a panic thrown from the pkg-config crate. The cause is that fitsio-sys always queries pkg-config with a combined "cfitsio >= 3.37" string, and some versions of pkgconf return empty output for --modversion when given that combined form (even though the plain pkg-config --modversion cfitsio works fine and correctly reports the installed version). The fix is a small wrapper script that strips the version constraint before forwarding to the real pkg-config binary, pointed to via the PKG_CONFIG environment variable. Ask in the repo if you hit this and need the wrapper.

Downloading a catalog

Each catalog has its own script under downloaders/:

python downloaders/<script_name>.py

Run it, and its files will show up under $OUTPUT_DIR. Some catalogs are large enough that this can take hours. Most of the scripts download in parallel and support resuming, so re-running after an interruption should just pick up where it left off rather than starting over.

Ingesting a downloaded catalog

Once the files are downloaded, run the binary matching that catalog's file format, pointing it at the type of data it contains, the MongoDB collection to write into, and the path to the downloaded files:

./target/release/add_<format>_catalog <type-name> <collection-name> <path-to-files> --uri "$MONGODB_URI" --db "$MONGODB_DB" --init-indexes

A quick breakdown of the arguments, since it's easy to mix up the first two:

• <type-name> picks which internal schema to parse each row into. It's not a free-form label, it has to match one of the catalog types this repo already knows about (see src/types.rs). The CLI name is the kebab-case version of that type's name, e.g. the LSDR10 type is invoked as lsdr10.
• <collection-name> is just the MongoDB collection name you want the data to land in. This one is your choice, and doesn't have to match the type name (though it's common to make them the same for clarity).
• <path-to-files> can be a single file or a directory; if it's a directory, the binary searches it recursively for files of the right extension.
• --init-indexes tells it to build the collection's indexes (for most catalogs, a 2dsphere geospatial index on coordinates) once, after every file has been inserted. Omit it if you're ingesting the same catalog across multiple runs and only want to build indexes on the last one.
• --drop-existing-collection wipes the collection first instead of appending to it. It's useful for a clean re-ingest, but leave it off if you're resuming a partial run.

One thing worth flagging: index builds run entirely inside MongoDB itself, and the terminal gives no progress output while one is happening. A large catalog can sit "quietly" for a long time after the last file finishes while the index actually builds. If it looks stalled, check progress directly in mongosh rather than assuming something's wrong:

db.currentOp({ "command.createIndexes": { $exists: true } })

You can also check MongoDB Compass which shows indexes that are currently being built. It also details the progression percentage.

A handful of catalogs are related to each other and need to be ingested in a specific order. PS1 PSC is the main example: it doesn't create its own collection at all, it patches a ps_score field onto documents that already exist in the PanSTARRS collection, matched by objID. So PanSTARRS has to be ingested first, and then:

./target/release/add_ps1_psc_scores <panstarrs-collection-name> <path-to-ps1-psc-fits-files> --uri "$MONGODB_URI" --db "$MONGODB_DB"

Adding support for a new catalog

If the catalog you want isn't one of the types this repo already knows about, extending it follows the same pattern as the existing ones:

1. Write a downloader. Add a script under downloaders/ that fetches the catalog's raw files and saves them into $OUTPUT_DIR, following the pattern of an existing script for the same source type.
2. Define its schema. In src/types.rs, add a struct (#[derive(Deserialize, Serialize)]) describing the catalog's fields, and implement whichever trait matches its file format so the ingestion code knows how to parse rows into it: ParquetRowBatch for Parquet, FitsRowBatch for FITS, FromAsciiRow for ASCII (see an existing CSV type for that format's pattern). If the catalog has ra/dec coordinates, also implement HasCoordinates so it automatically gets the 2dsphere geospatial index on ingest.
3. Register the type. Add your new struct as a variant on the enum matching its format (ParquetCatalogs, FitsCatalogs, CsvCatalogs, or AsciiCatalogs), also in src/types.rs. Its CLI name is derived automatically from the variant name (kebab-cased), so no extra wiring is needed there.
4. Dispatch to it. In the corresponding binary under src/bin/ (e.g. add_parquet_catalog.rs), add your new enum variant to the match statement, calling process_<format>::<YourType>(...) the same way the existing variants do.
5. Build and run it exactly like any other catalog, following the "Ingesting a downloaded catalog" steps above.