Representation learning of Gaia XP DR3

• Gaia Explorer
• Poster @ ADASS 2025
• Tutorial @ SPACE HPC Visualization Workshop

Install dependencies in a virtual environment

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

PEST: Data Preparation

Download the Gaia XP DR3 data from Gaia ESA Archive (total size: 3.2 TB) and place it in the data/csv directory. The following command downloads only the first 3111 files (1.3 GB).

wget -P data/csv https://cdn.gea.esac.esa.int/Gaia/gdr3/Spectroscopy/xp_continuous_mean_spectrum/XpContinuousMeanSpectrum_000000-003111.csv.gz

The data preparation step is handled by PEST.

python pest/gaia_calibrate.py

Spherinator: Model Training

The model training step is handled by Spherinator.

spherinator fit --config spherinator/gaia_vae_8_nll_normal.yaml

Convert the trained model to ONNX format

Path to the checkpoint file of the trained model (e.g., lightning_logs/version_0/checkpoints/<name>.ckpt).

python onnx/export_onnx.py <path_to_checkpoint>

HiPSter: Generate HiPS Representation and Catalog

The generation of the HiPS representation and catalog is handled by HiPSter.

hipster --config hipster/gaia_vae_8_nll_normal.yaml

Visualization

Serve the HiPSter representation

python -m http.server 8083 -d hipster_output

and open http://localhost:8083 in your web browser.