CF4 Cosmography Pipeline

A reproducible workflow to visualize Laniakea, the Great Attractor, and the CF4 flow field

📦 Version 1.0.0 - March 2026: This repository has been recently reorganized with a new modular structure. See CHANGELOG.md for details.

This repository provides a fully reproducible pipeline for generating 3D visualizations of the Cosmicflows-4 (CF4) flow field, including:

• Galaxies with assigned basins of attraction
• Streamlines following the reconstructed CF4/CF4gp velocity field
• Interactive 3D visualization of the flows, attractors, and large-scale structure

The pipeline reconstructs the watershed structure described in Tully, Courtois, Hoffman et al., Nature (2014) and makes it fully reproducible using open tools, NumPy, and pygfx (GPU-accelerated 3D rendering).

🌌 Overview: What This Pipeline Does

The Universe around us can be partitioned into basins of attraction—volumes where galaxies flow toward the same gravitational attractor. Examples include:

• Laniakea
• Great Attractor
• Perseus–Pisces
• Shapley
• and others…

This workflow reconstructs those basins by combining:

1. CF4 galaxy data (positions, distances, SG coordinates)
2. CF4/CF4gp velocity field (3D vector field reconstructed from peculiar velocities)
3. Basin-of-attraction grid (watershed segmentation of the velocity potential)
4. RK4 integration to compute streamlines inside each basin
5. Interactive 3D GPU-accelerated visualization (pygfx/WebGPU)

📂 Repository Structure

.
├── src/                          # 🆕 Shared Python modules (v1.0+)
│   ├── constants.py              # Physical constants and defaults
│   ├── utils/
│   │   ├── coordinates.py        # Coordinate transformations
│   │   └── interpolation.py     # RK4 integration & interpolation
│   └── rendering/
│       ├── colors.py             # HDR color utilities
│       └── config.py             # Unified configuration system
│
├── config/                       # 🆕 Configuration files (v1.0+)
│   ├── simple_CF4_plot.toml
│   ├── simple_CF4_animation.toml
│   ├── flythrough_CF4_animation.toml
│   └── flythrough_CF4_animation_motion.toml
│
├── 1_CF4_galaxies_table/         # Data pipeline: Galaxy preparation
│   ├── build_dataset.sh
│   ├── build_cf4_galaxies_csv.py
│   ├── assign_galaxies_to_basins.py
│   ├── EDDtable*.txt             # Raw CF4 galaxy table
│   └── CF4_galaxies_with_basin_id.csv  # Output
│
├── 2_CF4_streamlines/            # Data pipeline: Streamline generation
│   ├── build_streamlines.sh
│   ├── build_streamlines_cf4.py
│   ├── CF4_streams_streamlines.csv  # Output
│   └── CF4_streams_seeds.csv        # Output
│
├── 3_CF4_galaxies_motion/        # Data pipeline: Galaxy motion integration
│   ├── galaxies_motion_CF4.py
│   ├── run_galaxies_motion.sh
│   └── CF4_galaxies_motion_longterm.csv  # Output
│
├── output/                       # 🆕 Animation output (v1.0+, was 4_animations.nosync)
│   └── *.mp4                     # Rendered animations
│
├── simple_CF4_plot.py            # Interactive 3D viewer (real-time)
├── simple_CF4_plot_animation.py  # Orbital animation renderer (360° rotation)
├── flythrough_CF4_animation.py   # Flythrough animation (custom paths + galaxy motion)
├── draw_camera_orbits.py         # Camera path generator (Marimo notebook)
├── camera_positions.npy          # Pre-generated camera positions for flythrough
│
├── .gitignore                    # 🆕 Git ignore rules (v1.0+)
├── requirements.txt              # Python dependencies
├── CHANGELOG.md                  # 🆕 Version history (v1.0+)
├── LICENSE                       # MIT License
└── README.md                     # This file

Note: Data files (*.csv, *.npy, *.mp4) are .gitignored due to size. You must supply the original CF4 FITS files and BoA FITS files, which are converted to NumPy arrays.

🧠 Scientific Motivation

The cosmic velocity field reconstructed from CF4 allows us to identify:

• attractor basins via watershed segmentation
• flow trajectories (streamlines) inside each region
• galaxies belonging to each attractor

This reveals the dynamic organization of the cosmic web, not just its density structure.

This pipeline brings these ingredients together in a way that is:

• reproducible
• open-source
• interactive
• easy to modify and extend for research or teaching

🚀 Step-by-Step Pipeline

Step 1 — Build galaxy dataset and assign basins

Directory: 1_CF4_galaxies_table/

This step:

1. Loads the raw EDD/CF4 galaxy table
2. Extracts positions and distance moduli
3. Converts distance moduli → physical distances
4. Converts SG longitude/latitude → SGX, SGY, SGZ (Mpc)
5. Loads the watershed basin-of-attraction grid (128³)
6. Assigns each galaxy to the basin at its SG coordinates

Run:

cd 1_CF4_galaxies_table
./build_dataset.sh

Output:

• CF4_galaxies_SGX_SGY_SGZ.csv
• CF4_galaxies_with_basin_id.csv ← used later

Step 2 — Generate streamlines inside each basin

Directory: 2_CF4_streamlines/

This step integrates the CF4/CF4gp velocity field using RK4 to generate flow trajectories.

Seeds are generated inside each basin (from 128³ grid):

1. Seeds are placed at voxel positions within each basin
2. Each seed is mapped into the 64³ velocity field
3. A streamline is integrated until:
   • it leaves the volume, or
   • the field becomes undefined
4. SG coordinates are stored for visualization

Run:

cd 2_CF4_streamlines
./build_streamlines.sh

Key parameters:

• --mode: Choose between cf4 or cf4gp velocity field
• --velocity-file: Path to velocity field (.npy format, shape 3×64×64×64)
• --boa-file: Path to basin-of-attraction grid (.npy format, shape 128×128×128)
• --all-basins: Generate streamlines for all basins (except label 0)
• --basin-id: Generate streamlines for a specific basin only
• --max-seeds-per-basin: Upper limit for seeds per basin (default: 800)
• --min-seeds-per-basin: Minimum seeds for large basins (default: 50)
• --h-step: RK4 step size in index-space (default: 0.4)
• --nsteps: Maximum RK4 steps per streamline (default: 360)

Output:

• CF4_streams_streamlines.csv — all streamline vertices
• CF4_streams_seeds.csv — starting points

Step 3 — Interactive 3D GPU-Accelerated Visualization

This script provides a GPU-accelerated 3D visualization using pygfx and wgpu:

• Streamlines rendered as smooth 3D lines (colored by basin with HDR support)
• Galaxy positions as point clouds (with configurable size and transparency)
• Real-time camera controls (orbit, pan, zoom)
• Configurable via TOML file or command-line arguments

Features:

• HDR color rendering with intensity multiplier for vivid basin colors
• Alpha blending for transparent streamlines and galaxies
• TOML-based configuration with CLI override support
• GPU-accelerated rendering via WebGPU (wgpu)

Run Interactive Visualization:

Default configuration (reads from config/simple_CF4_plot.toml):

python3 simple_CF4_plot.py --config config/simple_CF4_plot.toml

Or with custom parameters (CLI overrides TOML):

python3 simple_CF4_plot.py \
    --config config/simple_CF4_plot.toml \
    --intensity 2.5 \
    --alpha 0.1 \
    --canvas-size 1024 1024 \
    --thickness 0.01 \
    --galaxy-size 1.0

Command-line arguments override TOML parameters.

Create Animation (MP4 video):

Use simple_CF4_plot_animation.py to render an orbital animation and save as video:

python3 simple_CF4_plot_animation.py \
    --config config/simple_CF4_animation.toml \
    --output-file output/laniakea_animation.mp4 \
    --fps 60 \
    --quality 8 \
    --canvas-size 1920 1920

Additional animation parameters:

• --orbit-radius: Camera distance from center (default: 2500.0)
• --fps: Frames per second (default: 60)
• --quality: Video quality 0-10, higher is better (default: 8)

The camera will orbit around the structure in a circular path, creating a 360° animation.

Create Flythrough Animation with Custom Camera Path:

Use flythrough_CF4_animation.py for advanced flythrough animations with custom camera trajectories:

python3 flythrough_CF4_animation.py \
    --config config/flythrough_CF4_animation.toml

With optional galaxy motion (time-dependent positions):

python3 flythrough_CF4_animation.py \
    --config config/flythrough_CF4_animation_motion.toml \
    --galaxies-motion-file 3_CF4_galaxies_motion/CF4_galaxies_motion_longterm.csv

Features:

• Custom camera paths loaded from camera_positions.npy (generated with draw_camera_orbits.py)
• Optional time-dependent galaxy positions showing cosmic evolution
• Synchronized camera motion with galaxy trajectories
• Configurable galaxy colors (can override basin colors with fixed color)

Configuration:

Parameters can be set in TOML configuration files:

simple_CF4_plot.toml (for interactive visualization):

• File paths: galaxies and streamlines CSV files
• Rendering: alpha, intensity, thickness, galaxy size
• Canvas: size, background color
• Basins: which basins to display and their colors (hex)

simple_CF4_animation.toml (for animation rendering):

• All parameters from simple_CF4_plot.toml, plus:
• output_file: Path for the rendered MP4 video
• fps: Frames per second (default: 60)
• quality: Video quality 0-10 (default: 9)
• orbit_radius: Camera distance from center (default: 2500.0)

Output:

• Interactive mode: Real-time 3D window with orbit controls
• Animation mode: MP4 video file saved to output/ directory
• Use mouse to rotate, pan, and zoom the visualization (interactive mode only)

📊 What You Will See

The visualization shows the cosmic flow network with:

• Streamlines: Flow trajectories colored by basin

  • Red = Laniakea
  • Pink = Great Attractor
  • Blue = Sculptor
  • Yellow = Local Sheet
  • etc. (configurable in TOML files)

• Galaxies: Point clouds showing galaxy positions, also colored by basin

• Reference Frame: Optional coordinate axes and bounding box

Streamlines trace how matter flows in 3D space, revealing the watershed structure of the cosmic web. You can rotate, zoom, and pan to explore the structure from any angle.

🎬 Visualization Scripts Comparison

The repository includes three complementary visualization scripts:

Script	Mode	Output	Camera Control	Galaxy Motion	Best For
simple_CF4_plot.py	Interactive	Real-time window	User-controlled (mouse orbit, pan, zoom)	Static positions	Exploring data interactively, presentations
simple_CF4_plot_animation.py	Animation	MP4 video	Automatic 360° orbital rotation	Static positions	Creating overview videos, social media
flythrough_CF4_animation.py	Animation	MP4 video	Custom path from camera_positions.npy	Optional time-dependent	Cinematic flythrough, showing galaxy evolution

When to use each:

• Interactive viewer (simple_CF4_plot.py): Start here to explore your data, adjust rendering parameters, and identify interesting viewpoints
• Orbital animation (simple_CF4_plot_animation.py): Create quick 360° overview videos with consistent circular camera motion
• Flythrough animation (flythrough_CF4_animation.py): Advanced use - custom camera paths (designed with draw_camera_orbits.py) and optional galaxy motion over time

Note: Use draw_camera_orbits.py (Marimo notebook) to generate custom camera trajectories for flythrough animations.

📦 Data Requirements

Data Sources

The pipeline requires data from the following sources:

• EDD (Extragalactic Distance Database): https://edd.ifa.hawaii.edu

  • Galaxy catalog with distance moduli, positions, and velocities
  • Download the CF4 galaxy table (EDDtable*.txt)

• Cosmicflows Project: https://projets.ip2i.in2p3.fr/cosmicflows/

  • CF4 velocity field reconstructions
  • Basin-of-attraction (BoA) grid files
  • FITS files for scientific analysis

Required Files

You need:

• CF4 velocity field FITS files (CF4gp_new_64-z008_velocity.fits)
• BoA basin-of-attraction FITS file (CF4_new_128-z008_BoA.fits)
• EDD/CF4 galaxy table (*.txt)

FITS to NumPy Conversion

Convert FITS → NumPy using standard tools:

from astropy.io import fits
import numpy as np

# Load FITS file
with fits.open('CF4_new_64-z008_velocity.fits') as hdul:
    velocity_data = hdul[0].data

# Save as NumPy array
np.save('CF4_new_64-z008_velocity.npy', velocity_data)

For batch conversion, you can create a simple shell script to process multiple files.

🧪 Scientific Consistency Checks

The pipeline is designed to be astrophysically consistent:

• SGX, SGY, SGZ coordinates all fall in ±500 Mpc
• BoA grid and velocity field share the same physical cube
• Index → SG mapping uses voxel centers
• Velocity field is interpolated with trilinear interpolation
• Streamlines use unit vectors (flow topology, not speed)
• RK4 integration is robust to out-of-bounds flow

🛠 Requirements

Core Requirements

• Python ≥ 3.11 (required for tomllib standard library)

Python Dependencies

Core pipeline (Steps 1-2 - Data processing):

• numpy>=2.2.6
• pandas>=2.2.3
• tqdm>=4.67.1
• astropy>=5.3 (for FITS conversion)

Visualization (Step 3 - GPU rendering):

• pygfx==0.15.3 (GPU-accelerated 3D rendering)
• rendercanvas==2.4.2
• wgpu==0.28.1 (WebGPU backend)
• glfw==3.4
• pyglfw==2.10.0

Optional (for additional analysis):

• imageio>=2.31.0 (video encoding)
• plotly>=5.14.0 (alternative visualization)

Installation

Install all dependencies:

pip install -r requirements.txt

This will install all dependencies with pinned versions for reproducibility.

Or install manually:

# Core dependencies
pip install numpy pandas tqdm astropy

# GPU rendering
pip install pygfx rendercanvas wgpu glfw pyglfw

# Optional
pip install imageio plotly

---

🧭 How to Explore Further

Suggested student tasks:

• Visualize only Laniakea (--basin-id 1)
• Change streamline density using --max-seeds-per-basin and --min-seeds-per-basin
• Adjust integration parameters (--h-step, --nsteps) to explore different flow patterns
• Compare CF4 vs CF4gp velocity fields (--mode cf4 vs --mode cf4gp)
• Trace flows toward or away from attractors (flip velocity sign in code)
• Plot isosurfaces of flow magnitude or divergence
• Superimpose density field or filament extraction
• Create animations of the 3D structure using simple_CF4_plot_animation.py
• Experiment with rendering parameters (alpha, intensity, thickness) in TOML files

🆕 What's New in v1.0.0 (March 2026)

Modular Architecture

The repository has been restructured with a professional software engineering approach:

New Module System (src/)

• src/constants.py: Centralized physical constants and default parameters
• src/utils/: Reusable coordinate transformations and numerical methods
• src/rendering/: Shared rendering functions and unified configuration

Benefits:

• ✅ ~850 lines of duplicate code eliminated (35% reduction)
• ✅ DRY principle: Single source of truth for all shared functions
• ✅ Easier testing: Modular functions can be unit tested
• ✅ Better maintainability: Bug fixes in one place benefit all scripts

Organized Configuration

• Configuration files moved to config/ directory
• TOML files now use consistent naming and structure
• Unified VisualizationConfig class supports all visualization modes

Improved Documentation

• Added CHANGELOG.md tracking all changes
• Comprehensive README with usage examples
• Updated .gitignore to exclude large data files

Updated Commands

Configuration files have moved:

# Interactive viewer
python simple_CF4_plot.py --config config/simple_CF4_plot.toml

# Orbital animation
python simple_CF4_plot_animation.py --config config/simple_CF4_animation.toml

# Flythrough animation
python flythrough_CF4_animation.py --config config/flythrough_CF4_animation.toml

# Flythrough with galaxy motion
python flythrough_CF4_animation.py --config config/flythrough_CF4_animation_motion.toml

Future Refactoring

The existing visualization scripts will be refactored in future versions to use the new shared modules, further reducing code duplication and improving maintainability.

For detailed changes, see CHANGELOG.md.

📚 References

Scientific Publications

• Tully, R. B., Courtois, H., Hoffman, Y., & Pomarède, D. (2014). The Laniakea supercluster of galaxies. Nature, 513(7516), 71-73. DOI:10.1038/nature13674

• Tully, R. B., et al. (2023). Cosmicflows-4: The Catalog of ~10,000 Tully-Fisher Distances. ApJ, 944, 94. DOI:10.3847/1538-4357/ac94d8

Data Sources

• EDD (Extragalactic Distance Database): https://edd.ifa.hawaii.edu

  • Maintained by R. Brent Tully (University of Hawaii)
  • Galaxy catalog with distances, velocities, and photometry

• Cosmicflows Project: https://projets.ip2i.in2p3.fr/cosmicflows/

  • Velocity field reconstructions and basin-of-attraction grids
  • FITS files and data products from CF4 analysis

🟦 License

MIT License