Argus-RL: Reinforcement Learning for Agile Earth-Observation Satellite Task Scheduling

Argus-RL is a research and hackathon project that demonstrates the application of Proximal Policy Optimization (PPO) to the problem of autonomous on-board task scheduling for agile Earth-observation (EO) satellites. An agent is trained to make real-time, irrevocable image-or-skip decisions as ground targets pass through the satellite's visibility window, subject to non-stationary constraints on on-board storage and battery state.

This repository contains a full end-to-end pipeline: a custom Gymnasium environment that formalizes the sequential decision problem, three hand-coded baseline policies, a vectorized PPO training harness, a controlled evaluation framework with shared-episode seeds, a comparative analysis dashboard, and a real-time orbital visualization demo for live presentations.

Developed as a SuperNova 2026 hackathon submission.

---

Demo

Table of Contents

1. Problem Formulation
2. Environment Technical Specification
3. Baseline Algorithms
4. RL Architecture and Training Setup
5. Evaluation Methodology
6. Quantitative Results
7. Project Structure
8. Installation
9. Reproduction Pipeline
10. Visualization Tools
11. Research Context and Motivation
12. Reproducibility Notes

---

1. Problem Formulation

The scheduling problem is cast as a finite-horizon discounted Markov Decision Process (MDP) formally described by the tuple <S, A, P, R, gamma, T>:

• Timesteps: One episode corresponds to one simulated day of LEO overflights, parameterized with T = 150 ... 300 (default 200) imaging opportunities in strict temporal order. Each target can be decided exactly once, at the moment of overflight.
• State space S: Per-opportunity observations in R^15. Section 2 provides the full breakdown.
• Action space A: Discrete binary. a_t = 0 corresponds to skipping the current target; a_t = 1 requests an imaging attempt.
• Transition kernel P(s_{t+1} | s_t, a_t): Deterministic except for the procedural target distribution (fixed per-episode seed). Resource state evolves via deterministic depletion (on successful imaging) followed by fixed regeneration.
• Reward function R(s_t, a_t, s_{t+1}): Shaped composite signal detailed in Section 2.4.
• Discount factor: gamma = 0.99 (standard for episodic tasks of this horizon).

The objective is to maximize the expected cumulative effective value: J(pi) = E[ sum_t priority_t * (1 - cloud_prob_t) * I{successfully_imaged_t} ] subject to the physical resource constraints at every step.

---

2. Environment Technical Specification

File: eo_scheduling_env.py implements EOSchedulingEnv(gymnasium.Env).

2.1 Observation Space

A 15-dimensional continuous Box observation normalized to [0, 1]:

Index Range	Dimension	Variable	Description
0:3	3	Current-target attributes	priority, cloud_prob, size / 10.0
3:6	3	Resource-time state	storage_remaining / max_storage, battery_remaining / max_battery, (T - t) / T
6:12	6	Lookahead window	(priority, cloud_prob) pairs for the next 3 targets (3 x 2). Enables the policy to condition decisions on upcoming quality and learn to reserve resources for better future opportunities.
12:15	3	Safety padding	Constant zero padding. Allows future lookahead expansion without breaking observation-shape contracts.

2.2 Action Space

gymnasium.spaces.Discrete(2):

• 0: Skip. Opportunity is permanently lost.
• 1: Image. Attempted subject to resource feasibility; may be overridden with a penalty if resources are insufficient.

2.3 Resource Dynamics

Parameters (defaults):

Symbol	Parameter	Value
C_s	Max storage capacity	100.0 units
C_b	Max battery capacity	100.0 units
k_s	Storage cost per unit size	1.0
k_b	Battery cost per image	8.0
rho_s	Storage downlink per step	0.3 units
rho_b	Battery regeneration per step	0.4 units

After each step t, resource state is updated first via depletion (if the action was a successful image) and then via regeneration:

size_cost_t   = size_t * k_s
batt_cost_t   = k_b
feasible_t    = (storage_t >= size_cost_t) AND (battery_t >= batt_cost_t)

if a_t = 1 AND feasible_t:
    storage_{t+1/2} = storage_t - size_cost_t
    battery_{t+1/2} = battery_t - batt_cost_t
else:
    storage_{t+1/2} = storage_t
    battery_{t+1/2} = battery_t

storage_{t+1} = min(C_s, storage_{t+1/2} + rho_s)
battery_{t+1} = min(C_b, battery_{t+1/2} + rho_b)

2.4 Reward Function

A three-term composite reward with tuned coefficients. Value weight W=10.0, invalid-action penalty P_invalid=-5.0, high-value-skip penalty coefficient P_skip=-1.0, high-value threshold tau_hv=0.7:

if a_t = 1 AND feasible_t:
    value_term     = priority_t * (1 - cloud_prob_t) * W
    resource_pen   = 0.05 * (size_cost_t / C_s + batt_cost_t / C_b)
    r_t            = value_term - resource_pen

elif a_t = 1 AND NOT feasible_t:
    r_t            = P_invalid

else (a_t = 0):
    eff_val        = priority_t * (1 - cloud_prob_t)
    if eff_val >= tau_hv:
        r_t        = P_skip * eff_val
    else:
        r_t        = 0.0

2.5 Procedural Episode Generator

Opportunities are sampled independently then shuffled in temporal order to simulate non-uniform orbit coverage:

• Disaster-response cohort (8%): priority ~ Uniform(0.75, 1.0), cloud_prob ~ Beta(2.0, 3.0), size ~ Uniform(4.0, 10.0).
• Routine-monitoring cohort (92%): priority ~ Beta(1.8, 4.0), cloud_prob ~ Beta(2.0, 2.0), size ~ Uniform(1.0, 7.0).

All draws are seeded per-episode for reproducible cross-policy comparisons.

---

3. Baseline Algorithms

File: baselines.py. Implements three non-learned policies plus a universal episode rollout helper compatible with both baseline and stable-baselines3-model APIs.

Policy	Decision rule
Greedy (threshold 0.45)	Image iff priority * (1 - cloud_prob) >= 0.45 AND feasible_t. Default threshold hand-tuned on 100 seed sweep.
Random (p=0.5)	If feasible_t, image with probability 0.5 using a fixed-RNG bit. Otherwise skip.
FCFS (First-Come First-Served)	Always attempt to image whenever feasible_t is True. No preference weighting.

---

4. RL Architecture and Training Setup

File: train_ppo.py. Uses stable-baselines3 PPO with vectorized subproc environments for throughput.

4.1 Neural Network Architecture

Feedforward MLP with separate policy and value trunks:

• Input layer: 15 (observation dim)
• Shared pre-processing: none
• Policy trunk: Linear(15, 128) -> Tanh -> Linear(128, 128) -> Tanh -> Linear(128, 2) -> Softmax
• Value trunk: Linear(15, 128) -> Tanh -> Linear(128, 128) -> Tanh -> Linear(128, 1)
• Parameter count: ~49k weights (model file = 482 KB on disk).
• Device: CPU (configured for laptop portability; no CUDA required).

4.2 PPO Hyperparameters

Hyperparameter	Value	Rationale
Learning rate	3e-4	Default Adam LR for PPO on continuous-box observation tasks.
n_envs (vec)	4	CPU-core-aligned parallel rollouts.
n_steps	2048	Per-env rollout length before gradient update. 4*2048 = 8192-sample batch minibatched into 64 slices of 128.
batch_size	128	Minibatch size.
n_epochs	10	Reuse ratio for on-policy samples.
gamma	0.99	Long-horizon credit assignment over 200-step episodes.
gae_lambda	0.95	Generalized Advantage Estimation smoothing.
clip_range	0.2	Standard PPO trust region clip.
ent_coef	0.01	Low entropy bonus to prevent premature convergence to deterministic skip-all policy.
Max timesteps	200k (default, CLI-overridable)	Sufficient for explained variance > 0.99 on laptop in ~3 min.

4.3 Training Instrumentation

A custom CSVLoggingCallback appends one row every 20 episodes to results/ppo_training_log.csv with columns: timesteps, episodes, mean_reward, mean_value. The CSV is consumed by the dashboard to render the learning curve.

---

5. Evaluation Methodology

File: evaluate.py.

5.1 Shared-Seed Protocol

The evaluation set is composed of N=20 episodes with deterministic seeds derived from a base seed S_base = 9000 incremented per episode: seed_i = S_base + i for i = 0 .. N-1. Every policy under evaluation receives the exact same procedural target schedule and resource dynamics within a given episode. This eliminates sampling variance in episode difficulty as a confounder when comparing policies.

5.2 Reported Metrics

For each policy, the following per-episode statistics are aggregated:

Metric	Definition
total_value	Cumulative priority * (1 - cloud_prob) of successfully imaged targets (primary objective).
total_reward	Cumulative environment reward including shaping penalties.
storage_util_pct	100 * total_storage_consumed / (episode_steps * max_storage).
battery_util_pct	100 * total_battery_consumed / (episode_steps * max_battery).
high_priority_missed	Count of opportunities where effective_value >= tau_hv AND the policy did not capture the target.
images_taken	Count of successful imaging actions (discounts failed no-resource attempts).

5.3 Output Artifacts

All outputs are written to the results/ directory:

• policy_comparison.csv: one row per (policy, episode) pair, 80 rows for N=20 x 4 policies. Used for statistical post-hoc tests.
• policy_comparison_mean.csv: grouped means and standard deviations per policy. Consumed directly by the Streamlit dashboard.
• timeseries_data.pkl: dict[(policy_name, seed) -> per-step arrays: time_steps, cumulative_value, storage_levels, battery_levels, actions, effective_values]. Drives the timeline plots in the dashboard.

---

6. Quantitative Results

6.1 Primary Comparison: 20 Evaluation Episodes (seed 9000, 200 opportunities)

PPO model trained for 204,800 environment timesteps on 4 vectorized envs (explained variance = 0.995 at end of training).

Policy	Mean total value	Std dev value	Mean HP missed	Mean images taken	Value vs Greedy	Value vs Random
PPO	10.713	0.470	0.3	21.6	+19.6 %	+157.2 %
Greedy	8.959	1.352	0.0	15.1	1.00x baseline	+115.0 %
FCFS	4.220	0.539	2.1	22.0	-52.9 %	+1.3 %
Random	4.166	0.683	2.2	22.0	-53.5 %	baseline

Interpretation:

1. The PPO policy dominates all baselines on total value captured while maintaining a low high-priority miss rate comparable to Greedy.
2. PPO exhibits substantially tighter variance (sigma = 0.47) than Greedy (sigma = 1.35), indicating more robust behavior across varied target distributions.
3. PPO matches FCFS/Random on image count (21.6 vs 22.0) but allocates those images to significantly higher-quality opportunities: a 2.5x efficiency gain per-image over Random.
4. Greedy, despite using zero learning, already captures the non-trivial structure of the problem and forms a strong baseline. PPO's ~+20% gain must come from lookahead-aware timing (e.g. saving battery for incoming high-priority batches that the fixed-threshold baseline cannot reason about).

6.2 Training Dynamics

• Rollout throughput: ~1300 env steps / second on a 2.4 GHz 8-core laptop CPU.
• Wall-clock convergence:
  • 50k timesteps: policy learns to avoid invalid actions, mean reward ~-10 / episode.
  • 100k timesteps: selective imaging emerges, mean reward ~+50 / episode.
  • 200k timesteps: plateau at mean reward ~+99.4 / episode, explained variance 0.995.
• Total wall-clock time (200k steps): 2 min 36 s.

---

7. Project Structure

Argus-RL/
├── eo_scheduling_env.py   Custom Gymnasium MDP environment.
├── orbit_view.py          Pure-SVG orbital visualization component.
├── baselines.py           Greedy, Random, FCFS policies + rollout helper.
├── train_ppo.py           SB3 PPO training harness + CSV callback.
├── evaluate.py            Shared-seed evaluator + CSV/ pickle outputs.
├── dashboard.py           Streamlit comparative analysis dashboard.
├── live_sim.py            Real-time orbital step simulator (Streamlit + matplotlib modes).
├── requirements.txt       Python dependency manifest.
├── .gitignore             Excludes regenerable artifacts (models/*.zip, results/*.csv, results/*.pkl) and caches.
├── models/
│   └── .gitkeep           Placeholder; trained model files regenerated by train_ppo.py.
└── results/
    └── .gitkeep           Placeholder; CSV / pickle artifacts regenerated by train_ppo.py + evaluate.py.

---

8. Installation

Tested on Python 3.13 (backwards compatible to 3.9).

git clone <repository-url>
cd Argus-RL

# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate        # Linux / macOS
# .\venv\Scripts\activate       # Windows (PowerShell)

# Install all dependencies
pip install -r requirements.txt

Dependency manifest (requirements.txt):

• gymnasium >= 0.29
• stable-baselines3 >= 2.0
• torch >= 2.0
• numpy >= 1.24
• pandas >= 2.0
• matplotlib >= 3.7
• streamlit >= 1.24

---

9. Reproduction Pipeline

Execute in order to reproduce all tables and dashboard artifacts:

# Step 1: Environment self-test. Produces a 50-step rendered trace on stdout.
python eo_scheduling_env.py

# Step 2: Baseline self-test. Compares Greedy/Random/FCFS on one seed.
python baselines.py

# Step 3: Train PPO (200k default timesteps, ~3 min laptop CPU).
#         Writes models/ppo_eo.zip and results/ppo_training_log.csv.
python train_ppo.py --timesteps 200000

# Step 4: Full evaluation. 20 shared episodes across PPO + 3 baselines.
#         Writes results/policy_comparison.csv, policy_comparison_mean.csv,
#         timeseries_data.pkl.
python evaluate.py --episodes 20 --opportunities 200 --seed 9000

CLI references:

• train_ppo.py --help exposes --timesteps, --n_envs, --learning_rate, --output, --csv_log, --tensorboard_log, --seed.
• evaluate.py --help exposes --episodes, --model, --opportunities, --seed, --output_dir.

---

10. Visualization Tools

10.1 Comparative Analysis Dashboard

streamlit run dashboard.py

Components rendered:

1. Tabular summary of per-policy mean metrics.
2. Three side-by-side bar charts:
   • Mean total value with standard deviation error bars.
   • Grouped storage + battery utilization percent.
   • Mean count of missed high-priority targets.
3. Cumulative effective value line chart: one trace per policy over a common episode.
4. Storage and battery trajectory plots: single-policy focus view + multi-policy overlay option.
5. PPO training curve: mean reward and mean effective value over training timesteps.

10.2 Live Tasking Simulator

streamlit run live_sim.py        # Web UI (recommended for demos)
python live_sim.py --mode matplotlib --policy PPO --speed 0.05   # Windowed matplotlib animation

Streamlit mode page layout (top to bottom):

1. Policy selector, opportunities-per-episode slider, episode-seed input, step-delay slider, pause toggle, restart-episode button.
2. Five-metric status row (policy, step count, total value captured, storage, battery).
3. Hero orbital SVG visualization (via orbit_view.py):
   • Radial-gradient shaded Earth with atmospheric halos.
   • Dashed circular orbit path.
   • Geometric satellite (bus + teal solar panels + antenna) animated tangentially along the orbit at an angle proportional to step / total_steps.
   • Continuous pulse-glow ring around satellite (SVG <animate> on radius and opacity).
   • History dots on the orbit: green for successful images, pulsing double-ring red for missed high-value targets (SVG opacity flash anim), gray for routine skips.
   • Inline legend + orbit-pass caption with percentage completion and high-value-overhead flag.
4. Current-target info card (priority, cloud, size, effective value, cost feasibility).
5. Three-panel matplotlib timeline: cumulative value, storage, battery, with red scatter markers on image-taken steps.
6. Decision log table (last 20 steps, reverse-chronological).

10.3 Orbit-View Module Reference

orbit_view.py exposes:

render_orbit_view(
    step: int,
    total_steps: int,
    history: list[dict],   # [{"step":int, "priority":float, "decision":"image"|"skip", "high_value_missed":bool}, ...]
    current_priority: float | None = None,
    width_px: int = 780,
    height_px: int = 430,
) -> str                   # Returns self-contained HTML document string.

Rendering is dependency-free beyond standard SVG/HTML, and is embedded into Streamlit pages via streamlit.components.v1.html.

---

11. Research Context and Motivation

Onboard autonomous scheduling is a critical unsolved capability for next-generation EO constellations:

• Operational latency: Ground-in-the-loop tasking requires uplink windows spaced 90+ minutes apart for LEO satellites, making them too slow for time-critical events such as wildfire ignition detection, flood progression monitoring, and maritime domain awareness alerts.
• Uncertainty: Cloud-cover forecasts and emergent-event priorities are non-stationary. Hand-authored scheduling rules (the industry-standard approach on most current missions) degrade rapidly under distribution shift.
• On-board compute envelope: Radiation-hardened flight computers in the sub-10 W class have sufficient FLOPS to run a ~50k-parameter MLP policy at < 1 ms per inference, making policy-gradient RL deployable in a way that large transformer or MIP-based planners are not.

Relevant references:

1. Zhao, Y. et al. (2023). Deep Reinforcement Learning for Autonomous Earth-Observation Satellite Task Planning. IEEE Transactions on Geoscience and Remote Sensing, 61. DOI: 10.1109/TGRS.2023.3287412.
2. Furgale, P. et al. (2022). Onboard Autonomy for Small Satellites: A Survey. Acta Astronautica, 198, pp. 410-432.
3. Cui, H. et al. (2024). Multi-Satellite Agile Scheduling via Multi-Agent PPO with Look-Ahead Heuristics. CEAS Space Journal.

---