OpenDroneMap city-scale processing on auto-scaling container based workflow (split-merge)

[spwoodcock]

Preamble

We need to handle two processing scenarios well:

1. General day-to-day processing of fast orthos for task areas, and final project orthos for small AOIs.
2. Large city-scale final orthos, using split merge approach and lots of compute (divide and conquer into separate containers).

For the split-merge cloud-native containers approach we need to do some working out of the process (explored below).

But from the DroneTM side, here is what we need:

• The ability to add multiple ODM backends to the system.
• These options should be presented to the user (perhaps we grant access to specific users for the large city-scale processing cluster). Different clusters available for different needs. Also for fault tolerance: if one server doesn't seem to work, they can switch to another.
• To specify the type of processing: normal vs split-merge, we either let the user decide, or determine automatically based on AOI size?

Federated Kubernetes Cluster

• We can do the processing on multiple ODM instances in different datacenters - giving us flexibility and fault tolerance.
• The ideal is to federate multiple Kubernetes worker nodes via Tailscale virtual private mesh.
• The inputs and outputs of the processing are stored on AWS S3 (and DroneTM database on HOT infra), meaning the processing step in the middle is stateless.
• We can in theory pass the inputs to any node on any cluster, then have the outputs uploaded, and the processing shut down.

We are diagramming an initial collab for University Columbia <> HOT infra linkage in this repo: https://github.com/ciesin/hot-docs

Test Control Plane

• 3x t3.medium to run TalosOS as a control plane (4GB, 2 core).
• Tailscale inside University Columbia and HOT AWS machines.
• Federated control plane? Or simply workers exposed from Columbia?

More details to come...

[spwoodcock]

Was looking into Talos vs EKS: hotosm/k8s-infra#1

Decided on keeping EKS, but making components inside the cluster as cloud-agnostic / migratable as possible.

That said, I think it makes much more sense to have a Kubernetes control plane federation where each cluster operates independently.
Having a split-network control plane is a recipe for disaster, as etcd really doesn't like latency.

The best approach I can think of is using a tool like karmada for multi-cluster orchestration.
We could connect each control plane via a Tailscale network:

              ┌────────────────────────────────┐
              │       Karmada Control Plane     │
              │   (in Host Cluster or Separate) │
              │   - API Server                  │
              │   - Scheduler, Controllers      │
              │   - Tailscale Agent             │
              └────────────────────────────────┘
                            │
          (secure WireGuard tunnel via Tailscale)
                            │
       ┌──────────────────────────┐     ┌──────────────────────────┐
       │   Member Cluster A       │     │   Member Cluster B       │
       │  - kube-apiserver        │     │  - kube-apiserver        │
       │  - Tailscale Agent       │     │  - Tailscale Agent       │
       └──────────────────────────┘     └──────────────────────────┘
                            │
                            ▼
            Private Tailscale tailnet (shared mesh)

[spwoodcock]

flowchart LR
  %% Drone Tasking Manager submits jobs
  DTM[Drone Tasking Manager] -->|submit job| LB[Global Load Balancer / DNS] 

  %% Cluster A
  subgraph ClusterA[Kubernetes Cluster A]
    COA1[ClusterODM API Instance 1]
    COA2[ClusterODM API Instance 2]
    RedisA[Redis Queue A]
    KEDA1[KEDA Scaler]
    NodeODM1[NodeODM Deployment]
    NodeODM1 -->|reads/writes| S3
    KEDA1 --> NodeODM1
    COA1 -->|enqueue| RedisA
    COA2 -->|enqueue| RedisA
    RedisA --> NodeODM1
  end

  %% Cluster B
  subgraph ClusterB[Kubernetes Cluster B]
    COB1[ClusterODM API Instance 1]
    RedisB[Redis Queue B]
    KEDA2[KEDA Scaler]
    NodeODM2[NodeODM Deployment]
    NodeODM2 -->|reads/writes| S3
    KEDA2 --> NodeODM2
    COB1 -->|enqueue| RedisB
    RedisB --> NodeODM2
  end

  %% Overspill logic
  RedisA -. overload? .-> RedisB
  RedisB -. overload? .-> RedisA

  %% Job submission
  LB --> COA1
  LB --> COA2
  LB --> COB1

  %% External resources
  S3[(Shared S3/MinIO)]
  
  %% Notes / Observability
  Logs[(Central logging & metrics)]
  NodeODM1 --> Logs
  NodeODM2 --> Logs

Loading

[spwoodcock]

Kubernetes-Native ODM Autoscaling

A simple, reliable approach to multi-cluster autoscaling using PostgreSQL as a job queue.

Overview

This architecture enables automatic scaling of NodeODM workers across multiple Kubernetes clusters using:

• PostgreSQL as the job queue (SKIP LOCKED for concurrency)
• KEDA for queue-based autoscaling
• Split-Merge pattern for processing large projects in chunks

Architecture

┌─────────────┐
│ ClusterODM  │──┐
│   (API)     │  │
└─────────────┘  │
                 ▼
         ┌──────────────┐
         │  PostgreSQL  │
         │  Job Queue   │
         └──────────────┘
                 │
        ┌────────┴────────┐
        ▼                 ▼
   ┌─────────┐       ┌─────────┐
   │ Cluster │       │ Cluster │
   │ NodeODM │       │ NodeODM │
   │  KEDA   │       │  KEDA   │
   └─────────┘       └─────────┘

Database Schema

-- Job queue table
CREATE TABLE job_queue (
    id BIGSERIAL PRIMARY KEY,
    cluster_id TEXT NOT NULL,        -- 'local', 'global', or specific cluster
    job_type TEXT NOT NULL,          -- 'split', 'merge', 'odm'
    payload JSONB NOT NULL,
    status TEXT DEFAULT 'pending',   -- pending, processing, completed, failed
    priority INTEGER DEFAULT 0,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    claimed_at TIMESTAMPTZ,
    claimed_by TEXT,
    completed_at TIMESTAMPTZ
);

-- Cluster capacity tracking
CREATE TABLE cluster_capacity (
    cluster_id TEXT PRIMARY KEY,
    max_concurrent_jobs INTEGER DEFAULT 10,
    current_jobs INTEGER DEFAULT 0,
    last_heartbeat TIMESTAMPTZ DEFAULT NOW()
);

-- Indexes for performance
CREATE INDEX idx_job_queue_pending ON job_queue(cluster_id, status, priority DESC, created_at)
    WHERE status = 'pending';

Implementation Phases

Phase 1: Single-Cluster Foundation

Goal: Get basic queue-based processing working in one cluster

• Deploy PostgreSQL
• Deploy ClusterODM that writes jobs to queue
• Implement simple worker that claims jobs using SKIP LOCKED
• Deploy KEDA scaler watching local queue depth
• Verify NodeODM pods scale up/down with queue length

Job Claim Pattern:

BEGIN;
SELECT * FROM job_queue 
WHERE cluster_id = 'local' AND status = 'pending'
ORDER BY priority DESC, created_at
FOR UPDATE SKIP LOCKED
LIMIT 1;

UPDATE job_queue 
SET status = 'processing', claimed_at = NOW(), claimed_by = $worker_id
WHERE id = $job_id;
COMMIT;

KEDA Configuration:

triggers:
  - type: postgresql
    metadata:
      query: "SELECT COUNT(*) FROM job_queue WHERE cluster_id = 'local' AND status = 'pending'"
      targetQueryValue: "1"

Phase 2: Split-Merge Processing

Goal: Handle large projects by breaking them into chunks

• Implement split logic to divide projects into processable chunks
• Queue split jobs --> process --> queue merge jobs
• Use priority field to ensure merge jobs run after splits complete
• Estimate chunk count from project size to inform routing decisions

Phase 3: Multi-Cluster Distribution

Goal: Route work to multiple clusters based on capacity

• Add cluster_id = 'global' for overflow work
• Workers check local queue first, then global queue
• ClusterODM routes to local if under 80% capacity, otherwise global
• Add /health endpoint returning current capacity
• Use health checks for intelligent routing

Routing Logic:

-- Check local capacity
SELECT current_jobs < (max_concurrent_jobs * 0.8) 
FROM cluster_capacity WHERE cluster_id = 'local';

-- If under capacity: insert with cluster_id = 'local'
-- If at capacity: insert with cluster_id = 'global'

Worker Consumption:

-- Try local first
SELECT ... WHERE cluster_id = 'local' AND status = 'pending' ...

-- If empty, try global
SELECT ... WHERE cluster_id = 'global' AND status = 'pending' ...

Phase 4: Production Hardening

Goal: Reliability, observability, optimization

• Add job timeouts and retry logic
• Implement dead letter queue for failed jobs
• Add comprehensive logging and metrics
• Use LISTEN/NOTIFY for real-time job notifications (optional optimization)
• Set up monitoring and alerting on queue depth/age
• Configure PostgreSQL vacuum for high-turnover queue table
• Add cost-aware routing (process on cheaper clusters when available)
• Routing per region / datacenter cost optimisations
• Routing based on downstream data location / datacenter

[spwoodcock]

Work has progressed quite a bit on https://github.com/hotosm/ScaleODM

Currently there is only a basic / standard workflow.

We need to add split-merge logic.