Onboarding tasks for new backend dev

[spwoodcock]

Direct request from @cgiovando, we need:

• imagery statistics, visualization, and monitoring
• pixel/sqkm coverage instead of bounding boxes

Imagery stats, viz, monitoring

Stats:

• We can probably gather a lot from eoAPI and other components using Prometheus.
• The data should be fed into a Grafana dashboard, which can be used to visualise.
• Perhaps we can also pull this into the OAM UI somehow too.
• AlertManager can be used via the existing slack integration to alert the team if system issues.

Monitoring:

• We can hook up OpenTelemetry tracing, coordinating with the new devops engineer.
• Each component can feed into Sentry for now, but we can set up a system in k8s at some point perhaps.
• Tracebacks / errors will be recorded for easier debugging.

Better STAC metadata

• The uploader-api needs to be tweaked here, to add extra stats as needed.
• The mentioned pixel/sqkm coverage stat.

[cgiovando]

Thanks, @spwoodcock - picking up the stats half of this, plus what came out of #326. I've put the user-facing half in #332.

With help from my agents, I went digging through k8s-infra to see what was already there, and a fair bit of this looks covered. Correct me if I've misread anything:

• apps/monitoring in k8s-infra runs kube-prometheus-stack cluster-wide already - Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics. Apps opt in with a ServiceMonitor, so I think the per-app side is a ServiceMonitor rather than a Prometheus deployment.
• Alertmanager already has the Slack path set up for #hot-tech-alerts, and there are DB alerts in apps/monitoring/database-alerts.yaml.
• eoapi-k8s v0.16.3 added ServiceMonitor support and wired in stac-fastapi-pgstac metrics. We pin 0.15.0 in apps/oam.yaml, so a chart bump might get us most of the way before anyone writes instrumentation, which is roughly where feat(stac): add optional Prometheus metrics export #326 ended up.
• uploader-api already has MONITORING, SENTRY_DSN and OTEL_ENDPOINT scaffolding in .env.example.

Observability and statistics

It might help to scope them separately.

Operational observability - is it up, how fast, what's erroring. Prometheus, Grafana, Sentry, OTel, short retention, per-process. Largely covered by the above as best I can tell.

Catalog and usage statistics - how much imagery, where, from whom, how much area, how much of it actually gets used. These are questions about the contents of the catalog rather than about processes, they need history over months and years, and the numbers have to be reproducible enough to publish.

Requirements for the statistics layer

S1. Counting definitions, implemented as queries. Hosted and federated imagery counted separately, never blended into one number. Some images were published as tiled ARD counted at scene level rather than tile level e.g. Maxar item IDs look like 19-033111220012-103001011E213900, so an item count and a scene count differ by more than an order of magnitude.

S2. Footprint quality. Area is only meaningful with true footprint geometry, so where we have bounding boxes it's overstated, badly for off-nadir captures - we need to know which items have real footprints before publishing any area figure. This may require backfill and a runtime step during uploads.

S3. Coverage metrics. Deduplicated union area, so repeat captures of the same place don't multiply, plus total pixels from raster metadata, which better reflects what a high-resolution survey is worth.

S4. History, not just current state. We need to answer how something changed over six months, not only what it is today, which means the figures have to be recorded periodically rather than computed on demand. Counting live against pgSTAC each time gives no trend, and I assume it doesn't scale either.

S5. Public statistics endpoints. Platform totals, per collection, provider, user and item. These are what the contributor profiles in #332 consume, and they're where I'd draw the boundary between the two threads.

S6. Usage, attributable. Tile requests and downloads resolvable back to an item, a provider and an uploader. Without attribution none of it can feed profiles, and which imagery actually gets used is probably the most useful thing we could learn about the catalog.

S7. Fix combined counts. #327 - density bucket counts are combined with min() across filters, so the map can advertise imagery matching neither filter.

Open questions

1. Where do rollups live: a table in the pgSTAC database, or separate so analytics don't compete with the API?
2. Tile attribution: parse TiTiler access logs, or instrument the tile path directly?
3. STAC-native aggregation extension, or a plain OAM statistics endpoint?
4. Retention for per-item usage, given it's effectively access logs.

---

I use AI tools to help draft and format; the ideas, content, and responsibility are my own.