Add dp3 sh CLI interface

[xsedla1o]

Add dp3 sh and extend the HTTP API for troubleshooting and telemetry

Summary

This PR adds a new shell-oriented DP3 client, dp3 sh, together with a generated per-app wrapper <APPNAME>sh.

It also extends the HTTP API so the CLI can cover common troubleshooting and operational reads without falling back to direct database access.

The main pieces are:

• new dp3 sh ... CLI
• generated <APPNAME>sh ... wrapper from dp3 config supervisor
• shell completion support
• HTTP API support for raw datapoint reads, snapshot paging, and broader telemetry reads
• docs updated to prefer dp3 sh / <APPNAME>sh for routine same-host workflows

The CLI is intentionally API-backed rather than DB-backed. The point is not just to have a convenient operator tool, but also to make the supported HTTP API broad enough to cover real shell workflows.

---

Why

Before this PR, common troubleshooting often fell into one of these buckets:

• hand-written curl requests
• direct mongosh queries
• ad hoc telemetry/exporter scripts

That worked, but it meant:

• no first-class CLI consumer of the DP3 API lived in-tree
• some useful reads were only available via database knowledge or one-off scripts
• docs had to jump quickly from high-level usage to raw HTTP or MongoDB examples

This PR addresses that by adding a proper CLI and by extending the HTTP API where needed.

---

What this PR adds

1. New CLI: dp3 sh

This PR adds a new command family:

dp3 sh ...

It is designed for shell use:

• JSON/NDJSON output
• command shapes that work well with pipes and jq
• entity-oriented inspection commands
• coverage for common entity, control, datapoint, and telemetry workflows

Example commands:

dp3 sh health
dp3 sh datapoints payload.json

dp3 sh entities | jq keys

dp3 sh entity device list --has-attr risk_score

dp3 sh entity device id device-123 master
dp3 sh entity device id device-123 attr risk_score get
dp3 sh entity device id device-123 snapshots -l 10

dp3 sh telemetry sources-validity
dp3 sh telemetry metadata -F ndjson | head

2. Generated app wrapper: <APPNAME>sh

dp3 config supervisor now also generates:

<APPNAME>sh

This gives deployed apps a ready-to-use wrapper with config resolution already wired in.

3. Shell completion

This PR adds shell completion support for:

• bash
• zsh
• fish

Completion is config-aware and supports the dp3 sh command structure.

4. HTTP API extensions

This PR extends the HTTP API so the CLI can expose more of the useful troubleshooting and telemetry surface.

Entity/raw inspection

• raw datapoint browsing endpoint for current raw datapoints
• snapshot paging support on entity snapshot reads

Telemetry/operational reads

• source age
• entities per attribute
• snapshot summary
• metadata browsing
• RabbitMQ queue telemetry

These API reads replace or subsume several existing script-style operational queries.

5. Documentation updates

Docs now prefer dp3 sh / <APPNAME>sh as the normal same-host workflow, while still keeping:

• curl as the raw HTTP fallback
• mongosh as the lower-level DB fallback where appropriate

---

HTTP API changes

Breaking change

This PR removes the old:

GET /entity/{etype}

That endpoint is the main intentional breaking change in this PR.

Additive/extensions

The rest of the HTTP API work is additive or extends existing reads, especially around:

• entity snapshot paging
• raw datapoint reads
• telemetry and metadata endpoints

In other words, the API surface is broadened for troubleshooting and observability, but the main compatibility change is the removal of the old entity-type endpoint above.

---

Design notes

API-backed on purpose

The CLI does not read the database directly. It goes through the HTTP API.

That is intentional because it:

• keeps the CLI aligned with supported API behavior
• forces useful troubleshooting reads to exist as explicit endpoints
• makes the same surface reusable by future clients
• avoids teaching routine users to start with MongoDB internals

Telemetry intent

The telemetry additions are not just for the CLI.

They also expose operational reads that had previously existed mainly as exporter/helper scripts, so they can be consumed in a supported way through the HTTP API.

---

Suggested reviewer focus

• whether dp3 sh feels like a good first shell client for DP3
• whether the HTTP API additions expose the right raw/telemetry surface
• whether the removal of GET /entity/{etype} is acceptable and documented clearly enough
• whether the docs now strike the right balance between dp3 sh, curl, and mongosh

[Copilot]

Pull request overview

This PR introduces a new shell-oriented operator CLI (dp3 sh) (plus a generated per-app wrapper <APPNAME>sh) and expands the DP3 HTTP API to support common troubleshooting/telemetry workflows (raw datapoint browsing, snapshot paging, and additional telemetry reads), with accompanying docs and integration/unit tests.

Changes:

• Add dp3 sh command family (health, datapoints, entities/entity inspection, control, telemetry) with argcomplete-based shell completion.
• Extend API/database capabilities for troubleshooting: raw datapoint reads, snapshot paging (skip/limit), telemetry endpoints (sources age, entities-per-attr, snapshot summary, metadata browsing, RabbitMQ queue telemetry).
• Update docs and CI scripts/workflows to prefer the new CLI and to run API integration tests via a local runner script.

Reviewed changes

Copilot reviewed 27 out of 43 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
dp3/bin/sh.py	New dp3 sh entrypoint, argcomplete completion support, API-backed client wiring.
dp3/bin/shcmd/**	Implements shell-oriented command families (entity/entity-type/entity-id, telemetry, etc.).
dp3/api/routers/entity.py	Adds raw datapoint endpoint and snapshot pagination; removes deprecated GET /entity/{etype}.
dp3/api/routers/telemetry.py	Adds telemetry endpoints (sources age, entities-per-attr, snapshot summary, metadata, RabbitMQ queues).
dp3/history_management/telemetry.py	Expands TelemetryReader to back new API telemetry reads (incl. RabbitMQ management polling).
dp3/database/database.py	Adds find_raw_datapoints, find_metadata, count_entities_per_attr.
dp3/database/snapshots.py	Adds paging to snapshot reads; refactors latest-snapshot query API; adjusts delete methods return types.
tests/test_common/test_sh_completion.py	Adds completion/arg parsing unit tests for dp3 sh.
tests/test_api/test_raw.py, tests/test_api/test_telemetry.py, tests/test_api/test_snapshots.py	Adds/updates API integration tests for new endpoints and paging.
docs/**/*.md, docs/api.md	Updates docs to prefer dp3 sh / <APPNAME>sh and documents new/changed API endpoints.
scripts/run_api_tests_local.sh, scripts/run_api_test_shard.py	Adds local/CI-friendly API test runner and sharding.
.github/workflows/tests.yml, .github/workflows/release.yml	Switches API integration test execution to the new runner script.
requirements.txt	Adds argcomplete.
docker/python/Dockerfile	Sets SETUPTOOLS_SCM_PRETEND_VERSION during image build.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dbnk0]

LGTM