diff --git a/docs/docs.json b/docs/docs.json index a88602ae..14c4b8c9 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -56,7 +56,23 @@ "usage/rich-objects", "usage/model-training", "usage/cli", - "usage/export" + "usage/export", + { + "group": "Platform", + "pages": [ + "platform/self-hosted/overview", + "platform/self-hosted/install", + "platform/self-hosted/quickstart", + "platform/self-hosted/usage", + "platform/self-hosted/configure", + "platform/self-hosted/commands", + "platform/self-hosted/environment", + "platform/self-hosted/versioning", + "platform/self-hosted/advanced-usage", + "platform/self-hosted/troubleshooting", + "platform/self-hosted/faq" + ] + } ] }, { diff --git a/docs/usage/platform/advanced-usage.mdx b/docs/usage/platform/advanced-usage.mdx new file mode 100644 index 00000000..cc650e88 --- /dev/null +++ b/docs/usage/platform/advanced-usage.mdx @@ -0,0 +1,175 @@ +--- +title: 'Advanced Usage' +description: 'Configure the Dreadnode Platform for remote deployments and custom environments' +public: true +--- + +The `dreadnode` Platform can be configured for advanced deployment scenarios such as remote databases, proxy hosts, and external ClickHouse clusters. +These options are managed via the environment files (`.dreadnode-api.env` and `.dreadnode-ui.env`). + + +Modifying these files will impact your local instance. Always back up your configurations before making changes. + + +--- +### Using a Proxy Host (Remote UI Access) + +If you are running the UI on a remote host (not `localhost`), configure the **proxy host** (the domain/IP that external users will access): + +```bash +# .dreadnode-ui.env +PROXY_HOST=platform.example.com +ALLOWED_HOSTS="platform.example.com" +``` + +* `PROXY_HOST` defines the hostname where users will access the UI. +* `ALLOWED_HOSTS` must include the same value to pass and is passed to the Content Security Policy. +* Update DNS or reverse proxy settings (e.g., Nginx, Caddy) to point traffic to the correct container. + +--- + +### Configuring a Remote Database + +By default, the API service connects to a local Postgres database running in Docker. +To use a **remote database**, update the following variables in `.dreadnode-api.env`: + +```bash +# .dreadnode-api.env +DATABASE_USER=myuser +DATABASE_PASSWORD=secure-password +DATABASE_NAME=platform +DATABASE_PORT=5432 +DATABASE_HOST=db.example.com +```` + +* `DATABASE_HOST` should be the hostname or IP of your Postgres server. +* Ensure that the Postgres server accepts external connections, is reachable from your Docker containers, and your firewall/security groups allow access. + + +Changing the database configuration will start with a blank schema unless you migrate existing data. Make sure you back up your local database if you need persistence. + + +--- + +By default, ClickHouse is run locally in Docker. To use a **remote ClickHouse cluster**, adjust `.dreadnode-api.env`: + +```bash +# .dreadnode-api.env +STRIKES_CLICKHOUSE_HOST=clickhouse.example.com +STRIKES_CLICKHOUSE_PORT=8443 +STRIKES_CLICKHOUSE_USER=clickuser +STRIKES_CLICKHOUSE_PASSWORD=secure-password +STRIKES_CLICKHOUSE_DATABASE=platform +``` + +* `STRIKES_CLICKHOUSE_HOST` should be your remote server hostname. +* Ensure the cluster is reachable from the host running the platform. + +--- + +## Back up & Restore Docker Volumes (Best Practices) + +The volumes used by Docker will persist on disk, but a reliable backup strategy is recommended such as: + +**Logical backups (application-aware)** — recommended for databases (e.g., `pg_dump` for Postgres, `BACKUP` for ClickHouse). + +Use logical backups for consistency and faster point-in-time recovery; use volume snapshots for belt-and-suspenders disaster recovery. + +### Identify Your Volumes + +```bash +# Show compose resources (project must be in this directory) +docker compose ls +docker compose config | grep -A2 volumes: +docker volume ls | grep +``` + +> Compose typically prefixes volumes using the project name (e.g., `dreadnode_api-data`, `dreadnode_db-data`). + +--- + +Logical Backups + +#### Postgres (pg\_dump) + +Back up using `pg_dump` from the running DB container or a throwaway client: + +```bash +# From host, run a temporary Postgres client container to dump the remote/local DB +docker run --rm \ + -e PGPASSWORD=$DATABASE_PASSWORD \ + postgres:16 \ + pg_dump -h $DATABASE_HOST -p ${DATABASE_PORT:-5432} \ + -U $DATABASE_USER -d $DATABASE_NAME \ + -Fc -f - > postgres_$(date +%Y%m%d_%H%M%S).dump +``` + +Restore: + +```bash +# Create target DB first if needed: +# createdb -h $DATABASE_HOST -p ${DATABASE_PORT:-5432} -U $DATABASE_USER $DATABASE_NAME + +cat postgres_YYYYMMDD_HHMMSS.dump | docker run -i --rm \ + -e PGPASSWORD=$DATABASE_PASSWORD \ + postgres:16 \ + pg_restore -h $DATABASE_HOST -p ${DATABASE_PORT:-5432} \ + -U $DATABASE_USER -d $DATABASE_NAME --clean --if-exists +``` + + +Use the same major version of `postgres` in your dump/restore container as your server for best compatibility. + + +#### ClickHouse (native BACKUP) + +If you use ClickHouse 21.8+ (typical), run native backups to S3/MinIO: + +```sql +-- From a clickhouse-client shell or HTTP API: +BACKUP DATABASE platform TO S3( + 's3://my-clickhouse-backups/platform/{timestamp}', + 'YOUR_KEY_ID', 'YOUR_SECRET', + 'us-east-1', 'https://s3.amazonaws.com' +); +``` + +Restore: + +```sql +RESTORE DATABASE platform FROM S3( + 's3://my-clickhouse-backups/platform/', + 'YOUR_KEY_ID', 'YOUR_SECRET', + 'us-east-1', 'https://s3.amazonaws.com' +); +``` + + +For consistent backups, prefer **logical** methods (pg_dump / ClickHouse BACKUP) rather than copying live database files. + + +--- + + +### Example: Hybrid Deployment + +For a resilient hybrid deployment, you might: + +* Run **API & UI** services in Docker on a cloud VM. +* Point the **database** to a managed Postgres (e.g., AWS RDS). +* Use a **remote ClickHouse cluster** (e.g., AWS ClickHouse Cloud). +* Store artifacts in **AWS S3** (with a **CloudFront** CDN set as `S3_AWS_EXTERNAL_ENDPOINT_URL`). +* Expose the UI via a **proxy host** with TLS termination. +* Schedule nightly **pg\_dump** & ClickHouse **BACKUP** to S3, weekly volume snapshots, and test restores monthly. + +--- + + +Whenever you make changes to `.dreadnode-api.env` or `.dreadnode-ui.env`, restart the platform with: + +```bash +dreadnode platform stop +dreadnode platform start +``` + + \ No newline at end of file diff --git a/docs/usage/platform/commands.mdx b/docs/usage/platform/commands.mdx new file mode 100644 index 00000000..bdff7c74 --- /dev/null +++ b/docs/usage/platform/commands.mdx @@ -0,0 +1,42 @@ +--- +title: 'Command Reference' +description: 'All Platform CLI commands and options' +public: true +--- + +```bash +dreadnode platform start [--tag ] +```` + +Start the platform. If `--tag` is omitted, uses the current version or downloads `latest-`. + +```bash +dreadnode platform stop +dreadnode platform down +``` + +Stop all platform containers for the current version and remove the generated `.env`. + +```bash +dreadnode platform download --tag +``` + +Download templates for a specific tag (supports `latest-` and versioned tags). + +```bash +dreadnode platform upgrade +``` + +Upgrade to the latest version for your architecture. Prompts for confirmation and optional env merge. + +```bash +dreadnode platform refresh_registry_auth +``` + +Log into all registries referenced by the current version. + +```bash +dreadnode platform configure [--service api|ui] +``` + +Open the per-service env file in your OS editor (`open -t` on macOS, `xdg-open` otherwise). \ No newline at end of file diff --git a/docs/usage/platform/configure.mdx b/docs/usage/platform/configure.mdx new file mode 100644 index 00000000..f9c7e87a --- /dev/null +++ b/docs/usage/platform/configure.mdx @@ -0,0 +1,154 @@ +--- +title: 'Configure' +description: 'Set persistent platform configuration via key-value overrides; list, unset, or apply one-off overrides at start' +public: true +--- + +Use the `configure` command to **persist platform overrides** (e.g., ports, proxy host) for the **current** platform version, or for a **specific image tag**. +You can also supply **one-off (ephemeral) overrides** directly to `start` for a single run—see **Start-time overrides** below. + + +Overrides are stored with the selected platform version and take effect the **next time you start or restart** the platform. You can modify or remove them at any time. + + +## TL;DR + +```bash +# Persist one or more overrides on the current version +dreadnode platform configure KEY VALUE [KEY2 VALUE2 ...] + +# Target a specific version by image tag when persisting +dreadnode platform configure KEY VALUE --tag v1.3.4 + +# List currently active persistent overrides +dreadnode platform configure --list + +# Unset (remove) one or more persistent overrides +dreadnode platform configure --unset KEY [KEY2 KEY3 ...] +```` + +### Examples + +```bash +# Configure an HTTP proxy host and expose the API on a custom port +dreadnode platform configure proxy-host myproxy.local api-port 8080 + +# Target a specific image tag (when you have multiple downloaded) +dreadnode platform configure proxy-host corp-proxy --tag v1.3.4 + +# Remove a single override +dreadnode platform configure --unset proxy-host + +# Remove multiple overrides at once +dreadnode platform configure --unset proxy-host api-port +``` + +## Behavior & File Outputs + +When you run `configure`: + +1. The command resolves the **selected platform version**: + + * If `--tag` is provided, it uses that local version. + * Otherwise, it uses your **current** platform version. +2. It **writes your overrides** to that version's configuration-overrides file. +3. It also ensures a Docker Compose override is prepared so changes apply cleanly on the next start. + +On start, the platform **regenerates** its runtime configuration (Compose files and a composite `.env`) and **applies your persistent overrides on top** of the base service configuration. + + +If **no current platform version** is set (and no `--tag` is provided), you'll be prompted to start or download a platform version first. + + +## Flags (configure) + +* `--list, -l` + Prints the currently configured persistent overrides for the selected version. Cannot be combined with positional `KEY VALUE` arguments. + +* `--unset, -u` + Removes the specified keys. Provide one or more keys (**values are not required** when using `--unset`). + +* `--tag ` + Apply changes to a **specific** downloaded image tag instead of the current version. + +## Input Rules (configure) + +* Without `--unset`, arguments must be provided as **key-value pairs**: + + ```bash + dreadnode platform configure KEY VALUE [KEY2 VALUE2 ...] + ``` +* With `--unset`, provide **one or more keys** (no values): + + ```bash + dreadnode platform configure --unset KEY [KEY2 ...] + ``` + +--- + +# Start-time overrides (one-off) + +You can pass **temporary overrides** directly to `start`. These apply **only to that run** and are cleared the next time you start **without** those flags. + +```bash +# Start the current (or resolved) version +dreadnode platform start + +# Start a specific image tag +dreadnode platform start --tag v1.3.4 + +# Start with one-off overrides (use --key value form) +dreadnode platform start --proxy-host myproxy.local --api-port 8080 +``` + +### How it works + +* `start` accepts arbitrary **`--key value`** pairs as environment overrides (e.g., `--proxy-host myproxy.local`). +* If any are provided, they are written to a **start-argument overrides file** for this run. +* If none are provided, the **start-argument overrides file is removed** before launching, returning you to your **persistent** configuration. + +### Precedence (highest first) + +1. **Start-time (one-off) overrides** passed via `dreadnode platform start --key value` +2. **Persistent overrides** set via `dreadnode platform configure` +3. **Base configuration** and service defaults + +This means a key like `proxy-host` passed to `start` will **override** the same key from `configure`—but **only for that run**. + +### Examples + +```bash +# Temporarily change API port for this run only +dreadnode platform start --api-port 9090 + +# Temporarily bypass proxy for this run, regardless of persistent settings +dreadnode platform start --proxy-host "" + +# Combine with a tag you haven't marked current yet +dreadnode platform start --tag v2.0.0 --api-port 8081 +``` + + +To make a one-off override permanent, set it with `dreadnode platform configure KEY VALUE`, then start without the flag. + + +## Start behavior + +* If Docker/Docker Compose requirements aren't met, the command exits with an error. +* With `--tag`, the specified tag is **downloaded (if necessary)** and marked current before running. +* Without `--tag`, if a current version exists it is used; otherwise the **latest** is downloaded and marked current. +* On success, you'll see the app URL(s) if available. + +--- + +## Migration Notes (from older docs) + +* **Old behavior:** `platform configure` opened per-service env files (e.g., `.dreadnode-api.env`, `.dreadnode-ui.env`) in your editor. +* **New behavior:** + + * `platform configure` **persists** key-value overrides (no editor). + * `platform start` can accept **ephemeral** key-value overrides via `--key value` flags for a single run. + + +You can still edit service env files directly if needed, but for repeatable, layered config, prefer `configure` for persistence and `start --key value` for safe one-offs. + diff --git a/docs/usage/platform/environment.mdx b/docs/usage/platform/environment.mdx new file mode 100644 index 00000000..4e465bd8 --- /dev/null +++ b/docs/usage/platform/environment.mdx @@ -0,0 +1,46 @@ +--- +title: 'Environment & Files' +description: 'Where files live and how envs are generated/merged' +public: true +--- + +## Local cache & manifest + +- Base directory: `~/.dreadnode/platform` +- Manifest: `versions.json` + - Tracks all downloaded versions and which one is **current** +- Per-version directory: `~/.dreadnode/platform/` + - `docker-compose.yaml` + - `.dreadnode-api.env`, `.api.example.env` + - `.dreadnode-ui.env`, `.ui.example.env` + - `.env` (generated on start, concatenation of `.dreadnode-api.env` + `.dreadnode-ui.env`) + +## Services + +Two primary services: + +- `api` — the Dreadnode backend API +- `ui` — the Dreadnode frontend + +## Composite `.env` + +On `start`, the CLI concatenates: + +1. `..env` +2. `..env` + +into a **generated** `.env` consumed by Docker Compose. + + +Don't edit the generated `.env` directly—your changes will be overwritten on next start. Edit `.dreadnode-api.env` and `.dreadnode-ui.env` instead. + + +## Upgrade-time env merge + +If you opt-in during `upgrade`: + +- **Local changes win** over remote defaults. +- **Removed keys** upstream are dropped only when your local value wasn't changed from the original. +- **New remote keys** are added. +- **New local keys** are preserved. +- The merge attempts to keep **section context and formatting** where possible. \ No newline at end of file diff --git a/docs/usage/platform/faq.mdx b/docs/usage/platform/faq.mdx new file mode 100644 index 00000000..33fb9f67 --- /dev/null +++ b/docs/usage/platform/faq.mdx @@ -0,0 +1,27 @@ +--- +title: 'FAQ' +description: 'Answers to common questions' +public: true +--- + +#### Where are my downloaded versions? + +Under `~/.dreadnode/platform`, one directory per tag, tracked by `versions.json`. + +#### Can I run a specific historical version? + +Yes: +```bash +dreadnode platform start --tag vX.Y.Z- +``` + + +#### How do registries get authenticated? + +The CLI fetches credentials from the API and runs `docker login` per unique registry. If creds in `~/.docker/config.json` are fresh (by mtime), it skips login. + +#### What if I'm on Windows? + +This has not been developed for, nor tested on Windows. + +You can try to use WSL2 + Docker Desktop. The CLI checks `docker` and `docker compose` availability regardless of OS. \ No newline at end of file diff --git a/docs/usage/platform/install.mdx b/docs/usage/platform/install.mdx new file mode 100644 index 00000000..94ee9bfd --- /dev/null +++ b/docs/usage/platform/install.mdx @@ -0,0 +1,40 @@ +--- +title: 'Requirements & Installation' +description: 'Install the Dreadnode SDK & prerequisites' +public: true +--- + +The `dreadnode` SDK includes the Platform CLI. + + +```bash pip +pip install -U dreadnode +```` + +```bash uv +uv pip install dreadnode +``` + +```bash uv (project) +uv add dreadnode +``` + +```bash poetry (project) +poetry add dreadnode +``` + + + +#### Prerequisites + +* **Docker** and **Docker Compose** must be installed and available on `PATH`. + +* **Dreadnode Account** (for registry credentials and templates). + + * The CLI will use your configured API client (`create_api_client()`) to fetch releases and credentials. + * You can create your account [here](https://platform.dreadnode.io). + + +To access the private Dreadnode Platform images, you need a Dreadnode account and a Platform license. +[Contact us](https://dreadnode.io/contact-us) for more information. + diff --git a/docs/usage/platform/overview.mdx b/docs/usage/platform/overview.mdx new file mode 100644 index 00000000..93c821e2 --- /dev/null +++ b/docs/usage/platform/overview.mdx @@ -0,0 +1,39 @@ +--- +title: 'Overview' +description: 'Self Hosted Platform' +public: true +--- + +Deploy Dreadnode's evaluation and observability platform entirely within your own infrastructure to maintain complete control over sensitive security data. + +#### Why self-host Dreadnode? + +- **Keep your sensitive data secure** + +- **Meet your compliance requirements** + +- **Control your evaluation environment** + +- **Connect to your data, tools, and models** + + +#### What you get + +- **Full-featured evaluation platform** — All the same capabilities as Dreadnode Cloud, running entirely in your environment +- **Simple deployment** — One-command setup that handles version management, container orchestration, and configuration +- **Seamless updates** — Upgrade to latest features while preserving your custom configurations and data +- **Architecture flexibility** — Runs in Docker on Linux or Mac (advanced cloud deployments coming soon) + +#### How it works + +The self-hosted platform uses a simple CLI to manage your local Dreadnode deployment. +Behind the scenes, it handles version downloads, Docker container management, environment configuration, and secure registry access — so you can focus on your security evaluations instead of infrastructure management. + + +Access all platform management through the `dreadnode platform` CLI commands (start, stop, upgrade, etc.). + + + +This is a feature available to Enterprise Users of Dreadnode Cloud. +Contact Support for more information. + \ No newline at end of file diff --git a/docs/usage/platform/quickstart.mdx b/docs/usage/platform/quickstart.mdx new file mode 100644 index 00000000..fc499565 --- /dev/null +++ b/docs/usage/platform/quickstart.mdx @@ -0,0 +1,31 @@ +--- +title: 'Quickstart' +description: 'Start using the platform' +public: true +--- + +## Log in to Dreadnode Cloud + +```bash +dreadnode login +``` + + +## Start the platform + +```bash +# Download the latest for your host architecture and start running +dreadnode platform start +``` + +## Create An Account in your Local Platform + +Visit [http://localhost](http://localhost) in your browser to create a local account. + +## Create a new Local SDK Profile + +```bash +dreadnode login --server http://localhost --profile local +``` + +You can now start shipping data to your local Dreadnode instance! \ No newline at end of file diff --git a/docs/usage/platform/troubleshooting.mdx b/docs/usage/platform/troubleshooting.mdx new file mode 100644 index 00000000..8a7491e7 --- /dev/null +++ b/docs/usage/platform/troubleshooting.mdx @@ -0,0 +1,58 @@ +--- +title: 'Troubleshooting' +description: 'Common issues and how to fix them' +public: true +--- + +#### Docker / Compose not found + +- Ensure Docker Desktop or Docker Engine + Compose V2 is installed. +- Verify: + + ```bash + docker --version + docker compose --version + ``` + +#### 403 Forbidden Error + +- Ensure you've completed `dreadnode login` to the Dreadnode server. +- Ensure you've gotten a Platform License from [Dreadnode](https://dreadnode.com/contact-us). + +#### Failing to log into container registry + +* Ensure your API client is configured and can retrieve registry credentials. +* Re-run: + + ```bash + dreadnode platform refresh_registry_auth + ``` +* If credentials look stale, delete `~/.docker/config.json` entries for the registry and try again. + +#### UI URL not printed + +* The CLI tries to read `ORIGIN` from the `dreadnode-ui` container: + + ```bash + docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' dreadnode-ui | grep ORIGIN + ``` +* If it's not set, check container logs and env configuration in `.dreadnode-ui.env`. + +#### Start succeeds but services are unhealthy + +* Inspect containers: + + ```bash + docker compose -f ~/.dreadnode/platform//docker-compose.yaml ps + docker compose -f ~/.dreadnode/platform//docker-compose.yaml logs -f + ``` +* Verify required env vars are populated in `.dreadnode-api.env` and `.dreadnode-ui.env`. + +#### Upgrade didn't apply my custom env changes + +* If you **skipped** the merge step, your new version uses stock envs. +* Re-run `upgrade` again (or manually port your changes), then `start`. + + +You can always edit `..env` and restart to apply configuration changes. + \ No newline at end of file diff --git a/docs/usage/platform/usage.mdx b/docs/usage/platform/usage.mdx new file mode 100644 index 00000000..b2af6019 --- /dev/null +++ b/docs/usage/platform/usage.mdx @@ -0,0 +1,90 @@ +--- +title: 'Usage' +description: 'Start, stop, download, and upgrade the platform' +public: true +--- + +Once installed, use the `platform` namespace: + +```bash +dreadnode platform --help +```` + +### Start the platform + +```bash +# Start current version, or download the latest for your architecture if none exists +dreadnode platform start + +# Start a specific tag +dreadnode platform start --tag v1.2.3-amd64 +``` + +What happens: + +1. Logs in to the remote container registries. +1. Downloads any necessary definition and configuration files. +1. Pulls the latest images. +1. Runs the containers. + +### Stop the platform + +```bash +dreadnode platform stop +# alias: +dreadnode platform down +``` + +Stops and removes all running application containers. + +**Note:** This will NOT remove any data or delete images. You can restart the containers again. + +### Download a specific version (without starting) + +```bash +dreadnode platform download --tag v1.2.3-amd64 +dreadnode platform download --tag latest-amd64 +``` + +This fetches templates and writes them under the [versioned environment](/platform/cli/environment) + +### Upgrade to the latest + +```bash +dreadnode platform upgrade +``` + +Flow: + +1. Resolves `latest-` and downloads it. +2. If the remote is **newer** than your current semver, prompts to continue. +3. Optionally **merges env files** from your current version into the new version + (local changes win; removed keys are dropped if unchanged; new remote keys are added). +4. Stops the current version and starts the upgraded one. + + +If you skip the merge, your new version will carry the stock `..env` files from its templates. + + +### Refresh registry auth (out-of-band) + +```bash +dreadnode platform refresh-registry-auth +``` + +If you are modifying or restarting the platform out of band (using Docker Compose), you may need to refresh credentials to the private image registries. + +### Connect the SDK +Once you've started the platform, you can connect the SDK by running: + +```bash +dreadnode login --server http://localhost --profile local +``` + + +Make sure to replace `http://localhost` with the appropriate URL if you are accessing the platform remotely. + + +You can now start shipping data to your local Dreadnode instance! + +or, if you are using the Python SDK, see [the Python SDK docs](/strikes/usage/config#self-hosted-platforms). \ No newline at end of file diff --git a/docs/usage/platform/versioning.mdx b/docs/usage/platform/versioning.mdx new file mode 100644 index 00000000..12d05ed8 --- /dev/null +++ b/docs/usage/platform/versioning.mdx @@ -0,0 +1,26 @@ +--- +title: 'Versioning & Architectures' +description: 'How tags are resolved and compared' +public: true +--- + +## Architectures + +Supported architectures: + +- `amd64` +- `arm64` + +On machines reporting `x86_64`/`AMD64` → `amd64` +On machines reporting `arm64`/`aarch64`/`ARM64` → `arm64` + +## Latest tags + +The CLI constructs **arch-aware** latest tags: + +```txt +latest-amd64 +latest-arm64 +``` + +When you run `start` without a current version, it resolves to the appropriate `latest-`.