confidence-metrics-sync

Manage Confidence metrics as code: define fact tables, measurements, and metrics in YAML in a git repository, validate every change on pull requests, and sync to Confidence on merge. Synced metrics appear in Confidence as repo-managed — read-only in the UI, owned by your repo and its review process.

confidence-metrics validate <path>   # check definitions, show what would change
confidence-metrics sync <path>       # apply definitions to Confidence

Quick start

1. Create an API client in Confidence and store its credentials as secrets in your repository (CONFIDENCE_CLIENT_ID, CONFIDENCE_CLIENT_SECRET). Use a dedicated API client per metrics repository.
2. Add metric definitions under metrics/:

# metrics/streaming.yaml
# yaml-language-server: $schema=https://raw.githubusercontent.com/spotify/confidence-metrics-sync/main/internal/schema/metric.schema.json

fact_tables:
  - display_name: Hourly Stream
    table: analytics.prod.hourly_stream
    timestamp_column: event_time
    entities:
      - entity: user
        column: user_id
    measures:
      - display_name: minutes_played
        column: ms_played / 60000

measurements:
  - display_name: Hourly Minutes Played
    fact_table: Hourly Stream
    entity: user
    measure: minutes_played
    operation: sum
    null_handling:
      replace_entity_null_with_zero: true
    metrics:
      - display_name: Minutes Played - Day 1
        measurement_window:
          aggregation_window: 86400s
        preferred_direction: increase

3. Add the workflow:

# .github/workflows/confidence-metrics.yaml
name: Confidence Metrics
on:
  pull_request:
    paths: ['metrics/**']
  push:
    branches: [main]
    paths: ['metrics/**']

concurrency:
  group: confidence-metrics-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
  validate:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: spotify/confidence-metrics-sync@v1
        with:
          path: metrics/
          mode: validate
          client-id: ${{ secrets.CONFIDENCE_CLIENT_ID }}
          client-secret: ${{ secrets.CONFIDENCE_CLIENT_SECRET }}

  sync:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: spotify/confidence-metrics-sync@v1
        with:
          path: metrics/
          mode: sync
          client-id: ${{ secrets.CONFIDENCE_CLIENT_ID }}
          client-secret: ${{ secrets.CONFIDENCE_CLIENT_SECRET }}

That's it: pull requests get inline annotations on definition errors plus a dry-run summary of what would change; merges to main reconcile Confidence with the repository.

The action derives --source-reference as github.com/<owner>/<repository> unless the source-reference input overrides it. It downloads the CLI release pinned by the selected action version and verifies the archive against the published SHA256 checksums before execution.

How it works

validate

Runs on pull requests. Makes no changes.

• Parses all YAML files under the given path and validates them against the metrics schema: structure, types, enums, and cross-references (measurements exist, measures belong to their fact table, no duplicate names).
• Runs each metric through Confidence's server-side validation.
• Shows a dry-run plan against your Confidence account: what would be created, updated, deleted, or left unchanged.
• With --check-warehouse (coming soon), additionally verifies each fact table against your warehouse via Confidence — the query compiles and every declared column exists with a supported type — without reading any rows.

Credentials are required by default: a missing CI secret fails the check loudly instead of silently passing a weaker one. For contexts without secrets (e.g. pull requests from forks), --offline explicitly opts into schema-only validation.

sync

Runs on merge. Reconciles Confidence with the repository:

• Creates and updates fact tables, measurements, and metrics to match the YAML. Synced metrics are marked repo-managed and become read-only in the Confidence UI.
• Archives metrics that this repository previously synced but that are no longer defined in the YAML — never hard-deletes. Re-adding the definition unarchives it.
• Refuses to take over a metric that already exists in Confidence but is not managed by this repository — a name collision can never silently overwrite another team's metric. Intentional migrations opt in with sync --adopt-from, which names the source being taken over, and validate always shows pending adoptions in the plan first.

Sync reports a summary of every run:

Sync complete:
  Created:   3 metrics, 1 fact table
  Updated:   1 metric
  Adopted:   0
  Archived:  0
  Unchanged: 12 metrics, 4 fact tables
  Errors:    0

Moving resources between repositories

Ownership follows --source-reference, so moving a metric to another repository is a transfer, in this order:

1. In the new repo, add the definitions and sync with --adopt-from <old-reference>. The plan reports each one as » metric "…" from <old-reference>.
2. In the old repo, delete them. Until it does, its syncs fail: it still declares resources that now belong elsewhere. That failure is the reminder, and only the old repo can clear it.

Do not reverse these steps. Removing the definitions from the old repo first archives the metric and deletes the measurement and fact table; the new repo then recreates those under fresh resource names, and anything that referenced the originals breaks. Adopting first is the only order that keeps resource names — and once ownership has moved, the old repo can no longer archive them, so step 2 is safe.

The same flag recovers from a mistyped --source-reference: sync once with --adopt-from <the-typo>.

Remove --adopt-from once the migration has landed. Sync warns when an entry adopted nothing, because a flag left standing in CI would silently take over whatever collides on a display name next.

Configuration

Environment variable	Purpose
CONFIDENCE_CLIENT_ID	API client ID (required for plan/sync)
CONFIDENCE_CLIENT_SECRET	API client secret (required for plan/sync)
CONFIDENCE_METRICS_URL	Metrics API override (default https://metrics.confidence.dev)
CONFIDENCE_IAM_URL	IAM/token API override (default https://iam.confidence.dev)

Credentials are only ever read from the environment — never from flags or files. Flags:

• --source-reference — identifier of this repository; scopes ownership and reconciliation (e.g. github.com/org/repo)
• --output text|json — machine-readable results for scripting (default text)
• --offline — schema-only validation without credentials (validate only)
• --adopt-from <source> — take ownership of matched resources currently owned by that source: api for resources no repository manages, or another repository's reference. Repeatable. There is no wildcard, and api is reserved as a --source-reference

Exit codes: 0 success · 1 validation findings or sync errors · 2 usage error · 3 authentication or transient API failure (safe to retry).

Other CI systems

The GitHub Action is a thin wrapper; any CI works the same way. Replace the version and platform below as needed:

VERSION=v1.0.0
curl -fsSLO "https://github.com/spotify/confidence-metrics-sync/releases/download/${VERSION}/confidence-metrics_linux_amd64.tar.gz"
curl -fsSLO "https://github.com/spotify/confidence-metrics-sync/releases/download/${VERSION}/SHA256SUMS"
sha256sum --check --ignore-missing SHA256SUMS
tar xzf confidence-metrics_linux_amd64.tar.gz
export CONFIDENCE_CLIENT_ID=... CONFIDENCE_CLIENT_SECRET=...
./confidence-metrics validate metrics/    # PR pipeline
./confidence-metrics sync metrics/        # main-branch pipeline

Release binaries are available for Linux, macOS, and Windows on amd64 and arm64. They ship with SHA256 checksums; the GitHub Action verifies them on download.

Editor support

The YAML schema is published from this repository at https://raw.githubusercontent.com/spotify/confidence-metrics-sync/main/internal/schema/metric.schema.json. Add the # yaml-language-server: $schema=... header (as in the example above) to get autocomplete and inline validation in VS Code, IntelliJ, and any editor with YAML language server support.

Development

Requires Go 1.26+.

make ci        # vet + test + build
make build     # binary at bin/confidence-metrics
make snapshot  # cross-platform builds via goreleaser

Pull requests and main run make ci on GitHub Actions. Release Please opens version PRs from Conventional Commit titles. Merging a release PR creates the tag and GitHub release; GoReleaser then builds the platform archives and publishes SHA256SUMS, and advances the matching major action tag such as v1 only after those assets are available.

License

Apache-2.0. See NOTICE for attribution information.