5.0.2

Fixes

• g3dt.utils.athena_utils.construct_data_import_order is back as a thin wrapper over g3dt.import_order.derive_import_order. The aws-gen3-pipeline template's release-export Glue job (write_data_release_to_json.py) imports that name to write each release's DataImportOrder.txt; 5.0.0 had dropped it, so every deployment pinned to toolkitVersion >= 5.0.0 failed its release export with ImportError: cannot import name 'construct_data_import_order' (#15)

Full Changelog: 5.0.1...5.0.2

5.0.1

Fixes

• g3dt dict status no longer reports DRIFT when the S3 stamp (1.2.0, from the JSON's _dict_version) and the declared git tag (v1.2.0) are the same version; the prefix is ignored on both sides (c000a4b)

Full Changelog: 5.0.0...5.0.1

5.0.0

Breaking Changes

• ArgoCD sync is opt-in on every restart and deploy flow: k8s restart-ms and synth deploy no longer run argocd app sync before the ETL. Pass --sync (now available on restart-schema, restart-etl, restart-ms, dict deploy and synth deploy) to keep the old behaviour (bc5000e)
• k8s restart-ms honours --restart-services; its fallback order is the documented sheepdog, peregrine, guppy, portal (bc5000e)
• metadata upload (single study) requires the typed production confirmation, like upload-all already did (74f4d43)

Features

• metadata upload --release <tag> and upload-all --release: upload a specific release of a study, resolved from the registry path and verified in S3 (DataImportOrder.txt plus node JSONs) before dispatch, on the laptop even for --on ec2 (74f4d43)
• study set --path verifies the new S3 prefix and gains --dry-run / --no-verify; new config study-set alias; the config group help, README and g3dt docs now point to it (9b60a8e)
• g3dt --version; g3dt dict status shows the declared dictionary version next to the one stamped on the deployed S3 object, with --strict exiting 1 on drift (228f68c)

Other

• Help-text polish from a new-developer audit: production gates stated wherever they run, laptop/browser notes and examples on every k8s command, --which explained on pipeline, jobs group help, a glossary in g3dt docs, one name for the CDK template repo, pipx install notes (d2fd70d)

Full Changelog: 4.3.0...5.0.0

4.3.0

Features

• Resolve the delete metadata node order automatically instead of requiring a DataImportOrder.txt in the working directory: explicit --import-order <path|s3://…> → the registered study's release bucket → cwd DataImportOrder.txt → a topological sort derived from the dictionary (the env's deployed schema by default, or a --dict-version tag's bundle cached under ~/.g3dt/schemas). Workers log which source ordered every run (ee00de5)
• Add --import-order and --dict-version to g3dt delete metadata; with --on ec2 only s3:// import-order URIs are accepted, since a laptop path could resolve to a different file on the box (ee00de5)

Full Changelog: 4.2.0...4.3.0

4.2.0

Features

• Add --synthetic to g3dt delete metadata: registry-free deletion of synthetic data — each --studies name is used directly as the Gen3 project id, bare names default to all versions (unskippable confirmation), and a specific --version matches records' data_version property via a new GraphQL worker, since synthetic uploads write no Athena receipts (b264219)
• Add --program-id for synthetic deletes under non-default Gen3 programs (b264219)
• Add g3dt synth generate --data-version to stamp generated records' data_version property, making future batches version-deletable (b264219)

Fixes

• Tolerate s3:// URIs (including an accidentally doubled scheme) and S3 endpoint https URLs in the schema_s3_uri app fact — values normalize to scheme-less bucket/key at resolve time, fixing dictionary deploy/upload failing with Invalid bucket name "s3:" (bd903e5)

Other

• Require gen3-metadata-simulator ^0.5.3 for the synth extra: 0.5.3 generates real date-times for oneOf-nullable and format: date-time properties instead of random words that sheepdog rejects (9d92052)

Full Changelog: 4.1.0...4.2.0

4.1.0

Breaking Changes

• The marker studies: block is no longer read. The study registry now lives in the env's SSM tree; a one-time notice points at g3dt study migrate (2921fa3)

Features

• Add the g3dt study command group — the registry that decides what metadata upload sends to a commons moves from s3://<metadata-bucket>/config/studies.yaml into per-study SSM parameters (/{project}/{env}/studies/<name>, one atomic JSON parameter each):
  • study list / study show — script-friendly listing and a per-study liveness check (DataImportOrder.txt + node-JSON count, the exact upload preflight)
  • study add / study set / study remove — validated, atomic per-study writes with typed production gates
  • study repoint --release <tag>|--latest — the release-cutover verb: swaps each path's vX.Y.Z segment, validates every target before writing anything (one bad target writes nothing), reads --latest from the releases ledger, supports --dry-run
  • study migrate — all-or-nothing import of the legacy studies.yaml, verified by re-reading SSM before the file is renamed to .migrated; idempotent
• Guided errors throughout: misses list the actually-registered studies with a did-you-mean; an empty registry prints copy-pasteable study add/study migrate commands; auth failures propagate with an aws sso login hint instead of masquerading as "no studies configured"
• Environment separation is structural: the SSM path is the env, retiring the {study}_{env} key-suffix convention and its bare-key fallback trap; lookups stay case-forgiving and accept the legacy suffixed wire form, so dispatched service scripts need no changes
• Mutating commands refuse to run while the legacy S3 registry still serves (writing to SSM would silently shadow the whole file) — study migrate is the only bridge
• config studies becomes an alias of study list; config show --study unchanged on the surface, new resolver underneath
• New normative design doc: docs/design/studies.md

Deprecations

• The S3 config/studies.yaml fallback remains readable with a once-per-process warning and is removed in 5.0, keeping EC2 job boxes on older pinned toolkits working until each env migrates

Full Changelog: 4.0.1...4.1.0

4.0.1

Patch release fixing the CI test failures that blocked 4.0.0 from reaching PyPI — this is the first published package carrying the 4.0.x changes. See the 4.0.0 release notes for the full changeset (local-first config contexts, single-profile discovery, scoped synth uploads, required --studies, consistent short aliases, widened production gates).

Fixes

• Make CLI test rendering deterministic on CI: rich force-enables ANSI color under GitHub Actions, so --help substring assertions passed locally but failed on CI with escape-code-laden output; the test suite now sets NO_COLOR and strips any remaining ANSI codes before asserting (8464a30)

Full Changelog: 4.0.0...4.0.1

4.0.0

A UX overhaul of the CLI driven by two live incidents and a full help-text audit: context listing is now offline-first, discovery is built around one SSO profile at a time, synth deploy can no longer pick up stale batches, and the option surface (aliases, required args, types) is consistent CLI-wide.

Breaking Changes

• g3dt config contexts no longer contacts AWS by default — it lists the local marker instantly and offline; pass --verify for the per-context deployed check (now with botocore SSO tracebacks suppressed and an aws sso login hint on failures). --no-verify remains accepted as the default's name (d39ca72)
• Bare g3dt config discover no longer verifies configured contexts — it prints the discovery model and your ~/.aws/config profiles with no network calls; use g3dt config contexts --verify to check registered contexts. The repeatable --profile option is deprecated and hidden (fab58ad)
• g3dt synth deploy now requires --studies; the hardcoded ACDC demo default set (and its per-study record counts) is removed (b33ffbb)
• The synth commands' silent env="test" fallback is removed — synth resolves the current context like every other command group and errors with guidance when nothing is configured (f567e3f)
• Short-alias reassignments for a one-letter-one-meaning convention: k8s --sync and config diff --file are long-only (freeing -s for studies and -f for --follow), and indexd check-download --limit moves from -n to -l (f567e3f)

Features

• g3dt config current prints the bare current context name (script-friendly; exit 1 when none) (d39ca72)
• Single-profile-first discovery: g3dt config discover <profile> validates the SSO session, offers aws sso login inline, then scans that one account; --all-profiles remains as the best-effort sweep (fab58ad)
• Production gates on the widest-blast-radius commands: dict upload, dict deploy, and all k8s restart-* now require the typed context/env confirmation on production targets, matching the synth and delete commands; dict pull and the CodeBuild-driven release write stay ungated (7b9e0a5)
• Consistent short aliases CLI-wide: new -c --ctx, -v --version, -o --on, -l --limit, -r --region, -d --dry-run, -i --import-order, -k --key-path, and -s extended to every --studies (f567e3f)
• Stronger option validation: file options are Path with existence checks (a typo'd path is a usage error, not a traceback), fixed choice sets are enforced enums (synth --provider random|llm, --llm-provider anthropic|openai, pipeline --which per command), and synth deploy/generate reject mismatched study/count lists before anything runs (f567e3f)

Fixes

• synth deploy step [6] now scopes the upload to the run's --studies (--projects on the upload service script): stale study directories from earlier batches are skipped with a log line instead of being submitted — previously a leftover directory from another project's run could 404 and abort the whole deploy before any requested study uploaded (b33ffbb)
• One failing project no longer aborts the rest of a synth upload batch: healthy studies upload first, then the run exits 1 naming the failures (b33ffbb)
• Wrapped service scripts run Python via ${G3DT_PYTHON} (the interpreter owning the g3dt install) instead of bare python3, fixing a latent wrong-interpreter bug on AL2023 (b33ffbb)

Other

• Help-text overhaul: the root and config help now define what a context is; --env has one shared phrasing across ~20 commands; stale references to the deprecated config set now point at synth set-key; config diff examples show the required --file; deploy scripts echo "environment" where they previously (mis)said "profile"; synth deploy docs use the script's real [1]–[7] step numbers (b05b08d)
• Docs updated throughout: docs/design/contexts.md (normative command table, prod-gate set) and the README bootstrap, synth, and k8s sections (8d72df9)

Full Changelog: 3.8.1...4.0.0

3.8.1

discover, reworked around how SSO actually behaves

g3dt config discover <aws-profile> is now the recommended flow: it checks
that profile's SSO session first, offers to run aws sso login right
there, then scans that one account for deployed /{project}/{env}
environments — no more sweep output that is mostly "go log in N times".

• Findings are deduplicated when several profiles see the same account
  (e.g. a default alias) — one suggestion/registration per context, first
  profile wins.
• --all-profiles remains for the opt-in full sweep (stale sessions are
  skipped with a login hint, never logged in automatically).
• Guidance strings and docs updated to the single-profile flow.

3.8.0

Contexts: one obvious answer to "what am I pointed at?"

A context is a named (project, env, profile, region) tuple. The command
surface reads as plain English (design doc: docs/design/contexts.md):

g3dt config discover --all-profiles --add   # find deployed infra, register it
g3dt config contexts                        # list them (current marked *)
g3dt config use acdc/staging                # act there from now on
g3dt --ctx acdc/prod <command>              # one-shot override

• Universal banner: every command prints the resolved context first (on
  stderr — eval $(g3dt config dbt-env ...) stdout stays clean).
• Production guards: prod contexts are marked [PROD]; switching to one
  asks for confirmation; destructive actions require typing the context name;
  --yes never bypasses a production prompt.
• Full back-compat: legacy markers (project/default_env/profiles), the
  3-key EC2 box marker, the file-less CodeBuild path (G3DT_PROJECT + --env),
  and the --env <base>_ec2 dispatch wire form all behave identically.
  config show/diff/dbt-env default to the current context; --env selects
  the matching context everywhere. config set is deprecated (hidden); the
  LLM key path moved to g3dt synth set-key <path>.

Fixes

• upload-all: ModuleNotFoundError 'g3dt' on the EC2 box — wrapped shell
  scripts now receive the CLI's own interpreter (G3DT_PYTHON); bare python3
  on AL2023 is 3.9 while g3dt installs under 3.11.
• metadata upload: empty submission results fail fast with the real
  per-chunk errors instead of retrying an impossible empty receipts write (#14).
• config show --full / config diff no longer try a laptop profile for
  _ec2 envs (latent ambient-credentials bug).
• config envs lists across all of a project's accounts instead of
  authorizing with an arbitrary first profile.
• synth install-simulator upgrades instead of install-once.