v1.1.0

The Docker / OCI registry, format plugins (Helm built in, NuGet ported to
one), the trusted-publishing token exchange, short-lived S3 credentials and a
local blob cache, and the fixes for the findings of the 1.0.0 release review.
The 1.0.0 known limitations still apply, except where an entry below says
otherwise.

Upgrade notes

• Startup refuses a server.staging_dir that does not belong to Forge's
  user, or that group or others can write unless it is sticky like /tmp.
  The container image and the Kubernetes manifests are not affected.
  (#127)
• Durations above 100 years (outbox.grace_seconds,
  retention.default_run_interval_seconds, retention.lease_seconds,
  crypto.cache.unwrapped_key_ttl_secs,
  crypto.rotation.dek_max_age_days, crypto.rotation.kek_max_age_days,
  and the new docker.upload_ttl_seconds,
  docker.unreferenced_blob_ttl_seconds and docker.sweep_interval_seconds)
  fail validation instead of panicking later.
  (#128)
• Migration 0032 deletes grants of the admin action on a single
  repository, which never had an effect, and the grants API refuses them
  with 400. Administration stays global.
  (#124)
• While a credential-forwarding realm is enabled, an account that an OIDC
  realm provisions is named <claim>@<realm>. Existing accounts keep their
  names. (#123)
• Proxy search now calls the upstream's search service, which may live on
  another host than the feed (on nuget.org, azuresearch-*.nuget.org).
  Where outbound traffic is restricted, allow that host, or proxy search
  keeps answering from the cache only.
  (#112)
• The production kustomize overlay adds a NetworkPolicy that admits only
  the ingress controller's namespace (set it in the overlay), and the Helm
  chart renders one as soon as networkPolicy.from lists a peer.
  (#129)
• Helm: the first upgrade rolls the Forge pods (when secretEnv is set)
  and restarts the in-cluster Redis once, because the checksum annotations
  are now salted. (#133)
• The published cosign verify commands also pin the tagged commit and the
  push trigger; use the commands of the release you verify.
  (#130)
• Migration 0033 adds the Docker / OCI registry's tables; it changes no
  existing data. To push images through a reverse proxy, allow request
  bodies of a docker repository's upload limit (1 GiB by default: docker
  sends a whole layer in one request) on /v2, and serve /v2 at the root
  of the host. The shipped Ingress now allows 1 GiB. The default
  server.max_staging_bytes (four times the largest item) grows with it to
  4 GiB; size the staging volume for it.
  (#40)
• An explicit server.max_staging_bytes keeps working as it is. When it is
  below the new 1 GiB docker defaults, they shrink to fit it (less
  nuget.max_nuspec_bytes): docker repositories without an upload limit of
  their own accept layers up to that size, and so do docker proxy
  downloads unless docker.max_blob_bytes is set; startup logs a warning.
  Raise the budget for larger layers. A budget too small for the package
  limits is refused as before, with an error that names the setting that
  decides the minimum. (#198)
• The container image's baked forge.toml no longer holds the placeholder
  minioadmin S3 keys. An image started without FORGE_S3__ACCESS_KEY /
  FORGE_S3__SECRET_KEY (and without short-lived credentials) still
  refuses to start, now with an error that names the missing keys.
  (#60)
• server.max_upload_bytes is deprecated: upload limits are now set per
  hosted repository, in the web UI or the admin API. The setting still
  works as the default of package repositories without a limit of their own,
  and startup logs a warning while it is set; give the repositories that
  need it their own limit, then remove it. A repository's limit may be at
  most server.max_staging_bytes (less nuget.max_nuspec_bytes); raise the
  staging budget first if a repository needs more.
• A workload identity realm's max_token_age_seconds defaults to 39600
  (11 hours) instead of 3600, so GitLab jobs with a timeout over an hour
  can publish. Existing realms keep the value they were saved with; to
  use the new default, clear the field in the realm form (or drop the key
  from the config) and save.
• The username oidc is reserved for workload identity (trusted
  publishing), and 1.0.0 allowed it. An account named oidc can no longer
  sign in with its password, in the browser or over Basic, and its API
  tokens work only as X-NuGet-ApiKey, no longer as the Basic password;
  startup logs a warning that names it. A config with
  auth.bootstrap_admin.username = "oidc" refuses to start. Rename the
  account before upgrading; there is no API for it, so use SQL:
  UPDATE app_user SET username = 'oidc-admin' WHERE username = 'oidc';
  (the account keeps its id, password, roles and tokens). Then set
  auth.bootstrap_admin.username to the new name if it was oidc.
  (#199)
• Migration 0034 adds the workload_api_key table for the trusted-publishing
  token exchange; it changes no existing data. /api/v2/token is a new
  route, so a reverse proxy that only forwards known prefixes needs /api.
  (#169)
• cache.l1_max_capacity counts an entry over 16 KiB once per started
  16 KiB, so it bounds the in-process cache's memory as well as its entry
  count (about 160 MiB of large entries at the default 10000). Small
  entries count once, as before.

Features

• Format plugins (preview): further package formats, served under
  /repository/{repo}/ (or a mount the plugin names) with the same
  storage, RBAC and package selectors, hosted, proxy and group
  repositories, retention, ranged downloads and audit log as NuGet. A
  plugin is a crate implementing one contract (forge-plugin-api's
  FormatPlugin, mirrored by the forge:format@0.3.0 WIT package; a
  plugin's upstream side tells the host what to fetch for its proxies) and
  runs either built into the server or as a WebAssembly component run by
  Wasmtime. Components are loaded from [plugins] only when pinned by
  SHA-256, and each call runs with fuel, memory and time limits and no WASI
  capabilities. Helm chart repositories (plugins/helm) are built in:
  index.yaml, chart downloads and ChartMuseum-compatible push, delete and
  listing, and proxies of other chart repositories. The index lists chart
  metadata with the types Helm reads, so a Chart.yaml with appVersion: 2
  or an annotation false does not break it
  (#206). A Helm proxy
  parses its upstream's index once while the upstream serves the same
  document, one parse at a time, instead of once per operation
  (#207). Parsed indexes
  are kept up to 128 MiB per instance, shared by all Helm proxies: about
  60 MB of index.yaml, enough for every upstream of a typical group. Past
  that, the indexes in use stay kept and only the others are parsed again
  for each operation
  (#279,
  #284). NuGet is a built-in plugin too (plugins/nuget), served at
  /v3/{repo}/ by the same dispatcher with the same paths, responses and
  proxy behaviour as before; its proxies read the upstream feed through the
  plugin's upstream side. GET /admin/formats lists the installed
  formats; the web UI offers them when creating a repository. See
  docs/format-plugins.md.

• Docker / OCI registry under /v2/ (OCI Distribution Spec v1.1,
  passing its conformance suite), for container images, Helm charts and
  other OCI artifacts, as a native format next to NuGet ("format": "docker", listed by GET /admin/formats) with all three repository
  kinds:

  • hosted: monolithic, chunked and cross-repository-mount pushes,
    deletes, the referrers API for cosign signatures and attestations, and
    optional immutable tags (with a latest exception);
  • proxy: a pull-through cache of Docker Hub (with its library/
    names), GHCR, Quay or any distribution-spec registry, with upstream
    credentials, tag re-validation by HEAD after metadata_ttl_seconds,
    and stale serving while the upstream is down or rate limits;
  • group: members by priority, pushes to a writable_member, and the
    prefer_hosted / reserved_prefixes / owner guards of NuGet groups.

  docker login goes through a Bearer token service (API tokens, passwords
  where allowed, or a workload OIDC token), anonymous pulls follow the
  repository's anonymous read, and repositories are addressed by path or by
  an optional host name of their own. Image retention rules (keep the last N
  tags, stale tags, untagged manifests; a HEAD of a manifest counts as a
  pull, #197) and a sweeper
  release layers only when no image references them. The web UI lists images, tags, manifests,
  platforms and referrers. New [docker] settings; see
  docs/docker.md.
  (#40)

• Per-repository upload limits: every hosted repository (NuGet, Helm,
  docker or a WebAssembly plugin format) has its own upload limit, the
  largest push it accepts (docker: per blob), editable as Maximum upload
  size in the web UI and as max_upload_bytes in the admin API. Without
  one, a repository accepts 512 MiB (docker 1 GiB per blob).
  docker.max_blob_bytes now bounds proxy downloads only.

• Browse by format: GET /browse/repos takes format and names each
  repository's client URL; the web UI's Browse screen filters by format,
  and NuGet and Helm repositories show their dotnet nuget and helm
  commands (other plugin formats, their URL) like docker ones show theirs.
  (#38)

• Short-lived S3 credentials from the platform's workload identity, so
  no long-lived S3 key has to be stored or rotated. s3.credentials
  (FORGE_S3__CREDENTIALS) is static (the default, s3.access_key /
  s3.secret_key as before), default_chain (the AWS SDK chain: IRSA, EKS
  Pod Identity, ECS task roles, EC2 instance profiles, AWS_* variables),
  or web_identity: a token file exchanged with STS
  AssumeRoleWithWebIdentity against AWS or MinIO / Ceph RGW
  ([s3.web_identity]: token_file, role_arn, sts_endpoint,
  session_name, duration_seconds). Credentials are refreshed before they
  expire and the token file is read again on each refresh. A failed fetch
  names the credential source and the STS error code, never the token. The
  short-lived modes refuse S3 keys, and startup warns about a plain-http
  sts_endpoint (the token and the temporary keys would cross the network
  in clear text). The Helm chart has s3.credentials and
  s3.webIdentity (with an optional projected service-account token), and
  the kustomize component s3-workload-identity mounts one. See
  docs/deployment.md.
  (#60)

• Trusted-publishing token exchange: POST /api/v2/token (nuget.org's
  path and request/response shape, so NuGet/login works with
  token-service-url) exchanges a CI job's OIDC token, validated by a
  workload_identity realm, for a short-lived workload API key
  (forge_wif.…). Each token is redeemed once (its jti is required), the
  key lives for the realm's new api_key_ttl_seconds (default an hour)
  whatever the token's lifetime, it authorizes and is audited as the
  token's workload, and any edit, disabling or deletion of the realm
  revokes its keys for good.
  dotnet nuget push --source <feed>/v3/<repo>/package --no-service-endpoint --api-key <key> publishes with the key alone; restore takes it as the
  Basic password of the username oidc.
  (#169)

• Reserved account names: oidc is the workload-identity Basic
  username. Basic oidc:<secret> is only ever checked as a workload
  credential (an OIDC token or a workload key), never as a user token or a
  password and never forwarded to a credential-forwarding upstream, and no
  account can be created, bootstrapped or provisioned under the name (an
  OIDC login whose username claim is oidc gets oidc-2).
  (#169)

• Proxy cache invalidation: POST /admin/repositories/{name}/proxy-cache/invalidate, and Invalidate
  cache on a proxy's page in the web UI, make a NuGet, Helm or docker proxy
  ask its upstream again before its TTLs run out, for example once a package
  it answered "not found" for is published upstream. The proxy forgets the
  upstream's "not found" answers (of one package with {"package": "<id>"},
  or of all) and the listings and documents it cached from the upstream, on
  every instance; invalidating the whole repository also refreshes the
  upstream's service index, and a docker proxy re-validates its tags (of one
  image, or of all) on their next pull. Downloaded packages, blobs and
  manifests stay cached, and so does the package-delay clock.

• Local blob cache (optional, [blob_cache]): a copy of downloaded and
  pushed blobs on a local disk, sized from the disk (max_bytes = "auto")
  or fixed, with FIFO eviction that survives a restart. Hot downloads skip
  S3, concurrent misses of one blob share a single S3 read, and cached
  packages keep resolving while S3 is unreachable: uncached downloads and
  pushes then answer 503 with Retry-After at once (a circuit breaker),
  /readyz stays 200 and reports the object store degraded, and an
  instance starts without S3. With the cache, package downloads also answer
  Range requests. Statistics at GET /admin/blob-cache; Kubernetes
  components local-cache (emptyDir) and local-cache-persistent (a
  PersistentVolume). The cache directory must belong to Forge's user and be
  writable by no one else (startup refuses it otherwise; a directory a
  Kubernetes fsGroup pass made group-writable is restored instead), and
  its entries are readable by that user alone. The kustomize base and the
  Helm chart set fsGroupChangePolicy: OnRootMismatch
  (#195). See
  docs/deployment.md.

Fixed

• Proxy and group search find packages nobody has restored through Forge
  yet: a proxy forwards the search to its upstream's search service,
  applies the caller's package filter, not-found markers and the package
  delay, and answers from its cache when the upstream search fails. The
  browse UI keeps listing what Forge holds.
  (#112)
• A proxy pointed back to its previous upstream within the metadata TTL
  served empty version listings.
  (#105)
• A proxied package id's casing could flip between reads; it is the
  newest version's, as on nuget.org.
  (#108)
• The package delay held back old unlisted upstream versions, such as ones
  a lockfile pins, for the whole delay. They now count from when the
  upstream created them, and old versions of large packages are dated at
  their first sighting too.
  (#111)
• A push with a version longer than 64 characters answered 500; it is
  refused with 400. (#134)
• An offline nested group was still served through its outer group; it now
  fails like an offline hosted member.
  (#106)
• When a group was reachable along two paths, a guard on one path hid
  packages the other path admits.
  (#107)
• Downloads are counted in memory and written in batches every few
  seconds. A burst of downloads of one version no longer queues on its row
  and holds every database connection.
  (#119)
• Creating a missing S3 bucket failed on AWS outside us-east-1, and a
  bucket name owned by another account counted as created.
  (#113)
• Unknown paths under /admin, /auth, /browse and /v3 answered 200
  with the web UI's page; they answer 404.
  (#121)
• PUT /admin/realms/order answered 500 when it raced another realm
  change; it answers 409. (#117)
• The browse API served offline hosted and group repositories; it answers
  503 like the feed, and the web UI says the repository is offline.
  (#120)
• A failed OIDC sign-in ended on a plain-text page; it returns to the
  login page with a message and a way to try again.
  (#109)
• Web UI: a proxy's version times are labelled "Cached", not "Published"
  (#110); disabling a
  sign-in realm asks first
  (#104); unflagging a
  group's last owner and removing a member ask first
  (#122); the user list
  shows how each user signs in
  (#123).

Security

• Behind an outbound HTTP(S)_PROXY, a proxy refuses an upstream host
  name that resolves to a link-local or cloud-metadata address, and checks
  every redirect hop the same way.
  (#125)
• Forged Redis invalidations can no longer grow every instance's memory:
  the invalidation bookkeeping is bounded.
  (#126)
• The staging directory is private to Forge's user, and a staged package
  is parsed and uploaded from the file that was hashed, not reopened by
  name. (#127)
• An OIDC-provisioned account can no longer take a credential-forwarding
  user's username. (#123)
• Only the ingress proxies can reach Forge when it trusts
  X-Forwarded-For from a pod network (NetworkPolicy, see above).
  (#129)
• Release: every cosign verify pins the tagged commit
  (#130); re-running a
  release reuses what it already pushed instead of moving the tags to a
  new build (#131); the
  build helpers and the dependency images are pinned by digest
  (#132); the Helm
  checksum annotations are salted
  (#133).
• Docker registry tokens are HS256 JWTs bound to public_base_url, live
  for docker.token_ttl_seconds (5 minutes), grant only the scopes the
  caller held when they were issued, and are accepted under /v2 only. A
  token request names at most 32 repositories
  (#200).
  Every registry request is authorized again against current permissions
  and content selectors, the user and API token behind a token must still
  be active, and a browser session never pushes or deletes. The token
  endpoint takes the same Basic credentials as the feeds, so it adds no new
  password path. The signing key is docker.token_secret, or a generated
  key kept in the database, sealed when [crypto] is enabled and re-sealed
  by DEK rotation; an instance replaces it only when it can never be opened,
  and the others pick the new key up within 30 seconds
  (#196).
• Registry digests are verified on every upload and every proxied blob and
  manifest. Blobs are linked per image: knowing a layer digest grants no
  read on another image's copy, and a mount needs read on the source image.
  A docker proxy sends upstream credentials only to its remote_url origin
  and the token realm the upstream's challenge names (over https when the
  upstream is https), and drops Authorization on a redirect to another
  host. Registry responses carry the feeds' hardening headers and
  Vary: Authorization, Cookie, Accept.

Documentation

• The deployment guide lists the S3 permissions and the PostgreSQL role
  Forge needs, and an S3 delete that is denied names the missing
  s3:DeleteObject. (#114)

Known limitations

• With a package delay, proxy search shows a version only once the proxy
  has recorded it and the delay has passed, because a search answer names
  no publish times. Totals of a proxy search are the upstream's, less the
  ids dropped from the returned page; for a caller whose content selectors
  hide packages, they count what it sees, plus one while more may follow.
• Behind an outbound proxy, a host name that resolves differently for the
  proxy than for Forge, or that only the proxy can resolve, still gets
  past the metadata guard; block those addresses on the proxy as well.
• Format plugins receive no request bodies other than the published
  package (which must be the whole body or one multipart part), accept only
  Basic credentials, and treat a version with a - as a prerelease; a
  response of a plugin may load at most 2,048 query answers. Helm refuses a
  Chart.yaml or index.yaml whose YAML aliases would add more than
  100,000 nodes, or that nests deeper than 64 levels. Their proxies
  read the upstream with plain GETs of whole documents (Basic credentials
  only, no token exchange).
• A docker proxy downloads a blob completely and verifies it before it
  answers, so a client waits for a large layer's first fetch; concurrent
  pulls of one layer share a download per instance only. A registry upload
  of several chunks is assembled into a new blob when it finishes (one-chunk
  uploads, what docker, podman and containerd send, are stored once). The
  registry has no catalog API (/v2/_catalog) and no per-repository
  connector ports.

Image

ghcr.io/schmitch/forge on GitHub Packages.

Built from commit cc1fbfd, which v1.1.0 points to. The cosign verify commands below accept only a signature by this repository's release workflow for that tag and that commit.

docker pull ghcr.io/schmitch/forge:1.1.0
# digest: sha256:8d3e49aee2c3af589efe2da8670a1f08569baf06e3ec58bec985c3cb245386c7
cosign verify ghcr.io/schmitch/forge@sha256:8d3e49aee2c3af589efe2da8670a1f08569baf06e3ec58bec985c3cb245386c7 \
  --certificate-identity https://github.com/schmitch/forge/.github/workflows/release.yml@refs/tags/v1.1.0 \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-github-workflow-sha cc1fbfd88e0670b22d7568e901e65ab01b3f0397 \
  --certificate-github-workflow-trigger push

Helm chart

oci://ghcr.io/schmitch/charts/forge on GitHub Packages.
Deploys exactly the image above (by digest).

helm install forge oci://ghcr.io/schmitch/charts/forge --version 1.1.0
# digest: sha256:9d0b4afccd3ff6948e0dfa5cf930a50cac43b502eeddc5e427bf7f117cbf4336
cosign verify ghcr.io/schmitch/charts/forge@sha256:9d0b4afccd3ff6948e0dfa5cf930a50cac43b502eeddc5e427bf7f117cbf4336 \
  --certificate-identity https://github.com/schmitch/forge/.github/workflows/release.yml@refs/tags/v1.1.0 \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-github-workflow-sha cc1fbfd88e0670b22d7568e901e65ab01b3f0397 \
  --certificate-github-workflow-trigger push