RFC: Split the OASF server and schema into separate repos

[akijakya]

Tracking epic: #537

Problem

Every ghcr.io/agntcy/oasf-server:vX.Y.Z image has one schema version baked into it (COPY --from=schema in server/Dockerfile). To serve several OASF versions you run several images, one per schema release line (v0.7.x, v0.8.x, v1.0.x, v1.1.x, …).

So a server CVE fix or bug fix has to be cherry-picked to every release branch, patch-released once per branch, and rolled out once per version in every deployment. The server is stable now, and this coupling costs more than it saves: security fixes can't ship quickly to all OASF versions.

We'd like to go back to the model OCSF uses:

• separate schema and server repos,
• one server image that can serve any OASF version,
• one server instance per schema version, with routing done by a reverse proxy / ingress.

After the split, a server fix is one server release plus one image tag bump, and it reaches every schema version at once.

How OCSF does it

As described in ocsf-server#131, the public OCSF site runs one server container per schema version. Nginx routes requests by URL prefix (e.g. /1.4.0/ → the 1.4.0 instance). Each container gets SCHEMA_DIR pointing at its own schema checkout, and SCHEMA_PATH set to its URL prefix. Our Helm chart already follows this pattern: one Deployment and Service per schema version, a SCHEMA_VERSIONS ConfigMap for the version switcher, and path-based ingress routing. The only real difference is where the schema comes from: a separate checkout there, baked into the server image here.

Feasibility check

• The server on main already serves every released schema. Run with SCHEMA_DIR pointing at the schema/ tree of v0.7.8, v0.8.7, v1.0.4 and v1.1.1, it starts cleanly for all four, with no errors or warnings. The UI pages and the main /api, /schema and /sample endpoints return 200 for each. A full output-parity comparison against the current bundled images is still to be done.
• The schema/ directory layout has been stable since 0.7.
• Schema content did change within patch lines (e.g. v1.0.0..v1.0.4), so schema artifacts must be keyed by the git tag, not only by version.json.

Target architecture

agntcy/oasf  (schema repo — keeps name, history, tags, issues, buf module)
  schema/, proto/, docs
  publishes: ghcr.io/agntcy/oasf-schema:<git tag>   (data-only OCI image)
             buf.build/agntcy/oasf                   (unchanged)

agntcy/oasf-server  (new — server history carried over)
  mix project at repo root, Dockerfile, Helm chart
  publishes: ghcr.io/agntcy/oasf-server:<server semver>   (no schema inside)
             oci://ghcr.io/agntcy/oasf-server/helm-charts/oasf

Kubernetes (per schema version, same server image):
  Deployment oasf-1-1-0 ─┬─ container: oasf-server:vS         SCHEMA_DIR=/schema
                         └─ volume: oasf-schema:v1.1.1  →  /schema
  Deployment oasf-1-0-0 ─── same server image, oasf-schema:v1.0.4
  ...                        ingress routing /<ver>/…, /api/<ver>/… unchanged

Proposed decisions

#	Decision	Proposal	Why
D1	Which repo keeps agntcy/oasf	The schema stays in agntcy/oasf; the server moves to a new agntcy/oasf-server	Schema history, tags, release branches, GitHub releases, issues, buf.build/agntcy/oasf and every github.com/agntcy/oasf/tree/main/schema/... link stay untouched.
D2	Server history	git filter-repo on a fresh clone of agntcy/oasf that keeps server/, install/ and the build/CI files, and moves server/ to the repo root	All server commits keep their authors and dates. The original repo is not rewritten.
D3	Old bundle tags in the server repo	Rename to bundle/vX.Y.Z and drop the release branches	Keeps a record of which server commit shipped in which bundle, and frees vX.Y.Z for the new independent server semver.
D4	Server versioning	Independent semver starting at v2.0.0	v1.2.0 would read as "the image for schema 1.2". A major version marks the packaging break: the image no longer contains a schema.
D5	Image names	Keep ghcr.io/agntcy/oasf-server and add ghcr.io/agntcy/oasf-schema	Existing consumers keep the same repository. Old bundle tags stay pullable.
D6	How the schema gets into the pod	A data-only OCI image per schema git tag (ghcr.io/agntcy/oasf-schema:vX.Y.Z), pinned by digest. The chart chooses the mount with schema.mount. The default is initContainer, which copies /schema into an emptyDir. imageVolume is opt-in and mounts the image with subPath: schema. Image volumes need Kubernetes ≥ 1.35, so the chart refuses to render that mode on older versions unless an override is set. Once the chart's minimum is 1.35, a later chart major makes imageVolume the default. Decided in #539.	No dependency on GitHub at pod start, content fixed by digest, signed with cosign, and dependency bots and GitOps tooling can track it like any other image. Image volumes are off by default before 1.35, and with the gate off the API server silently drops the volume, so the init container is the only mode that is safe everywhere today.
D7	Schema image base	One image for both mount modes: a static busybox base, with the schema under /schema, running as a non-root user. Decided in #539.	The init container needs cp, and with subPath the same image also works as an image volume, where the busybox layer is only mounted, never run. That makes one artifact per tag to build, sign, scan and pin.
D8	Chart location	Server repo, published to oci://ghcr.io/agntcy/oasf-server/helm-charts/oasf as chart v2.0.0 (breaking values change: one server image tag plus a list of schema versions)	Matches oasf-sdk and dir, and releases alongside the server.
D9	Schema release branches (v0.7.x … v1.1.x)	Keep them, but stop server work there. Schema-content fixes still land on them and are tagged vX.Y.(Z+1), which publishes a new schema image.	Server backports stop. Schema data fixes still have a path.

Schema delivery: alternatives considered

Option	How	Verdict
gitRepo volume	Built-in Kubernetes volume type	Rejected. Deprecated and turned off in recent Kubernetes versions.
git-sync init container	git-sync --one-time --depth=1 --ref=<tag or SHA> into an emptyDir	Rejected. Every pod start depends on GitHub, so an outage or rate limit means pods can't restart or scale. A tag can be moved unless you pin the SHA, and there's no signature or provenance check.
ConfigMap rendered from the repo	GitOps tool renders the schema files into a ConfigMap	Rejected. The schema is about 3.3 MB and a ConfigMap is limited to 1 MiB.
Schema OCI image from the tag	CI on vX.Y.Z packages schema/ into oasf-schema:vX.Y.Z	Proposed. Content is fixed and signed, it's pulled from GHCR like the server image, and it can be tracked like any other image.

Feedback welcome, especially on D1 (repo naming), D4 (server versioning), and which schema versions should stay supported once a single server serves them all.

[akijakya]

Decision: schema mount method and image base (D6/D7)

Recommendation: publish one schema image per tag, built on static busybox with the schema under /schema. The chart supports both mount methods through a value:

• schema.mount: initContainer (the default): copies /schema into an emptyDir. Works on every cluster.
• schema.mount: imageVolume: mounts the same image with subPath: schema. Requires a cluster with image volumes enabled.

We switch the default to imageVolume in a later chart major, once the chart's minimum Kubernetes version reaches 1.35.

This changes D7. The plan was two bases, scratch for image volumes and busybox for the init container. One busybox image serves both modes, so there is one artifact per tag to build, sign, scan and pin. In image-volume mode the busybox layer is only mounted, never run.

Support for image volumes

Kubernetes	ImageVolume gate	Notes
1.31–1.32	alpha, off
1.33–1.34	beta, off	subPath supported since 1.33
1.35	beta, on
1.36+	GA, on

The feature also needs container runtime support: containerd ≥ 2.1, or CRI-O with image volume support (≥ 1.31 for alpha). Many clusters in use today are below 1.35 or don't have the gate on:

• EKS 1.34 is in standard support until 2026-12-02.
• OpenShift 4.20 is based on Kubernetes 1.33.
• GKE 1.33 nodes ship containerd 2.0.

Making image volumes the only option would cut those users off.

What the kind prototype showed

I ran the current main server, built without a schema, against the v1.1.3 schema (3.3 MB, schema/test removed). I tested three variants on kind v0.33.0 (containerd 2.3.4). Pods ran as non-root with readOnlyRootFilesystem and all capabilities dropped.

Variant	Image size	1.35.8 (defaults)	1.34.11 (defaults)	1.34.11 (gate on)
A. Image volume, FROM scratch	4.2 MB	✅	❌	✅
B. Image volume, busybox image, subPath: schema	6.8 MB	✅	❌	✅
C. Init container (busybox → cp -R → emptyDir)	6.8 MB	✅	✅	✅

• Every passing variant serves the full schema, and /api/version reports schema 1.1.0. Image volumes are mounted read-only. The init-container variant also mounts read-only through volumeMounts[].readOnly.
• Silent failure below 1.35: when the gate is off, the API server accepts the pod but drops the image volume and its volumeMount without an error. The server then starts on an empty /app/schema and crash-loops (file /app/schema/dictionary.json does not exist). The field still shows in kubectl explain, so the chart can't detect support through API discovery. Instead, the chart should fail when schema.mount: imageVolume is set and .Capabilities.KubeVersion is below 1.35. A schema.imageVolume.allowBelow135-style override would cover clusters that turned the gate on early.
• In both modes the schema changes only when the pod spec changes. Bumping the schema tag rolls out new pods like any other image change. With IfNotPresent, reusing a tag does not pull the image again, so the chart should pin the schema image by digest, the way it pins the server image.

Side finding, not related to the mount method

The containerd in kind gives pods an open-file limit of about 1 billion, compared with about 1 million in Docker. BEAM sizes its port table from that limit, so each server used about 2.2 GB of RSS and three pods exhausted an 8 GB node. Setting ERL_MAX_PORTS=65536 brings it back to about 150 MB. Our test:e2e runs on kind, and any runtime with a very high nofile triggers the same thing. This is tracked in #554.

Sources:

• Feature gates, ImageVolume rows
• Use an image volume with a pod
• Image volume beta blog (1.33)
• ACK 1.35 release notes, containerd ≥ 2.1
• EKS version lifecycle
• GKE containerd 2 migration
• OpenShift 4.20 release notes

[akijakya]

D9 update: the schema release branches (v0.3.x … v1.1.x) are no longer used. They only existed to backport server fixes, and the server now lives in agntcy/oasf-server. Every schema release is a vX.Y.0 tag on main, which publishes ghcr.io/agntcy/oasf-schema:X.Y.0. Any schema change, including additions, bumps the minor version, and fixes to a published schema ship in the next version. The existing branches and patch tags stay for reference but get no new commits or tags. The rewritten RELEASE.md is in #556.

[akijakya]

The split is done: the server and its Helm chart now live in agntcy/oasf-server (v2.0.0), schemas are published as ghcr.io/agntcy/oasf-schema images, and schema.oasf.outshift.com runs the new setup. The announcement covers what changed for users, how to upgrade, and the deprecation of 0.7.0.