Add ExperimentalListPackageCustomSchemas gRPC endpoint#1981
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1981 +/- ##
==========================================
+ Coverage 57.90% 58.77% +0.87%
==========================================
Files 140 141 +1
Lines 13441 13405 -36
==========================================
+ Hits 7783 7879 +96
+ Misses 4470 4319 -151
- Partials 1188 1207 +19 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds support for streaming per-package “custom schema” FBC blobs over gRPC, backed by new cache storage primitives in both pogreb and JSON cache backends. This extends the opm serve gRPC surface to expose non-standard schema content without introducing new typed protobuf messages.
Changes:
- Adds
ListPackageCustomSchemas(schema, packageName)server-streaming gRPC RPC returninggoogle.protobuf.Struct. - Extends cache backends with meta blob storage/retrieval keyed by
(schema, packageName)and updates cache build to persist non-standard schemas. - Updates protoc/tooling wiring to include protobuf well-known types (
struct.proto) during codegen.
Reviewed changes
Copilot reviewed 13 out of 15 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/ensure-protoc.sh | Downloads protoc include files so well-known protos (e.g., struct.proto) can be imported during codegen. |
| README.md | Documents the new gRPC method in the public list of endpoints. |
| pkg/server/server.go | Implements the new gRPC streaming handler and converts stored JSON blobs into structpb.Struct. |
| pkg/server/server_test.go | Adds integration-style gRPC tests for streaming results, empty results, and invalid-argument behavior. |
| pkg/client/client_test.go | Updates the client stub interface to include the new RPC. |
| pkg/cache/pogrebv1.go | Adds meta blob Put/Send support in the pogreb backend. |
| pkg/cache/meta_key.go | Introduces meta key validation helpers for schema/packageName components. |
| pkg/cache/json.go | Adds meta blob directory layout and Put/Send support in the JSON backend. |
| pkg/cache/cache.go | Extends cache interface + build pipeline to store non-standard schema blobs and expose query method. |
| pkg/cache/cache_test.go | Adds unit tests covering custom schema storage/retrieval and packageless blob skipping. |
| pkg/api/registry.proto | Adds the new RPC and request message; imports google/protobuf/struct.proto. |
| pkg/api/registry.pb.go | Regenerated protobuf Go types (incl. new request type). |
| pkg/api/registry_grpc.pb.go | Regenerated gRPC service stubs (incl. new streaming method). |
| Makefile | Updates protoc include paths used by make codegen. |
| AGENTS.md | Documents the new gRPC method in internal agent docs. |
Files not reviewed (2)
- pkg/api/registry.pb.go: Language not supported
- pkg/api/registry_grpc.pb.go: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
perdasilva
force-pushed
the
generic-blob-endpoint-compressed
branch
2 times, most recently
from
9e6cfcf to
8f19197
Compare
perdasilva
force-pushed
the
generic-blob-endpoint-compressed
branch
from
8f19197 to
029eaa4
Compare
perdasilva
force-pushed
the
generic-blob-endpoint-compressed
branch
3 times, most recently
from
318e335 to
4226e11
Compare
perdasilva
force-pushed
the
generic-blob-endpoint-compressed
branch
from
4226e11 to
f45c501
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 16 changed files in this pull request and generated 2 comments.
Files not reviewed (2)
- pkg/api/registry.pb.go: Language not supported
- pkg/api/registry_grpc.pb.go: Language not supported
Comments suppressed due to low confidence (1)
pkg/cache/cache.go:323
- In Build’s WalkMetasFS callback, custom-schema metas with an empty packageName are still appended to byPackageReaders[""] and later written into the package index via
pkgs[pkgName] = pkgIndex[pkgName]. This will cause ListPackages to include an empty package name when packageless custom schema blobs exist. Consider short-circuiting after storing the meta (PutMeta) for non-standard schemas when packageName == "" so they don’t participate in package index construction (and don’t create a pkgs[""] entry).
switch meta.Schema {
case declcfg.SchemaPackage, declcfg.SchemaChannel, declcfg.SchemaBundle, declcfg.SchemaDeprecation:
default:
mk, err := newValidatedMetaKey(meta.Schema, packageName)
if err != nil {
return fmt.Errorf("invalid custom schema meta: %v", err)
}
if err := c.backend.PutMeta(ctx, mk, meta.Blob); err != nil {
return fmt.Errorf("store custom schema meta %v: %v", mk, err)
}
}
if _, err := tmpFile.Write(meta.Blob); err != nil {
return err
}
sr := io.NewSectionReader(tmpFile, offset, int64(len(meta.Blob)))
byPackageReaders[packageName] = append(byPackageReaders[packageName], sr)
offset += int64(len(meta.Blob))
f45c501 to
ba1392c
Compare
perdasilva
force-pushed
the
generic-blob-endpoint-compressed
branch
from
ba1392c to
fbe2335
Compare
perdasilva
force-pushed
the
generic-blob-endpoint-compressed
branch
3 times, most recently
from
26f2d3a to
3176657
Compare
|
@fgiudici: Overrode contexts on behalf of fgiudici: go-apidiff DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
perdasilva
changed the title
|
New changes are detected. LGTM label has been removed. |
perdasilva
force-pushed
the
generic-blob-endpoint-compressed
branch
2 times, most recently
from
2cc69cb to
07e8015
Compare
perdasilva
force-pushed
the
generic-blob-endpoint-compressed
branch
from
c5b9a80 to
ef719ce
Compare
perdasilva
force-pushed
the
generic-blob-endpoint-compressed
branch
from
ef719ce to
bb3283d
Compare
perdasilva
force-pushed
the
generic-blob-endpoint-compressed
branch
3 times, most recently
from
7bcc452 to
97a8931
Compare
Add a new ExperimentalRegistry gRPC service with a streaming ExperimentalListPackageCustomSchemas endpoint that returns custom schema blobs stored in FBC catalogs, filtered by schema name and optionally by package name. The endpoint returns an empty stream unless the caller passes the X-Acknowledge-Experimental gRPC metadata header with value "true". - Extend cache backends (pogreb, JSON) with meta blob storage keyed by (schema, packageName) - Add validated metaKey type to enforce schema/package constraints - Register ExperimentalRegistry service in all server entry points - Update README and AGENTS.md with usage documentation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
perdasilva
force-pushed
the
generic-blob-endpoint-compressed
branch
from
97a8931 to
9e45498
Compare
| } | ||
|
|
||
| api.RegisterRegistryServer(s, server.NewRegistryServer(store)) | ||
| api.RegisterExperimentalRegistryServer(s, server.NewExperimentalRegistryServer(store)) |
There was a problem hiding this comment.
Drop this? opm registry serve is deprecated/legacy and only serves sqlite DBs directly.
| s := grpc.NewServer() | ||
|
|
||
| api.RegisterRegistryServer(s, server.NewRegistryServer(store)) | ||
| api.RegisterExperimentalRegistryServer(s, server.NewExperimentalRegistryServer(store)) |
There was a problem hiding this comment.
Drop this? I think this is only used for catalog sources with type: ConfigMap
I don't remember how those work, but iirc they predate and don't directly support FBC catalogs.
| s := grpc.NewServer() | ||
|
|
||
| api.RegisterRegistryServer(s, server.NewRegistryServer(store)) | ||
| api.RegisterExperimentalRegistryServer(s, server.NewExperimentalRegistryServer(store)) |
There was a problem hiding this comment.
Drop this? registry-server (iirc) is a separate binary with just the sqlite serving bits extracted.
| // NOTE: pogreb does not support prefix/range scans, so this performs a full table scan. This scales | ||
| // linearly with the total number of entries (bundles + metas + index) and may be slow for large catalogs. |
There was a problem hiding this comment.
What if we stored a single blob for each package + schema? Then SendMetas could lookup a specific key and send everything under that key.
I suppose the tradeoff would be that if there are a large number of blobs for a single schema + package, we'd end up loading all of that into memory, as opposed to the current situation of a single blob in memory at a time.
| if err := json.Unmarshal(blob, &m); err != nil { | ||
| return status.Errorf(codes.Internal, "unmarshal custom schema blob: %v", err) | ||
| } |
There was a problem hiding this comment.
This seems like it could be expensive if we're able to use a non-JSON blob in the cache? iirc the pogreb cache stores the other blobs directly with proto serialization. Could we have the closure directly provide structpb.NewStruct(m), and let the implementation decide how to get the blob into that format?
| The server also exposes an experimental `ExperimentalRegistry` service: | ||
|
|
||
| ```sh | ||
| $ grpcurl -plaintext localhost:50051 list api.ExperimentalRegistry | ||
| ExperimentalListPackageCustomSchemas | ||
| ``` | ||
|
|
||
| Experimental endpoints require the `X-Acknowledge-Experimental: true` gRPC metadata header. Without it, the endpoint returns an empty response. | ||
|
|
||
| ```sh | ||
| grpcurl -plaintext -H 'X-Acknowledge-Experimental: true' \ | ||
| -d '{"schema":"custom.operator.io","packageName":"mypkg"}' \ | ||
| localhost:50051 api.ExperimentalRegistry/ExperimentalListPackageCustomSchemas | ||
| ``` |
There was a problem hiding this comment.
WDYT about leaving all of this undocumented?
There was a problem hiding this comment.
Same with AGENTS.md?
Someone developing a client may point their agent at this repo. To be fair, the agent can probably figure this out anyway if they see our repo, but maybe we don't make it easy?
| md, _ := metadata.FromIncomingContext(stream.Context()) | ||
| // Silently return an empty stream when the experimental header is absent | ||
| // so that unaware clients see no disruption. | ||
| if vals := md.Get("x-acknowledge-experimental"); len(vals) == 0 || vals[0] != "true" { |
Summary
Adds a new
ExperimentalListPackageCustomSchemasstreaming gRPC endpoint that returns custom schema blobs (arbitrary JSON documents) stored in FBC catalogs, filtered by schema name and optionally by package name.Key design decisions
ExperimentalRegistrygRPC service — the endpoint lives outside the stableRegistryAPI to signal its experimental status and allow independent evolutionX-Acknowledge-Experimental: truegRPC metadata; without it the endpoint silently returns an empty streammetaKey— schema and package name components are validated against^[a-zA-Z0-9][a-zA-Z0-9._-]*$to prevent path traversal and filesystem tricksPutMeta/SendMetasKnown limitations
SendMetasperforms a full table scan — pogreb does not support prefix/range scans, so queries iterate all entries in the database. This scales linearly with total catalog size and may be slow for very large catalogs.Test plan
metaKeyvalidation (valid keys, empty, path traversal, dot sequences, leading special chars)SendMetasimplementationsgo test ./pkg/cache/... ./pkg/server/...passes🤖 Generated with Claude Code