Build A MVP Persistent Caching Proxy

[sttts]

Certain data in a kcp cluster is critical for operation and SPOF in a multi-region/AZ environment, e.g. org workspaces. It is feasible to build a consistent cache hierarchy which

1. persists objects to a local etcd or
2. keep data in memory if one makes the proxies highly available through multiple instances
   and which serve consistent data by checking freshness of the cache on quorum reads. Using such a setup could give read-only availability to e.g. org workspaces.

Big question: do we run into time-travel problems as we do with pods and kubelets. So we must be super careful when answering consistent reads with potentially stale data in an outage situation. But for certain operation like the personal workspace virtual workspace from @davidfestal, a stale read is good enough.

Acceptance Criteria

We're just looking for an MVP implementation here:

• shards push data to the proxy
• no auth{n,z} needed
• APIExport + APIResourceSchema hard-coded as types to push
• dynamically determine which types to serve. For example, the built-in types (APIExports and APIResourceSchema) could be automatically served as CRDs.
• turn on the watch cache in the cache server
• have secondary informers in kcp reconcilers use the cache server

Items

• @p0lyn0mial create initial cmd/cache-server and pkg/cache/server/{config.go,server.go} and pkg/cache/server/options plumbing (cache-server scaffolding #1790)
  • follow options/config/server pattern tightly
  • options should be compatible and easily composable with those in pkg/server/options
• @p0lyn0mial instantiate apiextensions-apiserver and wire CRD lister similar to today's pkg/server/apiextensions.go with at least APIExports and APIResourceSchemas
  • use empty schemas, as dumb as possible: APIExport/APIResourceSchema CRDs can be static as in

    kcp/pkg/virtual/apiexport/schemas/apis.go

    Line 36 in 8d2ed6a

    if err := configcrds.Unmarshal(fmt.Sprintf("apis.kcp.dev_%s.yaml", resource), &crd); err != nil {

  • no admission, no validation, no conversion, no pruning, no SSA (if possible), for now no authorization
  • PRs: cache-server: starts empty apiextentions-server #1811, cache-server: expose apiresourceschemas and apiexports #1815, UPSTREAM: <carry>: collapse with extensions-apiserver udpates kubernetes#93
• @p0lyn0mial wire custom etcd prefix for "cached CRDs" returned by the CRD lister: /registry/group/cache:resource:identity:shard/… (cache-server: adds WithShardScope HTTP filter #1841)
• @p0lyn0mial assign shard names to objects (UPSTREAM: <carry>: assign shard names to objects kubernetes#100, cache-server: adds ShardNameFromObjectRoundTripper #1926, 🌱 cache: makes all internal clients shard and cluster aware #1928)
• @p0lyn0mial serve /services/cache/shards/*/clusters/* (🌱 cache: expose the cache-server under "/services/cache" path #1961)
• @p0lyn0mial serve /services/cache/shards/{shard-name}/clusters/{cluster-name} (🌱 cache: expose the cache-server under "/services/cache" path #1961)
• @p0lyn0mial instantiate cache-server inside of kcp binary. (🌱 cache: small refactor to make wiring into kcp easier #1949, 🌱 cache: moves common HTTP handlers to a shared pkg #1947, 🌱 Wire the cache server into the kcp server #1954)
• @p0lyn0mial wire --cache-url into kcp, defaulting to localhost. (🌱 kcp: add flags related to the cache server #1970)
• @p0lyn0mial create shard controller pushing (labeled to be published) apiexports, apiresourceschemas to cache (🌱 cache: introduce the replication controller #2024, 🌱 cache server: stop serving subresources for built-in resources #2063, UPSTREAM: <carry>: no graceful deletion of the cached objects kubernetes#102)
• create cache controller creating "cached CRDs" or create "cached CRD" on the fly in the CRD lister for all cached APIExports
• serve internal APIs through schema-less CRDs, wired statically
• wire secondary informers to cache instead of the other shard
• enable the watch-cache
• failing the optional hook should not render the main server unhealthy (🌱 cache: run the replication controller when the cache server is enabled #2132 (comment))

authorization

• shards should only be able to write their shard partition of the cache

data removal

• unused resources could be automatically removed

shard management, a few loose ideas:

what will decide when a new shard needs to be created?
which cluster it gets deployed to?
making other shards aware of the new shard?

on kcp server

• wire informer both ways this includes a controller to cope with not-yet-synced informers
• start the second informer (aka. TemporaryRootShardKcpSharedInformerFactory) in a dedicated post-start-hook

[stevekuznetsov]

2 seems quite a lot harder than 1 FWIW

[sttts]

The freshness problem we have either way. So I don't think one or the other is harder. The first is reboot safe, the second is operationally simpler.

[ncdc]

Clearing milestone to re-triage

[sttts]

Part of multi-release epic #1225

[stevekuznetsov]

@p0lyn0mial FYI for tracking, this is the issue for implementing the cache server, would be cool to link PRs to it as new ones come in

[ncdc]

@p0lyn0mial @sttts is this going to be completed this week for v0.10? It looks like we still have several outstanding items?

[p0lyn0mial]

@p0lyn0mial @sttts is this going to be completed this week for v0.10? It looks like we still have several outstanding items?

we need to finish the workspace refactoring before we can finish this feature.

[p0lyn0mial]

I think we can close this issue. Not all items have been implemented but it is unclear to me if we will use the cache server in the long-run. Perhaps it will be replaced by CRDB.

[ncdc]

👍
/close

[openshift-ci]

@ncdc: Closing this issue.

In response to this:

👍
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.