feat: Initial work refactoring noosphere-gateway to be generic over the spheres it manages.

[jsantell]

• trait ContextManager is provided to the gateway to access managed spheres' contexts.
• SingleSphereContextManager implementation for existing orb gateway and tests.
• All gateway routes now require X-SPHERE-IDENTITY header to specify the target sphere. Corresponding UCAN auth could apply to many spheres.
• In addition to GatewayAuthority synthesized for authenticated routes, there's now GatewaySphereContext which provides access to the targeted gateway sphere.
• Use TypedHeaders for parsing X-SPHERE-IDENTITY and UCAN headers into a proof chain. Note using the headers module, it should be the same version as axum in servers.
• Worker tasks naively loop through all managed sphere contexts before waiting on some timer. Not scalable, and could result in long worker cycles, but good enough for now.

cargo test --features test-kubo,helpers passing locally

[jsantell]

Updates:

• Requests are now scoped by hostname e.g. did:key:abc.subconscious.cloud.
  • Some tests updated to properly set counterpart in gateways
  • SingleSphereContextManager always has available its one sphere context, and does not use whatever's in the hostname. This enables existing cluster gateways, and possibly misconfigured self-served gateways to continue functioning. The real counterpart identity is fetched from sphere storage to reflect truth.
  • TBD on what formatting we want for the DID subdomain e.g. did_key_abc or elide the DID type altogether abc.subconscious.cloud.
• GatewayAuthority, in addition to handling auth, now contains the scoped sphere context. For unauthenticated requests (Did route), a GatewaySphereContext is used to access the sphere context. Rationale here is to not double up on processing on every request, and use one or the other in a route (with GatewayAuthority using an underlying GatewaySphereContext.
• X-SPHERE-IDENTITY header removed.

[jsantell]

Rebased on #685

[github-actions]

Test flake analysis

status	platform	features	toolchain
🟢	macos-13	test-kubo,headers,rocksdb	stable
🟢	ubuntu-latest	test-kubo,headers,rocksdb	stable
🟢	ubuntu-latest	test-kubo,headers	nightly
🟡	macos-13	test-kubo,headers	stable
🟡	ubuntu-latest	test-kubo,headers	stable
🟢	windows-latest	test-kubo,headers	stable

Flake summary for macos-13, test-kubo,headers, stable

     Summary [ 401.883s] 176 tests run: 176 passed (1 flaky), 1 skipped
   FLAKY 6/6 [  59.153s] noosphere::distributed_stress latency::clients_can_sync_when_there_is_a_lot_of_content

Flake summary for ubuntu-latest, test-kubo,headers, stable

     Summary [  45.391s] 176 tests run: 176 passed (1 flaky), 1 skipped
   FLAKY 2/6 [  16.842s] noosphere::distributed_stress multiplayer::orb_can_render_peers_in_the_sphere_address_book

[jsantell]

rebased

[cdata]

IIUC what you wrote here correctly:

GatewayAuthority, in addition to handling auth, now contains the scoped sphere context. For unauthenticated requests (Did route), a GatewaySphereContext is used to access the sphere context. Rationale here is to not double up on processing on every request, and use one or the other in a route (with GatewayAuthority using an underlying GatewaySphereContext.

This is actually a lot more expensive for the DID route, which currently is an unauthorized route that just response with a string:

noosphere/rust/noosphere-gateway/src/handlers/v0alpha1/did.rs

Line 5 in 39a58e6

Ok(gateway_identity.into())

Note that accessing a SphereContext effectively opens a database, and in a stateless world we face doing this on every request. This could lead to DoS by malicious clients hitting the DID route abusively.

[cdata]

Can you speak a little bit about what this is building towards? Have you chosen a serverless function stack?

[jsantell]

Can you speak a little bit about what this is building towards? Have you chosen a serverless function stack?

Step 1 is to genericize sphere context access in the gateway to share functionality between single-sphere gateways and a multi-tenant one, using the high-level start_gateway() function in noosphere-gateway. The minimum interface ContextManager supports this currently, with hand waving over .iter() for the worker threads. Landing this would let us begin continually integrating MTG progress in our cluster.

Step 2 involves more long term abstraction over how we want to extend the gateway. Using the high level start_gateway() doesn't provide extensions to the axum server/custom routing, nor worker configuration, but we can begin to sketch out a shared worker queue for all tasks, compatible with remote MQs, exporting axum routes, etc. -- none of this is needed yet to get a multi-tenant gateway up and running.

No opinions on serverless function stacks; the multi tenant gateway is designed to run as several instances operating over common data. Could become serverless functions if needed. Does not seem immediately relevant to spinning up multi-tenant gateways

This is actually a lot more expensive for the DID route, which currently is an unauthorized route that just response with a string:

We can add a more lightweight function in ContextManager for resolving the map from counterpart to sphere identity for this route; edit: all of our routes have the same weakness w/r/t accessing e.g. sphere state

[cdata]

Does not seem immediately relevant to spinning up multi-tenant gateways

It's relevant insofar as we are writing Send + Sync bounds e.g., in ContextManaager. If we are generalizing without expressing an opinion at this point, I would like to make sure we aren't ruling out wasm32-wasi.

[cdata]

This is a good first pass. I think some aspects need a bit of thought, but I'm excited about the direction it's going in.

[cdata]

Tentatively this looks like a candidate for merge, but let's talk F2F about some of my last comments!

[cdata]

Great work on this!