feat(middleware): capability end-point manifest

[twilson63]

Background:

Since there will likely be many different gateways with many different capabilities, it is helpful to implement a middleware component that provides capability insights to applications as they interact with the gateway.

ANS-101: Proposes this here: https://specs.g8way.io/#/view/hLSKTSwd5_3xB71zciyK_WFEpK9wVX2IeGzxk9Yl2xY

This could be implemented with a middleware attached to the /info/capabilities route and returns a JSON document describing the supported capabilities of this gateway. This could allow applications to let users choose their own gateways and enable features based on the capabilities of each gateway.

[TillaTheHun0]

This is a really awesome idea.

It seems to me that capabilities will often be implemented as other middleware themselves, and then simply mounted onto the "base" gateway express app. So in order for this capabilities middleware to know all the capabilities of the gateway, each middleware that is mounted will somehow need to "broadcast" the capabilities it is adding to the gateway, if any.

Each gateway has a SQLite Database. I propose the following approach:

1. Each middleware, on gateway startup and before mounting its routes, find or create a record into a SQLite database table ie. gateway_capabilities with the following schema:

name TEXT
version TEXT
metadata TEXT (serialized JSON?)

2. The Capabilities middleware would be part of the "base" gateway. It would query the gateway_capabilities table, map the rows to the shape described by the spec, using that to respond to clients. The implementation could fetch on gateway startup and cache the result from the DB, or lazily load on the first request.

G8way Middleware shape proposal

It would be great if each middleware was self-encapsulated, following a certain shape, and having certain dependencies injected to it by the gateway. A potential approach could be middleware that follows this signature:

type ArweaveG8WayMiddleware = (context: ArweaveG8WayContext) => Promise<(app: Express) => Express>

type ArweaveG8WayContext = {
  addCapability: (capability: { name: string, version: string, ...rest }): Promise<{ ok: boolean }>
  log: 
  ... // other apis the gateway would like to inject into middleware
}

by injecting addCapability, the gateway controls the impl, and doesn't leak details to middleware being mounted, all while making middleware impls more testable. Also it means that the impl can change, as the gateway sees fit, without breaking middleware, ie. instead of persisting to SQLite DB, the impl could simply populate an object in scope of the capabilities middleware.

A middleware implementation might look like this:

const awesomeArweaveG8WayMiddleware: ArweaveG8WayMiddleware = async ({ addCapability, log }) => {
  await addCapability({ name: 'awesome-capability', version: '1.2.3', some: 'other', data: 'here' }).then(...).catch(...)

  return (app) => {
    app.get(...)
    ....
    return app
  }
} 

Then finally, the base gateway can simply fold over middleware to gain capabilities:

const app = express()
... // base gateway things ie. mount built-in routes (just another fold with internal deps?)

const context = {
   addCapability: () => {...}
}

const middlewares = [
  awesomeArweaveG8WayMiddleware,
  anotherMiddleware
]

const $gateway = middlewares.reduce(
  async ($app, middleware) => middleware(context).then(m => m(await $app)),
  app
)

$gateway.then(gateway => gateway.listen())

These are just ideas and open to scrutiny and feedback of course. I wouldn't mind taking a stab at the implementation, in the base gateway, and the capabilities middleware itself, if folks thought this was a cool approach.

[djwhitt]

Thanks for the thoughtful proposal!

Before we get too far into implementation details, let's discuss the spec itself. As written, it looks like there's no way to support multiple versions of the same capability. Is the assumption that capabilities are always backwards compatible?

[TillaTheHun0]

Before we get too far into implementation details, let's discuss the spec itself.

Absolutely 👍

As written, it looks like there's no way to support multiple versions of the same capability

That's how I understood the spec. Though I could definitely see gateways wanting/needing to support multiple versions of the same capability; the use case being the ability for gateways to deprecate capability version support and allow migration to new versions, without breaking user-space.

Is the assumption that capabilities are always backwards compatible?

The spec describes version as being the "capability [Semver] version". This implies each capability could have its own versioned spec, or at the very least a documented behavior, and that a major version bump would indicate breaking changes. That buttresses the need for supporting multiple versions of the same capability -- the use case being the ability to deprecate and migrate without breaking user-space.

Maybe the spec should allow for versions as an array of Semvers?

Tangentially related to this question: what ought to be responsible for defining how a capability's different versions are exposed?

The gateway?
The specific capability spec?
ANS-101?

Right now, I lean towards the specific capability spec; all of the possible capability use-cases aren't clear to me. The obvious use-case is adding public routes, but some maybe could add support for certain headers, and then some may event augment internal behavior of the gateway. So until a pattern emerges, I lean towards keeping the constraints as local to the specific capability as possible.

Edits: wording clarification, typos

[djwhitt]

That's how I understood the spec. Though I could definitely see gateways wanting/needing to support multiple versions of the same capability; the use case being the ability for gateways to deprecate capability version support and allow migration to new versions, without breaking user-space.

I'd be inclined to say backwards compatibility is required and new capabilities should require a new name (e.g. append a '-v2'). In that context, I don't think semantic versions make much sense. A single number that increases as features are added would likely be better. (see this talk for some of the rationale for this)

Tangentially related to this question: what ought to be responsible for defining how a capability's different versions are exposed?

Mostly doesn't matter as long as we don't break backwards compatibility. We just allow a single version of each capability. Anything breaking becomes a new capability.

The obvious use-case is adding public routes, but some maybe could add support for certain headers, and then some may event augment internal behavior of the gateway. So until a pattern emerges, I lean towards keeping the constraints as local to the specific capability as possible.

We should be extremely cautious about extending or modify the core API. Headers might be a gray area since most probably will not conflict, but in general I think extensions should live under there own routes. That allows space for experimentation without breaking the core API. Eventually I imagine some will stabilize and their functionality can be integrated more deeply.

[TillaTheHun0]

All your points on versioning lgtm, and makes other issues w.r.t deprecating and migrating to new versions moot 👍 . The spec draft would need to be revised to reflect those points.

We should be extremely cautious about extending or modify the core API.

💯 absolutely. My proposal for core to inject apis into middleware stems from that desire, only allowing middleware to call black-box apis that are provided to it, by core. It helps ensure middleware can augment in ways that core allows. Only once a capability stabilizes and becomes almost ubiquitous with gateways ought it be considered for deeper integration in core (which could simply be core internally composing it instead of the gateway runner).

But being new here, I admit I don't know all of the use-cases for middleware 😄

in general I think extensions should live under there own routes

Seems reasonable. This could just be a convention for middleware developers to follow. But if it's decided that the core gateway ought to enforce that, then core would need to provide an api that each middleware could use ie. addRoute or something. Again, dep injection would work well here.

The impl of addRoute could ensure each capability had it's own "space" -- perhaps it's a route mount point on the public api. Furthermore, perhaps that mount point could be defined by the capability? In that case, core could fail fast, on startup, if it detected two middleware attempting to mount to the same place (i'm thinking of the scenario in which a gateway runner mistakenly tries mount two middlewares that both implement the same capability).

[djwhitt]

I wonder if there should be any distinction between capabilities and extensions. 🤔 It seems like we could have a completely self-contained extension mechanism the enables self-describing extensions (could be a single 'extensions' capability), and use capabilities as more of a way to describe features provided by the base protocol.

[djwhitt]

But being new here, I admit I don't know all of the use-cases for middleware 😄

Reminds me, I should have asked this earlier - do you have a particular use case in mind yourself?

[TillaTheHun0]

#26 and https://specs.g8way.io/#/view/lXLd0OPwo-dJLB_Amz5jgIeDhiOkjXuM3-r0H_aiNj0 is interesting as it would enable apps to not have to use hash-based client-side routing.

Also ANS-108 is really interesting as it means gateways could serve up applications designed to render certain kinds of content.

Worth noting that both of those are not standalone routes to be mounted on a gateway but change behavior of the gateway itself.

[djwhitt]

Gotcha. That context is helpful. I think we should probably split this into two tickets - one about future extension mechanisms and the other (this ticket) about the capabilities end-point. We can focus on the path manifests change first and the capabilities shortly capabilities end-point after that.

@TillaTheHun0, can you add a new issue about the future extension mechanism including a rationale for it and a list of requirements based on that rationale?

Re ANS-108, it's an interesting proposal, but it's also a breaking change. It both changes the behavior for existing content tagged with Render-With and breaks any existing data consumer that expects to retrieve unaltered without having to worry about how it's tagged. If we decide to implement it, we'll need to give plenty of warning about it in advance.

[twilson63]

Maybe we could consider an endpoint like `raw` but for rendering. Like`/render/:tx` or something or `/tx?render=true`, the capabilities endpoint can inform developers of gateways supporting the render option. Also, this could be an opt-in and not a breaking change. We could potentially experiment via a `/labs` route where proposed routes and functionality can be in a sandbox to iterate on design and implementation without impacting production. These experiments do not even have to be merged in to the main code base, they could implemented downstream in forks and can be evaluated in the real world. Just a thought.

…

On Tue, Aug 15, 2023 at 3:50 PM David Whittington ***@***.***> wrote: @TillaTheHun0 <https://github.com/TillaTheHun0> Gotcha. That context is helpful. I think we should probably split this into two tickets - one about future extension mechanisms and the other (this ticket) about the capabilities end-point. We can focus on the path manifests change first and the capabilities shortly capabilities end-point after that. @TillaTheHun0 <https://github.com/TillaTheHun0> Can you add a new ticket about the future extension mechanism including a rationale for the it and a list of requirements based on that rationale? Re ANS-108, it's an interesting proposal, but it's also a breaking change. It both changes the behavior for existing content tagged with Render-With and breaks any existing data consumer that expects to retrieve unaltered without having to worry about how it's tagged. If we decide to implement it, we'll need to give plenty of warning about it in advance. — Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFGLBXINMTOZFLOWN4PELXVPHJTANCNFSM6AAAAAA3KCMEWQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[TillaTheHun0]

I like @twilson63 suggestion for a /labs or some sort of sandbox that allows for experimentation, in the open.

As for Render-With functionality, I also like the idea of render defaulting to false. Yet another option is to leverage the Accept header and the gateway can respect that, which is a common web server behavior. If Accept contains certain MIME types ie. text/html, then the gateway can additionally fetch the Renderer for a Render-With transaction and perform the rendering. This would "just work" when using a browser to fetch a transaction with Render-With as most browsers have default Accept values, but programmatically fetching a transactions ie. with fetch would be opt-in via explicitly setting the Accept header on the request. Just a thought.

@TillaTheHun0, can you add a new issue about the future extension mechanism including a rationale for it and a list of requirements based on that rationale?

Sure. I can do that.

[djwhitt]

These experiments do not even have to be merged in to the main code base, they could implemented downstream in forks and can be evaluated in the real world.

I'll drop some thoughts on the new ticket when I get more time, but I agree the spirit of this. I'd like extension to mostly live outside the main code base and be loadable as modules.

[djwhitt]

We could potentially experiment via a /labs route where proposed routes
and functionality can be in a sandbox to iterate on design and
implementation without impacting production. These experiments do not even
have to be merged in to the main code base, they could implemented
downstream in forks and can be evaluated in the real world.

Good idea! An intentionally more awkward URL is proposed here: #34 (comment)

[djwhitt]

Continuing the discussion on how to describe capabilities:

I think we need the following updates to the existing ANS:

• Remove references to semantic versioning. Breaking changes should be implemented as new capabilities.
• Explicitly request that capabilities only add non-breaking changes. There is no way to enforce this, but making the intention clear is still helpful.
• Make version sorting explicit. We want to avoid breaking changes, but expanding capabilities in non-breaking ways is still okay. Given capabilities maintainers will likely choose different versioning schemes, we will need a reliable way to determine version order.
• Request that capabilities be namespaced (e.g. io.ar/amazing-capability) to help prevent name conflicts.

We should also continue thinking about:

• Should we recommend some way to include docs or API specs (JSON Schema, OpenAPI, or GraphQL schema)?
• Is there anything else we need to include?
• How can we safely support extensibility to our capability specifications?