Extensibility based on Tower

[BrynCooke]

It has been proposed that we shift our architecture to a Tower based pipeline to:

1. Support a clean model for extending functionality.
2. Provide us with off the shelf functionality common to proxy type software.
3. Reduce our architectural churn.

The router already uses Tower internally, so moving our existing components to services should be something we can do gradually.

A mockup to demonstrate how this could look has been created, but the public facing API need to be developed and we need to prove that Tower is going to work for us.

Execution is split in to phases after which we reevaluate if we need to pivot. After each phase the tasks for the following phase will be converted to tickets.

Phase 1 (1 week) (COMPLETED!):

• Mockup extensibility API. #304
• Mockup YAML configuration. #340
• Mockup architecture based on Tower services. #305

Phase 2 (1 month) (NEARLY!):

• Extensibility tests #336
• Convert existing components to services: #342
  • Query planner conversion to service. #343
  • Fetcher conversion to service. #344 *
  • Routing (Router) as a service. #346
  • Execution conversion to service. #345
  • Give each service it's own set of service types #417
• Integrate extensibility API. #347 *
• Load configuration in to pluggable service builder #381 *
• Fix introspection #382 *
• Check that request handling is correct #383
• Post-Tower Performance testing #386
• Response formatting #389
• Request and query plan validation #390
• Http structures improvement #396
• Fix tracing spans #401
• Dynamic configuration via YAML. #402
  • Reexamine plugin macro #406
  • Generic layer config #407
  • Plugin lifecycle startup/shutdown not called. #409
• Implement some reusable layers:
  • automated persisted queries (APQ) #130

* = broken but was working previously

Phase 3 (After merge)

Note that the following tickets need to be prioritized. They are just a bucket of ideas right now:

• Project structure documentation and development clarity #400
• Core plugins configuration #449
• Investigate plugin/layer config versioning #438
• Add examples for new API #419
• Re-examine use of buffer #387
• Runtime extension type discovery (Optional) #403
• Implement some reusable layers:
  • Query planning caching as a layer. #384 *
  • Instrumentation as a layer #398
• Implements some plugins
  • JWT auth #434
  • Biscuit auth #435
  • Oauth #436
  • Re-write Studio Explorer integration to use a different technique #388
  • Otel as a plugin #399
  • investigate a rhaiscript plugin #431