Plugin Widgets and Composability

[Rugvip]

Feature Suggestion

Plugins should be able to provide not only entire pages, but also smaller parts of a page, such as a card on an overview page, or an item in a header. This has always been desirable as it would allow plugins to be composed of components from other plugins, but with the addition of the service catalog and overview pages, the need has become even more clear and urgent.

Possible Implementation

We likely want to go beyond simply importing React components across plugins, as we at least want components to be wrapped in a context for plugin, and possibly provide some constraints for how and where components can be used. We also want to provide an API where the developers of each individual Backstage app can decide what plugin components go where. A very high level example of this is given in chapter 4 of #1233, but it is in no way decided that we want to go with that API.

Context

We've been looking at this issue for a while as a part of introducing the service catalog. The purpose of this issue is to have a more open place for discussion and for the community to be able to provide feedback and ideas.

[andrewthauer]

A few initial thoughts on this topic:

Display Rules - I'm assuming the RFC describes a registry of widgets a plugin exposes? This seems appropriate for describing where (location) and how (desired size/orientation) widgets would be displayed. I'm guessing a registry of metadata describing placement rules for each widget that a plugin exposes?

Tabs/Routes - Can widgets be included into tabs within a plugin? For instance, the catalog entity page might have tabs that contain widgets that are specific to the org/domain/entity. Perhaps the tab itself would be a customizable widget that just routes somewhere with some params. Or, maybe these would just be widgets on the same route.

Input Data - Many widgets will likely require some sort of input values to render the appropriate data within the context they exist. For instance, within the catalog entity page, it would need to know the entity being displayed. This would likely require some formal metadata that both the widget and widget host use to indicate what to render.

Host Containers - Do widget consumers expose slots where certain widgets can be inserted? If so, then perhaps there also needs to be metadata for what types of widgets a container can support.

Configuration - How are widgets and containers configured? Do they work auto magically via some conventions, or do they require full configuration. Ideally, there are smart defaults for widgets to avoid configuration in many cases. Perhaps some widgets can suggest being injected into well known containers (e.g. entity page) if the plugin itself is registered. Some widgets may be opt-in using a simple toggle. Others might require more complex configuration.

[Rugvip]

Display Rules - I'm assuming the RFC describes a registry of widgets a plugin exposes? This seems appropriate for describing where (location) and how (desired size/orientation) widgets would be displayed. I'm guessing a registry of metadata describing placement rules for each widget that a plugin exposes?

An alternative to a registry is just regular exports from the plugin package. Regarding layout I do feel we need some solution for setting constraints for how a plugin can be displayed and probably a method for choosing how to display them within those constraints.

Tabs/Routes - Can widgets be included into tabs within a plugin? For instance, the catalog entity page might have tabs that contain widgets that are specific to the org/domain/entity. Perhaps the tab itself would be a customizable widget that just routes somewhere with some params. Or, maybe these would just be widgets on the same route.

I'm thinking that we strive for an API that is flexible enough to do more than just info boxes, so also tabs and for example header items.

Input Data - Many widgets will likely require some sort of input values to render the appropriate data within the context they exist. For instance, within the catalog entity page, it would need to know the entity being displayed. This would likely require some formal metadata that both the widget and widget host use to indicate what to render.

Indeed, some early experiments in this repo used a React context for the entity information. Internally we pass props to widgets, which are embedded in the type of a "widget space".

Host Containers - Do widget consumers expose slots where certain widgets can be inserted? If so, then perhaps there also needs to be metadata for what types of widgets a container can support.

Internally we have the concept of "widget spaces", that are owned by plugins. Plugins register their widgets to different widget spaces at startup, along with the configuration that is required to add a widget to that space. The plugins can pick widget from the space and choose how to render them themselves. The widgets spaces also have user settings associated with them that allows customization of the widgets by the user.

It's a pretty nice and flexible way to build something like a "widget store" where users can select widget that they want to display themselves, but as a method for composition it doesn't really have everything that I think we need. You end up lacking a lot of control of where things end up being rendered, as it kinda just depends on the order plugins are being loaded or the user. It's also tricky to be able to decide other things through the app, like wanting to hide widgets on some pages. Another issue is that it also doesn't compose well, in case you want to do something like creating a widget that uses widgets from other plugins.

I do think a web components-like slots concept might be the way to go, and gonna dive into that a bit more. I'm also thinking that the API would be something for the app to use to compose everything, with the possibility of building higher-level APIs on top.

Configuration - How are widgets and containers configured? Do they work auto magically via some conventions, or do they require full configuration. Ideally, there are smart defaults for widgets to avoid configuration in many cases. Perhaps some widgets can suggest being injected into well known containers (e.g. entity page) if the plugin itself is registered. Some widgets may be opt-in using a simple toggle. Others might require more complex configuration.

I think we'll see about this, but most likely I imagine it'll work a lot like react props.

[shmidt-i]

Scope:

This RFC looks into the problem of composing different plugins together in Backstage, allowing them to extend each others functionality.

Definitions:

Extension - piece of functionality that integrates into other plugin. Can be Tab, InfoCard, Route or something else.

Host plugin - plugin that hosts other plugins extensions.

Widget - prepared component, exported from plugin that is intended to be used as an extension.

Extension point - defined by the plugin that allows others to "extend" it, e.g. adding additional tab or a card. Defines a contract on how this extension is going to be instantiated.

Actors:

Let’s define different type of users for our Backstage instance (APP below)

1. Plugin developer
2. APP admin that wants to install plugin
3. APP user that wants to use the plugin functionality in the applicable context

Motivations

Plugin developer:

I want to create a piece of software that will help the users. I envision several use cases, e.g. going into a separate route, having a simple card in the entity overview, having more complex view under a tab in the entity view, maybe having some information available for share with other plugins? I also respect other plugin developers and admins, so I want to offer nice experience on setting my plugin up.

Another thing - I'm looking in the future and want to be able to allow other devs to hook up into my plugin, therefore I want leave some extension points that people can use to do so. I also want to define those in a visible manner - typed and with easy-to-follow (i.e. standard) way of using them.

APP admin:

I want the installation process to be easy and straightforward. I need to be able to select which parts of the plugins are installed in APP. I need to able to configure plugins before deployment and adjust they behaviour runtime (e.g. based on a type of entity).

APP user:

I want to have the functionality that plugin promises to bring to be available in the most convenient way - information should be coupled in a sensible manner, e.g. in the entity view I want to see the overview of a plugin-provided data. Also, as an owner of projects A and B, I want to be able to configure parts of the plugins differently for each project (I don't care about some metrics for project A, but very interested in those for project B).

Proposed architecture:

1. ExtensionRegistry - singleton, reactive storage for extensions keyed by the extension points. Can be implemented as an observable, something like BehaviourSubject from rxjs. Best to keep it out of React scope, because then we can initialize it out of React-code related context.

2. ExtensionPoint - export from a Host Plugin, contains type - string and type definition for a factory function. This factory function is going to be called from the Host Plugin, when it prepares extension for rendering.

   // In plugin-catalog exports
   type TabExtension = (
     entity?: Entity,
   ) =>
     | {
         id: string;
         label: string;
         route: RouteRef;
         isActive?: (currentLocation: string) => boolean;
       }
     | undefined;
   
   export const tabsExtensionPoint = new ExtensionPoint<TabExtension>(
     'catalog/entity-tabs',
   );

3. Extension - export from different Plugin, can be anything, piece of functionality, that can fit particular ExtensionPoint (e.g. latest master build overview card from github-actions plugin). Represented by a factory function, that will be called by Host Plugin according to the ExtensionPoint contract. ❗️Factory function also acts as a predicate, meaning that based on this factory function during runtime you can decide whether you want to have this particular feature in this context or not. Example: GitHub Actions tab that is visible only if entity has corresponding annotation.

   // GitHub Actions plugin exports
   
   // entity here is being injected run-time by the host plugin
   export const tabExtension: TabExtension =
     entity =>
       entity.metadata.annotations[GITHUB_ACTIONS_ANNOTATION] && {
         id: 'overview',
         label: 'Overview',
         route: createRouteRef({ path: 'github-actions', title: 'GitHub Actions' }),
       };
   
   // Or if you need some config to be defined by APP Admins
   // Just do higher-order functions
   export const tabExtensionConfigurable: TabExtension = 
   	buildTimeConfig => 
   	  runTimeConfig =>
   	    runTimeConfig.entity.metadata.annotations[GITHUB_ACTIONS_ANNOTATION] && {
   	      id: 'overview',
   	      label: config.label ?? 'Overview'
   	      route: createRouteRef({ path: 'github-actions', title: 'GitHub Actions' }),
   	    };
   	

4. Registration - process of configuring Extension, integrating ExtensionPoint & Extension and registering it into ExtensionsRegistry

   // APP/extensions.ts
   import { catalogExtensionPoints } from '@backstage/plugin-catalog';
   import { tabExtensionConfigurable } from '@backstage/plugin-github-actions';
   registerExtension(
     catalogExtensionPoints.tabsExtensionPoint,
     tabExtensionConfigurable({ label: "My custom overview" })
   );
   	

   As an alternative, this is also allowed

   // APP/extensions.ts
   import { catalogExtensionPoints } from '@backstage/plugin-catalog';
   import { Widget, entitySupportsGithubActions } from '@backstage/plugin-github-actions';
   
   // Compose the extension directly in the APP
   registerExtension(
     catalogExtensionPoints.overviewCardExtensionPoint,
     entity =>
       entitySupportsGithubActions(entity) && {
         component: () => (
           <Grid item xs={3}>
             <Widget entity={entity} branch="master" />
           </Grid>
         ),
       },
   );

5. Host plugin fetches list of Extension factories from ExtensionRegistry, instantiates them according to the contract from (2), filters out the ones that didn't match their predicates and renders the extensions. There are helper hooks available, that hide complexity.

   // Host plugin
   import { useExtension } from '@backstage/core';
   import { tabsExtensionPoint } from '../extensions';
   
   const entity = ...  // get entity somehow
   const tabs = useExtension(tabsExtensionPoint)
       .map(tabFactory => tabFactory(entity))
       .filter(Boolean);
   
   // Use tabs to render stuff

Key moments:

• Order of Extensions for one ExtensionPoint is based on order of their registration, that's why it is good to have extensions registration as a separate from the plugin import/instantiation step. Example - multiple tabs for entity page in plugin-catalog
• User desire to configure plugins and/or extensions for himself is not touched by this RFC, though it is possible to have another storage for configs and use it inside the Extensions to fetch the corresponding config and adjust the functionality.
• Types of ExtensionPoints that I played around with in plugin-catalog + plugin-github-actions setup:
  • Route/Subroute
  • Tabs
  • Entity overview card
  • Left-side menu items

Conclusion:

This approach brings a lot of composability to the table, but with great power comes great responsibility😜. It may be that we want to do more in terms of safeguards and general rules.

Currently, the weakest part is our routing system. Nested routes, figuring out which routes are active while you are in several nested routes, support for breadcrumbs, routes that needs to be changed based on the placement (example: github-actions plugin widget has the link to the /:buildId if it's mounted under /ci-cd already , but it should be /ci-cd/:buildId if it's mounted in the entity root - all of that is suboptimal atm.

Also, the runtime configuration system for the plugins is not there yet, albeit there are some discussions happening.

[Rugvip]

@shmidt-i Great write-up and awesome to bring in some good terminology around the concepts, way better than what we have internally 😅

Could of thoughts/questions before we dive deeper:

In 1 you mention that the registry should be reactive, is there something in the design that makes that a requirement? Internally it's all just synchronously wired up before the app renders.

The conditions in 3 may be better to just leave to the Extension component itself, i.e. just return null from render. Definitely see that we could have it, but only if we have a clear idea of how it would be used and why it provides value. Otherwise I'd rather just leave space in the implementation to add it later and invoke yagni.

4 is super tricky and I'm not sure we can have a plugin or app-driven registration tbh. I'd like to explore a JSX-based API for the APP as the lowest-level composition API, on top of which we could possibly build some higher-level (config driven?) methods of composing plugins. It seems to me like you'll always end up with issues of order and separate configuration for different plugins unless you have very fine-grained registration of Extensions, at which point you're essentially building up a component tree using a flat JS API and you'd be better off with JSX.

Regarding the routing issues I'm hoping we can extend the RouteRef API to be able to handle those use-cases. The original RFC includes route parameters, which isn't implemented yet.

[freben]

Not sure if it needs repeating, but as you know I'm
in the JSX camp as well :) +1 to the above.

[shmidt-i]

So we had a discussion about this and came up with 2 things:

1. List of design constraints:

• Easy to get going with a new app.
• Easy to modify an existing app.
• Design for deletability.
• Easy to get a good overview of the app and page layouts in the app source code.
• PM should be able to modify the app

2. Rough idea on the consuming version of JSX-based api:

<CatalogPageTemplate type="service">
  <MyHeaderSlotConsumer slot="header" />
  <Tabs slot="content">
    <Tab title="Overview">
      <WidgetView>
        <WorkflowWidget />
      </WidgetView>
    </Tab>

    <Tab title="CI/CD">
      <WorkflowRunsTable/>
      <WorkflowRunDetails/>
    </Tab>
  </Tabs>
  <PageFooter slot="footer"/>
</CatalogPageTemplate>

const CatalogPageTemplate = () => {
  return (
    <Page>
      <Slot name="header">
        <Header/>
      </Slot>
      <Content>
        <Slot name="content"/>
      </Content>
      <Slot name="footer"/>
    </Page>
  )
}

And the next step being looking more into the JSX-based approach

[shmidt-i]

Another take is that we don't need a reactive registry from the beginning, it can be just a synchronous registering on the app startup and then passing that down through e.g. React.Context

[Rugvip]

Some output from another brainstorming session today. We came up with a couple of ideas/hypothesises that we want to try out in a more concrete implementation:

1. Different types of Component entities will want to have different cards and tabs shown.
2. Tab titles as part of JSX in the app improves readability and makes it easier to find location in the code corresponding to the website.
3. File structure that reflects the site map makes it easier to navigate the code.
4. Cards and other extensions need to be configured in the app, and the configuration type needs to vary based on the extension point.
5. Functional entity filers provide more flexibility and portability compared to more hardcoded values.

Let's keep filling in the above list as we progress.

Some other output was more high-level sample code from the app's point of view. We primarily focused on finding a good level of information and structure that would make it easy for anyone coming from the website to navigate the code. Particularly focusing on the use-case of finding for example the cards on a particular overview page, and then rearranging or reconfiguring them.

const App = (
  <>
    <CatalogPage>
      <ComponentList />
    </CatalogPage>

    {/* pages/components/service.tsx */}
    <EntityContext filters={[kind('Component'), type('service')]}>
      <TabbedPage>
        <CardPage>
          <ReadmeCard />
          <GitHubWorkflowsCard />
          <MetadataCard />
        </CardPage>

        <SpotifyCiCdPage />
      </TabbedPage>
    </EntityContext>

    {/* pages/components/website.tsx */}
    <EntityContext filters={[kind('Component'), type('website')]}>
      <TabbedPage>
        <TabbedPage.Tab title="Overview">
          <CardPage>
            <ReadmeCard size={2} />
            <LighthouseCard size={4} />
            <GitHubWorkflowsCard />
            <MetadataCard />
          </CardPage>
        </TabbedPage.Tab>

        <TabbedPage.Tab title="CI/CD">
          <SpotifyCiCdPage />
        </TabbedPage.Tab>
      </TabbedPage>
    </EntityContext>

    <ScaffolderPage>
      <ScaffolderTemplates />
    </ScaffolderPage>

    <GraphiQLPage>
      <GitHubEndpoint />
      <GitLabEndpoint />
      <BackstageEndpoint />
    </GraphiQLPage>
  </>
);

Another idea was switches for selecting content, where the first matching page for a given entity will be displayed:

const SpotifyCiCdPage = () => (
  <Switch>
    <GithubWorkflowPage>
      <GithubWorkflowListPage />
      <GithubWorkflowDetailsPage />
    </GithubWorkflowPage>

    <CircleCiWorkflowPage>
      <CircleCiWorkflowListPage />
      <CircleCiWorkflowDetailsPage />
    </CircleCiWorkflowPage>
  </Switch>
);

[Rugvip]

Quick update that this is still something we're looking at, specifically how to solve routing between different plugins, and different components exported from a plugin. Some more info here: #2565

[Rugvip]

Core team has been focusing on this along with a complete implementation of routing and RouteRefs of some form.

Going at it through a bunch of different experiments, all collected in this branch around here:

backstage/plugins/explore/src/components/Experiment11.tsx

Line 214 in 2488367

export const Experiment = () => {

Experiment 11 is a particular one where we started getting more happy with the implementation and simplicity. We're essentially allowing some custom data to be tied to individual react elements, and then use that for route and plugin discovery. Every component (and maybe other things too) that a plugin exports is wrapped up and exported as an "Extension", which is a point were we'll be able to let the core APIs wrap plugin components with additional functionality and context. It is also somethings that can be used for composability, as the API would be open for plugins to export their own create*Extension for other plugins to use.

This is still experiments though, no naming is settled, and the widget stuff you see in experiment 11 is not something we're planning to add now either 😅

Please reach out here or on Discord if this triggered any questions or ideas, or if you want to get involved in general!

[OrkoHunter]

The architecture proposed here is very exciting and looks great from TechDocs point of view (from me and @hooloovooo).

Examples of some widgets that TechDocs will most certainly use/create in coming future -

• Display number of open GitHub issues
  • If a user is looking at a docs site, it will show how many GitHub issues are currently open in the repository associated with it.
  • The widget can be used for a lot of similar GitHub/GitLab/etc. stats.
  • It might be useful to show a widget like this on the Entity page as well.
  • This widget can be "hosted" by a plugin which is much closer to the external provider. So the GitHub widget can live in a plugin that knows GitHub well. Same for GitLab, etc.
• A feedback widget
  • In TechDocs, it will be used to give feedback for a piece of docs on page. Users will select some text on a page, and will be able to easily create an issue/ticket with some feedback.
  • The feedback widget can be used to give feedback for an entity as well. So, it's something that can benefit from composability.
• Number of open Stackoverflow Enterprise questions based on topics
  • TechDocs can host this one for other plugins to use potentially.

(A lot more ideas will come in future.)

We will start implementing some of these widgets for TechDocs in 2021 Q1. If the Extension APIs, registry, etc. are available to use, we will make use of it. But even if they are not available timeline-wise, that is probably okay. We can always expose the widgets later on.

Looking forward to the great work on this. :) 🚀 🤞