[RFC] Backend System Evolution

[jhaals]

Status: Open for comments
Authors: @backstage/maintainers

Need

The new Backend system stems from a couple of needs identified prior to writing this RFC.

Some of the areas covered in this RFC need much more work, but we would like to open up for comments and feedback for this proposal before moving forward.

Simplified Installations

The perhaps most important of these is to make it easy for integrators to install new backend plugins without having to jump around in multiple files to wire up the plugin.

The current process is both time consuming and prone to errors. Ideally the installation should not require more than a few lines of code excluding the configuration itself.

Sane Defaults

It should be easy to stay up to date with new backend plugins when upgrading.

Today, plugins often have their dependencies constructed in the plugin's setup file, which leads to a high risk of breaking changes and manual labour during upgrades when a plugin starts taking additional dependencies or when constructor parameters change.

We need to be able to introduce new default APIs without having to change the existing code in order to consume them.

Extending Plugins

It should be easy for integrators to install extensions for existing plugins in order to gain new functionality.

Plugin developers should be able to provide extension points for other modules to use.

Developer Friendly

The concept of installing, extending and providing extension points should be similar across the system and easy for developers to understand.

The development experience should be easy and quick.

Proposal

The proposal is divided into several sections, as the use cases for integrators and plugin developers are different.

Plugin Usage

In its simplest form new plugins can be installed by adding them to the backend.

import { createBackend } from '@backstage/backend-common';
import { catalogPlugin } from '@backstage/plugin-catalog-backend';
import { catalogGithubModule } from '@backstage/plugin-catalog-github-module';

async function main() {
  const backend = createBackend();
  // Adds the catalogPlugin
  backend.add(catalogPlugin());
  // Installs GitHub org discovery, which adds the entity provider to the catalog
  backend.add(catalogGithubModule.orgDiscovery());
  await backend.start();
}

Plugin authors can expose options that can be used for altering the default setup.

backend.add(catalogPlugin({ disableProcessing: true }));

Plugin Authors

New plugins are created by exporting the result of createBackendPlugin.

createBackendPlugin accepts a register function which takes care of wiring up the plugin to the backend once it has been installed.

The register function is passed a backend environment (env) parameter. The environment can then be used to register the plugin's init function which is called on startup.

The init function uses a dependency injection system similar to the Utility APIs found in the frontend core library. For now we refer to these as "Services" rather than "APIs", but naming is to be determined. This is to help separate the concepts, as they don't function in the exact same way.

Dependencies to the plugin are provided in the deps section by mapping them to a name that then can be referenced in the init function and the corresponding ServiceRef. The backend framework takes care of initializing dependencies prior to calling the init function.

export const examplePlugin = createBackendPlugin({
  id: 'com.my-company.example',
  register(env) {
    env.registerInit({
      deps: {
        router: httpRouterServiceRef,
      },
      async init({ router }) {
        // plugin specific setup code.
        router.use('/hello', async (_req, res) =>
          res.json({ message: 'Hello World' }),
        );
      },
    });
  },
});

Plugins Providing Extension Points

There is a need for plugins to expose extension points which can be extended by other plugins. The software catalog is an example of such a plugin, which today is extended with custom processors and entity providers.

A plugin can register multiple extension points, each of which can be depended on by other plugins through a service reference.

Please note that the ServiceRef for the extension point is imported from the plugin's node package, as we should avoid plugin-to-plugin imports.

// exported from @backstage/plugin-example-hello-world-node

export interface GreetingsExtensionPoint {
  addGreeting(greeting: string): void;
}

export const greetingsExtensionPoint =
  createServiceRef<GreetingsExtensionPoint>({
    id: 'com.my-company.example.greetings',
  });

import { greetingsExtensionPoint } from '@backstage/plugin-example-hello-world-node';

export const examplePlugin = createBackendPlugin({
  id: 'com.my-company.example',
  register(env) {
    // implements the GreetingsExtensionPoint
    const greetingsExtensions = new GreetingsExtensions();

    env.registerExtensionPoint(greetingsExtensionPoint, greetingsExtensions);

    env.registerInit({
      async init() {
        // The greetingsExtensions instance is in scope as it's created above.
        // Note how this uses the private .greet() method which is
        // not exposed in the public API.
        greetingsExtensions.greet();
      },
    });
});

Providing Services

There is going to be a set of default services provided by the framework and surrounding packages, for the purposes of for example routing, logging and many other aspects. These are common utilities that are intended to be usable by all backend plugins. Regardless of what this system ends up looking like, we intend for there to be a package that brings together sensible defaults for the average backend, just like @backstage/app-defaults does for the frontend. These services can be overridden with custom implementations or configurations at the backend level.

What's common for many of these services is that there is a need to scope the service implementation to each plugin. For example the logging service is more useful when it's able to tell which plugin is producing a particular log line. To accommodate this use case, the service factory always creates an instance scoped to the plugin requesting the service.

Service factories are created using the createServiceFactory function connecting the factory implementation to the ServiceRef. The factory function returned is expected to produce an instance of the service given a plugin ID.

export const loggerServiceRef = createServiceRef<LoggerService>({
  id: 'core.logging',
});

const loggerServiceFactory = createServiceFactory({
  api: loggerServiceRef,
  deps: {},
  factory: async () => {
    const rootLogger = new RootLogger();
    return async (pluginId = 'root') => rootLogger.child({ pluginId });
  },
});

// The backend is then passed the loggerFactory to register it with the backend.
const backend = createBackend({
  apis: [loggerServiceFactory],
});

Similar to frontend Utility APIs, service factories may depend on other services. The dependency mechanism is slightly different though, as we receive a factory function rather than the concrete implementation of the service dependency. This factory function is then used to request a service instance for a given plugin ID. Services may also support un-scoped instances, which are retrieved by omitting the plugin ID.

export const dbServiceRef = createServiceRef<DbService>({
  id: 'core.db',
});

const dbServiceFactory = createServiceFactory({
  api: dbServiceRef,
  deps: {
    loggerFactory: loggerServiceRef,
  },
  factory: async ({ loggerFactory }) => {
    const rootLogger = await loggerFactory();
    const dbManager = new DbManager(rootLogger);

    return async (pluginId?: string) => {
      if (!pluginId) {
        throw new Error('DB Service must be scoped to a plugin');
      }
      const logger = await loggerFactory(pluginId);
      return dbManager.forPlugin(pluginId, { logger });
    };
  },
});

Developer Experience

Many installations have several backend plugins running together in the same main process. To accommodate for a leaner and less noisy development experience it is desirable to have an option to run only a specific subset of plugins wired up to any given backend instance.

For example this would just start the catalog and scaffolder backend.

yarn backstage-cli start-backend --backend catalog --backend scaffolder

Alternatives

Several experiments on different setups where conducted in the backend-system-exploration repository where we looked at different approaches for wiring up plugins and providing dependencies.

Roadie have also helped out prototyping what the backend system would look like with an off-the-shelf dependency injection framework added on top of the existing backend system. See #9165 for more information.

Risks

Massive Backend Plugin Migration

The intent is to gradually let existing plugins implement the new Backend API while still exposing the old API for backwards compatibility.

There should be a way to install and wire a "legacy" plugin into the new backend system. We don't see a risk supporting that use case as the existing plugin setups are mostly relying on the old environment for configuration.

Restrictions Introduced by the Framework

The current way of building and structuring backend plugins is extremely open, with very few restrictions or guardrails. This means that you might not be fighting the framework that much, but it also makes it difficult for the framework to help you. The suggestions in this RFC changes this balance by a fairly large margin. It introduces a lot more structure, benefitting utility and easy of installation, but at the cost of flexibility and stricter boundaries enforced by the framework.

[kuangp]

Amazing proposal! I really like how it encapsulates much of the same pattern from the frontend system which provides integrators the same familiar interface and a consistent configuration experience throughout the entire system.

Can you elaborate on the plugin extensions point design a bit more ie. what would an example look like for a catalog extended with a custom processor/provider?

I also resonate with the framework restrictions risk. For us, one of the main obstacles for other teams helping implement custom plugins is the learning curve involved with learning the "Backstage framework". Backend plugins were a lot easier to "sell" since you only really needed to know how to implement an Express router (though according to this proposal, it shouldn't be that different).

[Rugvip]

Amazing proposal! I really like how it encapsulates much of the same pattern from the frontend system which provides integrators the same familiar interface and a consistent configuration experience throughout the entire system.

Thank you! Yep that is indeed part of what we were going for, making sure existing knowledge of the Backstage frontend can be reused.

Can you elaborate on the plugin extensions point design a bit more ie. what would an example look like for a catalog extended with a custom processor/provider?

They're a completely open interface, a blank canvas, but we do have some plans. As part of the backend system design we've explored a couple of different concepts for bridging together plugins and modules. One of them is "hooks", another is "feature collections". Both are things that we thought could potentially be the interaction point, but in the end we realized that it was the wrong abstraction level and we should let plugins choose instead.

So as part of this system we would likely include a couple of convenient building blocks for creating extension points. I'll give a super quick intro to the two concepts above, but let's please keep the focus of the discussions in this RFC around the original proposal and not these APIs as it could quite easily derail otherwise.

Hooks

A way to replace or add behavior deep inside plugins, inspired by the Webpack architecture.

// in catalog register()
const engineExtensionPoint = {
  hooks: {
    stitchEntity: createAsyncHook<{entity: Entity}>(),
    beforeEntityProcessing: createSyncHook<{entity: UnprocessedEntity}>(),
  },
}

env.registerExtensionPoint(catalogEngineExtensionPoint, engineExtensionPoint);

// inside catalog stitcher
await engineExtensionPoint.hooks.stitchEntity.callHook({entity})

// inside a different module
({
  deps: { catalogEngine: catalogEngineExtensionPoint },
  init({ catalogEngine }) {
    catalogEngine.hooks.stitchEntity.tap(async ({ entity }) => {
      // custom logic
    })
  }
})

Feature collections

Inspired by all different places where we have various features that we want to add to or replace, such as catalog processors, entity providers, search collators, etc.

// in catalog register()
const ingestionExtensionPoint = {
  providers: createFeatureCollection<EntityProvider>()
}

env.registerExtensionPoint(catalogIngestionExtensionPoint, ingestionExtensionPoint);

// inside catalog init
const providers = ingestionExtensionPoint.providers.get()

// inside a different module
({
  deps: { catalogIngestion: catalogIngestionExtensionPoint },
  init({ catalogIngestion }) {
    catalogIngestion.providers.add(new GitHubOrgProvider())
    // or, perhaps more likely to be called directly in the backend setup
    catalogIngestion.providers.remove('url-reader')
  }
})

I also resonate with the framework restrictions risk. For us, one of the main obstacles for other teams helping implement custom plugins is the learning curve involved with learning the "Backstage framework". Backend plugins were a lot easier to "sell" since you only really needed to know how to implement an Express router (though according to this proposal, it shouldn't be that different).

Yep, well strive to keep things lean with hopefully minimal amount of code needed for a simple express route plugin. No fancy wrapping of the express API or anything like that. Hopefully this system will in the end make it easier to build even a simple plugin with few external interactions. Very much aware of the risk here though

[anitsh]

Great proposal! Understood why and what's expected with examples.

To be honest I haven't used Backstage and haven't gone through the code nor the system design docs. But with some experience in using other frameworks with similar problems (i guess), the proposed way would help the community, strengthen contributors a lot.

Thanks.

[Rugvip]

Implementation is underway, tracked through a meta issue at #13081

[cowboyd]

Have we given any thought to having teardown in addition to setup be a core part of the API. I think this is critical to be able to do things that most mature plugin/addon systems have that allow you to programmatically add/remove or restart it. To use the example from the readme, this could look like:

import { greetingsExtensionPoint } from '@backstage/plugin-example-hello-world-node';

export const examplePlugin = createBackendPlugin({
  id: 'com.my-company.example',
  register(env) {
    // implements the GreetingsExtensionPoint
    const greetingsExtensions = new GreetingsExtensions();

    env.registerExtensionPoint(greetingsExtensionPoint, greetingsExtensions);

    env.registerInit({
      async init() {
        // The greetingsExtensions instance is in scope as it's created above.
        // Note how this uses the private .greet() method which is
        // not exposed in the public API.
        greetingsExtensions.greet();
      },
      async drop() {
        // do teardown specific to this exertion point.
      },
    });
  },
  async unregister() {
    // remove databases, clear out custom directories, etc...
  },
});

[Rugvip]

@cowboyd other than for local development we didn't (seriously) consider any kind of system for hot-swappable plugins or anything like that. I don't agree that most mature plugin systems have the ability to add/remove plugins at runtime. Are there any particular use-cases you have in mind?

We did consider startup/teardown hooks though, but are currently thinking that we can expose those as services if needed. So init would depend on for example lifecycleServiceRef and you could do something like lifecycle.onShutdown(() => ...) from within init.

[cowboyd]

One example is testing. In our harnesses, we currently have to put everything at an arms length in a separate process so that it can shutdown cleanly. It would be nice to be able to stop services, servers, and even objects, all within the same process without experiencing a bunch of negative consequences. If we had that, we could draw our testing boundaries at almost any level of integration that we wanted without having to require any changes from the framework itself, or setup complex mock environments to tear down afterwards.

Another use-case that we've wanted is the ability to restart just a single plugin during development, without having to pay the cost of restarting the entire backstage server.

A highly restartable system could, I think, be used to really improve the story around file-watching and HMR.

Apart from these three use cases though, I think an important feature is a general culture of system composability: the idea that any operation that defines a method of accumulating state can (and should) also define the method of discarding that state. Operations like that are zero sum, and are therefore composable. Operations that don't, aren't, and a generally composable system means the door is not shut to use-cases we have not yet imagined.

[Rugvip]

Yep we'll have a look at local development more properly before settling things and happy to bounce ideas in that area too. We did some early experiments where we concluded that the HMR that we have right now may be a bit overkill, and restarting the entire server on every change might be alright. The backend has to be and generally is stateless, with all state stored in the DB. An exception to that during development is SQLite, which we'd need some form of solution for.

[cowboyd]

The backend has to be and generally is stateless, with all state stored in the DB.

I should qualify what I mean by state. I'm talking about things like database connections, bound server ports, open file descriptors, and active TCP connections. Viewed from that perspective, the backend is extremely stateful.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[freben]

Closing this as the first versions of this are in alpha already, and the rest of the work is tracked in other issues.