[RFC] [Scaffolder] Have a global environment context that is accessible by all templates

[dhenneke]

Status: Open for comments

Need

Have a common set of arguments that are provided to every template that is executed in a Backstage installation. These arguments are config driven, i.e. the values could differ between different Backstage installations (Company A, Company B, ...) or deployment stages (dev, staging, prod). This enables Templates that behave slightly different in each installation without the need to maintain multiple versions of it.

This goes beyond providing a custom action (see Alternative 1) that provides outputs, because it would also be neat to access these properties in the parameters so they are part of the input form.

Context

We provide Backstage to our company-internal developers. We also provide Backstage as part of a product where we prepare a cloud-based development environment (SCM, CI/CD, Kubernetes, ...) for our customers. Both use the Scaffolder to create new software for the respective (development) environments. In both use cases, we want to reuse the same templates so we ourselves but also our customers always use the most up-to-date and reliable template without too much maintenance overhead. However, while using the same technologies and tools, each environment slightly differs between each other: They have different GitHub organizations, different container registries, might not use all SAST tools, different secret names in the CI/CD environment, ….

Prior to the scaffolder, we started with a set of selectable presets in cookiecutter. But that had its' own organizational problems such as not wanting to share infrastructure details between customers.

A cookiecutter example

{
  "_environment_configuration": {
    "sdase": {
      "github_org": "SDA-SE",
      "image_registry": "our.registry.com",
      "image_registry_org": "sdase",
      "some_secret_id": "some-secret-name",
      "..."
    },
    "customer-1": {
      "github_org": "...",
      "image_registry": "docker.a-custom-host.com",
      "image_registry_org": "customer-1",
      "some_secret_id": "some-other-secret-name",
      "..."
    },
    "..."
  },
  "base_configuration": ["sdase", "customer-1", "..."],
  "..."

  "github_org": "{{ cookiecutter._environment_configuration[cookiecutter.base_configuration].github_org }}",
  "image_registry": "{{ cookiecutter._environment_configuration[cookiecutter.base_configuration].image_registry }}",
  "image_registry_org": "{{ cookiecutter._environment_configuration[cookiecutter.base_configuration].image_registry_org }}",
  "github_secret_id": "{{ cookiecutter._environment_configuration[cookiecutter.base_configuration].github_secret_id }}",
}

Instead, we would like to have the possibility of providing some kind of environment context to the Backstage instance that is available in each template:

A Scaffolder example

The context:

# app-config.yaml

scaffolder:
  additionalContext:
    githubOrg: SDA-SE
    imageRegistry: our.registry.com
    imageRegistryOrg: sdase
    someSecretId: some-secret-name

The scaffolder schema:

apiVersion: backstage.io/v1beta2
kind: Template
metadata:
  name: my-template
spec:
  parameters:
    - title: Configure Your Service
      required:
        - name
      properties:
        name:
          type: string
          title: Name
          description: The name of the service.
        githubOrg:
          type: string
          title: GitHub Organization
          # the parameters are templated by the '*/parameter-schema' endpoint
          default: '{{ ctx.environment.githubOrg }}'
          ui:readonly: true

  steps:
    - name: Render Service Template
      action: fetch:cookiecutter
      input:
        url: "template"
        values:
          project_name: '{{ parameters.name }}'
          # the environment is available in the handlebars context
          image_registry: '{{ ctx.environment.imageRegistry }}'
          image_registry_org: '{{ ctx.environment.imageRegistryOrg }}'
          some_secret_id: '{{ ctx.environment.someSecretId }}'

Proposal

1. Add a new parameter to the scaffolder router. An integrator might decide to source this from the config:

  export interface RouterOptions {
    logger: Logger;
    config: Config;
    reader: UrlReader;
    database: PluginDatabaseManager;
    catalogClient: CatalogApi;
    actions?: TemplateAction<any>[];
    taskWorkers?: number;
    containerRunner: ContainerRunner;
+   templateEnvironmentContext?: Record<string, string>;
  }

2. Decide a handlebars variable name such as {{ ctx.environment.<property-name> }}.

3. Provide the context as handlebars variable to the parameters schema:

# ...
  parameters:
    - title: ...
      properties:
        my-property:
          type: string
          # the parameters are templated by the '*/parameter-schema' endpoint
          default: '{{ ctx.environment.some-setting }}'
          ui:field: GitHubGroupPicker
          ui:options:
            # could be used to configure custom fields
            organizationUrl: 'https://github.com/{{ ctx.environment.githubOrg }}'
# ...

4. Provide the settings as handlebars variable to the steps:

# ...
  steps:
    - name: Render Service Template
      action: fetch:cookiecutter
      input:
        url: "template"
        values:
          # the environment is available in the handlebars context
          image_registry: '{{ ctx.environment.imageRegistry }}'

Alternatives

Alternative 1: Custom Action

Skip templating the parameters-schema but provide the context as a custom action:

apiVersion: backstage.io/v1beta2
kind: Template
metadata:
  name: my-template
spec:
  #...

  steps:
    - name: Access environment context
      id: context
      action: get-context

    - name: Render Service Template
      action: fetch:cookiecutter
      input:
        url: "template"
        values:
          # the environment is available in the handlebars context
          image_registry: '{{ steps.context.outputs.<property> }}'

If an integrator wants to access the context in the UI, one could wrap a frontend field extension that automatically sets certain settings prior to executing the original field extension (example: SdaOwnerPicker vs. OwnerPicker).

Or we could write a custom express middleware for our instance, that hooks into the parameters-schema endpoint and post processes it before returning the response.

Alternative 2: New apiVersion

Do a larger refactoring and move parameters into a step:

apiVersion: backstage.io/v1beta3
kind: Template
metadata:
  name: my-template
spec:
  steps:
    - name: Access environment context
      id: context
      action: get-context

    - name: Configure Your Service
      action: input:form
      input:
        required:
          - name
        properties:
          name:
            type: string
            default: {{ steps.context.outputs.<property> }}
          # ...

    - name: Render Service Template
#...

This could also allow user inputs after some actions have been executed. Examples are: manually trigger a deployment, wait for the merge of a PR, …

Risks

There settings are not part of the template and can't be validated. It is also not possible to document them somewhere. There is the /actions route that could show a list of these environment specific settings.

Some company might want to use this feature to provide secrets via this new context. However, as they are also accessible in the parameters-schema, there is a risk of (accidentally) spoiling them to the frontend. This would be safer in e.g. Alternative 1 where settings for the frontend must be present as a config prop with @visibility: frontend.

We would love to hear your thoughts on this topic 😃.

[dhenneke]

Some updates: When I wrote the "Risks" section above, I already noticed that I actually don't really like my proposal. So we went with implementing Alternative 1 instead. We created a new sdase:getEnvironment with a proper output schema so all properties are visible in /create/actions.

More tricky is the idea of wrapping existing field-extensions since the return value of createScaffolderFieldExtension(...) doesn't contain a reference to the original React Element. But we only needed this feature in our custom fields, yet.

But something like the following would work:

import { getComponentData } from '@backstage/core-plugin-api';
import {
  createScaffolderFieldExtension,
  EntityPickerFieldExtension,
  FieldExtensionOptions,
  scaffolderPlugin,
} from '@backstage/plugin-scaffolder';
import { FieldProps } from '@rjsf/core';
import { merge } from 'lodash';
import React from 'react';

export const CustomEntityPicker = ({
  uiSchema,
  ...props
}: FieldProps<string>) => {
  const componentData = getComponentData(
    <EntityPickerFieldExtension />,
    'scaffolder.extensions.field.v1',
  ) as FieldExtensionOptions;

  if (componentData?.component) {
    return (
      <componentData.component
        {...merge(
          {
            uiSchema: {
              'ui:options': {
                allowedKinds: ['API', 'Template'],
                defaultKind: 'API',
              },
            },
          },
          props,
        )}
      />
    );
  }

  // TODO: some error!
  return <></>;
};

export const CustomEntityPickerFieldExtension = scaffolderPlugin.provide(
  createScaffolderFieldExtension({
    component: CustomEntityPicker,
    name: 'CustomEntityPicker',
  }),
);

Maybe we could think about improving the composability of ScaffolderFieldExtensions?

[freben]

I am not sure how I personally could not have seen this before. But just acknowledging it now, so it doesn't seem unattended to :)

[dhenneke]

Let's close this. We are quite happy with our selected approach and don't think extra changes are needed if no one else runs into this particular problem.