Provide some default values + secrets for the Scaffolder context

[benjdlambert]

Following on from the closed #6632, it would be great if we could define some default values that can be used in the template manifest (template.yaml) and also in fetch:template.

Feature Suggestion

The aforementioned issue was closed because the solution of having an action that could use ctx.output('config', configApi.get('scaffolder.environment')); was enough to satisfy the request.

The template author could use the output of one action to template values into the next using ${{ ctx.outputs.myStep.blah }} to template that into a succeeding action.

The problem that we have with that solution is there is no first class support for secrets, and how to have some default secrets context that can be used in the templating manifest file or in the actions under ctx.secrets.

Possible Implementation

So first off, I'm contemplating removing the ctx.secrets way of accessing secrets in actions, and thinking that it would be better if these things were passed in through inputs to the action and are just templated in rather than the action assuming that secrets exist. It's going to make action writing a little less coupled to the context and things that might or might not exist there. Can probably address this issue separately though.

In order to supply some default parameters, and default secrets, we could go two ways. One I think is simpler, and one uses some more of the more known capabilities of config that backstage integrators and plugin developers might be aware of.

Open Config approach

We could have the option to provide a default environment that would get injected into the nunjucks templating in app-config.yaml

# app-config.yaml 
...
scaffolder:
  defaultEnvironment:
    parameters:
      region: eu-west-1
    secrets:
      AWS_ACCESS_KEY: ${AWS_ACCESS_KEY}
...

The above is a pretty simple config, and pretty clear distinction between secrets and parameters that could provide some defaults at the templating level in template.yaml.

${{ parameters.region }} would be resolved to eu-west-1 unless the parameters in the template.yaml had an option to override that. And the same for ${{ secrets.AWS_ACCESS_KEY }}, unless this was overridden when creating the task by using the SecretsContext or/and a CustomFieldExtension.

config.d.ts approach

It would also be possible for developers to just define a defaultEnvironment, and the keys under that object would be subject to secret or backend visibility part of a config.d.ts.

Usage

The idea here, is that in the betav3 versions of templates, you could set up your defaultEnvironment either way, and be able to do things like this:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: v1beta3-demo
  title: Test Action template
  description: scaffolder v1beta3 template demo
spec:
  owner: backstage/techdocs-core
  type: service

  parameters:
    ...

    - title: Choose a location
      properties:
        region:
          title: Region
          type: string
    ...

  steps:
    ...

    - id: template
      name: Create Repo
      action: fetch:template
      input:
        region:  ${{ parameters.region }}
    - id: push
      name: Push to S3
      action: publish:s3
      input:
        region:  ${{ parameters.region }}
        accessToken: ${{ secrets.AWS_ACCESS_TOKEN }}
    ...

[benjdlambert]

As an aside here also, and maybe this changes the design a little bit, what if we were able to set default values that you could also use as default values in the parameters block. Maybe we don't want to store the default values on the parameters namespace then, as it's gonna get super confusing for people as to what values they can and can't use in the template schema, only being allowed to use parameters in the parameters bit if it's a default value etc.

Maybe we could have something like:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: v1beta3-demo
  title: Test Action template
  description: scaffolder v1beta3 template demo
spec:
  owner: backstage/techdocs-core
  type: service

  parameters:
    ...

    - title: Choose a location
      required: ['region']
      properties:
        region:
          title: Region
          type: string
          default: ${{ environment.region }}
    ...

  steps:
    ...

    - id: template
      name: Create Repo
      action: fetch:template
      input:
        region:  ${{ parameters.region }}
    - id: push
      name: Push to S3
      action: publish:s3
      input:
        region:  ${{ parameters.region }}
        accessToken: ${{ secrets.AWS_ACCESS_TOKEN }}
    ...

And have the default values under environment instead

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[peimanja]

still valid

[ivd-git]

For us this is also of interest. Our use case is providing the name of the GitHub org in which to create a new repository to our scaffolding template.

When we could configure the GitHub org in app-config.yaml and use its value in the scaffolder template, then that template can be re-used without modification between dev / test and prod.

I believe in our case the Open Config approach might be sufficient. With the remark that I think exposing those values as ${{ environment.xxx }} would be the nicest option.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[peimanja]

still valid

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[peimanja]

still valid

[benjdlambert]

Also going to tie this together with #16275 for the templating part, might be something we do together.

[benjdlambert]

Related #16996

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dfl-aeb]

still needed

[Raffaello-Ferrari-NTT]

still valid

[benjdlambert]

Gonna add this to the #10430 but it's probably going to come after the release. We ideally want a separate backlog or roadmap for the scaffolder plugin. Watch this space!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[supergnomey]

Still very much validito.

[regicsolutions]

One workaround for this is to just set default values and using either the ui:readonly true if you would like to see the value or ui:widget: hidden if you don't

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: v1beta3-demo
  title: Test Action template
  description: scaffolder v1beta3 template demo
spec:
  owner: backstage/techdocs-core
  type: service

  parameters:
    ...

    - title: Choose a location
      properties:
        region:
          type: string
          # Hidden Option
          ui:widget: hidden 
          # Read Only Option
          ui:readonly: true 
          default: us-west

    ...

  steps:
    ...

    - id: template
      name: Create Repo
      action: fetch:template
      input:
        region:  ${{ parameters.region }}
    ...

[kyleshrader]

One workaround for this is to just set default values and using either the ui:readonly true if you would like to see the value or ui:widget: hidden if you don't

I don't think this is a workaround for cases such as the one mentioned by @ivd-git . We have the same needs, in which app-config.local.yaml could refer to a specific organization but the deployed instance of backstage could refer to another, removing the need for updating the template between local development and merging for release.

Another need we have in a solution is that it could be used for templating the inputs. environment === dev might provide enum values; where as prod might provide other enum values for input.

[floric]

We consider a pretty complex solution right now to resolve the template by ourself with custom code and then use sed based rules to replace environment specific values. I also checked the resolve logic in Backstage for templates but it's pretty difficult to expose this logic as a plugin endpoint for resolving templates directly with the correct logic in Backstage. Maybe someone has a better idea instead of reimplementing it?