[Feedback Wanted] Evaluate scaffolder templates through dry-run API

[Pike]

Status: Open for comments

Need

Template developers want to test their in-development templates against their installation of backstage.

Proposal

Have a CLI script that uses the dry-run API of a backstage installation, and serialize the created workdir to local disk for the developer to test.

With some auth trickery, this could also be used for CI on template repositories.

Check out a possible variant of such a script in the contrib documentation, https://github.com/backstage/backstage/tree/master/contrib/scaffolder.

Alternatives

• Just rendering a template with nunjucks.
  • This would not utilize all the non-fetch-template parts
• Write a wrapper around the actual dry-run logic to be executed locally.
  • This would require some way to get all custom steps to execute, and would require quite a bit of mocking of backstage infrastructure.

Risks

This doesn't actually test the survey/questionnaire part of the template.

[freben]

Hi,

Just checking, are you aware of the /create/edit path that's typically available on your Backstage installation? Also reachable from the top right "kebab menu" in the Create section.

Does this correlate with what you need?

[Pike]

That's the functionality that the work is based on, but in an automatable fashion.

I've not done a lot of testing of the editor, as it often bailed on the upload limit for our instance and templates. Once we update to 1.7, I'll do more testing. E.g., I just tested downloading the edited files, and the .git directory got busted ;-) . They're excluded in the 1.7, so I'll wait for that.

The way I think about it is that the editor is really going to shine if you're editing the questionnaire, while the cli tool would shine if you made edits on the template skeleton, and you want to verify that linters and tests and all that work for a given input.

And then there's the CI part, and we did contemplate to use something like cypress to go through the UI on the edit interface. But then we'd need to see how to log a CI user in, which I ran away from.

[jhaals]

Can this be closed with #13991 merged in?

[Pike]

Can this be closed with #13991 merged in?

This issue is to collect feedback on that document, so no.

[Rugvip]

Forgot about the relation between things here, got a bit confused by the format I think x) I updated the title to hopefully be a bit more clear about the intent. Perhaps it would be best to update the issue body with references to the code in contrib as well?

[haho16]

Hey all,
New to Backstage. Still learning and experimenting with the scaffolder templating. Quick doubt about this:
Is that a standalone tool or does it have a dependency to be run from within a Backstage node? I.e. does it have some system/node dependencies?

[Pike]

Yes, it does have node.js dependencies, but it's not required to be run on a machine with backstage installed. To avoid saying node twice for different things. The machine you run it on does need network access to the machine running backstage, though. So it's low on local dependencies, but whether you call that stand-alone or not would be up to you ;-)

[haho16]

Hey, @Pike, basically I am having issues with running this on local machine, due to following errors:

import { dirname, join, relative } from 'node:path';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:360:18)
    at wrapSafe (node:internal/modules/cjs/loader:1088:15)
    at Module._compile (node:internal/modules/cjs/loader:1123:27)
    at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:23:47

I suspected I am missing a package.json file whereby I specify something like:

{
  "type": "module"
}

and then I get

import type { ScaffolderDryRunResponse } from '@backstage/plugin-scaffolder';
            ^
SyntaxError: Unexpected token '{'
    at Loader.moduleStrategy (internal/modules/esm/translators.js:145:18)
    at async link (internal/modules/esm/module_job.js:64:21)

I know I am doing something wrong or missing some npm dependencies, but not sure which?

I am on Node.js v18.12.1. I am running it in the following way:

node scaffolder-dry.js http://remotehost/ <template-directory> template.yml <output-directory>

Excuse my ignorance if it is something obvious :)

[brandonphan]

Would also like some clarification on how to use the script, not super familiar with typescript and couldn't get it working while messing around with it. My apologies if something is obvious.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Pike]

Let's keep this open.

[jessebye]

For a bit more context, with Cookiecutter we had a simple way to test templates locally: just run cookiecutter and examine the output. This was critical because when building new templates, you need to iterate rapidly and don't want to add any extra steps or dependencies like running the template through an active Backstage instance.

We also used Cookiecutter in CI pipelines to build some basic tests. At least, making sure the template compiled and didn't have any errors.

With the switch to nunjucks, we don't have any good way to test locally or in CI. This is a big step backward for us in terms of functionality.

[Gladdstone]

Could really use some more details on how this is meant to be run. Not clear on why this is a .md. As a .ts the path to the package.json may have changed since this was created, but the package.json it appears to be referencing is not configured properly to execute this as a module. Not currently able to get this to function.

[rmartine-ias]

@Gladdstone I got it working, here's how I modified it: https://gist.github.com/rmartine-ias/4fe26f27b6d520806bd26fa0f1607fa2

If this is missing anything, let me know. It's not super pretty, but it runs.

yarn install, and run with
npx ts-node --esm src/index.ts http://localhost:7007/ path/to/template/folder values.yml out --token "$FRONTEND_TOKEN" (also kind of hacky, but it runs)

I got FRONTEND_TOKEN by going to backstage in my browser, and inspecting the request headers to authenticated resources (e.g. asking about entities) -- copy the JWT after "Bearer: " and then export FRONTEND_TOKEN="whatever". values.yml should include all required values for the parameters step.

[electrofelix]

I've been bumping into an issue recently using the template editor directly, it appears that when using replace(' ', '-') to set up a value, trying to click Create in the preview to run through the scaffolder dry-run results in a 403 error. Switching this to replace(' ', '\-'), gets through fine and it appears that using the escaping has no ill effects when done for real.

The unescaped form works fine when using templates for real, it only appears to cause an issue when sent to the dry-run endpoint. I'm guessing a serialisation issue perhaps? Using roadie and they confirmed they weren't seeing anything in their logs regarding the forbidden error.

Happy to report this as a separate issue, just figured some people might end up i this thread.

Btw, having this is fantastic, so I'm hopeful that work will continue and build out a CLI tool to support along the lines of the scripts referenced above.

[freben]

@electrofelix Could you make a separate issue with some reproduction steps for that?

[sp71]

Any chance we can increase the priority for this CLI to be official? It's an amazing tool

[benjdlambert]

@sp71 yep it's something that's on our radar, I think we probably want to wait for some of #15999 before to settle before taking on some of the CLI work. Thinks like authentication etc need to be settled for CLI's first.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rmartine-ias]

not stale, still want this

[Rugvip]

@Pike what do you think about moving this to an official @backstage/plugin-scaffolder-cli package? (👀 @benjdlambert)

[Pike]

Sounds like a good idea to me. There might be issues with the permissions framework, but I'm not sure if they're bound to me often folding the backend token back in. I've got a local branch where I remove both the identity and the permissions from my local backend plugin.

Another thing I end up doing often is to actually create a git commit. The publish:github is getting skipped, IIRC, so your local setup isn't fully matching.

Also the user is different in dry run.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Pike]

naggitynagnag

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Pike]

naggitynagnag

[benjdlambert]

@Pike that aught to keep the stalebot away. Still blocked on the auth issue at the minute though, which we should hopefully get some time in Q1 to start taking a look at.