feat(cli): support dynamic imports from cjs

[sennyeya]

Hey, I just made a Pull Request!

might fix #8242?

Context: I ran into the ESM module support issue while testing out ink + a new Backstage CLI. It would be great to start the ball rolling on adding support and this seems like the first blocker.

This PR adds a new plugin that handles rollup's resolveDynamicPlugin and renderDynamicPlugin methods. Currently, I enabled it just for non-relative files as the bundler doesn't pick up relative .mjs files directly. Given the context of where this would be used, I doubt that that is a necessary usecase as well. I did see that the commonjs plugin is set up to run against just node_modules,

       commonjs({
          include: /node_modules/,
          exclude: [/\/[^/]+\.(?:stories|test)\.[^/]+$/],
        }),

Not sure if that impacts this work. From what I can tell, the resolutions for the dynamic imports work despite this, but it may require further investigation. Interestingly, I only needed to change the build config, and was able to leave the webpack config untouched.

Testing plan: I'm not quite sure what else should be tested here. I ran both the build and start commands with dynamic imports and they worked as expected (like the snapshot in the rollup test). Would love ideas here.

✔️ Checklist

• A changeset describing the change and affected packages. (more info)
• Added or updated documentation
• Tests for new functionality and regression tests for bug fixes
• Screenshots attached (for UI changes)
• All your commits have a Signed-off-by line in the message. (more info)

[backstage-goalie]

Missing Changesets

The following package(s) are changed by this PR but do not have a changeset:

• @backstage/backend-dynamic-feature-service
• @backstage/repo-tools
• @backstage/plugin-app-backend
• @backstage/plugin-auth-backend-module-guest-provider
• @backstage/plugin-catalog-backend-module-unprocessed
• @backstage/plugin-catalog-backend

See CONTRIBUTING.md for more information about how to add changesets.

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/backend-dynamic-feature-service	packages/backend-dynamic-feature-service	none	v0.2.10-next.2
example-backend	packages/backend	none	v0.0.26-next.1
@backstage/cli	packages/cli	minor	v0.26.5-next.1
@backstage/repo-tools	packages/repo-tools	none	v0.9.0-next.2
@backstage/plugin-app-backend	plugins/app-backend	none	v0.3.66-next.1
@backstage/plugin-auth-backend-module-guest-provider	plugins/auth-backend-module-guest-provider	none	v0.1.4-next.1
@backstage/plugin-catalog-backend-module-unprocessed	plugins/catalog-backend-module-unprocessed	none	v0.4.5-next.2
@backstage/plugin-catalog-backend	plugins/catalog-backend	none	v1.22.0-next.2

[github-actions]

Uffizzi Ephemeral Environment - Virtual Cluster

☁️ deploying cluster pr-22638

⚙️ Updating now by workflow run 8608047205.

Download the Uffizzi CLI to interact with the upcoming virtual cluster
https://docs.uffizzi.com/install

[sennyeya]

The E2E test failure is definitely related to this work,

[yarn new --select plugin --option id=test].err: TypeError: actionFunc is not a function

I'll investigate this week.

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[Rugvip]

Awesome! 🎉 👏

Big step towards module support. Gonna play it safe and ship this after the next main-line.

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[freben]

@Rugvip this was still of interest right

[Rugvip]

@freben yep 👍

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[aramissennyeydd]

@Rugvip (sorry for the late response here) I think I misread the rollup option and assumed it was doing both imports. When I was initially working on this, I noticed some fairly odd behavior around relative dynamic imports with rollup. I just pushed a commit with such a broken state in plugins/catalog-backend and packages/backend-next. The output of plugins/catalog-backend doesn't respect the dynamic import and in fact converts my MJS to CJS, packages/backend-next explodes with a ERR_UNSUPPORTED_DIR_IMPORT error. I'm pretty squarely out of my depth here 🙃, so not too sure what good next steps are here. Was thinking of opening issues outlining what I was running into in both projects.

[Rugvip]

@aramissennyeydd alright, thank you for the update 👍

Could let it sit for the time being if you want, I might poke around at this some more at some point too.

Regarding Webpack we'll get rid of that in not too long for backend builds, it'll be part of deprecating support for the old backend system.

[aramissennyeydd]

Ah, sorry, had meant rollup 😅 . I'll try and wade through the rollup/webpack/swc miasma next week and see if I can come up with anything. The mismatch between documented options and output is bugging me a little.

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[aramissennyeydd]

@Rugvip I think I'm getting closer, the SWC maintainer pointed me to an ignoreDynamic option that keeps the imports as is - which breaks some functionality on our end 😅 . Namely the CLI importing pattern and allowing relative dynamic imports in the backend definition,

lazy(() => import('./start').then(m=>m.command))

and

backend.add(import('./authGithubProvider'))

Those were both being compiled to require statements with the correct file mappings (not necessarily the same as the source files). I don't think we can have it both ways of using imports and expecting some compiled and some uncompiled code.

It also means we'll have to adjust package definitions to better support ESM resolution, TS is not well behaved here (TypeStrong/ts-node#1997). The error on our side looks like,

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /Users/aramis.sennyey/Projects/backstage/backstage/plugins/catalog-backend-module-scaffolder-entity-model/src/index.ts

I did to a few PRs here, basically adjusting the main field to point to a dist/file.cjs.js file instead of the TS allows us to use dynamic imports. Unfortunately, that would end up being a breaking change as you couldn't use any plugins with dynamic import that don't have this correctly defined. Open to ideas here, I know there's a similar in between step of publishConfig which points to these files as you would expect.

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!