chore: Allow external extensions to include their own package.json files

[kgabryje]

SUMMARY

Superset supports external frontend extensions by registering them in superset-frontend/src/setup/setupExtensions.ts. However, the extensions could not easily add their own dependencies to superset's dependencies. This PR allows external extensions to install new dependencies by including package.json file within superset-frontend/src/setup tree (not necessarily root level, it will work with path such as superset-frontend/src/setup/my-cool-extension-dir/package.json).

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

1. Run npm install or npm ci when there is no package.json file under src/setup dir - there should be no changes to node_modules nor package-lock.json
2. Create a "fake" extension dir under src/setup and create a package.json file in it, e.g. src/setup/my-extension/package.json. Add some dependencies to it and run npm install or npm ci. Verify that the new dependencies were installed correctly

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[john-bodley]

@kgabryje how do we perceive handling dependencies conflicts (due to custom extensions) with Superset dependencies?

CC: @justinpark

[kgabryje]

@kgabryje how do we perceive handling dependencies conflicts (due to custom extensions) with Superset dependencies?

@john-bodley The way we do it in Preset is we use Superset's dependencies as extension's peer dependencies and generate a package-lock.json. That ensures that there are no new deps added.
However, if you don't do that and there are some conflicting dependencies (like your extension has a dependency that uses react 18 as its dependency), then a separate node_modules for your extension will get generated, which will contain the deps that it could not resolve from the main superset's package-lock.

[john-bodley]

@justinpark and @michael-s-molina you're probably better suited for processing @kgabryje's #26004 (comment).

[michael-s-molina]

@kgabryje I noticed that we have many setup files in src/setup that are not related to extensions but general setup of Superset. What do you think about creating a src/extensions folder to put the extensions? This folder could contain a README file with instructions on how to setup the extensions, how to handle dependencies, etc.

[kgabryje]

@kgabryje I noticed that we have many setup files in src/setup that are not related to extensions but general setup of Superset. What do you think about creating a src/extensions folder to put the extensions? This folder could contain a README file with instructions on how to setup the extensions, how to handle dependencies, etc.

I agree that extensions deserve their own folder, but I'm afraid that such change of structure might be a breaking change for some orgs. For example, Preset's build process assumes that extension-related files are in src/setup. I propose we add this refactor to a list of tasks for 4.0

[michael-s-molina]

LGTM