feat(install-dynamic-plugins): port from Python to TypeScript/Node.js#4574
Open
gustavolira wants to merge 10 commits intoredhat-developer:mainfrom
Open
feat(install-dynamic-plugins): port from Python to TypeScript/Node.js#4574gustavolira wants to merge 10 commits intoredhat-developer:mainfrom
gustavolira wants to merge 10 commits intoredhat-developer:mainfrom
Conversation
Code Review by Qodo
1.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Review Summary by QodoPort install-dynamic-plugins from Python to TypeScript/Node.js with performance improvements
WalkthroughsDescription• **Complete rewrite**: Replaces Python-based install-dynamic-plugins.py with a TypeScript/Node.js
implementation (18 modules, 105 Jest tests)
• **Performance improvements**: Incorporates parallel OCI downloads, shared image cache, and
streaming SHA verification from the install-dynamic-plugins-fast.py POC
• **New package structure**: scripts/install-dynamic-plugins/ with TypeScript sources, bundled to
single dist/install-dynamic-plugins.cjs (~412 KB)
• **Runtime contract unchanged**: Same dynamic-plugins.yaml input schema, same output format, same
lock-file behavior, same {{inherit}} semantics
• **Resource-conscious concurrency**: Respects cgroup CPU limits with default workers `max(1,
min(floor(cpus/2), 6)), configurable via DYNAMIC_PLUGINS_WORKERS`
• **Memory-efficient**: Streaming tar extraction and SHA verification; typical 10-plugin run uses
20–80 MB peak RSS
• **Security parity**: Path traversal prevention, zip-bomb detection, symlink validation, device
file rejection, SRI integrity verification, registry fallback
• **CI updated**: Replaced pytest with npm run tsc && npm test plus bundle freshness
verification
• **Container image**: Updated Containerfile to copy .cjs bundle instead of .py; wrapper shell
script execs node instead of python
• **Deleted**: install-dynamic-plugins.py (1288 LOC), test_install-dynamic-plugins.py (3065
LOC), pytest.ini
• **Documentation**: Comprehensive README, updated user docs and CI references
Diagramflowchart LR
A["Python Script<br/>install-dynamic-plugins.py"] -->|"Replaced by"| B["TypeScript/Node.js<br/>18 modules + 105 tests"]
B -->|"Bundled to"| C["dist/install-dynamic-plugins.cjs<br/>~412 KB"]
C -->|"Copied in"| D["Container Image<br/>Containerfile"]
E["Parallel OCI<br/>Downloads"] -->|"Incorporated"| B
F["Shared Image<br/>Cache"] -->|"Incorporated"| B
G["Streaming SHA<br/>Verification"] -->|"Incorporated"| B
H["pytest"] -->|"Replaced by"| I["npm test<br/>Jest 105 tests"]
File Changes1. scripts/install-dynamic-plugins/src/index.ts
|
Contributor
|
The container image build workflow finished with status: |
Contributor
|
my browser cannot even load the 11,5k-line file 😅 |
Member
Author
@rostalan No need to review that file, it's the auto-generated esbuild bundle of src/. Just pushed a commit marking it as linguist-generated so GitHub collapses it in the diff. Only src/ and tests/ need review; CI verifies the bundle is up-to-date on every push. |
Contributor
|
The container image build workflow finished with status: |
gustavolira
added a commit
to gustavolira/rhdh
that referenced
this pull request
Apr 13, 2026
… feedback Addresses review feedback on PR redhat-developer#4574: - Extract shared helpers into `src/util.ts`: `fileExists`, `isInside`, `isPlainObject`, `isAllowedEntryType`, `markAsFresh`. Removes 4x duplication across `index.ts`, `catalog-index.ts`, `installer-oci.ts`, `merger.ts`, and `tar-extract.ts`. - Unify `catalog-index.ts` tar filter with `tar-extract.ts`: oversized entries now throw `InstallException` instead of being silently dropped; `OldFile` and `ContiguousFile` are accepted (were previously excluded by mistake). Uses the `isAllowedEntryType` helper. - Extract `markAsFresh(installed, pluginPath)` helper used by both installers to drop stale hash entries after a successful install. - `installer-npm.ts`: use `npm pack --json` instead of parsing the last line of text stdout (warnings on stdout would shift the filename). Also simplify the integrity-check flow — one gate that throws on missing hash, then one verify call (was two overlapping conditionals). - Split `Plugin` → `PluginSpec` (YAML schema) + `Plugin` (internal, with `_level`/`plugin_hash`/`version`). Makes it explicit which fields originate from user YAML vs runtime state. - `lock-file.ts`: add `DYNAMIC_PLUGINS_LOCK_TIMEOUT_MS` (default 10 min) so a stale lock from a `kill -9`'d process no longer wedges the init container forever. New test covers the timeout path. - Drop the broken `lint:check` script — it had `|| true` silencing every lint error and there is no ESLint config in the package. - `README.md`: remove stale reference to non-existent `cli.ts`, document the new lock-timeout env var, mention `util.ts`. Tests: 115 (was 114). Bundle rebuilt (413.4 KB).
Contributor
|
The container image build workflow finished with status: |
Contributor
|
The container image build workflow finished with status: |
gustavolira
added a commit
to gustavolira/rhdh
that referenced
this pull request
Apr 13, 2026
Resolves the 21 issues flagged by SonarQube on PR redhat-developer#4574: Prototype pollution (CodeQL, merger.ts) - `deepMerge` now assigns via `Object.defineProperty` (bypasses the `__proto__` setter on `Object.prototype`) in addition to the existing `FORBIDDEN_KEYS` guard. CodeQL recognizes this pattern. Redundant type assertions - `index.ts:180`: drop `pc as Record<string, unknown>` — use the `isPlainObject` type guard already imported from `util.ts`. - `installer-npm.ts:37`, `installer-oci.ts:35`: replace `(plugin.pluginConfig ?? {}) as Record<string, unknown>` with a typed local variable. - `installer-oci.ts:41,51,71,78`: drop `as string` casts by restructuring the `isAlreadyInstalled` helper with proper `undefined` checks. - `merger.ts:136-140`: replace `.slice(-1)[0] as string` with `.at(-1) ?? ''`. - `merger.ts:215`: `ReadonlyArray<keyof Plugin | string>` collapses to `ReadonlyArray<string>`. Cognitive complexity reductions - `installOciPlugin` (17 → ~10): extract `resolvePullPolicy` and `isAlreadyInstalled` helpers. - `mergeOciPlugin` (20 → ~12): extract `resolveInherit`. - `npmPluginKey` (16 → ~7): extract `tryParseAlias`, `isGitLikeSpec`, `stripRefSuffix`. - `ociPluginKey`: extract `autoDetectPluginPath`. Modern JS / readability (es2015-es2022) - `integrity.ts`: `charCodeAt` → `codePointAt` (es2015). - `oci-key.ts`: use `String.raw` for the regex pieces containing `\s`, `\d`, `\]`, `\\` instead of escaped string literals (es2015). - `oci-key.ts:escape`: `.replace(/.../g, ...)` → `.replaceAll(...)` (es2021). - `plugin-hash.ts`: pass an explicit code-point comparator to `sort` so deterministic-hash behavior is spelled out. `localeCompare` is NOT used — it varies per-locale and would break hash stability. All 115 tests still pass. Bundle rebuilt (415.1 KB).
Contributor
|
The container image build workflow finished with status: |
| * gives CodeQL the pattern it recognizes for prototype-pollution safety. | ||
| */ | ||
| function safeSet(dst: Record<string, unknown>, key: string, value: unknown): void { | ||
| Object.defineProperty(dst, key, { |
Member
Author
|
/review |
PR Reviewer Guide 🔍(Review updated until commit ee984d5)Here are some key observations to aid the review process:
|
Contributor
|
The container image build workflow finished with status: |
gustavolira
added a commit
to gustavolira/rhdh
that referenced
this pull request
Apr 13, 2026
… feedback Addresses review feedback on PR redhat-developer#4574: - Extract shared helpers into `src/util.ts`: `fileExists`, `isInside`, `isPlainObject`, `isAllowedEntryType`, `markAsFresh`. Removes 4x duplication across `index.ts`, `catalog-index.ts`, `installer-oci.ts`, `merger.ts`, and `tar-extract.ts`. - Unify `catalog-index.ts` tar filter with `tar-extract.ts`: oversized entries now throw `InstallException` instead of being silently dropped; `OldFile` and `ContiguousFile` are accepted (were previously excluded by mistake). Uses the `isAllowedEntryType` helper. - Extract `markAsFresh(installed, pluginPath)` helper used by both installers to drop stale hash entries after a successful install. - `installer-npm.ts`: use `npm pack --json` instead of parsing the last line of text stdout (warnings on stdout would shift the filename). Also simplify the integrity-check flow — one gate that throws on missing hash, then one verify call (was two overlapping conditionals). - Split `Plugin` → `PluginSpec` (YAML schema) + `Plugin` (internal, with `_level`/`plugin_hash`/`version`). Makes it explicit which fields originate from user YAML vs runtime state. - `lock-file.ts`: add `DYNAMIC_PLUGINS_LOCK_TIMEOUT_MS` (default 10 min) so a stale lock from a `kill -9`'d process no longer wedges the init container forever. New test covers the timeout path. - Drop the broken `lint:check` script — it had `|| true` silencing every lint error and there is no ESLint config in the package. - `README.md`: remove stale reference to non-existent `cli.ts`, document the new lock-timeout env var, mention `util.ts`. Tests: 115 (was 114). Bundle rebuilt (413.4 KB).
gustavolira
force-pushed
the
feat/install-dynamic-plugins-ts
branch
from
b76ee6a to
dd84f75
Compare
gustavolira
added a commit
to gustavolira/rhdh
that referenced
this pull request
Apr 13, 2026
Resolves the 21 issues flagged by SonarQube on PR redhat-developer#4574: Prototype pollution (CodeQL, merger.ts) - `deepMerge` now assigns via `Object.defineProperty` (bypasses the `__proto__` setter on `Object.prototype`) in addition to the existing `FORBIDDEN_KEYS` guard. CodeQL recognizes this pattern. Redundant type assertions - `index.ts:180`: drop `pc as Record<string, unknown>` — use the `isPlainObject` type guard already imported from `util.ts`. - `installer-npm.ts:37`, `installer-oci.ts:35`: replace `(plugin.pluginConfig ?? {}) as Record<string, unknown>` with a typed local variable. - `installer-oci.ts:41,51,71,78`: drop `as string` casts by restructuring the `isAlreadyInstalled` helper with proper `undefined` checks. - `merger.ts:136-140`: replace `.slice(-1)[0] as string` with `.at(-1) ?? ''`. - `merger.ts:215`: `ReadonlyArray<keyof Plugin | string>` collapses to `ReadonlyArray<string>`. Cognitive complexity reductions - `installOciPlugin` (17 → ~10): extract `resolvePullPolicy` and `isAlreadyInstalled` helpers. - `mergeOciPlugin` (20 → ~12): extract `resolveInherit`. - `npmPluginKey` (16 → ~7): extract `tryParseAlias`, `isGitLikeSpec`, `stripRefSuffix`. - `ociPluginKey`: extract `autoDetectPluginPath`. Modern JS / readability (es2015-es2022) - `integrity.ts`: `charCodeAt` → `codePointAt` (es2015). - `oci-key.ts`: use `String.raw` for the regex pieces containing `\s`, `\d`, `\]`, `\\` instead of escaped string literals (es2015). - `oci-key.ts:escape`: `.replace(/.../g, ...)` → `.replaceAll(...)` (es2021). - `plugin-hash.ts`: pass an explicit code-point comparator to `sort` so deterministic-hash behavior is spelled out. `localeCompare` is NOT used — it varies per-locale and would break hash stability. All 115 tests still pass. Bundle rebuilt (415.1 KB).
Contributor
|
The container image build workflow finished with status: |
Member
Author
|
/review |
|
Persistent review updated to latest commit ee984d5 |
Contributor
|
The container image build workflow finished with status: |
gustavolira
added a commit
to gustavolira/rhdh
that referenced
this pull request
Apr 13, 2026
…tions Two focus-area suggestions from the Qodo reviewer guide on redhat-developer#4574: 1. Path handling (index.ts) - Resolve `dynamic-plugins.yaml` to an absolute path at startup and log it — the cwd-dependency is now explicit in the operator log. - Resolve `includes[]` entries relative to the config file's directory (was implicitly relative to cwd). Absolute include paths are preserved untouched so existing deployments that pass a full path still work. 2. Segment-based path-traversal check (tar-extract.ts) - Replace the coarse `pluginPath.includes('..')` filter with a segment-based scan: split on `/` and `\\` and reject any segment that is empty, `.`, or `..`. Absolute paths are still rejected up front. - Benign filenames that happen to contain `..` inside a segment (e.g., `my..plugin/`) are now accepted; path-traversal via `foo/../bar`, leading `.`/`..` segments, or empty segments (`foo//bar`) is still blocked. Added 3 new Jest cases covering: - `foo/../bar` rejected - `./plugin-one` rejected - `foo//bar` rejected - `my..plugin` accepted (regression guard for the stricter check) Total tests: 117 (was 115). Bundle rebuilt.
Replace the Python implementation of install-dynamic-plugins (used by the RHDH init container to install dynamic plugins from OCI/NPM/local sources) with a TypeScript/Node.js implementation, distributed as a single bundled CommonJS file consumed directly by the runtime container. Why - Node.js 22 is already in the runtime image (it runs Backstage), so no new system dependencies are needed; Python is no longer pulled in for the init container path (the techdocs venv still uses python3.11 separately). - Easier to maintain alongside the rest of the TS/Node ecosystem in the repo. - Picks up the parallelization improvements from PR redhat-developer#4523 from the start (parallel OCI downloads, shared OCI image cache, streaming SHA verification). Runtime contract preserved - Same 'dynamic-plugins.yaml' input schema (includes, plugins, package, pluginConfig, disabled, pullPolicy, forceDownload, integrity). - Same output 'app-config.dynamic-plugins.yaml', plugin directory layout, dynamic-plugin-config.hash / dynamic-plugin-image.hash files, lock file. - {{inherit}} semantics, OCI path auto-detection from the io.backstage.dynamic-packages annotation, registry.access.redhat.com→quay.io fallback, and SRI integrity (sha256/sha384/sha512) all preserved. Resource-conscious for OpenShift init containers - Streaming tar extraction via node-tar (no full layers in memory). - Streaming SHA via node:crypto pipeline. - Worker count via os.availableParallelism()/2 capped at 6 (honours cgroup CPU limits); override with DYNAMIC_PLUGINS_WORKERS. - NPM 'npm pack' calls stay sequential to avoid registry throttling. Security checks ported with parity - Per-entry MAX_ENTRY_SIZE cap (zip bomb protection). - Symlink/hardlink realpath validation. - Device file / FIFO / unknown entry-type rejection. - 'package/' prefix enforced for NPM tarballs. - Plugin path traversal (.., absolute) rejected. Container build - build/containerfiles/Containerfile copies dist/install-dynamic-plugins.cjs instead of install-dynamic-plugins.py. The bundle (~412 KB, built by esbuild) is committed to the repo so the hermetic build doesn't need network access — same pattern as .yarn/releases/yarn-*.cjs. - install-dynamic-plugins.sh now execs 'node install-dynamic-plugins.cjs'. CI - .github/workflows/pr.yaml replaces 'pytest' with 'npm install && npm run tsc && npm test' (105 Jest tests with full parity to the pytest suite), plus a freshness check on dist/install-dynamic-plugins.cjs. Cleanup - Deletes install-dynamic-plugins.py (1288 LOC), test_install-dynamic-plugins.py (3065 LOC), pytest.ini. - Updates docs and AI rule references that pointed at the old Python script. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Marks `scripts/install-dynamic-plugins/dist/install-dynamic-plugins.cjs` as `linguist-generated=true` so GitHub collapses it by default in PR diffs and excludes it from language statistics. The 11.5k-line esbuild bundle is not meant to be reviewed line-by-line — reviewers should look at `src/` instead.
Three fixes + regression tests from automated PR review: 1. `MAX_ENTRY_SIZE` NaN fallback (Qodo bug #1) `Number(process.env.MAX_ENTRY_SIZE)` returned NaN for non-numeric values, which silently disabled the zip-bomb / oversized-entry size check (every `stat.size > NaN` evaluates false). Extracted to a `parseMaxEntrySize()` helper that uses `Number.parseInt` + validates the result is a finite integer >= 1, else falls back to the default. 2. Boundary-safe OCI plugin-path filter (Qodo bug #2) `extractOciPlugin` used `filePath.startsWith(pluginPath)` which also matched sibling directories with the same prefix (e.g. requesting `plugin-one` would also extract `plugin-one-evil/`). Now requires either exact match or a path-separator boundary. 3. Prototype-pollution guard in `deepMerge` (CodeQL finding) User-supplied YAML could contain `__proto__`, `constructor`, or `prototype` keys. `deepMerge` blindly copied them via `dst[key] =`, allowing prototype-chain pollution. These keys are now skipped in both `deepMerge` and `copyPluginFields`. Added regression tests: prefix-collision extraction, prototype-pollution merge, 7 tests for `parseMaxEntrySize` covering NaN/empty/negative/zero inputs. Total: 114 tests (was 105). Bundle rebuilt via `npm run build`.
… feedback Addresses review feedback on PR redhat-developer#4574: - Extract shared helpers into `src/util.ts`: `fileExists`, `isInside`, `isPlainObject`, `isAllowedEntryType`, `markAsFresh`. Removes 4x duplication across `index.ts`, `catalog-index.ts`, `installer-oci.ts`, `merger.ts`, and `tar-extract.ts`. - Unify `catalog-index.ts` tar filter with `tar-extract.ts`: oversized entries now throw `InstallException` instead of being silently dropped; `OldFile` and `ContiguousFile` are accepted (were previously excluded by mistake). Uses the `isAllowedEntryType` helper. - Extract `markAsFresh(installed, pluginPath)` helper used by both installers to drop stale hash entries after a successful install. - `installer-npm.ts`: use `npm pack --json` instead of parsing the last line of text stdout (warnings on stdout would shift the filename). Also simplify the integrity-check flow — one gate that throws on missing hash, then one verify call (was two overlapping conditionals). - Split `Plugin` → `PluginSpec` (YAML schema) + `Plugin` (internal, with `_level`/`plugin_hash`/`version`). Makes it explicit which fields originate from user YAML vs runtime state. - `lock-file.ts`: add `DYNAMIC_PLUGINS_LOCK_TIMEOUT_MS` (default 10 min) so a stale lock from a `kill -9`'d process no longer wedges the init container forever. New test covers the timeout path. - Drop the broken `lint:check` script — it had `|| true` silencing every lint error and there is no ESLint config in the package. - `README.md`: remove stale reference to non-existent `cli.ts`, document the new lock-timeout env var, mention `util.ts`. Tests: 115 (was 114). Bundle rebuilt (413.4 KB).
Sonar typescript:S5852 flagged `/^[A-Za-z0-9+/]+={0,2}$/` and `/=+$/` in
`integrity.ts` as ReDoS hotspots (super-linear backtracking risk). In
practice these are O(n) because the character classes are flat, but
Sonar is conservative about any `+`/`*` quantifier.
Replaces both regexes with `isBase64Shape()` (char-code scan, one pass)
and `stripTrailingEquals()` (char-code trim). Same linear complexity,
no regex engine involved, Sonar-clean.
Resolves the 21 issues flagged by SonarQube on PR redhat-developer#4574: Prototype pollution (CodeQL, merger.ts) - `deepMerge` now assigns via `Object.defineProperty` (bypasses the `__proto__` setter on `Object.prototype`) in addition to the existing `FORBIDDEN_KEYS` guard. CodeQL recognizes this pattern. Redundant type assertions - `index.ts:180`: drop `pc as Record<string, unknown>` — use the `isPlainObject` type guard already imported from `util.ts`. - `installer-npm.ts:37`, `installer-oci.ts:35`: replace `(plugin.pluginConfig ?? {}) as Record<string, unknown>` with a typed local variable. - `installer-oci.ts:41,51,71,78`: drop `as string` casts by restructuring the `isAlreadyInstalled` helper with proper `undefined` checks. - `merger.ts:136-140`: replace `.slice(-1)[0] as string` with `.at(-1) ?? ''`. - `merger.ts:215`: `ReadonlyArray<keyof Plugin | string>` collapses to `ReadonlyArray<string>`. Cognitive complexity reductions - `installOciPlugin` (17 → ~10): extract `resolvePullPolicy` and `isAlreadyInstalled` helpers. - `mergeOciPlugin` (20 → ~12): extract `resolveInherit`. - `npmPluginKey` (16 → ~7): extract `tryParseAlias`, `isGitLikeSpec`, `stripRefSuffix`. - `ociPluginKey`: extract `autoDetectPluginPath`. Modern JS / readability (es2015-es2022) - `integrity.ts`: `charCodeAt` → `codePointAt` (es2015). - `oci-key.ts`: use `String.raw` for the regex pieces containing `\s`, `\d`, `\]`, `\\` instead of escaped string literals (es2015). - `oci-key.ts:escape`: `.replace(/.../g, ...)` → `.replaceAll(...)` (es2021). - `plugin-hash.ts`: pass an explicit code-point comparator to `sort` so deterministic-hash behavior is spelled out. `localeCompare` is NOT used — it varies per-locale and would break hash stability. All 115 tests still pass. Bundle rebuilt (415.1 KB).
- `merger.ts isEqual` (complexity 20 → ~6): extract `isArrayEqual` and `isObjectEqual` helpers. Each branch now dispatches to a single- purpose function so the main `isEqual` is a flat type-dispatch instead of nested loops. - `plugin-hash.ts compareCodePoint`: replace the nested ternary `a < b ? -1 : a > b ? 1 : 0` with three early returns. Same behaviour, no nested conditional. 115 tests still pass.
…d context
The root `.dockerignore` has `**/dist` which also filters out the
committed esbuild bundle at
`scripts/install-dynamic-plugins/dist/install-dynamic-plugins.cjs`.
The runtime stage of `build/containerfiles/Containerfile` copies that
file, so the Hermetic Build Image job fails with:
no items matching glob
".../scripts/install-dynamic-plugins/dist/install-dynamic-plugins.cjs"
copied (1 filtered out using .dockerignore)
Add negation patterns so the bundle (and only the bundle) is
re-included in the build context, matching the `.yarn/releases/yarn-*.cjs`
pattern the repo already uses for committed bundled entry points.
…tions Two focus-area suggestions from the Qodo reviewer guide on redhat-developer#4574: 1. Path handling (index.ts) - Resolve `dynamic-plugins.yaml` to an absolute path at startup and log it — the cwd-dependency is now explicit in the operator log. - Resolve `includes[]` entries relative to the config file's directory (was implicitly relative to cwd). Absolute include paths are preserved untouched so existing deployments that pass a full path still work. 2. Segment-based path-traversal check (tar-extract.ts) - Replace the coarse `pluginPath.includes('..')` filter with a segment-based scan: split on `/` and `\\` and reject any segment that is empty, `.`, or `..`. Absolute paths are still rejected up front. - Benign filenames that happen to contain `..` inside a segment (e.g., `my..plugin/`) are now accepted; path-traversal via `foo/../bar`, leading `.`/`..` segments, or empty segments (`foo//bar`) is still blocked. Added 3 new Jest cases covering: - `foo/../bar` rejected - `./plugin-one` rejected - `foo//bar` rejected - `my..plugin` accepted (regression guard for the stricter check) Total tests: 117 (was 115). Bundle rebuilt.
gustavolira
force-pushed
the
feat/install-dynamic-plugins-ts
branch
from
a0032b4 to
add3f1d
Compare
Member
Author
|
/test ? |
Member
Author
|
/test e2e-ocp-helm-nightly |
Member
Author
|
/test ? |
|
@gustavolira: The following commands are available to trigger required jobs: The following commands are available to trigger optional jobs: Use DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Member
|
One thing I noticed in the review — the error handling behavior changed from the Python version. In Python, when a plugin fails to install, the script stops immediately. The config file is never written, nothing is cleaned up. It's messy but clearly "not finished". In the new TS version, when a plugin fails, the script continues with the rest. It writes The exit code 1 prevents the main container from starting, so in normal Kubernetes/Compose flow this is fine. But the problem is that you end up with a state that looks "almost valid" — most plugins installed, config file written, cleanup done. If someone troubleshooting manually restarts just the main container (skipping the init), or if the deployment doesn't enforce init container success, they could get a partially working instance with missing plugins. Also plugin dependencies could be an issue — a frontend plugin might be installed and configured while its backend dependency failed. The app would start but with broken UI. I think the idea of collecting all errors and showing them at once is great (much better than fix-one-restart-fix-one-restart with 50 plugins). But maybe don't write the config file when there are errors? That way you get the diagnostic benefit without leaving a valid-looking state on disk. |
Member
Author
@kadel tyvm for your review! Just pushed a fix that on any plugin failure we now skip both the Skipped cleanup too (not just the write) because otherwise the previous run's config would remain on disk referencing plugin dirs we just deleted, worse than the original bug. Added unit tests for the success and failure paths, and verified end-to-end with a bogus OCI ref: exit 1, no config written, prior sentinel preserved. |
5 tasks
Member
|
I took a moment to try this out locally by grabbing the dynamic-plugins.default.yaml from the latest catalog image and used this dynamic-plugins.yaml: Script worked fine but it didn't appear to me like it was much faster than the python script. So I opted to time it's execution as compared to the python script, it seems to take about the same amount of time, at least in my environment: I suppose most of this is spent doing |
Member
|
One other observation, flipping between scripts reveals that the two scripts use different methods to evaluate differences in existing dynamic-plugins-root content. I figured I'd do an additional test where I'd run each script with an already populated dynamic-plugins-root. For both the python script and this javascript version I actually had to run each script a couple of times before getting to a point where the scripts would not download and reinstall an existing plugin. I grabbed the execution time for this case once the contents of the dynamic-plugins-root folder settled: |
Member
Author
My test was using sanity-plugin check which loads every plugin that we are shipping |
jonkoops
reviewed
Apr 16, 2026
| run: yarn run test --continue --affected | ||
|
|
||
| - name: Run Python tests | ||
| - name: Test install-dynamic-plugins (TypeScript) |
Contributor
There was a problem hiding this comment.
I think mentioning the technology involved doesn't really provide value here.
Suggested change
| - name: Test install-dynamic-plugins (TypeScript) | |
| - name: Test install-dynamic-plugins |
| run: pytest scripts/install-dynamic-plugins/test_install-dynamic-plugins.py -v | ||
| working-directory: scripts/install-dynamic-plugins | ||
| run: | | ||
| npm install --no-audit --no-fund |
Contributor
There was a problem hiding this comment.
NPM has a dedicated command for installations on CI, see npm ci.
Suggested change
| npm install --no-audit --no-fund | |
| npm ci |
| - name: Verify install-dynamic-plugins bundle is up to date | ||
| if: ${{ steps.check-image.outputs.is_skipped != 'true' }} | ||
| working-directory: scripts/install-dynamic-plugins | ||
| run: | |
Contributor
There was a problem hiding this comment.
I see there is a build step to bundle all of this up into a single file. I assume that this is to reduce the size of the final image?
Would it not be acceptable to npm install --production to only install the production dependencies and then bring in the source + node_modules instead?
Note that Node.js also supports single executable applications, perhaps that would be fit for purpose here?
…exists + skip parallel for no-ops Three perf/correctness fixes targeting the no-op "already-installed" restart path that Stan Lewis benchmarked at 17.77s vs 13.72s for Python in redhat-developer#4574 (comment): 1. plugin-hash: byte-compatible with Python (correctness) The previous TS implementation used different field names and JSON separators than Python, so install hashes never matched across the migration — switching from Python to TS triggered a full reinstall of every plugin on the first run. Fixes: - Rename internal field `_level` → `last_modified_level` (Python's name) - Keep `last_modified_level` in the hash (Python keeps it, we used to strip) - Rename `_local` → `_local_package_info`, plus inner field renames (`_pj` → `_package_json`, `_pj_mtime` → `_package_json_mtime`, `_mtime` → `_directory_mtime`, `_missing` → `_not_found`, `_err` → `_error`) - Convert `mtimeMs` to seconds (Python's `os.path.getmtime` returns float seconds) - `stableStringify` now emits Python-style `, ` and `: ` separators so the serialized JSON is byte-identical to `json.dumps(..., sort_keys=True)` Two regression tests pin the hash to known Python-computed SHA256 values; if these break, cross-compat is silently lost. 2. Skopeo.exists: cache forks across calls (perf) `exists()` was the only Skopeo wrapper without an in-memory cache — `image-resolver.ts` calls it per OCI plugin to probe the RHDH registry. With N plugins from the same image, that's N redundant `skopeo inspect --no-tags` forks each restart. Now memoized like `inspect()` / `inspectRaw()` (promise cache for in-flight dedup). For a 30-plugin / 1-image catalog: 30 forks → 1. 3. installOci: skip parallel pool for definitely-no-op plugins (perf) The IF_NOT_PRESENT no-op path was already a fast hash lookup, but it went through `mapConcurrent` + Semaphore + Promise wrapping for every plugin. Added a synchronous pre-pass that filters out "hash present + not forced + pull-policy ≠ Always" plugins before spinning up the worker pool. Saves the entire Promise machinery for the common restart case where every plugin is steady-state. ALWAYS-pull plugins (`:latest!` or explicit `pullPolicy: Always`) still go through the regular install path because they need a `skopeo inspect` to compare local vs remote digest — that's where Fix #2's exists-cache delivers the win. 124 tests now (was 117): +5 Python-compat hash assertions, +2 Skopeo exists-cache assertions. Bundle 418.1 KB (was 415.7).
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the Python-based
install-dynamic-plugins.pyinit-container script with a TypeScript/Node.js implementation, incorporating all improvements from theinstall-dynamic-plugins-fast.pyPOC (#4523): parallel OCI downloads, shared image cache, streaming SHA verification.Motivation: The Python script was a longstanding duplication target for export overlays (see rhdh-plugin-export-overlays#2231). A Node.js implementation removes the Python dependency from the init-container runtime, enables reuse across RHDH core and overlays, and adopts the faster parallel architecture natively.
What changed
scripts/install-dynamic-plugins/— 18 TypeScript modules + 9 Jest test files (105 tests)dist/install-dynamic-plugins.cjs(~412 KB), committed and verified fresh in CI (same pattern as.yarn/releases/yarn-*.cjs).cjsbundle instead of.py; wrapper shell script execsnodeinstead ofpythonnpm run tsc && npm test+ bundle freshness check (replacespytest)install-dynamic-plugins.py(1288 LOC),test_install-dynamic-plugins.py(3065 LOC),pytest.iniKey design decisions
Runtime contract is unchanged
Same
dynamic-plugins.yamlinput schema, sameapp-config.dynamic-plugins.yamloutput, samedynamic-plugin-config.hash/dynamic-plugin-image.hashfiles, same lock-file behaviour, same{{inherit}}semantics and OCI path auto-detection.Resource-conscious concurrency
availableParallelism()respects cgroup CPU limits (init containers often get 0.5 CPU)max(1, min(floor(cpus/2), 6))— cap avoids exhausting registry/networkDYNAMIC_PLUGINS_WORKERS=<n>Memory: streaming everywhere
node-tarstreams extraction — no full-archive read into RAMnode:cryptopipeline for SHA integrity — chunks through the hashSecurity parity with Python
.., absolute)tar-extract.tsMAX_ENTRY_SIZE)tar-extract.ts,catalog-index.tstar-extract.tstar-extract.tspackage/prefix enforced for NPM tarballstar-extract.tssha256/sha384/sha512)integrity.tsregistry.access.redhat.com/rhdh→quay.io/rhdhimage-resolver.tsTest plan
npm run tscpasses (strict mode,noUncheckedIndexedAccess)npm test— 105 Jest tests pass (npm-key, oci-key, integrity, tar-extract, merger, concurrency, lock-file, image-resolver, plugin-hash)npm run buildproduces freshdist/install-dynamic-plugins.cjs.cjs(CI will verify)fast.pybaseline (~2:42 for full catalog)Compatibility
skopeois still installed for OCI inspectioninstall-dynamic-plugins.shwrapper contract unchanged (./install-dynamic-plugins.sh /dynamic-plugins-root)Related