OSGi cluster plugin deploy — Stage 2: eliminate the shared-load-folder restart race (Mode B)

[dsolistorres]

Follow-up to #36434 / PR #36503 (Stage 1).

Context

Stage 1 (#36434, PR #36503) fixed Mode A — the persistent failure where a cluster plugin upload wrote the new package to the shared osgi-extra.conf but no node ever restarted, so the bundle stayed unresolved forever. The restart decision is now taken under the osgi_restart_lock against each node's live export snapshot and can no longer be raced away.

Stage 1 deliberately left Mode B open. This issue covers it.

Problem still present after Stage 1 (Mode B — transient)

The user (felix) and system frameworks share /data/shared/felix/load, and fileinstall on every node watches that shared folder. When the origin node moves a freshly-uploaded plugin jar into the shared load folder, a remote node's fileinstall can auto-start that jar against its still-stale framework (whose system.packages.extra doesn't yet include the newly-exported package) before that node processes the pub/sub restart request — producing, transiently:

BundleException: Unable to resolve <bundle>: missing requirement
  osgi.wiring.package; (&(osgi.wiring.package=<pkg>)(version>=0.0.0))

It is self-healing (the node recovers once it restarts), but it still emits the exact error the customer reports, so it reads as a failure in the logs. Root cause: a shared load folder combined with independent per-node framework state and no version gate on when a node may start a bundle.

Goal

Separate persisting a config/plugin change from applying it on each node, so deployment is an idempotent, crash-recoverable operation that scales to N nodes and does not depend on timing:

• A node that crashes after the config is persisted but before it restarts should recover automatically on next boot.
• No node should start a bundle whose required exports are not yet live on that node.

Proposed design (for discussion)

1. Node-local load folder. Keep fileinstall, but make each node's load (fileinstall watch dir) node-local rather than shared. A node only populates its own load after it has applied the required exports (restarted). This removes the shared-folder race entirely. (Decision from Stage 1 planning: keep fileinstall, make load node-local.)
2. Shared monotonic "exports version". Persist a cluster-wide version that increments whenever osgi-extra.conf changes. Each node records the version it has applied.
3. Poll the version, not the upload folder. A node that sees the shared version ahead of its applied version restarts, marks the version applied, then installs the pending bundle(s) locally. Replaces the fragile upload-folder polling.
4. Verify exports are live before auto-start. Before starting a bundle from load, confirm the required exports are present in the current framework; otherwise defer until after the pending restart. (Directly addresses Mode B even before the folder is fully node-local.)
5. Consider cleaning the felix bundle cache on restart when exports changed (org.osgi.framework.storage.clean), or otherwise forcing re-resolution.

The liveExportedPackages snapshot introduced in Stage 1 is a natural stepping stone toward this apply-model.

Acceptance criteria

• On a 2+ node cluster sharing /data/shared, uploading a plugin that imports a not-yet-exported dotCMS package resolves to ACTIVE on all nodes with no wiring.package error in any node's log (not even transiently).
• Killing a node between "config persisted" and "restart" and bringing it back leaves it consistent (bundle ACTIVE, exports live) with no manual intervention.
• No dependency on debounce timing or poller interleaving for correctness.

Test plan

• Extend the 2-node Docker harness used to validate Stage 1 (reintroduce the OSGI_RESTART_REMOTE_DELAY_MILLIS fault-injection seam locally to force the remote node to lag) and assert no transient resolve error on the delayed node.
• Crash-recovery scenario as above.

Refs: #36434, #36503

[dsolistorres]

Scope clarification (from PR #36503 review)

The transient pre-restart window this issue tracks affects both remote nodes and the origin node, not only remote nodes as the description's Mode B framing implies.

Origin-node case: after the upload is processed, the jar is moved into the shared, fileinstall-watched load folder before the async cluster-wide restart republishes the new exports. The origin node's own fileinstall can therefore briefly auto-start the bundle against its still-stale framework and log the wiring.package=…(version>=0.0.0) error until its (immediately submitted) restart runs. It is transient/self-healing and does not reproduce reliably (the restart usually outruns fileinstall's poll), but it is the same root cause.

The Stage 2 remedy (node-local load + verify-exports-before-start / version-gated apply) closes the window on all nodes, origin included. PR #36503 documents this window in a code comment at the jar-move site.