[DEFECT] OSGi plugin fails to resolve on a cluster (osgi.wiring.package … version>=0.0.0) because the system export is written to osgi-extra.conf but the framework restart is skipped

[dsolistorres]

Freshdesk: https://dotcms.freshdesk.com/a/tickets/37894

Code references below are pinned to main @ 5efb47a8d9fa26cab0911f25eb5fbe8525026d5a.

Problem statement

On a clustered dotCMS (2+ nodes, Evergreen/SaaS or self-hosted with a shared /data/shared), uploading an OSGi plugin that imports a dotCMS package not yet in the exported-packages list can leave the plugin permanently unresolved on one or more nodes, with:

org.osgi.framework.BundleException: Unable to resolve <bundle>: missing requirement
[<bundle>] osgi.wiring.package; (&(osgi.wiring.package=<pkg>)(version>=0.0.0))

…even though the Exported Packages list (osgi-extra.conf) does contain <pkg>;version=0. Customers "fix" it by adding <pkg>;version="0.0.0" and re-deploying, which makes it look like a version-matching problem. It is not.

This is NOT a version-string problem

In OSGi, version=0 parses to 0.0.0, and an import with no floor compiles to (version>=0.0.0), which a bare version=0 export satisfies. A bundle importing pkg;version=0 resolves to ACTIVE when pkg is exported as bare version=0, and fails only when pkg is absent from the running framework at resolve time. The ;version="0.0.0" workaround works only because editing exports / re-deploying triggers a fresh framework restart — the value is incidental. (Regression tests demonstrating this will accompany the fix PR.)

Root cause

The real cause is a restart being skipped (or applied too late) on a cluster, so a package that is in osgi-extra.conf is not live in the running framework when the bundle is resolved. Two variants, both reproduced:

Mode A — restart skipped entirely (persistent failure) — primary

OSGIUtil.restartOsgi() only restarts if the upload folder still contains jars (if (UtilMethods.isSet(pathnames))).

On a cluster, both nodes run the upload-folder poller (startWatchingUploadFolder → checkUploadFolder) against the shared upload folder. The race:

1. Node B's poller processes the jar → processOsgiPackages sees new packages → writes osgi-extra.conf and schedules a debounced restartOsgi. Jar stays in upload.
2. Node A's poller runs the reload ~fractions of a second later. The export is already written, so it hits the containsAll == true branch and moves the jar from the shared upload → load — no restart.
3. Node B's debounced restartOsgi fires, finds upload empty (isSet(pathnames) false), and silently does nothing.

Net result: export written to the shared file, jar deployed to load, no node ever restarts → the package is never published to the running framework → the bundle fails to resolve on every node, persistently, until an unrelated restart. fileinstall then auto-starts the jar from the shared load folder against the stale framework and logs the (version>=0.0.0) error.

Mode B — remote restart applied too late (transient failure) — secondary

Even when a restart does happen, the origin node publishes a cluster restart request over pub/sub (restartOsgiClusterWide; handled at OsgiRestartTopic). A remote node's fileinstall can auto-start the jar from the shared load folder before that node processes the restart, so it resolves against its still-stale system.packages.extra → same error, until the node restarts (then it recovers).

Contributing factor for both: org.osgi.framework.storage.clean=onFirstInit — the felix bundle cache is not cleaned on a restart, only on JVM first init.

Steps to reproduce

Reproduced on a 2-node cluster sharing /data/shared (shared felix upload/load/undeployed + osgi-extra.conf). A docker-compose harness + minimal plugin will accompany the fix PR.

1. Bring up a 2-node cluster sharing /data/shared.
2. Upload a plugin (with a Bundle-Activator, so OSGi resolves it on start) that imports a real dotCMS package not currently in osgi-extra.conf, e.g. Import-Package: com.dotcms.ai.model;version=0.
3. Mode A: with normal timing, observe that the export is written to osgi-extra.conf and the jar lands in load, but no Restarting OSGI Cluster Wide occurs → the bundle fails to resolve on both nodes and stays failed.

Observed evidence

The node that wrote the export, then the skipped restart:

17:01:48  There are a new changes into the exported packages
17:01:48  OSGI Extra Packages Saved
17:01:51  Debouncing : restartOsgi after 3000 MILLISECONDS
17:01:51  Restarting OSGI            <-- entered, but upload folder already emptied by the other node → guard fails, no restart

osgi-extra.conf contains com.dotcms.ai.model;version=0 the whole time, yet:

BundleException: Unable to resolve com.dotcms.osgi.repro-37894d:
  osgi.wiring.package; (&(osgi.wiring.package=com.dotcms.ai.model)(version>=0.0.0))

No Restarting OSGI Locally after the upload on either node → package never applied.

Expected vs actual

• Expected: once a package is added to osgi-extra.conf, every cluster node restarts (or otherwise re-publishes system exports) before the plugin is resolved, so it resolves consistently across the cluster.
• Actual: the restart is skipped (Mode A) or races the plugin start on a remote node (Mode B), leaving the plugin unresolved despite the export being present.

Affected versions

• Current Evergreen 26.06.30-01.
• Requires a cluster (2+ nodes) sharing /data/shared. Not reproducible on a single node.

Suggested fixes (for discussion)

1. Don't gate restartOsgi() on the upload folder. Track whether exports changed (or whether jars were moved to load) and restart on that, so a concurrent containsAll move on another node can't cancel the restart.
2. Make the export-change → restart atomic per cluster, coordinated by the existing OSGI_RESTART_LOCK_KEY cluster lock, so exactly one node owns "write exports + move jars + trigger cluster-wide restart" for a given upload.
3. Before auto-starting a bundle from the load folder, verify the required exports are live in the current framework; if not, defer until after the pending restart (addresses Mode B).
4. Consider cleaning the felix bundle cache on restart when exports changed (storage.clean), or otherwise forcing re-resolution.

Diagnostics (proposed alongside the fix)

• Make felix.log.level Config-driven (e.g. FELIX_LOG_LEVEL, default 3) so 4 can be set to capture the Felix resolver's candidate evaluation during OSGi restarts — directly useful for diagnosing this class of issue in the field.

External Links

https://dotcms.freshdesk.com/a/tickets/37894

[github-actions]

PRs linked to this issue

• fix(osgi): restart on live-export state, not the shared upload folder (#36434) by @dsolistorres

[dsolistorres]

📌 Scope note for this issue (and for QA)

This issue is being resolved in two stages. The fix in PR #36503 covers Stage 1 only — Mode A (the persistent failure).

✅ In scope here (Stage 1 — Mode A)

The persistent cluster failure where an uploaded plugin's new package is written to the shared osgi-extra.conf but no node ever restarts, so the bundle stays permanently unresolved (osgi.wiring.package=...;(version>=0.0.0)) until an unrelated restart.

QA focus: on a 2+ node cluster sharing /data/shared, upload a plugin that imports a dotCMS package not yet in the Exported Packages list. Expected: the framework restarts cluster-wide and the bundle reaches ACTIVE on every node, with no persistent resolve error.

Verified during development on a 2-node Docker cluster: bundle ACTIVE on both nodes, no wiring.package error.

⏭️ NOT in scope here — tracked in #36504 (Stage 2 — Mode B)

The transient, self-healing case where a remote node's fileinstall starts the jar from the shared load folder before that node processes the restart, briefly logging the same wiring.package error before it recovers on restart.

QA note: a brief/transient wiring.package error on a remote node that clears after that node restarts is the known Mode B behavior and is expected until Stage 2 (#36504). It should not be filed as a regression against this issue. Only a persistent unresolved bundle indicates a Stage 1 failure.

Stage 2 (node-local load folder + version-gated apply) is tracked separately in #36504.