fix(ci): drop dead build artifact, guard an NPE in a catch block, and make integration tests report their failures - #36943
fix(ci): drop dead build artifact, guard an NPE in a catch block, and make integration tests report their failures#36943wezell wants to merge 14 commits into
Conversation
…36914) Category Content ran ~40m with ContentTypeResourceTests alone taking 14m37s of it, while the Template job finished in ~10m (mostly boot overhead). Rebalancing the collection across existing groups cuts the Postman critical path ~15m without adding a job or paying another ~12m dotCMS boot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ained (#36914) The 'ensure' tests assert a Video content type exists, but nothing creates it eagerly at startup — it only existed because collections that previously ran before this one in the category-content group created it indirectly. Moving the collection to the template group exposed this (Video missing, dotAsset present). Add a create-if-missing setup request so the collection passes regardless of group placement or ordering. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… artifact The PR pipeline's wall clock is bounded by its slowest test job. Two jobs were tied at the top -- Integration MainSuite 1a (38.9m) and Postman GraphQL (37.8m) -- so rebalancing either one alone moved nothing. This rebalances both, plus removes an artifact that was uploaded on every build and consumed by nothing. Measured from GitHub Actions job timings and failsafe/newman reports: Integration 7 shards, 130.8m of test time, max 38.9m, ideal 26.5m Postman 11 shards, 147.8m of test time, max 37.8m, ideal 22.8m fixed cost per shard: ~7.6m (IT), ~9.4m (Postman) Changes: * Integration: repack all 557 classes across 7 MainSuite shards by measured per-class time (LPT bin-packing) instead of by accretion. Every shard is now 18.4m of test time (spread 0.0m), vs 14.2m-31.1m before. Adds MainSuite3b and MainSuite4a. Class count is a poor proxy for time and was the reason the old split drifted -- one shard held 298 classes/31m, another 52 classes/14m. Suites now use fully-qualified class names so rebalancing does not churn imports. The "run FIRST on purpose" full-scan block from #36911 is preserved at the head of each shard. * Postman: regroup on measured newman time, 11 shards -> 9, all at 16.4m. GraphQLTests was a single 25.6m collection and the binding constraint on the whole Postman tail -- no regrouping could get below 25.6m + 9.4m overhead. It is now folder-sharded via a new `folders` key in config.json rather than by splitting the 518KB collection file. Four micro-groups (ai, pp, container, template) that were ~85% fixed overhead are merged away. * index.js: support the `folders` key, and hard-fail on a folder name that does not exist. Newman runs zero requests for an unknown folder and still exits green, so a typo would silently delete coverage. `errors` never affected the exit code -- only `failures` did -- so this validates up front and exits 1, consistent with how an unknown groupname is already handled. * verify-config.js: checks config.json against the collections on disk -- missing collections, unknown folders, double-claimed collections, and folders covered by no shard. It caught a real bug while writing this: a folder named "Related content with condition / query" contains a slash, and deriving names by splitting report labels on " / " silently truncated it. * maven-job: drop the `build-classes` artifact. It was uploaded on every build (~0.5m on the serial critical path, since the build gates every test job) and downloaded by nobody -- the only `restore-classes: true` caller is deploy-javadoc, which no workflow invokes. Net job count is unchanged: integration 7 -> 9 shards, postman 11 -> 9. Expected tail: 38.9m -> ~26m. Wall clock above that floor is queue time, which this does not address. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NH4Pes5S9d1AQorJBNeFJU
|
Claude finished @wezell's task in 3m 38s —— View job Code ReviewReviewed the diff against New IssuesNo issues found. Notes (non-blocking)
Overall: clean, well-reasoned changes with unusually good explanatory comments. Nothing blocking. |
…with awaitility
Two of the slowest integration tests, both spending their time waiting rather
than testing.
Task240306MigrateLegacyLanguageVariablesTest -- 7.26m, the single most
expensive IT class (5.6% of all integration test time). The cost is not
executeUpgrade: testDataTaskIdempotency runs the upgrade twice and costs the
same as testExecuteUpgrade running it once. It is the teardown. cleanup()
walked the migration summary and did a find + destroy per inode, and a run
produces well over a hundred Language Variables (en=38, fr-fr=23, es-es=34,
plus others). destroy() is @WrapInTransaction, so that was ~150 transactions,
each with its own ES delete and cache invalidation, three times over.
Now batched through findContentlets(List) and destroy(List) -- one query and
one transaction. Teardown must not leave content behind for the next test, so
a batch failure falls back to the original per-contentlet loop, which tolerates
individual failures. removeExistingLanguageVariables() got the same treatment.
No test semantics change; only how the fixtures are torn down.
ImportUtilTest -- 34s of literal Thread.sleep, the most in the suite:
30s polling for content to appear in the index. It slept 30s BEFORE the
first check, so it always paid the full cost even when the index was
already caught up, and its retry bound (100 iterations) allowed a
50-minute worst case.
2s waiting for an async import to land three contentlets
1s x2 waiting after addContentToIndex for content to become searchable
All four replaced with awaitility, following the pattern already used in
SiteSearchJobImplTest and other ITs: poll every 200ms, fail at 30s. The waits
now cost what they actually need rather than a fixed guess, and a genuinely
stuck index fails fast with a clear timeout instead of hanging.
Also leaves a note-worthy bug untouched, deliberately: removeLanguageAndContent
returns false immediately after destroying its first contentlet, so it never
reaches deleteLanguage. It is used by one test whose assertions depend on the
current return value, so fixing it is a behaviour change and belongs in its
own PR.
Not run locally (needs the full IT stack); verified by test-compile and by the
integration suites in this PR.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NH4Pes5S9d1AQorJBNeFJU
…way round BrowserAPITest was the third most expensive integration test at 4.06m. Its pagination tests picked a page size and then built a dataset several times larger to cross the boundaries - 25 folders + 30 file assets to test "page 1 holds every folder plus one contentlet", and so on. Every FileAssetDataGen writes a temp file, persists a contentlet and indexes it with WAIT_FOR, so the dataset is nearly all of the runtime. None of those assertions depend on absolute numbers. They depend on relationships: page size vs folder count, and how much content remains after the cursor. Scaling both sides down preserves every invariant exactly: page 1 fills with folders then tops up 25f + 30c -> 5f + 6c later page, fewer than a page remain 10f + 25c -> 2f + 5c mid-stream page, more remains 15f + 50c -> 3f + 10c exhaustive scan across permission gaps 20c -> 10c scan limit stops the loop 20c -> 5c That is 145 file assets down to 36. The sizes are now named constants derived from the page size, so the relationship under test is visible and the next person has no reason to re-inflate the dataset. Three tests are renamed: their old names hardcoded the counts and would have become lies. FolderResourceTest got the same treatment, but only where it was safe. Three of its five large loops are load-bearing and are left alone: moreThan20Folders (25) regression test against a former hardcoded 20 cap defaultLimit40 (45) pins the @DefaultValue("40") on the limit param limitMinusOne (50) proves limit=-1 bypasses that same 40 cap Shrinking any of those below its cap would make the test pass trivially and silently stop testing anything. Only withCustomLimit and withOffset - which pin no production constant - are reduced, 30 subfolders each to 10. Not run locally (needs the full IT stack); verified by test-compile and by the integration suites in this PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NH4Pes5S9d1AQorJBNeFJU
…rder-dependent tests
Re-sharding the integration suites surfaced two tests that only passed because of
where they sat in the run order. Both are genuine defects, so they are fixed
rather than pinned back into place.
AWSS3PublishingEndPoint - a NullPointerException thrown from inside a catch
block. validatePublishingEndPoint() catches S3 connection failures and reports
them to the user, but built that notification with:
PortalUtil.getUser().getUserId()
PortalUtil.getUser() returns null whenever no request is bound to the thread:
final HttpServletRequest req = HttpServletRequestThreadLocal.INSTANCE.getRequest();
return req == null ? null : getUser(req);
So for any non-request caller - scheduled jobs, background tasks, tests - the
error handler NPE'd, replacing the S3 error it was trying to report with a
confusing NullPointerException and defeating the catch entirely. Now null-checked;
the notification is a UI concern, so with no user to notify the logged warning
above it is the whole story.
PublishingEndPointTest depended on that NPE not happening, which in turn depended
on some earlier test in the same suite leaving an HttpServletRequestThreadLocal
behind. In MainSuite1a something did; scheduled elsewhere, nothing does. The
tests also swallowed the exception:
catch (Exception e) { Assert.assertTrue("No Exception should be thrown", false); }
which discards the cause - four retries produced four identical, contentless
messages and told us nothing. They now let the exception surface, so a failure
names itself. Renamed from *_returnException to *_returnsWithoutThrowing, since
they assert the opposite of what they claimed, and dropped the dead
`exceptionCatched` locals. The third test in the file had the identical pattern
and is fixed too - it has not failed yet, but only because of where it runs.
RuleBundlerTest created its Rules in a static @dataProvider. Providers are
evaluated when the suite is CONSTRUCTED - MainBaseSuite builds every runner up
front - so those Rules existed long before the test ran, and the ~65 test classes
scheduled in between could destroy them, giving "Cannot invoke Rule.getGroups()
because rule is null". The provider now returns only a flag and the Rule is
created inside the test, immediately before use. TestCase also gained a toString
so failures read "rule attached to a page" instead of "TestCase@5c2004ac".
Not run locally (needs the full IT stack); both modules test-compile clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NH4Pes5S9d1AQorJBNeFJU
… catch block
A scan of all 845 integration test files found 18 catch blocks that fail the
test while discarding the exception that caused it. Three were fixed in the
previous commit after they cost a full CI run to diagnose; this fixes the
remaining 15.
The shape:
catch (Exception e) { Assert.fail("Should work"); }
The test fails, but nothing about the actual cause reaches the log. Surefire
retries produce N identical, contentless messages - which is exactly what
happened with PublishingEndPointTest: four retries, four copies of "No Exception
should be thrown", and no way to tell what threw without reproducing locally.
Two of these were worse still - a bare `Assert.fail()` with no message at all
(FiltersTest:709, Task220413IncreasePublishedPushedAssetIdColTest:63).
All 15 now use:
catch (Exception e) { throw new AssertionError("Should work", e); }
which fails the test identically but chains the cause, so the stack trace
survives into the CI log. Existing messages are preserved verbatim; the two
message-less sites get "Unexpected exception". Where the catch did other work
first - TailLogResourceTest re-interrupting the thread - that work is kept.
This is timely rather than incidental: 14 of the 15 sit in classes this PR moves
to a different shard. If any of them trips on a latent order dependency, it will
now say why instead of sending the next person on the same archaeology.
For balance, the same scan found 90 catch blocks that already pass the exception
along, so the codebase mostly gets this right - these were the outliers. It also
counted 163 empty catch blocks, but only 8 are the legitimate
`try { x; fail(); } catch (Expected e) {}` idiom; the rest are a mix of
deliberate best-effort cleanup and genuine hiding that a regex cannot separate.
Left alone rather than guessed at.
Not run locally (needs the full IT stack); test-compiles clean, and a re-scan
reports zero remaining.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NH4Pes5S9d1AQorJBNeFJU
One conflict: MainSuite3a.java. This branch rewrites every MainSuite to use fully-qualified class names and redistributes all 576 integration test classes across 7 shards by measured time, while main added CacheResourceIntegrationTest to MainSuite3a in #36917. Resolved by keeping the repacked suites and re-adding the new class, so it is still scheduled. Verified the merged suites contain exactly the same set of test classes as origin/main - 576 on both sides, none dropped, none duplicated across shards - and that both :dotcms-core and :dotcms-integration compile. This kind of conflict is expected while this branch is open: any PR that adds a test class touches a MainSuite file. The resolution is always the same - keep the repacked shards, re-add the new class to whichever shard is lightest.
…ependencies REVERT BEFORE MERGE - the restore value is in the comment above the setting. Re-sharding the integration suites exposes tests that only passed because of which suite-mates ran before them. Three found so far, each in a different shard, and each cost a full ~40 minute run to find because fast-fail cancels the other 25 jobs at the first failure. Turning it off for this branch converts that into a single run that reports every affected shard at once, so the remaining coupling can be fixed in one batch instead of one per CI cycle.
…sk240306 test testDropThenRecreateLanguageVariableContentType deleted the Language Variable content type through ContentTypeAPI, then asserted that checkContentType() recreated it. Removed, along with the now-orphaned removeLanguageVariableContentType() helper and two imports it was the only user of. That delete should not be possible in the first place. The Language Variable content type underpins all i18n, and losing it makes LanguageVariableAPIImpl throw NotFoundInDbException and language resolution fall back to emitting raw keys site-wide. #36958 tracks marking it `system` so ContentTypeFactoryImpl's if (type.system()) throw new DotDataException(...) guard refuses the delete. Once that lands this test cannot work as written, and until then it is a test deliberately exercising the destructive path we are trying to close off - and it does so with DELETE_CONTENT_TYPE_ASYNC forced to false, so the type really is gone for whatever runs next in that suite. Nothing else is lost: its remaining assertions (executeUpgrade succeeds, summary present, no failures) duplicate testExecuteUpgrade. What IS lost is coverage of checkContentType() recreating the type when genuinely absent - a real situation for legacy installs, which is why the upgrade task performs that check at all. A comment at the removal site records this, and #36958 tracks re-covering it by seeding the missing state directly rather than by calling the delete API. Two side benefits: - This was one of three ~2.4m tests in the single most expensive integration test class (7.26m, 5.6% of all IT test time), so it takes roughly a third off it. - A grep of all 845 integration test files now finds zero API-level deletes of this content type: 24 references remain and every one is a read. Not run locally (needs the full IT stack); test-compiles clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NH4Pes5S9d1AQorJBNeFJU
…nt fixes
The rebalance works — measured on run 31203516115, with fail-fast disabled so
every shard reported:
IT shards 22.2 - 26.2m (was 38.9m)
Postman 18.6 - 28.8m (was 37.8m)
But collecting it costs more than this PR can carry. 3 of 7 integration shards
and 5 of 9 Postman shards failed, all from tests that were only passing because
of which suite-mates ran before them.
The Postman breakage was the bigger misjudgement on my part: those group names
were not labels, they were dependency clusters. `category-content` grouped
Category + ContentResourceV1 + Content_Resource because those collections share
data. Rebalancing purely on measured time shattered every one. The GraphQL
folder split broke the same way - "Page API - Testing 'page' field with inline
fragments" needs setup performed by the "Page API" folder, which landed in the
other shard.
Reverted here: test-matrix.yml shard lists, the seven repacked MainSuite files,
MainSuite3b/4a, dotcms-postman/config.json, index.js folder support, and the
temporary fail-fast:false.
Kept, because none of it depends on the rebalance:
- the dead build-classes artifact (0.5m off the serial prefix, consumed by
nothing)
- AWSS3PublishingEndPoint: NullPointerException thrown from inside a catch
block whenever no request is bound to the thread
- 15 catch blocks that failed a test while discarding the cause
- Task240306 teardown batched (~150 transactions -> 1) and its Language
Variable content type delete removed (see #36958)
- ImportUtilTest: 34s of Thread.sleep -> awaitility
- pagination fixtures sized to the page size rather than the reverse
The full rebalance is preserved on `issue-36942-shard-rebalance-wip` at
f612b27 so it can be resumed rather than rebuilt. It is worth resuming: it
is an effective detector of tests that depend on their neighbours, and every
failure it surfaced so far has been a real defect - including a production NPE
and a deletable Language Variable content type. That is a test-independence
programme though, not a prerequisite for the fixes above.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NH4Pes5S9d1AQorJBNeFJU
…#36961) Closes part of #36947. **Stacked on #36960** — review that one first; this PR's diff is against it. ## What Gives the Apache Maven Build Cache Extension somewhere to keep its results: a shared S3 bucket on OVH object storage. The extension was **already installed and enabled** (`.mvn/extensions.xml`, v1.2.0) — but local-only, so on a fresh CI runner it cached nothing. The only missing piece was shared storage. Scoped to the **Initial Artifact Build**, the serial prefix that gates every test job. Test phases are deliberately *not* cached: a memoised green run would mean "we did not run", and this suite has measured flakes plus a hang that has burned a 122m job timeout. ## Transport The extension speaks HTTP `PUT`/`GET`/`HEAD`; S3 needs SigV4. Rather than add a Maven S3 wagon — the available ones aren't maintained (`seahen` 1.3.3 is 2021, `gkatzioura` 2.3 is 2019, both AWS SDK v1) — [`aws-sigv4-proxy`](https://github.com/awslabs/aws-sigv4-proxy) runs as a container and signs on the way out, and Maven talks to `127.0.0.1`. **No new Maven dependency.** ## Security A build cache untrusted code can write is a supply-chain vector: a poisoned entry is replayed as a build output on a trusted ref. This is [CVE-2025-36852](https://www.cve.org/CVERecord?id=CVE-2025-36852) ("CREEP"), which killed Nx's `@nx/s3-cache` and its siblings. **The control is the credential, not the client** — a job holding a writable key can bypass Maven entirely with one `aws s3 cp`, so `remote.save.enabled` is defence in depth, not the boundary. | Ref | Key | Writes | |---|---|---| | PR | `..._ACCESS_KEY_RO` (GetObject only) | no | | merge queue / trunk | `..._ACCESS_KEY` | yes | | fork PR | none — builds uncached | no | Plus: - `remote.save.final=true` — an existing entry is never overwritten. - The action **asserts** its read-only key is read-only (one `PUT`, expects `403`) instead of assuming it. A writable "read-only" key looks identical to a correct setup until abused. - Writing builds record a `provenance.json` beside each entry (ref, sha, run id, actor), first-writer-wins — nothing else in a bucket says which ref produced a hash. ## The subtle one: `alwaysRunPlugins` Load-bearing, not tuning. On a cache hit the extension skips cached plugin executions **including `install:install`** — measured: 1 jar in `~/.m2/repository` after a cold build, **0 after a hit**. This job exists to publish that repository as the `maven-repo` artifact ~25 test jobs consume. Same story for `docker-maven-plugin:build`, which writes the `docker-build.tar` the next step uploads. Both would have looked perfectly green on the cold populate run and broken everything on the first *warm* one. ## Verification Against MinIO before any of this was wired: | Check | Result | |---|---| | `PUT` / `GET` / `HEAD` through the proxy | `200` / `200` / `200` | | Missing key | `404` (a `403` reads as a hard error, not a miss) | | 10 MB body round-trip | byte-identical | | Build with an **empty** local cache | `Found cached build, restoring … by checksum` | | Remote unreachable | build exits `0`, logs an error, rebuilds | And separately, because `cicd_comp_build-phase.yml` fails a PR on a dirty tree while `openapi.yaml` is a tracked file generated at compile: built `:dotcms-core --am` twice — 12 modules restored including `dotcms-core`, `openapi.yaml` md5 identical, `git status` unchanged. ## Releases are not affected Release, LTS, nightly, manual-deploy and CLI-release workflows don't pass the secrets, and **no workflow in this repo uses `secrets: inherit`** (verified). The action sees empty credentials and exports an empty `BUILD_CACHE_ARGS`; those pipelines build from scratch exactly as today. A release is the build where "we didn't actually compile this" is least acceptable and the saving is worth least. ## Turning it off | Scope | How | |---|---| | One PR | label `CI: no build cache` | | Everything, now | repo/org variable `BUILD_CACHE_DISABLED=true` | | Local build | `-Dmaven.build.cache.enabled=false` | | Force rebuild, still publish | `-Dmaven.build.cache.skipCache=true` | ## Expected effect, honestly Ceiling is the ~7.0m of Maven time inside a 14.2m build job, against a 74–103m PR wall clock. Real but modest — the twin-tail shard rebalance (#36943) is still the bigger lever. `dotcms-core-web` only participates once #36960 lands. **The first merge-queue run is the canary**: it's what proves the OVH SigV4 handshake and the region derived from the endpoint host. If either is wrong the cache disables itself with a warning rather than failing the build. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014a2iJy9JXRBSVdKBbmoZ2S --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes #36942 (partially — the shard rebalance is deferred, see below).
A set of independent CI and test fixes, none of which depend on each other.
Build
Dead
build-classesartifact removed. It was uploaded on every build and downloaded by nobody — the onlyrestore-classes: truecaller isdeploy-javadoc, which no workflow invokes. Costs ~0.5m on the serial critical path that gates all ~25 test jobs.A production bug
AWSS3PublishingEndPoint.validatePublishingEndPoint()catches S3 connection failures to report them to the user, then builds that notification withPortalUtil.getUser().getUserId(). But:Null whenever no request is bound to the thread — scheduled jobs, background tasks, tests. So the error handler NPE'd, replacing the S3 error it was trying to report and defeating the catch entirely. Now null-guarded.
Tests that could not report their own failures
A scan of all 845 integration test files found 18 catch blocks that fail a test while discarding the exception:
Surefire retries then produce N identical, contentless messages. This cost a full CI run to diagnose on
PublishingEndPointTest— four retries, four copies of "No Exception should be thrown", no way to tell what threw. Two were worse: a bareAssert.fail()with no message at all.All 18 now
throw new AssertionError(msg, e)— same failure, cause chained. The scanner is included as.github/scripts/test-balance/find_swallowed.py; a re-scan reports zero remaining. For balance, 90 catch blocks already did this correctly — these were the outliers.Test performance
Task240306...Testfind+destroyper inode over ~150 Language Variables × 3 tests, each its own transaction → one query, one transactionImportUtilTestThread.sleepBrowserAPITestFolderResourceTestgot the same treatment, but only where safe — three of its five large loops are load-bearing (they pin the former 20-item cap, the@DefaultValue("40")limit, and thatlimit=-1bypasses it). Shrinking those would make them pass trivially.Also removed
Task240306's deliberate delete of the Language Variable content type — see #36958, which tracks marking itsystemso that delete becomes impossible.Why the shard rebalance is not here
It works. Measured on run 31203516115 with fail-fast disabled so every shard reported:
But 3 of 7 IT shards and 5 of 9 Postman shards failed — all tests that only passed because of which suite-mates ran before them.
The Postman group names turned out not to be labels but dependency clusters:
category-contentgroupedCategory+ContentResourceV1+Content_Resourcebecause those collections share data. Rebalancing on time alone shattered them.Preserved on
issue-36942-shard-rebalance-wip(f612b27) so it can be resumed rather than rebuilt. It is worth resuming — it is an effective detector of tests that depend on their neighbours, and every failure it surfaced was a real defect, including the production NPE above and the deletable content type in #36958. But that is a test-independence programme, and it should not gate these fixes.Verification
Both
:dotcms-coreand:dotcms-integrationcompile. Not run locally — needs the full IT stack.