fix(viewtools): keep aggregations in $json.generate() reflection JSON (#36435)

[fabrizzio-dotCMS]

Summary

The problem: When a VTL template does $json.generate($rawResults.response) to turn search results into JSON and then reads the aggregations, the JSON came out without the aggregations. The #foreach printed nothing — no error, no log. A customer hit this because their sitemap.vtl rendered only the base URL (Freshdesk 38225).

The cause: During the ES→OpenSearch migration, getAggregations() was annotated @JsonIgnore (so Jackson wouldn't duplicate the tree in the API JSON). But the serializer behind $json.generate() also honours that annotation → it skipped the aggregations → they vanished from the generated JSON.

The fix: Move that suppression from a method-level @JsonIgnore to a class-level @JsonIgnoreProperties. Jackson still honours it → the API JSON is unchanged. The $json.generate() serializer does not honour it → the aggregations show up again. Extended to tookInMillis and suggest, which shared the same latent bug.

The breaking change: The legacy .get("asMap") hop no longer works (it was an Elasticsearch artifact). The correct navigation is now direct: aggregations.<name>.buckets.

Verification: unit tests 14/14 ✅, end-to-end integration 10/10 ✅ (real Velocity engine + Elasticsearch).

---

Problem

Templates that round-trip a raw search response through $json.generate($rawResults.response) and then navigate the resulting JSONObject (e.g. $results.aggregations.<name>.buckets) silently got no aggregations after the ES→OpenSearch migration. A customer's sitemap.vtl rendered only the base URL — every aggregation-driven loop (folders, pages, urlmaps) iterated zero times, with no exception and no log line (Freshdesk 38225).

This is a distinct path from #36026/#36027, which restored direct Velocity object navigation ($rawResults.aggregations...). This bug shows up only when the template serializes the response through $json.generate() first.

Root cause

JSONTool.generate(Object) is literally new com.dotmarketing.util.json.JSONObject(bean) — a reflection/bean serializer that:

1. only reads public zero-arg getX()/isX() accessors, and
2. honours Jackson's @JsonIgnore (JSONObject.isIgnorable()).

ContentSearchResponse.getAggregations() carried a method-level @JsonIgnore (added so the neutral Jackson wire is single-sourced from aggregationTree and not double-emitted). Because the vendored JSONObject also honours @JsonIgnore, it skipped the getter too — so the aggregations vanished from the generated JSON and the customer's #foreach iterated nothing.

An audit of the object graph found two more Velocity back-compat aliases with the exact same latent regression: getTookInMillis() and getSuggest() also carried a method-level @JsonIgnore. Pre-migration the raw ES SearchResponse exposed both via getters, so $json.generate() templates reading timing/suggestions regressed the same way (lower blast radius — no ticket yet).

Fix

Move the Jackson suppression from method-level @JsonIgnore on the getters to a class-level @JsonIgnoreProperties({"aggregations", "tookInMillis", "suggest"}):

Serializer	Reads method-level @JsonIgnore?	Reads class-level @JsonIgnoreProperties?
Jackson (API wire)	yes	yes → still suppresses all three; wire shape unchanged (tree=aggregationTree, timing=tookMillis, suggest omitted)
vendored JSONObject ($json.generate())	yes	no → the getX() aliases are visible again → data restored

Net: $json.generate() templates work again, and the neutral Jackson JSON gains no duplicate aggregations / tookInMillis / suggest key.

Rest of the graph audited clean: SearchHits, SearchHit, Aggregation, AggregationBucket, TotalHits and ContentSearchResults carry no @JsonIgnore getters, so no other attribute is affected.

Test plan

Automated

• AggregationDomainTest (unit, no search engine): drives the exact new JSONObject(response) reflection path and asserts aggregations (down to each bucket's key/docCount), tookInMillis and suggest all survive; the Jackson-shape test asserts none of the three leak a duplicate wire key (tookMillis/aggregationTree stay; tookInMillis/suggest/aggregations absent). 14/14 green. Verified failing on the pre-fix source (1 failure) and passing with the fix.
• ContentSearchToolTest (integration): drives the customer's $json.generate($rawResults.response)-then-navigate idiom through the real dotCMS Velocity engine — the path Aggregation return-type change breaks existing VTL templates accessing $results.aggregations #36026/fix(viewtools): restore aggregation tree for VTL $estool.search (#36026) #36027 did not cover.

Manual (QA)

TC1 — aggregations (the reported bug):

1. On a site with published content, create a VTL page/widget:

   #set($esQuery = '{"aggs":{"folders":{"terms":{"field":"contentType","size":20}}},"size":0,"query":{"bool":{"filter":[{"term":{"live":true}}]}}}')
   #set($rawResults = $estool.search($esQuery))
   #set($results = $json.generate($rawResults.response))
   RAW JSON: $results
   #foreach($group in $results.aggregations.folders.buckets)
     key: $!{group.key} — docCount: $!{group.docCount}
   #end

2. Expected: RAW JSON contains an "aggregations" block, and the loop prints one key: … docCount: … line per bucket.
3. Before the fix: RAW JSON has no aggregations block and the loop prints nothing.

TC2 — tookInMillis & suggest (the two siblings fixed in the same PR):

1. On any site, create a VTL page/widget:

   #set($esQuery = '{"size":1,"query":{"match_all":{}}}')
   #set($rawResults = $estool.search($esQuery))
   #set($results = $json.generate($rawResults.response))
   RAW JSON: $results
   tookInMillis: $!{results.tookInMillis}
   suggest key present: $!{results.suggest}

2. Expected: RAW JSON contains both a "tookInMillis" number and a "suggest" key (an object, {} when the query has no suggester); tookInMillis: renders a number and suggest key present: renders (not blank).
3. Before the fix: RAW JSON has neither tookInMillis nor suggest; both output lines render blank.
4. (Optional, to see suggest populated) add a suggester to the query, e.g. "suggest":{"my-suggestion":{"text":"lorem","term":{"field":"title"}}}, and confirm the suggest block carries the suggester's entries.

QA Note

• Breaking-detail for affected templates: the legacy .get("asMap") hop (e.g. $results.aggregations.get("asMap").folders.buckets) only ever worked because the pre-migration object was an ES Aggregations (which had getAsMap()). The neutral tree is flatter, so the correct navigation is now aggregations.<name>.buckets directly — drop the asMap hop. Worth a docs/KB note for customers using the json.generate()-then-navigate idiom.
• No REST/wire contract changes: the /api/es/raw and /api/es/search neutral JSON shape is unchanged (verified by the Jackson no-duplicate assertions).

Closes #36435.

🤖 Generated with Claude Code

Rollback safety (H-8 — reviewed)

The rollback-safety bot flags this as a VTL viewtool contract change (H-8, HIGH): the PR restores the $json.generate($rawResults.response).aggregations (and .tookInMillis / .suggest) contract, so rolling back to N-1 would silently re-break any template that relies on it. This is inherent to the forward-fix and is expected:

• Templates already using this idiom (e.g. Freshdesk 38225) are already broken on the current release; rollback leaves them no worse off.
• The only new exposure is templates authored/updated to adopt the idiom during this cycle, which would silently regress on rollback (zero-iteration #foreach, no error/log).
• No REST/wire contract change — the neutral /api/es/raw & /api/es/search JSON is unchanged (pinned by the Jackson no-duplicate assertions).

Release-notes action for ops: rolling back this release silently reintroduces the $json.generate(...).aggregations regression (Freshdesk 38225) for any template adopting the restored idiom this cycle. Weigh that in any rollback decision.

[claude]

Claude finished @fabrizzio-dotCMS's task in 5m 43s —— View job

---

Rollback-Safety Analysis Complete

• Read docs/core/ROLLBACK_UNSAFE_CATEGORIES.md
• Get full PR diff (afcd170a...40e0a862, matches PR's +188/-13 across 3 files)
• Analyze diff against every unsafe category
• Post verdict (comment + label)

Verdict: Unsafe to rollback — Category H-8 (VTL Viewtool Contract Change), Risk: 🟠 HIGH

ContentSearchResponse (the record handed to Velocity via $estool.raw()/$dotcontent.raw()) had getAggregations(), getTookInMillis(), and getSuggest() moved from method-level @JsonIgnore to a class-level @JsonIgnoreProperties. Jackson still suppresses them from the neutral wire (no contract change there), but the vendored reflection-based com.dotmarketing.util.json.JSONObject (used by $json.generate()) no longer honors the suppression — restoring visibility. Any customer template that adopts the now-working $json.generate($rawResults.response).aggregations/tookInMillis/suggest idiom during this release would silently break again on a rollback to N-1 (the exact Freshdesk 38225 regression this PR fixes). No data loss, N-1 boots fine, but rendering breaks silently for affected templates.

Full details posted as a separate PR comment. Label AI: Not Safe To Rollback applied.

[claude]

Pull Request Unsafe to Rollback!!!

• Category: H-8 — VTL Viewtool Contract Change
• Risk Level: 🟠 HIGH
• Why it's unsafe: ContentSearchResponse is the record handed to Velocity via $estool.raw()/$dotcontent.raw()/$estool.search() ($rawResults.response), and this PR restores three of its accessors — getAggregations(), getTookInMillis(), and getSuggest() — to visibility on the reflection-based com.dotmarketing.util.json.JSONObject bean constructor behind JSONTool.generate(Object). Before this PR, all three carried a method-level @JsonIgnore, which the vendored JSONObject also honors, so $json.generate($rawResults.response) silently dropped aggregations, tookInMillis, and suggest (Freshdesk 38225). This PR moves the suppression to a class-level @JsonIgnoreProperties({"aggregations", "tookInMillis", "suggest"}), which Jackson still honors (wire shape unchanged) but the vendored JSONObject does NOT — restoring all three to the reflection path. If a customer template is authored or updated during this release cycle to rely on the now-working $json.generate(...).aggregations/tookInMillis/suggest idiom (exactly the fix this PR ships to enable, e.g. sitemap.vtl), a rollback to N-1 silently reintroduces the method-level @JsonIgnore and the exact customer-facing regression this PR fixes: #foreach loops over aggregations.<name>.buckets iterate zero times with no exception and no log line, and any $results.tookInMillis/$results.suggest reference in a $json.generate()-derived object goes silently missing. N-1 boots fine and no data is lost, but rendering for any template adopting this idiom breaks silently — the textbook H-8 HIGH scenario. Likelihood can't be scoped to a single template since $json.generate() is a general-purpose viewtool available to any VTL across the site.
• Code that makes it unsafe: dotCMS/src/main/java/com/dotcms/content/index/domain/ContentSearchResponse.java — class-level @JsonIgnoreProperties({"aggregations", "tookInMillis", "suggest"}) added above the record declaration (line 54), and removal of the method-level @com.fasterxml.jackson.annotation.JsonIgnore from getAggregations() (line 126), getTookInMillis() (line 110), and getSuggest() (line 138) — all three previously carried the method-level annotation, now removed.
• Alternative (if possible): This is largely unavoidable here since the PR is itself the two-phase-style fix (warming the contract): the safer framing per H-8 is to treat this as intentionally "warming" the $json.generate()-visible contract now, and explicitly document in release notes that rolling back this release will silently reintroduce the $json.generate($rawResults.response) regression (Freshdesk 38225) for aggregations, tookInMillis, and suggest alike, for any template that adopts the new idiom during this cycle, so ops can weigh that against the rollback decision.

View job run

[fabrizzio-dotCMS]

Thanks — the H-8 flag is understood and addressed as a documentation/ops concern, not a code change, since (as the bot notes) restoring the $json.generate() contract is inherent to the forward-fix and largely unavoidable.

Actions taken:

• Added a Rollback safety (H-8) section to the PR description: rolling back this release silently reintroduces the $json.generate($rawResults.response).aggregations regression (Freshdesk 38225) for any template that adopts the restored idiom this cycle → flagged for release notes.
• Clarified scope: templates already using the idiom are already broken on the current release, so rollback leaves them no worse off; the only new exposure is templates authored to adopt it during this cycle.
• Confirmed no REST/wire contract change — the neutral /api/es/raw & /api/es/search JSON is unchanged, pinned by the Jackson no-duplicate assertions in AggregationDomainTest.

No safer code alternative exists without abandoning the fix itself, so proceeding with the documented rollback caveat.

[wezell]

Good test.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request