Version bump to OpenSearch 3.7 (Jackson 2 → 3 parser API + _source excludes serialization)#5361
Merged
RyanL1997 merged 8 commits intoopensearch-project:mainfrom Apr 20, 2026
Conversation
OpenSearch 3.7.0 migrated JsonXContentParser's constructor to accept tools.jackson.core.JsonParser (Jackson 3) instead of com.fasterxml.jackson.core.JsonParser (Jackson 2), breaking compilation since JsonNode.traverse() still returns the Jackson 2 parser. Switch to the public JsonXContent.createParser(registry, handler, String) API, which has a stable signature across both Jackson versions. Signed-off-by: Kai Huang <ahkcs@amazon.com>
ahkcs
requested review from
LantaoJin,
RyanL1997,
Swiddis,
acarbonetto,
anirudha,
dai-chen,
joshuali925,
mengweieric,
noCharger,
penghuo,
ps48,
qianheng-aws,
songkant-aws,
vamsimanohar,
ykmr1224 and
yuancu
as code owners
Signed-off-by: Kai Huang <ahkcs@amazon.com>
ahkcs
added
skip-diff-analyzer
OpenSearch 3.7.0's JsonXContentParser constructor requires Jackson 3's tools.jackson.core.JsonParser, but call sites were passing Jackson 2's com.fasterxml.jackson.core.JsonParser (via JsonFactory.createParser). Switch all such call sites to JsonXContent.jsonXContent.createParser (and YamlXContent.yamlXContent.createParser in HintFactory), which take a plain String and hide Jackson's type in their signatures, so the underlying migration is invisible to callers. Signed-off-by: Kai Huang <ahkcs@amazon.com>
ahkcs
changed the title
In OpenSearch 3.7, FetchSourceContext no longer serializes an empty "excludes":[] array. Strip it from the expected explain output in the PPL endpoint and optimization docs to match the new format. Signed-off-by: Kai Huang <ahkcs@amazon.com>
Jackson 3 reports duplicate JSON fields as "Duplicate Object property \"name\"" instead of Jackson 2's "Duplicate field 'name'". Adjust the async-query converter test to match the new phrasing surfaced via OpenSearch 3.7's xcontent layer. Signed-off-by: Kai Huang <ahkcs@amazon.com>
OpenSearch 3.7's FetchSourceContext no longer emits "excludes":[] when empty. Update the two top_hits assertions in MetricAggregationBuilderTest to match the new serialization. Signed-off-by: Kai Huang <ahkcs@amazon.com>
OpenSearch 3.7's FetchSourceContext no longer emits "excludes":[] when the list is empty. Sweep remaining fixtures (expectedOutput/**, legacy explain fixtures, yamlRestTest) and user docs to drop the stale field so textual comparisons match the new server output. YAML fixtures using `|` literal block style (most calcite/*) apply cleanly; a few legacy ppl/*.yaml fixtures that use folded double-quoted strings may need fold-offset regeneration once those ITs surface. Signed-off-by: Kai Huang <ahkcs@amazon.com>
ahkcs
changed the title
ahkcs
changed the title
Swiddis
previously approved these changes
Apr 16, 2026
Collaborator
Swiddis
left a comment
Swiddis
left a comment
There was a problem hiding this comment.
diff too small, plz also add an autoformatting change or something
The earlier sweep stripped `,"excludes":[]` textually but left the
surrounding YAML double-quoted string folds at their old column
positions. Once the embedded JSON shrinks, the server's emitter folds
at different offsets, so the checked-in fixtures diverged from actual
output. Regenerate the 47 affected legacy-path fixtures under
expectedOutput/ppl/{,big5/} from a live 3.7-SNAPSHOT cluster so fold
offsets match byte-for-byte.
Signed-off-by: Kai Huang <ahkcs@amazon.com>
Swiddis
approved these changes
Apr 16, 2026
RyanL1997
approved these changes
Apr 20, 2026
8 tasks
This was referenced Apr 20, 2026
Member
|
@ahkcs Thanks for working on this - I was blocked on this. :) |
ahkcs
added a commit
to ahkcs/sql
that referenced
this pull request
Apr 29, 2026
…cludes serialization) (opensearch-project#5361) Signed-off-by: Kai Huang <ahkcs@amazon.com>
ahkcs
added a commit
to ahkcs/sql
that referenced
this pull request
Apr 30, 2026
…cludes serialization) (opensearch-project#5361) Signed-off-by: Kai Huang <ahkcs@amazon.com>
4 tasks
ahkcs
added a commit
that referenced
this pull request
Apr 30, 2026
* Version bump to OpenSearch 3.7 (Jackson 2 → 3 parser API + _source excludes serialization) (#5361) Signed-off-by: Kai Huang <ahkcs@amazon.com> * Update vendored analytics JARs from 3.6 to 3.7 Replace libs/ JARs with 3.7.0-SNAPSHOT versions built from the OpenSearch sandbox. Update build files to reference 3.7 file names. Keep files() references instead of Maven coordinates so CI can resolve dependencies without publishToMavenLocal. Signed-off-by: Ahmed Khatib <ahkcs@amazon.com> Signed-off-by: Kai Huang <ahkcs@amazon.com> * Async QueryPlanExecutor + drop stub and stub-only ITs The analytics-framework 3.7 QueryPlanExecutor interface is async (void execute(plan, ctx, ActionListener<Stream>)). Update AnalyticsExecutionEngine to dispatch via the listener instead of the old synchronous return-value form, and rework the unit test mocks likewise. Drop StubQueryPlanExecutor and its callers (TransportPPLQueryAction, SQLPlugin, RestUnifiedQueryActionTest); the SQL plugin now expects the real analytics-engine executor to be supplied via Guice cross-plugin injection (a small AnalyticsExecutorHolder bridges the binding to the SQLPlugin REST handler that runs before Node injection). Drop AnalyticsPPLIT and AnalyticsExplainIT (and their expectedOutput/analytics fixtures): these tested stub canned data which no longer exists. Signed-off-by: Kai Huang <ahkcs@amazon.com> --------- Signed-off-by: Kai Huang <ahkcs@amazon.com> Signed-off-by: Ahmed Khatib <ahkcs@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Unblocks the OpenSearch 3.7.0-SNAPSHOT bump (#5296) end-to-end. Two coordinated API drifts in 3.7 needed fixes across this plugin:
Jackson 2 → 3 migration. OpenSearch 3.7 migrated its internal Jackson dependency from Jackson 2 (
com.fasterxml.jackson.*) to Jackson 3 (tools.jackson.*). Both Jacksons coexist on the classpath (different packages), but the publicJsonXContentParserandYamlXContentParserconstructors leak the Jackson type in their signatures, so every call site that built these parsers by handing in a Jackson 2JsonParser/YAMLParserfails to compile against 3.7. Jackson 3 also rephrased its duplicate-field error fromDuplicate field 'name'toDuplicate Object property \"name\".FetchSourceContextserialization. 3.7 no longer emits\"excludes\":[]when the list is empty, so every golden-file fixture, doctest, and assertion that captured the older serialization drifted.Fix
new JsonXContentParser(..., new JsonFactory().createParser(...))forJsonXContent.jsonXContent.createParser(registry, handler, String)(andYamlXContent.yamlXContent.createParserfor YAML). The factory's signature takes a plainString, hiding Jackson's type so the underlying 2→3 migration is invisible to callers.CreateAsyncQueryRequestConverterTestto match the new phrasing.excludessweep. Drop,\"excludes\":[](and its pretty/YAML-folded variants) from golden fixtures (expectedOutput/**, legacy explain fixtures,yamlRestTest), top-hits assertions inMetricAggregationBuilderTest, doctest markdown/rst, and dev/user docs.opensearch.versiondefault to3.7.0-SNAPSHOTso CI exercises the fix end-to-end; [AUTO] Increment version to 3.7.0-SNAPSHOT #5296 can be closed in favor of this PR.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.