[refactor] Catalog spi 413 upgrade compat ut - #66214
Open
morningman wants to merge 4 commits into
Open
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
morningman
force-pushed
the
branch-catalog-spi
branch
from
July 29, 2026 09:35
09f86f0 to
f672e78
Compare
morningman
requested review from
924060929,
Gabriel39,
dataroaring,
englefly,
gavinchou,
liaoxin01,
luwei16,
morrySnow,
mymeiyi,
seawinde and
starocean999
as code owners
July 29, 2026 09:35
morningman
force-pushed
the
branch-catalog-spi
branch
from
July 29, 2026 13:18
f672e78 to
0da96f1
Compare
…ing replay Replaying OP_CREATE_CATALOG builds the connector synchronously, on the thread that is starting the FE, and EditLog's fallback turns anything thrown there into System.exit(-1). So a connector that rejects a property an older FE stored without validating -- or a half-installed plugin throwing NoClassDefFoundError -- keeps the whole FE from starting instead of making one catalog unusable. The image path never had this problem because it builds the connector lazily. Catch on the replay arm only and fall through to the degraded registration that already exists for an unclaimed type; the failure resurfaces at first access as an ordinary query error. Interactive CREATE CATALOG still fails loud, because the user is waiting for the error. Catching RuntimeException | Error rather than Exception is deliberate: a half-installed plugin throws NoClassDefFoundError, not a RuntimeException. Covered by Legacy413JournalReplayTest#connectorConstructorThatThrowsMustNotTakeTheFeDown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mework
Adds golden FE metadata fixtures produced by real Doris 4.1.3 bytecode, and
the tests that replay them on this branch. Goal: a catalog created on 4.1.3
must migrate onto the connector SPI and remain usable, through both the image
and the edit-log path.
The fixtures are generated, never hand-written. 4.1.3's wire format has at
least five details a hand-authored blob gets wrong: HTML escaping is on
(jdbc_url persists '=' and '&' escaped), serializeNulls is off (the resource
key vanishes when null), taap is polymorphic ({} when empty, an array of
2-tuples when not), clazz is written first, and the create path rewrites the
property map before it is persisted. The generator, its provenance and the
regeneration recipe live next to the fixtures; output is byte-deterministic.
Fixture families, and why the resource-backed one is not optional: with a
"type" property present, deleting the migration entirely leaves every
assertion green, because getType() just reads what was already on the wire.
Only the resource-backed catalogs -- which persist no "type" and recover it
from logType -- can fail when the migration is removed. Mutation-verified.
Tests:
Legacy413ImageMigrationTest all 20 legacy catalog labels migrate;
engine-owned types (doris) do not; type
recovery incl. the trino-connector hyphen;
comment/lastUpdateTime/taap/jdbc_url fidelity
Legacy413LabelRegistryTest every label 4.1.3 could write still resolves,
compared registry-to-registry rather than
against a hand-maintained list
Legacy413JournalReplayTest edit-log replay, property fidelity through to
the connector, degraded paths
Legacy413CloudImageTest cloud mode; one guard, not a second matrix,
because the cloud and non-cloud modules differ
in exactly one entry
Legacy413ProviderTypeContractTest the recovered type must be a string a REAL
shipped provider answers to -- the two halves
are literals in different modules
fe-core gains test-scope-only dependencies on the connector modules, matching
the existing fe-filesystem-* arrangement, solely so that last test can compare
both halves. Production still loads connectors from plugins/connector/.
Anti-vacuity: five mutations were applied and each turned the predicted test
red -- disabling the type backfill, dropping the TRINO_CONNECTOR case,
deleting a GSON label, reverting the replay guard, and removing a connector
from the shipped set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
morningman
force-pushed
the
catalog-spi-413-upgrade-compat-ut
branch
from
July 29, 2026 13:22
790206f to
95b7435
Compare
…vn test` can compile against them fe-connector-hms-hive-shade and fe-connector-paimon-hive-shade carry no sources: their Hive metastore content exists only once maven-shade has run. With both maven-jar-plugin and maven-shade-plugin bound to `package`, any build that stops earlier leaves those modules with nothing but an (empty) target/classes, and that is what the reactor then hands to their consumers. run-fe-ut.sh builds with `mvn test`, which stops before `package`. Once fe-core took test-scope dependencies on the connector impl modules, the FE UT reactor grew from 40 to 59 modules and began pulling fe-connector-hms in, where every org.apache.hadoop.hive.* import failed to resolve (TeamCity FE UT 1009434: "package org.apache.hadoop.hive.metastore.api does not exist", build aborted before fe-core ran a single test). fe-connector-paimon would have failed the same way on HiveConf one module later. Move both executions to process-classes -- the last phase still ahead of test-compile -- keeping maven-jar-plugin declared first so the main artifact is present before shade, which is what the package-phase pin was protecting under a parallel reactor. Cost is ~21s per FE UT run (paimon 17s, hms 4s), since the build cache is disabled at project level for both modules. Verified: the full 59-module `mvn -pl fe-common,fe-core -am test-compile` succeeds and reproducibly fails on fe-connector-hms without this change; both shade jars rebuild identical to the previous ones (same byte size, same 7940 / 55313 entry listings); `package` still runs each shade exactly once and the hive / paimon / iceberg / hudi plugin zips each carry their shade jar. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019JZp6xJNzwavabuea7bsAP
…-classes) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019JZp6xJNzwavabuea7bsAP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.