refactor(common)!: reorganize hudi-common packages into core and common namespaces#19193
Merged
vinothchandar merged 12 commits intoJul 6, 2026
Merged
Conversation
Member
Author
|
No code changes, just classes being moved around packages for a better grouping. Specifically, I am splitting classes into |
…tion.lock Relocates InProcessLockProvider and NoopLockProvider out of org.apache.hudi.client.transaction.lock in hudi-common, pre-staging the future hudi-common/hudi-core module split so the eventual module move stays FQN-stable. - InProcessLockProvider: the old FQN is a documented, user-configured value of hoodie.write.lock.provider resolved via reflection, so a deprecated stub is kept at the old FQN, marked with the new @CompatAlias annotation (hudi-io) recording the replacement and the 1.3.0 deprecation release. The stub preserves the reflective (LockConfiguration, StorageConfiguration) constructor contract. - NoopLockProvider: internal-only (upgrade/downgrade nested-lock path), moved without a stub. - Sites that string-compare the configured lock provider class now use InProcessLockProvider.isInProcessLockProvider(), which accepts both the current and legacy FQN: the 1ms clock-skew inference in HoodieTimeGeneratorConfig, the metadata-table multi-writer guard in HoodieMetadataWriteUtils, and the single-writer auto-adjust in HoodieWriteConfig. Without this, configs carrying the legacy FQN would silently lose those behaviors. Behavior-preserving otherwise. Legacy-FQN test configs are deliberately left in place as regression coverage for the stub path; a reflective load test and a legacy-FQN clock-skew row are added.
…pache.hudi.core.validator Relocates BasePreCommitValidator and ValidationContext from org.apache.hudi.client.validator to org.apache.hudi.core.validator in hudi-common, continuing the core-package staging for the future hudi-common/hudi-core module split. BREAKING CHANGE: both classes shipped in 1.2.0 as @PublicAPIClass(EVOLVING) and are the extension point for custom pre-commit validators on the Flink path (hoodie.precommit.validators). A custom validator compiled against 1.2.0 fails at load with NoClassDefFoundError and must be recompiled with the new import. This is an accepted EVOLVING-API change: the config only ever names the user's concrete class (FQN unchanged), all shipped validators move together, and no compatibility stub is provided deliberately - a stub at the old FQN would turn externally compiled overrides of validateWithMetadata(ValidationContext) into dead overloads once the parameter type moves, silently skipping the user's validation, which is worse than the loud load failure. Engine subclasses and utils in hudi-client-common, hudi-utilities and hudi-flink update their imports; StreamingOffsetValidator and its test previously resolved the base types via same-package access and now import them explicitly. The test also accessed the protected config field cross-package, which the move surfaced; it now goes through an accessor on the test's mock subclass. Verified with the client-common, utilities and flink validator suites.
…mon.config.metrics Relocates the seven metrics config classes (HoodieMetricsConfig and the CloudWatch/Datadog/Graphite/Jmx/M3/Prometheus variants) from org.apache.hudi.config.metrics to org.apache.hudi.common.config.metrics, joining HoodieCommonConfig and HoodieMetadataConfig in the hudi-common config namespace. The old package was a leftover from the classes' original hudi-client-common home; the new location also avoids adding a split package under hudi-io's org.apache.hudi.common.metrics. BREAKING CHANGE: source/binary break for code that imports these classes to build configs programmatically (e.g. HoodieWriteConfig.Builder#withMetricsConfig) - fails loudly with NoClassDefFoundError, fixed by an import change. Regular usage is unaffected: all hoodie.metrics.* config keys are unchanged, reporter selection is enum-based (no class names in config values), and custom reporters via hoodie.metrics.reporter.class are instantiated with a (Properties, MetricRegistry) constructor that never exposes these types. No compatibility stub: inherited static builder methods would fail exact-descriptor resolution on the nested Builder types anyway. Note: mixing a new engine bundle with an older hudi-aws-bundle fails at metrics init for CloudWatch (reflective constructor lookup); bundle versions must match, as already required. Verified across hudi-common, client-common, spark/java clients, hudi-spark, utilities, sync-common, hive-sync and hudi-aws with the metrics config/reporter test suites.
…re.io Relocates the 19 domain-aware file I/O contract classes in hudi-common from org.apache.hudi.io[.storage] to org.apache.hudi.core.io[.storage]: HoodieFileReader/Writer and their factories, HoodieIOFactory, HFile/bootstrap readers, HoodieParquetConfigInjector, and the ORC/HFile config holders. This establishes org.apache.hudi.core as the namespace for upper-layer code that depends on both hudi-io primitives and hudi-common domain types (HoodieRecord, HoodieSchema), disambiguating it from the hudi-io module's low-level org.apache.hudi.io namespace. Engine implementations (org.apache.hudi.io.storage.hadoop and the Spark/Flink packages) do not move; hoodie.io.factory.class defaults and values are unchanged. BREAKING CHANGE: source/binary break for externally compiled HoodieIOFactory subclasses (@PublicAPIClass EVOLVING, plugged via hoodie.io.factory.class) and HoodieParquetConfigInjector implementations (hoodie.parquet.write.config.injector.class, since 1.2.0) - both fail loudly at load with NoClassDefFoundError and are fixed by an import change plus recompile. Config values themselves are untouched in both cases (they name the user's or the shipped implementation class). Regular property-based usage is unaffected; no config key, reflection, serialization or bundle surface changes. Same-package consumers of the moved contracts (Spark reader/writer classes remaining in org.apache.hudi.io.storage, and tests) now import them explicitly. The hudi-trino-plugin import updates are verified by its own CI (module is not in the default reactor), as is the hudi-integ-test import and the non-active spark version modules. Verified with -Dspark4.1 across common, hadoop-common, client-common, spark/java clients, hudi-spark, hadoop-mr and utilities, and with -Dflink2.1 across the flink modules; HFile reader/factory, file writer factory and parquet config injector suites pass.
Deletes org.apache.hudi.parquet.io.ByteBufferBackedInputFile and with it the org.apache.hudi.parquet.io package, which contained nothing else. The class has zero references in the repository - orphaned by an earlier refactor of the parquet log-block read path. BREAKING CHANGE: the class shipped in releases up to 1.2.0, so externally compiled code importing this helper would fail loudly at class load. It is a 30-line self-contained utility; any such user can inline it. Never referenced by any config value or reflection path.
Relocates org.apache.hudi.ParquetAdapter to org.apache.hudi.stats, the package of its only two consumers (ValueType, ValueMetadata). The interface is a parquet-version compatibility shim serving column-stats typing exclusively; its two implementations remain in org.apache.parquet.schema (deliberate namespace residency for parquet API access) and are loaded reflectively via class literals, so they follow the move automatically. BREAKING CHANGE: source/binary break for external code importing this interface - unlikely in practice, since the static getAdapter() factory hard-wires the only two shipped implementations. Fails loudly at class load if it exists anywhere; never referenced by config or reflection by name.
…ore.read Relocates the engine-agnostic file index base (and its unit test) from the org.apache.hudi root package to org.apache.hudi.core.read, establishing the read-side home in the core namespace for domain-aware query/listing abstractions (Phase 4 read-path work lands alongside). It is a consumer of the metadata subsystem, not part of it, so org.apache.hudi.metadata was deliberately not chosen. BREAKING CHANGE: source/binary break for external code importing BaseHoodieTableFileIndex or its nested types (PartitionPath, FileStatusCache) directly. Engine subclasses shipped by Hudi keep their FQNs (HiveHoodieTableFileIndex, SparkHoodieTableFileIndex), and engine bundles are self-consistent, so existing Presto/Trino deployments are unaffected at runtime; connector source trees that import the base class directly see a compile-time import fix at their next Hudi version bump. Never a config value or reflection target. The move surfaced several same-package couplings, now explicit: the spark-common scala classes in package org.apache.hudi (file index and index-support family) referenced the base bare and now import it, HoodieBaseRelation read the protected metadataConfig field directly and now uses the getter, and the read-only introspection methods getAllQueryPartitionPaths/areAllPartitionPathsCached used by tests across the old package boundary are now public. Verified across common, hadoop-common, hadoop-mr, spark modules and utilities with -Dspark4.1; file index suites pass.
…expression Relocates the engine-agnostic predicate/expression AST (Expression, Predicates, Literal, NameReference, bind visitors, StructLike/ArrayData) from org.apache.hudi.expression to org.apache.hudi.common.expression. The package has no Hudi domain awareness - its only dependencies are the internal type system and Option - and its consumers span the read path, engine bridges and metastore sync tools alike, making it shared substrate (common) rather than domain machinery (core). Keeps the dependency layering acyclic for the future module split: types -> expressions -> domain -> engines. BREAKING CHANGE: source/binary break for external code constructing Hudi expressions programmatically, e.g. callers of HoodieTableMetadata.getPartitionPathWithPathPrefixUsingFilterExpression. Fails loudly; fixed by an import change. Expression classes are never named in configs and never loaded reflectively.
…ileIndex Follow-up to the core.read move: scala brace-group imports name classes without their package prefix, so the FQN rewrite missed this one test. Split the moved class out of the brace group.
…ema.internal Relocates the schema-evolution type system (InternalSchema, Type/Types, table-change actions, avro converters, visitors, and the file-based schema storage manager) from org.apache.hudi.internal.schema to org.apache.hudi.common.schema.internal, nesting it under the existing org.apache.hudi.common.schema namespace. Like the expression AST that sits on top of it, this is shared substrate consumed across every engine and the sync tools - common, not domain core - and "internal" as a top-level package name was an artifact of its origin. BREAKING CHANGE: source/binary break for external code importing these types, e.g. schema-evolution integrations using InternalSchema or SerDeHelper directly. Fails loudly at compile/class-load; fixed by an import change. No persisted format impact: serialized schemas (commit metadata and .schema files) carry no Java class names, and none of these classes are config values or reflection targets. All engine integrations (spark, flink, hive, utilities) move in lockstep in-repo. Verified with the full schema-evolution unit suites (SerDe, avro evolution, table changes, converter, schema storage manager) across the spark4.1 chain; version-shim modules for other spark/flink versions are import-only changes covered by CI.
…released hudi hudi-trino-plugin is a vendored copy of the Trino connector that compiles against released Hudi artifacts (dep.hudi.version=1.0.2, parent io.trino:trino-root), not against the in-repo modules. The org.apache.hudi.core.io.storage package introduced by the file I/O contract move does not exist in 1.0.2, so the earlier blanket import rewrite would have broken the plugin build. Restores the two files to the released-version imports; the plugin picks up the new package names whenever dep.hudi.version is bumped to a release containing them.
CI (TestSevenToEightUpgrade cases with hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.NoopLockProvider) showed the old NoopLockProvider FQN is exercised as a user config value in the table-upgrade path, invalidating the earlier internal-only assessment that let it move without an alias. Restores a deprecated @CompatAlias stub at the old FQN, mirroring the InProcessLockProvider treatment, and adds a reflective-load regression test covering the exact LockManager code path that failed.
vinothchandar
force-pushed
the
code-sweep-hudi-common
branch
from
bbd230a to
518ad60
Compare
Collaborator
vinothchandar
commented
Jul 6, 2026
vinothchandar
left a comment
vinothchandar
left a comment
Member
Author
There was a problem hiding this comment.
Made a full pass. mostly import changes outside of the few places where FQN of new lock provider is wired etc.
Member
Author
|
@bhasudha can you please stamp this. CI is passing now |
This was referenced Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
hudi-commonhas accumulated classes in packages that no longer reflect their role: engine-facing domain machinery sitting in the rootorg.apache.hudipackage, shared substrate under a top-levelinternal.*namespace, write-client config classes outside the common config namespace, and lock/validator/file-IO contracts underorg.apache.hudi.client.*/org.apache.hudi.io.*where they collide conceptually with other modules' namespaces.This PR establishes a deliberate two-tier package taxonomy inside
hudi-common, without changing the module structure:org.apache.hudi.core.*— domain-aware code implementing core read/write behavior (depends on bothhudi-ioprimitives and Hudi domain models).org.apache.hudi.common.*— shared substrate used across all modules and engines (type system, expressions, configs, utilities), with no domain awareness.This pre-stages a future
hudi-common/hudi-coremodule split so that the eventual module move is FQN-stable, and removes one piece of dead code found along the way.Summary and Changelog
No behavior changes; this is a package reorganization (423 files, overwhelmingly import-line churn). One commit per relocation:
refactor(common):InProcessLockProvider,NoopLockProvidermoved too.a.h.core.transaction.lock. Because the oldInProcessLockProviderFQN is a documentedhoodie.write.lock.providerconfig value resolved via reflection, a deprecated stub is kept at the old FQN, marked with a new@CompatAlias(of, since)annotation (added next to@PublicAPIClassinhudi-io). The three sites that string-compare the configured provider class (clock-skew inference, MDT multi-writer guard, single-writer auto-adjust) now accept both old and new FQNs via one predicate.refactor(common)!:BasePreCommitValidator,ValidationContextmoved too.a.h.core.validator.refactor(common)!: 7 metrics config classes moved too.a.h.common.config.metrics, joiningHoodieCommonConfig/HoodieMetadataConfig; avoids a split package withhudi-io'so.a.h.common.metrics.refactor(common)!: 19 file I/O contract classes (HoodieFileReader/Writerand factories,HoodieIOFactory, HFile/bootstrap readers,HoodieParquetConfigInjector) moved too.a.h.core.io[.storage]. Engine implementations (o.a.h.io.storage.hadoop.*, Spark/Flink packages) do not move.refactor(common)!: removedByteBufferBackedInputFileand itso.a.h.parquet.iopackage — zero references repo-wide (orphaned by an earlier parquet log-block refactor).refactor(common)!:ParquetAdaptermoved too.a.h.stats, next to its only consumers.refactor(common)!:BaseHoodieTableFileIndexmoved too.a.h.core.read.refactor(common)!:expressionpackage moved too.a.h.common.expression.refactor(common)!:internal.schema(schema evolution type system, 21 classes) moved too.a.h.common.schema.internal, nested under the existingo.a.h.common.schema.fix(trino):hudi-trino-pluginwas excluded from the sweep — it compiles against released Hudi 1.0.2, not the in-repo modules; its imports update whendep.hudi.versionis bumped.The moves surfaced several hidden same-package couplings, now made explicit: spark-common scala classes importing the file-index base explicitly instead of bare same-package references;
HoodieBaseRelationusing themetadataConfiggetter instead of cross-package protected field access; two read-only file-index introspection methods made public; a validator test accessing protected state through an accessor on its mock. No code was copied.Impact
No config keys, defaults, or semantics change. No storage-format impact: none of the moved FQNs are persisted (
hoodie.properties, commit metadata, and serialized schemas carry no Java class names), verified per move.Breaking (source/binary) for externally compiled code only, always failing loudly (
NoClassDefFoundError/compile error), never silently:BasePreCommitValidator(EVOLVING API, shipped only in 1.2.0; thehoodie.precommit.validatorsconfig names the user's own class, which is unchanged).withMetricsConfig(..)builders,.key()constants). Property-based metrics configuration is unaffected; reporter selection is enum-based.HoodieIOFactorysubclasses /HoodieParquetConfigInjectorimplementations (both EVOLVING-era plug points; the config values name the user's or shipped implementation class, unchanged — e.g. defaultorg.apache.hudi.io.storage.hadoop.HoodieHadoopIOFactory).BaseHoodieTableFileIndex(engine subclass FQNsHiveHoodieTableFileIndex/SparkHoodieTableFileIndexare unchanged and bundles are self-consistent, so existing Presto/Trino deployments are unaffected at runtime), HudiExpressionconstruction, andInternalSchema/SerDeHelperusage in custom integrations.Users configuring Hudi via
hoodie.*properties see zero impact; existing configs carrying the oldInProcessLockProviderFQN keep working via the compatibility stub. No performance impact.Risk Level
low — mechanical relocations with compatibility handling where user configs are involved. Verification: full 52-module reactor build green (
-Dspark4.1 -Dflink2.1, checkstyle/scalastyle enforced); ~330 targeted unit tests across all moved areas (lock provider incl. reflective load of the legacy FQN, validators across client-common/utilities/flink, metrics configs/reporters, HFile reader/writer factories, file index, expressions, full schema-evolution suites); functional verification viaTestHoodieClientMultiWriter(47 tests, concurrent writers through LockManager) and transaction/lock-manager/time-generator suites covering every reflection path touched.Documentation Update
none — no config keys or defaults change. The config reference regenerates class-location mentions automatically (doc strings referencing moved FQNs were updated in-repo).
Contributor's checklist