docs(troubleshooting): note that a plain set only reaches an embedded metastore - #19554
Open
deepakpanda93 wants to merge 2 commits into
Open
docs(troubleshooting): note that a plain set only reaches an embedded metastore#19554deepakpanda93 wants to merge 2 commits into
deepakpanda93 wants to merge 2 commits into
Conversation
… metastore
The Hive Sync entry for "The following columns have types incompatible with
the existing columns in their respective positions" recommends
set hive.metastore.disallow.incompatible.col.type.changes=false;
but that statement writes only to the session HiveConf. SetProcessor routes a
variable to Hive#setMetaConf -> IMetaStoreClient#setMetaConf only when it
carries the metaconf: prefix, and HiveAlterHandler#alterTable evaluates the
check against handler.getConf(), the metastore's own configuration. Against a
remote metastore the recommended set therefore has no effect and the error
persists.
Keep the existing advice, which is correct for an embedded metastore, and add
the two options that work against a remote Hive metastore service: the
server's hive-site.xml, or a per-connection metaconf: override. The property
is a metaConfVars entry, so the server accepts the override.
Applied to next and to version-1.2.0, the current released docs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Testing the guidance against Hive 3.1.3 showed the metaconf: form is not universal across Hive clients. Over HiveServer2 (Beeline) it works: `set metaconf:...=false` followed by `ALTER TABLE ... REPLACE COLUMNS` succeeded, taking the table from (id, name, ts) to (id, age, name, ts). The legacy Hive CLI is different. It does reach the metastore -- reading the value back with `set metaconf:<key>` returns false where it returned true before -- but the subsequent ALTER is still rejected with "The following columns have types incompatible with the existing columns in their respective positions", because the connection that runs the DDL is not the one carrying the override. A plain `set` fails on both clients, as already documented. Name Beeline explicitly and point Hive CLI users at the hive-site.xml option. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hudi-agent
reviewed
Aug 7, 2026
hudi-agent
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the docs update! This change clarifies that a plain set hive.metastore.disallow.incompatible.col.type.changes=false only affects an embedded metastore and adds correct guidance for the remote case (metastore hive-site.xml or a metaconf:-prefixed Beeline session), including the Hive CLI caveat. The technical claims align with how the Hive metastore evaluates this check server-side, and the change is purely additive across both the next and 1.2.0 docs. Please have a Hudi committer or PMC member give it a final review before merging.
cc @yihua
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
The Hive Sync troubleshooting entry for
tells the reader to run:
That statement writes only to the client's session
HiveConf. The metastore evaluates this check against its ownconfiguration —
HiveAlterHandler#alterTablereadshandler.getConf()— so against a remote metastore the recommendedsethas no effect and the error persists. The reader does the one thing the page tells them to do, sees no change, andhas nothing to go on.
No GitHub issue for this one; it surfaced while working on #19551, which documents the same property from the Schema
Evolution side.
Summary and Changelog
The existing paragraph and its snippet are left untouched — they are correct for an embedded metastore. Appended a short
paragraph plus one snippet covering the remote case:
hive-site.xmland restart the service, orset metaconf:..., which pushes the value to the metastore for thatconnection.
Plus a one-line caveat that the
metaconf:form applies over HiveServer2 and not the legacy Hive CLI (see below).Applied to
website/docs/troubleshooting.md(next) andwebsite/versioned_docs/version-1.2.0/troubleshooting.md(current released docs), per the next-plus-current conventionused in #19473. The nine older versioned copies carry the same text and were left alone — happy to widen if preferred.
Purely additive: no existing sentence, snippet, or heading was changed or removed, so inbound links and anchors are
unaffected.
Where this comes from in the code
Hive
rel/release-3.1.3:HiveAlterHandler.java:164-169— the check readshandler.getConf(), the metastore's conf, not the client's.SetProcessor.java:202-210— only ametaconf:-prefixed variable is routed toHive#setMetaConf; a plainsetjustupdates the session conf.
Hive.java:4625-4631—setMetaConfforwards toIMetaStoreClient#setMetaConf.MetastoreConf.java:275—DISALLOW_INCOMPATIBLE_COL_TYPE_CHANGESis inmetaConfVars, so the server accepts theoverride.
Reproduction
Verified against a standalone Hive 3.1.3 metastore in Docker. Table starts as
(id int, name string, ts bigint);the statement under test is
ALTER TABLE ts_test.t REPLACE COLUMNS (id int, age int, name string, ts bigint), whichshifts an
intinto a position astringoccupied — incompatible perColumnType#areColTypesCompatible.set ...=falseset metaconf:...=falsedescribereturnsid, age, name, tsThe plain-
setrows are the point of this PR: the currently-documented remedy does not work against a remote metastoreon either client.
Separately verified that the other two remedies do work against the same remote metastore: adding the property to the
metastore's
hive-site.xmland restarting lifts the check with no client-side setting at all, and a rawHiveMetaStoreClientcallingsetMetaConfthenalter_tableon one client object succeeds. Both were measured whiletesting #19551.
Why the caveat about the Hive CLI
Worth spelling out, because the Hive CLI does not fail at the protocol level — the metastore visibly accepts the value:
…and the very next
ALTERin that same session is still rejected.setMetaConfbinds to the metastore connection thatreceived it —
HMSHandlerholds it in a thread-local conf — and the CLI's DDL path does not run on that connection.HiveServer2 does keep one session on one connection, which is why Beeline works. Since a user could easily read the
value back, conclude it took effect, and be stuck, the page names Beeline explicitly and points Hive CLI users at the
hive-site.xmloption.Site verification
npm run buildpasses with the warning set byte-identical to a baseline build of the same base commit — no new brokenlinks or anchors.
/docs/troubleshootingand/docs/next/troubleshootingwere loaded fromnpm run serve; the newparagraph and
sqlsnippet render in the right section, the surrounding####anchors (hive-sync,sqlexception-following-columns-have-types-incompatible, and the followinghoodiehivesyncexception-...) are allintact, and the
<h4>count is unchanged from the untouched older versions.Impact
Documentation only. No code, config, or behaviour change.
Risk Level
none
Documentation Update
This PR is the documentation update — the Troubleshooting page,
/docs/troubleshootingand/docs/next/troubleshooting.Contributor's checklist