Skip to content

docs(troubleshooting): note that a plain set only reaches an embedded metastore - #19554

Open
deepakpanda93 wants to merge 2 commits into
apache:asf-sitefrom
deepakpanda93:docs/troubleshooting-metaconf-remote-hms
Open

docs(troubleshooting): note that a plain set only reaches an embedded metastore#19554
deepakpanda93 wants to merge 2 commits into
apache:asf-sitefrom
deepakpanda93:docs/troubleshooting-metaconf-remote-hms

Conversation

@deepakpanda93

Copy link
Copy Markdown
Collaborator

Describe the issue this Pull Request addresses

The Hive Sync troubleshooting entry for

Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions : col1,col2

tells the reader to run:

set hive.metastore.disallow.incompatible.col.type.changes=false;

That statement writes only to the client's session HiveConf. The metastore evaluates this check against its own
configuration — HiveAlterHandler#alterTable reads handler.getConf() — so against a remote metastore the recommended
set has no effect and the error persists. The reader does the one thing the page tells them to do, sees no change, and
has nothing to go on.

No GitHub issue for this one; it surfaced while working on #19551, which documents the same property from the Schema
Evolution side.

Summary and Changelog

The existing paragraph and its snippet are left untouched — they are correct for an embedded metastore. Appended a short
paragraph plus one snippet covering the remote case:

  • Set the property in the metastore's hive-site.xml and restart the service, or
  • override it for a single Beeline session with set metaconf:..., which pushes the value to the metastore for that
    connection.

Plus a one-line caveat that the metaconf: form applies over HiveServer2 and not the legacy Hive CLI (see below).

Applied to website/docs/troubleshooting.md (next) and
website/versioned_docs/version-1.2.0/troubleshooting.md (current released docs), per the next-plus-current convention
used in #19473. The nine older versioned copies carry the same text and were left alone — happy to widen if preferred.

Purely additive: no existing sentence, snippet, or heading was changed or removed, so inbound links and anchors are
unaffected.

Where this comes from in the code

Hive rel/release-3.1.3:

  • HiveAlterHandler.java:164-169 — the check reads handler.getConf(), the metastore's conf, not the client's.
  • SetProcessor.java:202-210 — only a metaconf:-prefixed variable is routed to Hive#setMetaConf; a plain set just
    updates the session conf.
  • Hive.java:4625-4631setMetaConf forwards to IMetaStoreClient#setMetaConf.
  • MetastoreConf.java:275DISALLOW_INCOMPATIBLE_COL_TYPE_CHANGES is in metaConfVars, so the server accepts the
    override.

Reproduction

Verified against a standalone Hive 3.1.3 metastore in Docker. Table starts as (id int, name string, ts bigint);
the statement under test is ALTER TABLE ts_test.t REPLACE COLUMNS (id int, age int, name string, ts bigint), which
shifts an int into a position a string occupied — incompatible per ColumnType#areColTypesCompatible.

Client plain set ...=false set metaconf:...=false
Beeline (HiveServer2) ❌ fails succeedsdescribe returns id, age, name, ts
Hive CLI ❌ fails ❌ fails

The plain-set rows are the point of this PR: the currently-documented remedy does not work against a remote metastore
on either client.

Separately verified that the other two remedies do work against the same remote metastore: adding the property to the
metastore's hive-site.xml and restarting lifts the check with no client-side setting at all, and a raw
HiveMetaStoreClient calling setMetaConf then alter_table on one client object succeeds. Both were measured while
testing #19551.

Why the caveat about the Hive CLI

Worth spelling out, because the Hive CLI does not fail at the protocol level — the metastore visibly accepts the value:

hive> set metaconf:hive.metastore.disallow.incompatible.col.type.changes;
metaconf:hive.metastore.disallow.incompatible.col.type.changes=true
hive> set metaconf:hive.metastore.disallow.incompatible.col.type.changes=false;
hive> set metaconf:hive.metastore.disallow.incompatible.col.type.changes;
metaconf:hive.metastore.disallow.incompatible.col.type.changes=false

…and the very next ALTER in that same session is still rejected. setMetaConf binds to the metastore connection that
received it — HMSHandler holds it in a thread-local conf — and the CLI's DDL path does not run on that connection.
HiveServer2 does keep one session on one connection, which is why Beeline works. Since a user could easily read the
value back, conclude it took effect, and be stuck, the page names Beeline explicitly and points Hive CLI users at the
hive-site.xml option.

Site verification

npm run build passes with the warning set byte-identical to a baseline build of the same base commit — no new broken
links or anchors. /docs/troubleshooting and /docs/next/troubleshooting were loaded from npm run serve; the new
paragraph and sql snippet render in the right section, the surrounding #### anchors (hive-sync,
sqlexception-following-columns-have-types-incompatible, and the following hoodiehivesyncexception-...) are all
intact, and the <h4> count is unchanged from the untouched older versions.

Impact

Documentation only. No code, config, or behaviour change.

Risk Level

none

Documentation Update

This PR is the documentation update — the Troubleshooting page, /docs/troubleshooting and /docs/next/troubleshooting.

Contributor's checklist

  • Read through contributor's guide
  • Enough context is provided in the sections above
  • Adequate tests were added if applicable

deepakpanda93 and others added 2 commits August 7, 2026 12:17
… metastore

The Hive Sync entry for "The following columns have types incompatible with
the existing columns in their respective positions" recommends

    set hive.metastore.disallow.incompatible.col.type.changes=false;

but that statement writes only to the session HiveConf. SetProcessor routes a
variable to Hive#setMetaConf -> IMetaStoreClient#setMetaConf only when it
carries the metaconf: prefix, and HiveAlterHandler#alterTable evaluates the
check against handler.getConf(), the metastore's own configuration. Against a
remote metastore the recommended set therefore has no effect and the error
persists.

Keep the existing advice, which is correct for an embedded metastore, and add
the two options that work against a remote Hive metastore service: the
server's hive-site.xml, or a per-connection metaconf: override. The property
is a metaConfVars entry, so the server accepts the override.

Applied to next and to version-1.2.0, the current released docs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Testing the guidance against Hive 3.1.3 showed the metaconf: form is not
universal across Hive clients.

Over HiveServer2 (Beeline) it works: `set metaconf:...=false` followed by
`ALTER TABLE ... REPLACE COLUMNS` succeeded, taking the table from
(id, name, ts) to (id, age, name, ts).

The legacy Hive CLI is different. It does reach the metastore -- reading the
value back with `set metaconf:<key>` returns false where it returned true
before -- but the subsequent ALTER is still rejected with "The following
columns have types incompatible with the existing columns in their respective
positions", because the connection that runs the DDL is not the one carrying
the override. A plain `set` fails on both clients, as already documented.

Name Beeline explicitly and point Hive CLI users at the hive-site.xml option.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added docs size:S PR with lines of changes in (10, 100] labels Aug 7, 2026

@hudi-agent hudi-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the docs update! This change clarifies that a plain set hive.metastore.disallow.incompatible.col.type.changes=false only affects an embedded metastore and adds correct guidance for the remote case (metastore hive-site.xml or a metaconf:-prefixed Beeline session), including the Hive CLI caveat. The technical claims align with how the Hive metastore evaluates this check server-side, and the change is purely additive across both the next and 1.2.0 docs. Please have a Hudi committer or PMC member give it a final review before merging.

cc @yihua

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs size:S PR with lines of changes in (10, 100]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants