HIVE-29754: Remove empty HIVE_UNION_SUBDIR moved directories after INSERT OVERWRITE UNION ALL

[Indhumathi27]

What changes were proposed in this pull request?

Added cleanup of the empty -tmp.HIVE_UNION_SUBDIR_N.moved staging directory
in Utilities.mvFileToFinalPath() when a UNION ALL branch produces 0 rows.

Change: a single else if (!avoidRename) block added after the
if (statuses != null && statuses.length > 0) block in mvFileToFinalPath()
to delete the .moved staging directory when it contains no output files.

Why are the changes needed?

When executing INSERT OVERWRITE ... UNION ALL with the Tez execution engine,
if one UNION ALL branch produces 0 rows (e.g. an anti-join that returns no
results), an empty directory named -tmp.HIVE_UNION_SUBDIR_<N>.moved is left
behind in the final partition directory after the query completes successfully.

Root cause in Utilities.mvFileToFinalPath():

1. Line 1474: `-tmp.HIVE_UNION_SUBDIR_1` is renamed to
   `-tmp.HIVE_UNION_SUBDIR_1.moved` unconditionally as a fencing mechanism
   to prevent runaway Tez task attempts from writing new files during commit.

2. Line 1482: getFileStatusRecurse() returns an empty array because the
   branch produced no output files.

3. Line 1485: if (statuses != null && statuses.length > 0) evaluates to
   false — the entire block is skipped.

4. The `.moved` directory that was created in step 1 is never deleted.
   MoveTask subsequently moves it to the final partition location.

The success=false path at line 1515 correctly calls fs.delete(tmpPath, true),
but the equivalent cleanup was missing from the success=true + empty output path.

This does not occur with MapReduce execution engine.

Does this PR introduce any user-facing change?

No

How was this patch tested?

q file has been added

[Indhumathi27]

@deniskuzZ / @abstractdog Please help to review. Thanks

[Indhumathi27]

@ayushtkn / @zabetak can you help to review the PR. Thanks

[Copilot]

Pull request overview

This PR addresses HIVE-29754 by ensuring Tez INSERT OVERWRITE ... UNION ALL commits do not leave behind empty -tmp.HIVE_UNION_SUBDIR_<N>.moved staging directories when a UNION branch produces zero rows, and adds a regression qtest to validate the behavior.

Changes:

• Add success-path cleanup for empty .moved staging directories in Utilities.mvFileToFinalPath().
• Refactor mvFileToFinalPath() into smaller helpers (computeAvoidRename, renameTmpIfNeeded, removeDuplicatesAndFillBuckets, moveKeptFiles) and update call sites to the adjusted signature.
• Add a new Tez positive qtest (query + expected output) and include it in the minitez test list.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java	Implements empty-branch .moved staging directory cleanup and refactors commit/move logic into helper methods.
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java	Updates mvFileToFinalPath invocation to match the new signature.
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java	Updates mvFileToFinalPath invocation to match the new signature.
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/truncate/ColumnTruncateMapper.java	Updates mvFileToFinalPath invocation to match the new signature.
ql/src/test/queries/clientpositive/union_all_empty_branch_no_tmp_dir.q	Adds a Tez query test covering both non-empty and empty UNION branch scenarios (including flattened subdirs mode).
ql/src/test/results/clientpositive/tez/union_all_empty_branch_no_tmp_dir.q.out	Adds the expected output baseline for the new Tez qtest.
itests/src/test/resources/testconfiguration.properties	Registers the new qtest in the minitez query list.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sonarqubecloud]

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.3% Duplication on New Code

See analysis details on SonarQube Cloud