HIVE-29805: Write header/footer lines in HiveIgnoreKeyTextOutputForma… - #6684
Open
dengziming wants to merge 1 commit into
Open
HIVE-29805: Write header/footer lines in HiveIgnoreKeyTextOutputForma…#6684dengziming wants to merge 1 commit into
dengziming wants to merge 1 commit into
Conversation
…t for skip.header/footer.line.count
For a text table with `skip.header.line.count=N` (and/or
`skip.footer.line.count=M`), `INSERT` / `INSERT OVERWRITE` writes data files
that do not contain N header lines or M footer lines. On read,
`SkippingTextInputFormat` skips the first N lines of each file and the last M
lines of each file. When new files are appended, the first N data lines of each
appended file are skipped as if they were headers, causing data loss.
Reproduce with:
CREATE TABLE t (a string, b string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE
TBLPROPERTIES ('skip.header.line.count'='1');
INSERT OVERWRITE TABLE t VALUES ('x','y'),('a','b'),('c','d');
SELECT * FROM t; -- returned only ('a','b'), ('c','d') before the fix
This mirrors how Spark native CSV handles `header=true`: the writer emits a
header line before the data rows (see Spark's CsvOutputWriter). Hive already
has the reader-side skip (`SkippingTextInputFormat`, `RecordReaderWrapper`);
this change adds the writer-side header/footer emission to
`HiveIgnoreKeyTextOutputFormat`.
Spark writer reference:
https://github.com/apache/spark/blob/751f29f12933ef3626518e8ed7a13a225c37d326/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CsvOutputWriter.scala#L42-L44
The header line is built from the table column names using the table's delimiter
and quote properties; OpenCSVSerde tables quote all header fields by default,
other text SerDes quote only when required. Footer lines are emitted before the
file is closed.
Also updates the `skip_header_footer_proj.q` golden output to reflect that all
inserted rows are now returned, and adds `TestHiveIgnoreKeyTextOutputFormat` to
verify the round-trip.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
<!--
Thanks for sending a pull request! Here are some tips for you:
-->
What changes were proposed in this pull request?
For a text table with
skip.header.line.count=N(and/orskip.footer.line.count=M),INSERT/INSERT OVERWRITEwrites data files that do not contain N header lines or M footer lines. On read,SkippingTextInputFormatskips the first N lines of each file and the last M lines of each file. When new files are appended, the first N data lines of each appended file are skipped as if they were headers, causing data loss.Reproduce with:
This mirrors how Spark native CSV handles
header=true: the writer emits a header line before the data rows (see Spark's CsvOutputWriter). Hive already has the reader-side skip (SkippingTextInputFormat,RecordReaderWrapper); this change adds the writer-side header/footer emission toHiveIgnoreKeyTextOutputFormat.Spark writer reference:
https://github.com/apache/spark/blob/751f29f12933ef3626518e8ed7a13a225c37d326/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CsvOutputWriter.scala#L42-L44
Why are the changes needed?
This is a bug that would lead to data loss
Does this PR introduce any user-facing change?
Yes, fix
How was this patch tested?
Update the
skip_header_footer_proj.qgolden output to reflect that all inserted rows are now returned, and addTestHiveIgnoreKeyTextOutputFormatto verify the round-trip.