Skip to content

ORC-248. PhysicalFsWriter sometimes passes negative padding down to shims#259

Merged
asfgit merged 1 commit intoapache:masterfrom
omalley:orc-248
May 1, 2018
Merged

ORC-248. PhysicalFsWriter sometimes passes negative padding down to shims#259
asfgit merged 1 commit intoapache:masterfrom
omalley:orc-248

Conversation

@omalley
Copy link
Contributor

@omalley omalley commented Apr 30, 2018

The PhysicalFsWriter sometimes pass down negative sizes for padding, which leads to exceptions.

My fix:

  • Add a new option to enable writing variable length HDFS blocks "orc.write.shortened.blocks", which is off by default.
  • Modified the shim to shortenBlock so that the PhysicalFsWriter can track the offsets.
  • shortenBlock isn't gated by the padding tolerance, although it is called when the new stripe would cross a block boundary.

@omalley omalley force-pushed the orc-248 branch 2 times, most recently from 9423c71 to 09d71b5 Compare April 30, 2018 22:29
omalley added a commit to omalley/orc that referenced this pull request Apr 30, 2018
requested padding to the shims.

Fixes apache#259

Signed-off-by: Owen O'Malley <omalley@apache.org>
omalley added a commit to omalley/orc that referenced this pull request May 1, 2018
requested padding to the shims.

Fixes apache#259

Signed-off-by: Owen O'Malley <omalley@apache.org>
IS_SCHEMA_EVOLUTION_CASE_SENSITIVE("orc.schema.evolution.case.sensitive", "orc.schema.evolution.case.sensitive", true,
"A boolean flag to determine if the comparision of field names in schema evolution is case sensitive .\n")
"A boolean flag to determine if the comparision of field names in schema evolution is case sensitive .\n"),
WRITE_SHORTENED_BLOCKS("orc.write.shortened.blocks", null, false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might as well call it what it is "orc.write.variable.length.blocks"

omalley added a commit to omalley/orc that referenced this pull request May 1, 2018
requested padding to the shims.

Fixes apache#259

Signed-off-by: Owen O'Malley <omalley@apache.org>
requested padding to the shims.

Fixes apache#259

Signed-off-by: Owen O'Malley <omalley@apache.org>
@asfgit asfgit merged commit 837a1bd into apache:master May 1, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants