ORC-477: BloomFilter for ACID table does not get created #373
Conversation
deniskuzZ
force-pushed
the
branch-1.5
branch
3 times, most recently
from
bd0d862
to
5e19bdb
Compare
| for (String column: selectedColumns.split((","))) { | ||
| TypeDescription col = findColumn(column, fieldNames, fields); | ||
| TypeDescription col = findColumn((isAcid ? "row." + column : column), fieldNames, fields); |
There was a problem hiding this comment.
This code should ideally be in the Hive codebase and ORC should interpret the selectedColumns there.
There was a problem hiding this comment.
ACID schema generation code is under ORC SchemaEvolution class
public static TypeDescription createEventSchema(TypeDescription typeDescr) {
TypeDescription result = TypeDescription.createStruct()
.addField("operation", TypeDescription.createInt())
.addField("originalTransaction", TypeDescription.createLong())
.addField("bucket", TypeDescription.createInt())
.addField("rowId", TypeDescription.createLong())
.addField("currentTransaction", TypeDescription.createLong())
.addField("row", typeDescr.clone());
return result;
}
There was a problem hiding this comment.
I am thinking on simplifying this a bit by introducing extra param (isAcid) in findColumn method.
if (resultTypeDesc.getCategory() == TypeDescription.Category.STRUCT && isAcid) {
//unfold
resultTypeDesc = findColumn(columnName, resultTypeDesc.getFieldNames(), resultTypeDesc.getChildren(), false);
}
| } else { | ||
| if (resultTypeDesc.getCategory() == TypeDescription.Category.STRUCT | ||
| && fieldName.equalsIgnoreCase(StringUtils.substringBefore(columnName, DOT))) { |
There was a problem hiding this comment.
This should already work if Hive sends "row." in selected columns?
There was a problem hiding this comment.
not completely, column decr are embedded under row struct, so we need to unfold it.
deniskuzZ
force-pushed
the
branch-1.5
branch
3 times, most recently
from
6fbd9b0
to
325d582
Compare
|
@t3rmin4t0r do you know how to configure the AppVeyor build? The build phase is set to "MSBuild" mode (default), but no Visual Studio project or solution files were found in the root directory. If you are not building Visual Studio project switch build mode to "Script" and provide your custom build command. |
deniskuzZ
force-pushed
the
branch-1.5
branch
3 times, most recently
from
caa49c2
to
e2db246
Compare
No description provided.