[CARBONDATA-3054] Fix Dictionary file cannot be read in S3a #2876
Conversation
|
@ravipesala , @kunal642 : please review |
ajantha-bhat
changed the title
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1122/ |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1334/ |
|
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9386/ |
|
retest this please |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1126/ |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1339/ |
|
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9390/ |
|
retest this please |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1135/ |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1347/ |
|
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9399/ |
ajantha-bhat
force-pushed
the
master_new
branch
from
4928236
to
9563057
Compare
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1140/ |
|
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9405/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1352/ |
|
|
||
| val exprs = child.output.map { exp => | ||
| ExpressionCanonicalizer.execute(BindReferences.bindReference(exp, child.output)) | ||
| } | ||
| ctx.currentVars = input | ||
| val resultVars = exprs.zipWithIndex.map { case (expr, index) => | ||
| if (dicts(index) != null) { | ||
| ThreadLocalSessionInfo.setConfigurationToCurrentThread(conf.value.value) |
There was a problem hiding this comment.
Remove its usage from this place
|
|
||
| var dictionary: Dictionary = null | ||
|
|
||
| var dictionaryLoader: DictionaryLoader = _ | ||
|
|
||
| def getDictionaryValueForKeyInBytes (surrogateKey: Int): Array[Byte] = { | ||
| if (dictionary == null) { | ||
| ThreadLocalSessionInfo.getOrCreateCarbonSessionInfo().getNonSerializableExtraInfo |
There was a problem hiding this comment.
Move this in doConsume method inside if (CarbonDictionaryDecoder.isRequiredToDecode(getDictionaryColumnIds)) { as wrapper is not the correct place for setting ThreadLocalSessionInfo
There was a problem hiding this comment.
cannot me moved. If moved problem in serialization with code gen
…onDictionaryDecoder.doConsume() codeGen
ajantha-bhat
force-pushed
the
master_new
branch
from
9563057
to
6f1da20
Compare
|
LGTM...can be merged once build passes |
|
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9418/ |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1366/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1154/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1165/ |
|
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9429/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1378/ |
|
LGTM |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1173/ |
|
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9437/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1386/ |
…onDictionaryDecoder.doConsume() codeGen problem: In S3a environment, when queried the data which has dictionary files, Dictionary file cannot be read in S3a with CarbonDictionaryDecoder.doConsume() codeGen even though file is present. cause: CarbonDictionaryDecoder.doConsume() codeGen doesn't set hadoop conf in thread local variable, only doExecute() sets it. Hence, when getDictionaryWrapper() called from doConsume() codeGen, AbstractDictionaryCache.getDictionaryMetaCarbonFile() returns false for fileExists() operation. solution: In CarbonDictionaryDecoder.doConsume() codeGen, set hadoop conf in thread local variable This closes #2876
[CARBONDATA-3054] Fix Dictionary file cannot be read in S3a with CarbonDictionaryDecoder.doConsume() codeGen
problem: In S3a environment, when queried the data which has dictionary files,
Dictionary file cannot be read in S3a with CarbonDictionaryDecoder.doConsume() codeGen even though file is present.
cause: CarbonDictionaryDecoder.doConsume() codeGen doesn't set hadoop conf in thread local variable, only doExecute() sets it.
Hence, when getDictionaryWrapper() called from doConsume() codeGen,
AbstractDictionaryCache.getDictionaryMetaCarbonFile() returns false for fileExists() operation.
solution:
In CarbonDictionaryDecoder.doConsume() codeGen, set hadoop conf in thread local variable
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
Any interfaces changed? NA
Any backward compatibility impacted? NA
Document update required? NA
Testing done
done
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA