[CARBONDATA-2753] Fix Compatibility issues while querying on 1.3 Store #2534
Conversation
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6128/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7368/ |
|
Retest this please |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7370/ |
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6131/ |
|
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5941/ |
|
retest this please |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7372/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6133/ |
|
retest sdv please |
|
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5945/ |
| indexFiles.put(indexFilePath, null); | ||
| } | ||
| } catch (IOException e) { | ||
| LOGGER.error(e.getMessage()); |
There was a problem hiding this comment.
- Don't LOG the error here, throw IO exception back to caller
- Mention the compatibility version number and provide a detailed comment in this method for file Existence check including the version numbers
dhatchayani
force-pushed
the
CARBONDATA-2753_indexFiles
branch
from
e257177
to
24b8662
Compare
dhatchayani
changed the title
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7385/ |
|
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6146/ |
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5960/ |
|
LGTM |
Problem: Currently,in the segmentFile we are writing the index files list in files field, only if it exists, otherwise it will be empty(in case if it is merged to merge index file). But in the old store, we were writing both the files and mergeFileName fields even if the index files are merged. Solution: While querying we have to check the physical existence of the index files listed in the files field. If it physically exists, then we have to consider that. This closes #2534
Problem:
Currently,in the segmentFile we are writing the index files list in files field, only if it exists, otherwise it will be empty(in case if it is merged to merge index file). But in the old store, we were writing both the files and mergeFileName fields even if the index files are merged.
Solution:
While querying we have to check the physical existence of the index files listed in the files field. If it physically exists, then we have to consider that.
Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Manual Testing with Old Store
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.