[CARBONDATA-2767][CarbonStore] Fix task locality issue #2528
Conversation
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7315/ |
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6079/ |
dependency
QiangCai
changed the title
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7388/ |
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6149/ |
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5962/ |
| @@ -1882,6 +1882,13 @@ | |||
|
|
|||
| public static final String CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT = "true"; | |||
|
|
|||
| /** | |||
| * config carbon scan task locality | |||
There was a problem hiding this comment.
Please provide more detail, like what scheduling behavior will be used for true and false
| @@ -87,6 +87,8 @@ class CarbonScanRDD[T: ClassTag]( | |||
| } | |||
| private var vectorReader = false | |||
|
|
|||
| private val isTaskLocality = CarbonProperties.isTaskLocality | |||
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7444/ |
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6199/ |
store/sql/pom.xml
Outdated
| <dependency> | ||
| <groupId>com.amazonaws</groupId> | ||
| <artifactId>aws-java-sdk-s3</artifactId> | ||
| <version> 1.10.6</version> |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7472/ |
| .split | ||
| .value | ||
| .getLocations | ||
| .filter(_ != "localhost") |
There was a problem hiding this comment.
What will happen if I configure TaskLocality and run the job in local machine or local pseudo distributed mode?
Besides, if you really want to exclude local machine, except 'localhost', why host name of local machine is not considered?
There was a problem hiding this comment.
It doesn't want to exclude local machine.
Because maybe all machines have configured "127.0.0.1 localhost", "localhost" is unuseful.
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6229/ |
pom.xml
Outdated
| @@ -110,7 +110,7 @@ | |||
| <properties> | |||
| <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> | |||
| <snappy.version>1.1.2.6</snappy.version> | |||
| <hadoop.version>2.7.2</hadoop.version> | |||
| <hadoop.version>2.8.3</hadoop.version> | |||
There was a problem hiding this comment.
please add this in Horizon profile only
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7492/ |
|
LGTM |
If the Spark cluster and the Hadoop cluster are two different machine cluster, the Spark tasks will run in RACK_LOCAL mode. This closes #2528
|
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6253/ |
If the Spark cluster and the Hadoop cluster are two different machine cluster, the Spark tasks will run in RACK_LOCAL mode. This closes apache#2528
If the Spark cluster and the Hadoop cluster are two different machine cluster, the Spark tasks will run in RACK_LOCAL mode. This closes apache#2528
If the Spark cluster and the Hadoop cluster are two different machine cluster, the Spark tasks will run in RACK_LOCAL mode. This closes apache#2528
If the Spark cluster and the Hadoop cluster are two different machine cluster, the Spark tasks will run in RACK_LOCAL mode.
So no need to provide the preferred locations to the task.
Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.