Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KYLIN-3714 Register kryo for spark spilling process. #487

Merged
merged 1 commit into from Feb 27, 2019

Conversation

Sidonet
Copy link

@Sidonet Sidonet commented Feb 26, 2019

After patch was applied in my env, i test it.

In my test case, i set properties
kylin.engine.spark-conf.spark.driver.memory=2G
kylin.engine.spark-conf.spark.executor.memory=512M

Launch build a huge cube, with no mapreduce.input.fileinputformat.split at all.

The most huge data goes to executor 20 (4.6G) and spark spilling process goes here:

2019-02-26 17:55:13 INFO ShuffleBlockFetcherIterator:54 - Getting 3898 non-empty blocks out of 3898 blocks
2019-02-26 17:55:13 INFO ShuffleBlockFetcherIterator:54 - Started 6 remote fetches in 39 ms
2019-02-26 17:55:15 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (1 time so far)
2019-02-26 17:55:40 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (2 times so far)
2019-02-26 17:56:02 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.6 MB to disk (3 times so far)
2019-02-26 17:56:25 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (4 times so far)
2019-02-26 17:56:54 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (5 times so far)
2019-02-26 17:57:18 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (6 times so far)
2019-02-26 17:57:42 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 98.4 MB to disk (7 times so far)
2019-02-26 17:58:09 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (8 times so far)
2019-02-26 17:58:34 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (9 times so far)
2019-02-26 17:58:54 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (10 times so far)
2019-02-26 17:59:17 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (11 times so far)
2019-02-26 17:59:46 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (12 times so far)
2019-02-26 18:00:13 INFO AbstractHadoopJob:511 - KylinConfig cached for : kylin_metadata@hdfs,path=hdfs://apachai1.apm.local:8020/kylin/kylin_metadata/kylin-5774d00a-bd56-ac28-e867-f9f5cb5d24f3/Test_Cube_2_clone/metadata
2019-02-26 18:00:13 INFO SparkFactDistinct:707 - Partition 19 handling column DEFAULT.SIDA_CASHIER_1.CASHIER_NAME, buildDictInReducer=true
2019-02-26 18:00:13 INFO SparkFactDistinct:716 - Received value: Николенко Наталья Николае

Step finished succesfully.

spilling

@asfgit
Copy link

asfgit commented Feb 26, 2019

Can one of the admins verify this patch?

Copy link
Contributor

@shaofengshi shaofengshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@shaofengshi shaofengshi merged commit e5c7c0d into apache:master Feb 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants