[Report] TPC-H Benchmark report of Kylin 4.0.0 #6
Assignees
Labels
Comments
Cluster Resource
aws emr create-cluster --applications Name=Hadoop Name=Hive Name=Spark Name=ZooKeeper Name=Tez Name=Ganglia \
--tags 'Project=Kylin_Benchmark' 'Owner=xiaoxiang.yu' \
--ec2-attributes '{"KeyName":"?","AdditionalSlaveSecurityGroups":["?"],"InstanceProfile":"?","SubnetId":"?","EmrManagedSlaveSecurityGroup":"?","EmrManagedMasterSecurityGroup":"?","AdditionalMasterSecurityGroups":["?"]}' \
--release-label emr-5.31.0 \
--instance-groups '[{"InstanceCount":4,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":400,"VolumeType":"gp2"},"VolumesPerInstance":2}]},"InstanceGroupType":"CORE","InstanceType":"m5.4xlarge","Name":"Hadoop Workers"},{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"m5.4xlarge","Name":"Hadoop Master"}]' \
--configurations '[{"Classification":"hdfs-site","Properties":{"dfs.replication":"1"}},{"Classification":"mapred-site","Properties":{"mapreduce.map.memory.mb":"3584","mapreduce.reduce.memory.mb":"8192","mapreduce.map.java.opts":"-Xmx3072m","mapreduce.reduce.java.opts":"-Xmx7168m"}},{"Classification":"yarn-site","Properties":{"yarn.nodemanager.resource.cpu-vcores":"13","yarn.nodemanager.resource.memory-mb":"52428","yarn.scheduler.maximum-allocation-mb":"52428","yarn.app.mapreduce.am.resource.mb":"2048"}}]' \
--auto-scaling-role EMR_AutoScaling_DefaultRole \
--ebs-root-volume-size 100 \
--service-role EMR_DefaultRole \
--name 'Kylin4 Benchmark' \
--scale-down-behavior TERMINATE_AT_TASK_COMPLETION \
--region cn-northwest-1 |
Cubing duration
|
Storage Size
|
Response Time (thread = 1)
|
Kylin Configurationkylin.metadata.url=benchmark_kylin312@jdbc,url=jdbc:mysql://ip-172-31-11-46.cn-northwest-1.compute.internal:3306/hive,username=hive,password=nzSqiiWPGj5Gqzp3,maxActive=10,maxIdle=10,driverClassName=org.mariadb.jdbc.Driver
kylin.env.zookeeper-connect-string=localhost:2181
## Disable retry
kylin.engine.max-retry-time=1
## Build Engine Resource
kylin.engine.spark-conf.spark.executor.cores=2
kylin.engine.spark-conf.spark.executor.instances=25
kylin.engine.spark-conf.spark.executor.memory=7GB
kylin.engine.spark-conf.spark.executor.memoryOverhead=1GB
## Query Engine Resource
kylin.query.spark-conf.spark.master=yarn
kylin.query.spark-conf.spark.driver.cores=1
kylin.query.spark-conf.spark.driver.memory=8GB
kylin.query.spark-conf.spark.driver.memoryOverhead=1G
kylin.query.spark-conf.spark.executor.cores=1
kylin.query.spark-conf.spark.executor.instances=40
kylin.query.spark-conf.spark.executor.memory=4G
kylin.query.spark-conf.spark.executor.memoryOverhead=1G
kylin.query.spark-conf.spark.sql.parquet.filterPushdown=false
## Disable canary
kylin.canary.sparder-context-canary-enabled=false
## Shard setting
kylin.storage.columnar.shard-size-mb=75
kylin.storage.columnar.shard-rowcount=1200000
kylin.storage.columnar.shard-countdistinct-rowcount=600000
kylin.storage.columnar.repartition-threshold-size-mb=75
kylin.engine.spark-conf.spark.hadoop.parquet.block.size=268435456 |
This comment has been minimized.
This comment has been minimized.
hit-lacus
changed the title
This comment has been minimized.
This comment has been minimized.
Screenshots of KylinCube Meta
Job Meta
HDFS
Yarn UI
Executor tab of Spark UI
|
Metrics of Cube BuildingFirst Segment of LineitemCube
Second Segment of LineitemCube
|
Storage (Size of each shard under specific cuboid dir)
[hadoop@ip-172-31-11-46 apache-kylin-4.0.0-SNAPSHOT-bin]$ hadoop fs -du -h /kylin/benchmark_kylin312/tpch/parquet///customer_vorder_cube/FULL_BUILD_AZC/31
94.7 M /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00000-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00001-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00002-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00003-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00004-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00005-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
94.7 M /kylin/benchmark_kylin312/tpch/parquet/customer_vorder_cube/FULL_BUILD_AZC/31/part-00006-816fd3a3-b27f-4301-9c2c-a82ce663a7e8-c000.snappy.parquet
[hadoop@ip-172-31-11-46 apache-kylin-4.0.0-SNAPSHOT-bin]$
[hadoop@ip-172-31-11-46 apache-kylin-4.0.0-SNAPSHOT-bin]$ hadoop fs -du -h /kylin/benchmark_kylin312/tpch/parquet/customer_cube/FULL_BUILD_SMD/7
38.7 M /kylin/benchmark_kylin312/tpch/parquet/customer_cube/FULL_BUILD_SMD/7/part-00000-42a8e12e-dbc2-4a04-a72a-dcc2c577e05d-c000.snappy.parquet |
Screenshots of Load Test (thread=1, 20 minutes)
Node Level
|
Screenshots of Load Test (thread=4, 10 minutes)
|
hit-lacus
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Table of Content
The text was updated successfully, but these errors were encountered: