Repository navigation
TPC-DS 10 TB benchmark questions: DataNode/NodeManager co-location and RSS configuration? #2597
Unanswered
fightBoxing
asked this question in
Q&A
Replies: 1 comment
|
Second Rerun Optimization，The specific results are as follows: 8.1 Q1–Q50
8.2 Q51–Q99
|
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Background
We are evaluating Auron (
9.0.0-SNAPSHOT,auron-spark-3.5_2.12-release-linux-x86_64) on top of Spark 3.5.8 in an on-premises Hadoop/YARN + Kyuubi deployment, using TPC-DS at 10 TB scale (ORC tables). We ran the full 99-query suite twice on the same cluster and the same data: once with Auron enabled (native extension + Auron shuffle manager), and once with the same Spark build without vectorization (plain Spark SQL).Result (sum of the 99 queries):
In our environment Auron ends up ~14% slower than plain Spark at this scale, which is the opposite of the speed-ups reported in the README. Before going deeper into per-query profiling we would like to make sure we are comparing against the same kind of setup the community benchmarks use.
Questions
spark.memory.offHeap.size,spark.sql.shuffle.partitions, AQE settings) that we should align with? Our current settings are listed below.Our submit command
/usr/local/service/kyuubi/bin/beeline -u 'jdbc:hive2://${KYUUBI_HOST}:10009/default;principal=hadoop/_HOST@${KRB_REALM};#kyuubi.engine.share.level=USER;kyuubi.engine.type=SPARK_SQL;spark.dynamicAllocation.enabled=false;spark.driver.memory=16G;spark.driver.cores=2;spark.executor.instances=200;spark.executor.cores=4;spark.executor.memory=8G;spark.executor.memoryOverhead=4G;spark.memory.offHeap.enabled=true;spark.memory.offHeap.size=20G;spark.security.credentials.hbase.enabled=false;spark.sql.shuffle.partitions=1000;spark.driver.maxResultSize=2g;spark.sql.adaptive.advisoryPartitionSizeInBytes=128m;spark.sql.extensions=org.apache.spark.sql.auron.AuronSparkSessionExtension;spark.shuffle.manager=org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager;spark.auron.enabled=true;spark.driver.extraClassPath=/usr/local/service/spark/jars/auron-spark-3.5_2.12-release-linux-x86_64-9.0.0-SNAPSHOT.jar;spark.executor.extraClassPath=./auron-spark-3.5_2.12-release-linux-x86_64-9.0.0-SNAPSHOT.jar;spark.yarn.dist.files=/usr/local/service/spark/jars/auron-spark-3.5_2.12-release-linux-x86_64-9.0.0-SNAPSHOT.jar;'Any pointers on the above — or on how the published benchmark numbers were measured — would be very helpful. Thanks!
All reactions