When running NDS 2.0 on Dataproc, the Spark eventlog may have duplicate query IDs

[viadea]

When running NDS 2.0 on Dataproc, the Spark eventlog may have duplicate query IDs.

For example, the Spark eventlog may show as:
query66
query66
query23_part2
query23_part2

This is very confusing. I can reproduce each time on Dataproc.

[wjxiz1992]

Is this only happening in Dataproc? Is it possible to have a look at the Spark history page for it, I'm wondering if it's related to https://github.com/NVIDIA/spark-rapids-benchmarks/blob/branch-22.06/nds/nds_power.py#L208-L209

[wjxiz1992]

22/06/28 08:09:58 ERROR org.apache.spark.scheduler.AsyncEventQueue: Dropping event from queue shared. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.

I saw such error when running on Dataproc environment. Still not sure if it's the cause.

[wjxiz1992]

This should be related to #37

[viadea]

So far i only saw this in dataproc. My standalone cluster looks good.

[wjxiz1992]

#41 resolves this issue according to my latest tests on Dataproc. It's due to the dropped event error mentioned above. I'm running more rounds tests to confirm, but it's pretty much like that.