[BUG] NDS power run fails with TypeError: 'JavaPackage' object is not callable

[jihoonson]

Describe the bug
The power_run script fails with the error below.

...
found com.nvidia.spark.rapids.listener.Manager 'JavaPackage' object is not callable
TaskFailureListener is registered.
...
(run the first query)
...
Traceback (most recent call last):
  File "/tmp/ab/spark-rapids-benchmarks/nds/nds_power.py", line 672, in <module>
    run_query_stream(args.input_prefix,
  File "/tmp/ab/spark-rapids-benchmarks/nds/nds_power.py", line 475, in run_query_stream
    summary = q_report.report_on(run_one_query,warmup_iterations,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ab/spark-rapids-benchmarks/nds/PysparkBenchReport.py", line 144, in report_on
    listener.unregister()
  File "/tmp/ab/spark-rapids-benchmarks/nds/python_listener/PythonListener.py", line 48, in unregister
    manager.unregister(self.uuid)
TypeError: 'JavaPackage' object is not callable

This bug seems to have been introduced in #226.

Steps/Code to reproduce bug

Run the nds power_run script.

Expected behavior
It should not fail.

Environment details (please complete the following information)

I ran the script in my workstation with Spark 3.5.5. I did not use the listener as it is optional.

[wbo4958]

Hi @jihoonson, Can you give more details on how to repro this issue?

[jihoonson]

spark-submit --master 'local[16]' --conf spark.driver.maxResultSize=2GB --conf spark.driver.memory=50G --conf spark.executor.cores=16 --conf spark.executor.instances=1 --conf spark.executor.memory=16G --conf spark.sql.files.maxPartitionBytes=2gb --conf spark.sql.adaptive.enabled=true --conf spark.plugins=com.nvidia.spark.SQLPlugin --conf spark.rapids.memory.host.spillStorageSize=16G --conf spark.rapids.memory.pinnedPool.size=8g --conf spark.rapids.sql.concurrentGpuTasks=3 --conf spark.rapids.sql.explain=all --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=/tmp/spark-events --conf spark.sql.legacy.createHiveTableByDefault=false --packages io.delta:delta-spark_2.12:3.3.1 --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --conf spark.driver.extraClassPath=/tmp/ab/rapids-4-spark_2.12-25.12.0-SNAPSHOT-cuda12-spark355.jar:/tmp/nds-benchmark-listener-1.0.SNAPSSHOT.jar --conf spark.executor.extraClassPath=/tmp/ab/rapids-4-spark_2.12-25.12.0-SNAPSHOT-cuda12-spark355.jar:/tmp/nds-benchmark-listener-1.0.SNAPSSHOT.jar /tmp/ab/spark-rapids-benchmarks/nds/nds_power.py /path/to/data /tmp/ab/query_0.sql /tmp/test-test-1.csv --json_summary_folder /tmp/test-run-1 --input_format delta --warmup_iterations 0 --iterations 1 --output_format parquet --delta_unmanaged

Hi @wbo4958, this is the command I used.

[wbo4958]

Hi @jihoonson,

I didn't repro it using the command

spark-submit \
  --master 'local[16]' \
  --conf spark.driver.maxResultSize=2GB \
  --conf spark.driver.memory=28G \
  --conf spark.sql.files.maxPartitionBytes=2gb \
  --conf spark.sql.adaptive.enabled=true \
  --conf spark.plugins=com.nvidia.spark.SQLPlugin \
  --conf spark.rapids.memory.host.spillStorageSize=16G \
  --conf spark.rapids.memory.pinnedPool.size=8g \
  --conf spark.rapids.sql.concurrentGpuTasks=3 \
  --conf spark.rapids.sql.explain=all \
  --conf spark.sql.legacy.createHiveTableByDefault=false \
  --conf spark.driver.extraClassPath=/home/xxx/github/mytools/jupyter/rapids-4-spark_2.12-25.06.0-cuda12.jar:/home/xxx/work.d/spark-rapids-benchmarks/nds/jvm_listener/target/nds-benchmark-listener-1.0-SNAPSHOT.jar \
  --conf spark.executor.extraClassPath=/home/xxx/github/mytools/jupyter/rapids-4-spark_2.12-25.06.0-cuda12.jar:/home/xxx/work.d/spark-rapids-benchmarks/nds/jvm_listener/target/nds-benchmark-listener-1.0-SNAPSHOT.jar \
  nds_power.py \
  /home/xxx/work.d/spark-rapids-examples/examples/spark-connect-gpu/data/nds \
  /home/xxx/work.d/spark-rapids-examples/examples/spark-connect-gpu/client/nds/query_0.sql \
  test-test-1.csv \
  --json_summary_folder test-run-1 \
  --warmup_iterations 0 \
  --iterations 1 \
  --output_format parquet \

[wbo4958]

I can repro it if I remove the jvm listener package. Made a PR to fix it #230