【Question】catboost for spark how to exit

[chencjiajy]

Problem: catboost for spark application not exit after finished, it hangs....., spark.stop() not work。 how to make the application exit after training. （ps:both IDEA and spark-submit not exit)
catboost version:1.0.6

example

[shuchih]

same problem here

[minhdang241]

I got the same problem? is there any work around guys ? :/

[tqtensor]

We have the same problem.

[art777nn]

We have the same problem.

Tried to run different versions catboost-spark:
catboost-spark_3.4_2.13:1.2.2, 1.2, 1.2.1
catboost-spark_3.4_2.12:1.2.2, 1.2, 1.2.1

Context:
Kubernets 1.27.7
Spark 3.4.2
Scala 2.13 or 2.12
Catboost 1.2.2
PySpark 3.4.2

Driver does not stop after fit call.

Attached extended logs that are an output after the python script is completed. You can see that at the end of logs PoolingHttpClientConnectionManager attempts to close idle connections infinitely.

Log line output:
PoolingHttpClientConnectionManager: Closing connections idle longer than 60000 MILLISECONDS
log_catboost.txt

[andrey-khropov]

Fixed in 2cdb8a2, the good news is that is was just auxiliary threads not shut down at the end of the program execution, the main computation logic had not been affected, the training results should be valid.

[art777nn]

Fixed in 2cdb8a2, the good news is that is was just auxiliary threads not shut down at the end of the program execution, the main computation logic had not been affected, the training results should be valid.

Thank you! This commit cured the application! Can you please tell me when we can expect a patch in maven?

[andrey-khropov]

Can you please tell me when we can expect a patch in maven?

This week.