[SUPPORT]RejectedExecutionException FutureTask rejected from ThreadPoolExecutor[Terminated...]

[liijiankang]

Describe the problem you faced
We use Structured Streaming to subscribe to the data in Kafka, and then write the data to the hoodie, the program will stop abnormally after running for a period of time.

Environment Description

• Hudi version :

• Spark version :

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) :HDFS

• Running on Docker? (yes/no) :no

Add the stacktrace of the error.

[liijiankang]

Hudi version :0.6.0

Spark version :2.4.0+cdh6.2.0

Hive version :2.1.1+cdh6.2.0

Hadoop version :3.0.0+cdh6.2.0

[bvaradar]

This does look like some exception coming from cleaner. Can you look around the logs to see if there are more stack traces related to this ?

Can you try setting hoodie.clean.async=False and see ?

[liijiankang]

Thanks，I will test it .If this exception comes from a cleaner, do we have to set hoodie.clean.async = false?

[nsivabalan]

@bvaradar : once you respond, can you please remove "awaiting-user-response" label for the issue. If possible add "awaiting-community-help" label.

[bvaradar]

@liijiankang : No, My intention was to make the cleaner synchronous so that the spark job can fail immediately after encountering the exception. It is purely to debug the issue and not a solution.

[liijiankang]

@bvaradar I set hoodie.clean.async to false and this exception did not occur again.

[bvaradar]

@liijiankang : This exception is also seen when shutting down service which has async threads running. With async cleaning, there is a separate executor service setup to handle cleaning. Is it possible that you were doing ctrl-C and then saw this exception.

[liijiankang]

@bvaradar Our spark streaming program is submitted to yarn to run in cluster mode, and ctrl-C is not executed during the operation.

[n3nash]

@liijiankang Does this issue come back when you turn on async cleaning ? If yes, can you file a JIRA ticket and ping back the ticket here so we can look into this ?

[liijiankang]

@n3nash https://issues.apache.org/jira/browse/HUDI-1855

[nsivabalan]

can you please update the jira w/ stack trace as well.