Hudi reported an error, but the SparkStreaming still works

[wangfeigithub]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

A clear and concise description of the problem.

To Reproduce

Steps to reproduce the behavior:

1.Run Hudi on SparkStreaming
2.When Hudi reports an error such as Hive failed to connect
3.The SparkStreaming, however, is still running and will not actively stop the task
4.

Expected behavior

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version :0.8

• Spark version :2.4.8

• Hive version :3.1

• Hadoop version :3.1

• Storage (HDFS/S3/GCS..) :

• Running on Docker? (yes/no) :

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.

[wangfeigithub]

In this case, how do you ensure that you actively stop the SparkStreaming task or stop the YARN task if you write to a HUDI table in error

[nsivabalan]

sorry for late turn around.
@wangfeigithub : so, is the expectation, you wanna fail the spark streaming job whenever any issues w/ hive sync?

[vinothchandar]

2.When Hudi reports an error such as Hive failed to connect
This is intended by design. For e.g you may be reading from streaming jobs, which should still be able to access the hudi commits. When connectivity to Hive is restored, hudi's incremental sync will keep Hive updated properly