SparkStructureStreamingRunner issue due to spark3.*

[damo1991]

We are getting below error on using spark 3 and above version:

java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/DataSourceV2

This error is due to SparkStructuredStreamingRunner internally using DataSourceV2 api from spark 3.* this class is not found, as it is tightly coupled with DataSourceV2 class we are facing this issue:

Please see below code snippet for the same.

public class DatasetSourceBatch implements DataSourceV2, ReadSupport {

@OverRide
public DataSourceReader createReader(DataSourceOptions options)

{ return new DatasetReader<>(options); }