[SUPPORT] HoodieDeltaStreamer - Spark master shouldn't have a default option (AWS Glue)

[Neuw84]

Describe the problem you faced

When trying to run HoodieDeltaStreamer on AWS Glue I found that the Spark master has no option to inherit from the environment as it defaults to local[2]. In these kind of Serverless environments where you do not have access to the master this configuration should be inherited

This can be seen on line 329 on HoodieDeltaStreamer.

public String sparkMaster = "local[2]";

This should be changed for supporting this kind of scenarios, a JavaSparkContext option where no Spark master is defined should be there.

Expected behavior

The Spark master shouldn't have a default as there are some environments (usually serverless such as AWS Glue) where it will be inherited.

Environment Description

• Hudi version : 0.9.0

• Spark version : Spark 3.1.1

• Storage (HDFS/S3/GCS..) : S3

• Running on Docker? (yes/no) : no

Additional context

If required I think I could work on this as I have quite good Java experience.

[yihua]

@Neuw84 The HoodieDeltaStreamer takes the --spark-master option to override the default Spark master to use, i.e., the same variable you mentioned, when you use spark-submit to run HoodieDeltaStreamer. I think your point is that, if the HoodieDeltaStreamer constructor is used, the Spark master config can be overwritten by what's configured in JavaSparkContext jssc, is that the case?

Feel free to create a Jira ticket here for this feature inquiry and work on it.

[Neuw84]

Hi @yihua,

The thing is that no default value should be in that class in order to be able to inherit to whatever the master is in serverless Spark engines such as AWS Glue where the master is not known. The idea is to remove that default value (local[2] ), add an option to build a Java Spark Context without the default value and change the documentation in the cli options.

Is a small change, but that will enable to use the DeltaStreamer in managed environments such as AWS Glue. Have opened this jira ticket to work on this.

https://issues.apache.org/jira/browse/HUDI-3994

Thanks

[yihua]

Got you. Sg. Let's track the progress there. Closing this issue.