[SPARK][FLINK][JAVA] Support yaml config files together with SparkConf/FlinkConf

[pawel-big-lebowski]

Problem

Client-java documentation says it can read configs directly from openlineage.yaml file, while Spark integration reads config entries from SparkConf only. Flink integration can read it from file and Flink conf exclusively. This is confusing for the user.

Problems:

• There is no way to add flink specific config to Flink integration.
• Some config entries in Flink conf will not work if passed through yaml file.
• Spark integration does not support reading conf from yaml file at all.

Solution

• rename OpenlineageYaml -> OpenlineageConfig
• create SparkOpenlineageConfig, FlinkOpenlineageConfig extending the above to keep entries specific to integration
• modify code to use only OpenLineageConfig classes.
• update doc to mention that both ways can be used interchangeably and final documentation will be a merge of all values provided.

Please describe your change as it relates to the problem, or bug fix, as well as any dependencies. If your change requires a schema change, please describe the schema modification(s) and whether it's a backwards-incompatible or backwards-compatible change, then select one of the following:

Note: All schema changes require discussion. Please link the issue for context.

• Your change modifies the core OpenLineage model
• Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

One-line summary:

Checklist

• You've signed-off your work
• Your pull request title follows our guidelines
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• Your comment includes a one-liner for the changelog about the specific purpose of the change (if necessary)
• You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
• You've added a header to source files (if relevant)

---

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

[pawel-big-lebowski]

added to prevent serialising the field within debug facet

[mobuchowski]

👍

[mobuchowski]

I see you've added this mechanism, but what's the actual purpose of it? Can't we just create new config instance rather than making config mutable if we need to merge config entries from multiple sources?

[pawel-big-lebowski]

I think we can go with the approach you describe: change the interface to return a new value instead
will prepare a change for this.

[mobuchowski]

👍

[d-m-h]

I'm on a phone right now, so it's a bit difficult to see the changes, so forgive me if I ask something that is obvious.

When converting from flat properties to a config object, how is this done?

To we try and infer a hierarchy from using dot separated properties? For example, if we do spark.openlineage.transport.foo.bar, do we assume that the object has the hierarchy of:

transport:
\tfoo:
\t\tbar: $value

If so, this could pose a problem in my use case, as I have letting the custom transport act as a pass through for the values the spark integration provides. For example, I have the following property:

spark.openlineage.transport.properties.cluster.federation

The cluster.federation is a property that the library my transport wraps expects. So if this code tries to infer hierarchy, this would break my integration.

[pawel-big-lebowski]

@d-m-h Actually this is already done within this piece of code -> https://github.com/OpenLineage/OpenLineage/blob/main/integration/spark/app/src/main/java/io/openlineage/spark/agent/ArgumentParser.java#L88

I don't think this PR changes this behaviour anyway.

[mobuchowski]

Amazing job @pawel-big-lebowski. That change was very much needed. I'm approving but please wait with merging till we hear if @d-m-h's concerns are answered.

One thing you can improve in the meantime is improving comments on MergeConfig class - I believe we can explain the semantics better and in simple language.

[pawel-big-lebowski]

@mobuchowski fixed docs for MergeConfig

[dolfinus]

I see that changelog entry was added, but for 1.12.0 instead of Unreleased. Fix: #2636

[pawel-big-lebowski]

@dolfinus thank you, sry