21.47.0

Added

• Flink: Experimental version for flink native lineage listener. #3099 @pawel-big-lebowski
  New flink listener to extract lineage through native Flink interfaces. Supports Flink SQL. Requires Flink 2.0.
• dbt: add support for consuming dbt structured logs for test and build commands . #3362 @MassyB
  New option for dbt integration now can handle test and build commands too.
• Spark: allow attaching custom facets to RDDExecContext events. #3379 @ssanthanam185
  Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.
• spec: add DatasetTypeDatasetFacet. #3390 @ssanthanam185
  Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.

Changed

• Python: allow adding additionalProperties to Python facets #3391 @JDarDagran
  Adds with_additonal_properties method that allows to create modified instance of facet with additional properties.
• Spark: allow attaching custom facets to RDDExecContext events. #3379 @ssanthanam185
  Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.
• Spark: SerializedFromObject events aren't filtered for not-delta plans. #3403 @ssanthanam185
  Those events shouldn't be filtered outside Databricks/Delta ecosystem.

Fixed

• Spark: fixed ClassLoader handling for OpenLineageExtensionProvider. #3368 @ddebowczyk92
  Fixes ClassNotFoundException issue when using the openlineage-spark integration alongside a Spark connector that implements the spark-extension-interfaces due to class loader conflicts.
• SQL: add minimal support for Snowflake LATERAL. #3368 @cisenbe
  SQL parser won't error on Snowflake's LATERAL keyword.
• dbt: handle errors in parse_assertions #3311 @dsaxton-1password
  dbt integration won't fail when looking at tests on seeds.
• Spark: fix infinite loop in RDD flatten & perf optimization. #3379 @ssanthanam185
  Spark integration now correctly handles complex jobs that have cycles and nested RDD trees.
• Python: FileTransport now correctly attaches json file extension. #3404 @kacpermuda
  When append=False, the json file extension wasn't properly added before.

21.46.0

Added

• Flink: Experimental version for flink native lineage listener. #3099 @pawel-big-lebowski
  New flink listener to extract lineage through native Flink interfaces. Supports Flink SQL. Requires Flink 2.0.
• dbt: add support for consuming dbt structured logs for test and build commands . #3362 @MassyB
  New option for dbt integration now can handle test and build commands too.
• Spark: allow attaching custom facets to RDDExecContext events. #3379 @ssanthanam185
  Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.
• spec: add DatasetTypeDatasetFacet. #3390 @ssanthanam185
  Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.

Changed

• Python: allow adding additionalProperties to Python facets #3391 @JDarDagran
  Adds with_additonal_properties method that allows to create modified instance of facet with additional properties.
• Spark: allow attaching custom facets to RDDExecContext events. #3379 @ssanthanam185
  Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.
• Spark: SerializedFromObject events aren't filtered for not-delta plans. #3403 @ssanthanam185
  Those events shouldn't be filtered outside Databricks/Delta ecosystem.

Fixed

• Spark: fixed ClassLoader handling for OpenLineageExtensionProvider. #3368 @ddebowczyk92
  Fixes ClassNotFoundException issue when using the openlineage-spark integration alongside a Spark connector that implements the spark-extension-interfaces due to class loader conflicts.
• SQL: add minimal support for Snowflake LATERAL. #3368 @cisenbe
  SQL parser won't error on Snowflake's LATERAL keyword.
• dbt: handle errors in parse_assertions #3311 @dsaxton-1password
  dbt integration won't fail when looking at tests on seeds.
• Spark: fix infinite loop in RDD flatten & perf optimization. #3379 @ssanthanam185
  Spark integration now correctly handles complex jobs that have cycles and nested RDD trees.
• Python: FileTransport now correctly attaches json file extension. #3404 @kacpermuda
  When append=False, the json file extension wasn't properly added before.

21.45.0

Added

• Flink: Experimental version for flink native lineage listener. #3099 @pawel-big-lebowski
  New flink listener to extract lineage through native Flink interfaces. Supports Flink SQL. Requires Flink 2.0.
• dbt: add support for consuming dbt structured logs for test and build commands . #3362 @MassyB
  New option for dbt integration now can handle test and build commands too.
• Spark: allow attaching custom facets to RDDExecContext events. #3379 @ssanthanam185
  Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.
• spec: add DatasetTypeDatasetFacet. #3390 @ssanthanam185
  Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.

Changed

• Python: allow adding additionalProperties to Python facets #3391 @JDarDagran
  Adds with_additonal_properties method that allows to create modified instance of facet with additional properties.
• Spark: allow attaching custom facets to RDDExecContext events. #3379 @ssanthanam185
  Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.
• Spark: SerializedFromObject events aren't filtered for not-delta plans. #3403 @ssanthanam185
  Those events shouldn't be filtered outside Databricks/Delta ecosystem.

Fixed

• Spark: fixed ClassLoader handling for OpenLineageExtensionProvider. #3368 @ddebowczyk92
  Fixes ClassNotFoundException issue when using the openlineage-spark integration alongside a Spark connector that implements the spark-extension-interfaces due to class loader conflicts.
• SQL: add minimal support for Snowflake LATERAL. #3368 @cisenbe
  SQL parser won't error on Snowflake's LATERAL keyword.
• dbt: handle errors in parse_assertions #3311 @dsaxton-1password
  dbt integration won't fail when looking at tests on seeds.
• Spark: fix infinite loop in RDD flatten & perf optimization. #3379 @ssanthanam185
  Spark integration now correctly handles complex jobs that have cycles and nested RDD trees.
• Python: FileTransport now correctly attaches json file extension. #3404 @kacpermuda
  When append=False, the json file extension wasn't properly added before.

21.44.0

Full Changelog: 0.8.2...1.2138.0

OpenLineage#3221
#3221 @JDarDagran
@JDarDagran

21.43.0

Full Changelog: 0.8.2...1.2138.0

OpenLineage#3221
#3221 @JDarDagran
@JDarDagran

21.42.0: x

Full Changelog: 0.8.2...1.2138.0

#3221 @JDarDagran

1.2138.0

Full Changelog: 0.8.2...1.2138.0

@JDarDagran #13

21.40.0

Full Changelog: 21.38.0...21.40.0

** TEST: @JDarDagran

21.39.0

What's Changed

• Test release notes. by @JDarDagran in #13

New Contributors

• @JDarDagran made their first contribution in #13

Full Changelog: 1.20.3...21.39.0

21.38.0

What's Changed

• Test release notes. by @JDarDagran in #13

New Contributors

• @JDarDagran made their first contribution in #13

Full Changelog: 1.20.3...21.38.0