spark: get column-level lineage from JDBC dbtable option #2284
Conversation
mobuchowski
force-pushed
the
spark-cll-dbtable
branch
4 times, most recently
from
c9359c5
to
343db5a
Compare
integration/spark/app/build.gradle
Outdated
| @@ -364,7 +368,7 @@ shadowJar { | |||
| relocate 'org.apache.commons.beanutils', 'io.openlineage.spark.shaded.org.apache.commons.beanutils' | |||
| relocate 'org.apache.http', 'io.openlineage.spark.shaded.org.apache.http' | |||
| relocate 'org.yaml.snakeyaml', 'io.openlineage.spark.shaded.org.yaml.snakeyaml' | |||
| relocate 'org.slf4j', 'io.openlineage.spark.shaded.org.slf4j' | |||
| // relocate 'org.slf4j', 'io.openlineage.spark.shaded.org.slf4j' | |||
There was a problem hiding this comment.
We need to dive deep into that relocate, since it makes internal logging not work... probably just make it compileOnly and add it in tests?
| void testJdbcColumnDbtable() { | ||
| GenericContainer mysql = SparkContainerUtils.makeMysqlContainer(network); | ||
| try { | ||
| mysql.start(); |
There was a problem hiding this comment.
Wouldn't it be better to create separate test suite like SparkJdbcIntegrationTest to have mysql docker shared for all such tests?
Wouldn't it be better to run Spark in memory and access spark session within test method instead running whole docker? Such tests are a way faster than running spark in docker and do not require mock server container.
There was a problem hiding this comment.
Rewrote to Spark-in-memory test.
...rc/main/java/io/openlineage/spark/agent/lifecycle/plan/column/ColumnLevelLineageBuilder.java
Outdated
Show resolved
Hide resolved
integration/spark/app/src/test/resources/spark_scripts/spark_jdbc_column.py
Outdated
Show resolved
Hide resolved
mobuchowski
force-pushed
the
spark-cll-dbtable
branch
3 times, most recently
from
a3f733c
to
012a13d
Compare
mobuchowski
force-pushed
the
spark-cll-dbtable
branch
from
012a13d
to
bcdc5a8
Compare
There was a problem hiding this comment.
Looks good to me. Two minor comments added: pls make sure they don't relate to some debug.
@mobuchowski Super happy to see you back contributing to Spark integration 🥇
integration/spark/shared/src/main/java/io/openlineage/spark/agent/util/JdbcUtils.java
Outdated
Show resolved
Hide resolved
integration/spark/app/src/test/java/io/openlineage/spark/agent/SparkContainerUtils.java
Outdated
Show resolved
Hide resolved
mobuchowski
force-pushed
the
spark-cll-dbtable
branch
from
bcdc5a8
to
5196ab0
Compare
|
@mobuchowski Some tests are still failing |
mobuchowski
force-pushed
the
spark-cll-dbtable
branch
8 times, most recently
from
3c8a8bc
to
d8795ea
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #2284 +/- ##
=========================================
Coverage 81.41% 81.41%
Complexity 125 125
=========================================
Files 90 90
Lines 3804 3804
Branches 33 33
=========================================
Hits 3097 3097
Misses 668 668
Partials 39 39 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
mobuchowski
force-pushed
the
spark-cll-dbtable
branch
from
d8795ea
to
f7ade8d
Compare
There were several problems with JDBC column level lineage.
First, it did not support dbtable option - which ment that one-to-one relationships weren't supported by column lineage collector.
Second, the
JdbcColumnLineageCollectordid not report the lineage when there was only one single input column.Third, it used wrong, naive dataset name straight from the parser results, rather than fixed one from
JdbcUtils.Those changes fix that.