You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"Registered listener io.openlineage.." - this one appears
"OpenLineageContext: Init OpenLineageContext:" - this one is missing
"AsyncEventQueue: Process of event SparkListenerApplicationStart" - this one appears
Logs
Please include any Spark code being ran that generates this error spark.openlineage.version v1 spark.openlineage.namespace <adb_workspace_id>#<cluster_id> spark.openlineage.host https://<functionapp_name>.azurewebsites.net/ spark.openlineage.url.param.code {{secrets/<scope>/<function_defualt_key}}
Init Scripts Path updated in the cluster:
Type: Workspace
File path: /Users/user_name/init_scripts/open-lineage-init-script.sh
Open Lineage script's absolute path - /Users/user_name/init_scripts/open-lineage-init-script.sh
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
OS: Windows
OpenLineage Version: 0.18.0
Databricks Runtime Version: 11.3
Cluster Type: All compute cluster
Cluster Mode: Single User Access
Using Credential Passthrough: N/A
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered:
Describe the bug
When the DB compute cluster starts, OpenLineage doesn't initialize as expected, and no events are produced.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
By the instructions - https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks#initialization-logs - we should see 3 log entries for intiatlization
"Registered listener io.openlineage.." - this one appears
"OpenLineageContext: Init OpenLineageContext:" - this one is missing
"AsyncEventQueue: Process of event SparkListenerApplicationStart" - this one appears
Logs
Please include any Spark code being ran that generates this error
spark.openlineage.version v1
spark.openlineage.namespace <adb_workspace_id>#<cluster_id>
spark.openlineage.host https://<functionapp_name>.azurewebsites.net/
spark.openlineage.url.param.code {{secrets/<scope>/<function_defualt_key}}
Init Scripts Path updated in the cluster:
Type: Workspace
File path: /Users/user_name/init_scripts/open-lineage-init-script.sh
Open Lineage script's absolute path - /Users/user_name/init_scripts/open-lineage-init-script.sh
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: