You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With the fix, I was able to create a Python File job in ADF and have it run on a cluster configured to work with the solution accelerator and then observe lineage in the Microsoft Purview UI.
Describe the bug
We have some ADF orchectrated python activity jobs which trigger a python file in dbfs.
Lineage for the same is not getting reflected in purview. I can see some events related to that is coming to purview out function with some errors.
To Reproduce
Steps to reproduce the behavior:
Logs
{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.18.0/integration/spark","_schemaURL":"https://openlineage.io/spec/1-0-5/OpenLineage.json#/$defs/RunFacet","spark-version":"3.0.1","openlineage-spark-version":"0.18.0"}}},"job":{"namespace":"adb-2636379187030124.4#default","name":"adb-2636379187030124_4_azuredatabricks_net_execute_create_data_source_table_as_select_command_smds_C_ALT_CODE","facets":{}},"inputs":[{"namespace":"dbfs","name":"/mnt/datalake-prod/raw/smds/initial_reload/MDM_INFM_SMDS.C_ALT_CODE","facets":{"dataSource":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.18.0/integration/spark","_schemaURL":"https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet","name":"dbfs","uri":"dbfs"},"schema":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.18.0/integration/spark","_schemaURL":"https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet","fields":[{"name":"ROWID_OBJECT","type":"string"},{"name":"CREATOR","type":"string"},{"name":"CREATE_DATE","type":"string"},{"name":"UPDATED_BY","type":"string"},{"name":"LAST_UPDATE_DATE","type":"string"},{"name":"CONSOLIDATION_IND","type":"string"},{"name":"DELETED_IND","type":"string"},{"name":"DELETED_BY","type":"string"},{"name":"DELETED_D 2023-02-28T05:18:29Z [Information] Executed 'Functions.PurviewOut' (Succeeded, Id=cb9445d3-a201-4f65-b2e3-c54fcdcb51c7, Duration=117ms) 2023-02-28T05:18:29Z [Error] AdbClient-GetSingleAdbJobAsync: error, message: Unexpected character encountered while parsing value: {. Path 'tasks[0].libraries[0].pypi', line 1, position 1516. 2023-02-28T05:18:32Z [Information] Executing 'Functions.PurviewOut' (Reason='(null)', Id=181ec998-7e09-4ba1-ad01-b5474fdb9e6e) 2023-02-28T05:18:32Z [Information] Trigger Details: PartionId: 0, Offset: 124585262552, EnqueueTimeUtc: 2023-02-28T05:18:31.5100000Z, SequenceNumber: 3045, Count: 1 2023-02-28T05:18:32Z [Verbose] Sending invocation id:181ec998-7e09-4ba1-ad01-b5474fdb9e6e 2023-02-28T05:18:32Z [Verbose] Posting invocation id:181ec998-7e09-4ba1-ad01-b5474fdb9e6e on workerId:a70f5206-8b51-40ea-9b85-2916ca30d772
Screenshots
If applicable, add screenshots to help explain your problem.
Python file location is in dbfs
The text was updated successfully, but these errors were encountered: