Skip to content

Commit

Permalink
Enable Hive Managed Tables in Demo Environment
Browse files Browse the repository at this point in the history
To enable hive managed tables in demo environment, the config must have a
spark.openlineage.namespace that matches the pattern adb-xxx.azuredatabricks.net#yyy
  • Loading branch information
wjohnson committed Oct 9, 2022
1 parent b5a0cc7 commit 07404d7
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion deployment/infra/openlineage-deployment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,9 @@ else
fi

# creating json for cluster configuration
# Demo Cluster configuration should include adb_ws_url in namespace to ensure
# support for managed hive tables out of the box
adb_ws_url_id=$(echo $adb_ws_url | sed 's/.azuredatabricks.net//g')
cat << EOF > create-cluster.json
{
"cluster_name": "$CLUSTERNAME",
Expand All @@ -233,7 +236,7 @@ cat << EOF > create-cluster.json
"num_workers": 1,
"spark_conf": {
"spark.openlineage.version" : "v1",
"spark.openlineage.namespace" : "adbpurviewol1#default",
"spark.openlineage.namespace" : "$adb_ws_url_id#default",
"spark.openlineage.host" : "https://$FUNNAME.azurewebsites.net",
"spark.openlineage.url.param.code": "{{secrets/purview-to-adb-kv/Ol-Output-Api-Key}}"
},
Expand Down

0 comments on commit 07404d7

Please sign in to comment.