Skip to content

Commit

Permalink
Fix formatting in the code snippet
Browse files Browse the repository at this point in the history
Signed-off-by: Damien Hawes <d-m-h@users.noreply.github.com>
  • Loading branch information
d-m-h committed Feb 21, 2024
1 parent 014ccca commit dd1a930
Showing 1 changed file with 7 additions and 2 deletions.
9 changes: 7 additions & 2 deletions docs/integrations/spark/spark.md
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,12 @@ The below example demonstrates how to set the properties directly in your applic
constructing
a `SparkSession`.

:::warning
The setting `config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener")` is
**extremely important**. Without it, the OpenLineage Spark integration will not be invoked, rendering
the integration ineffective.
:::

<Tabs groupId="spark-app-conf">
<TabItem value="scala" label="Scala">

Expand Down Expand Up @@ -355,8 +361,7 @@ from pyspark.sql import SparkSession

spark = SparkSession.builder
.appName("OpenLineageExample")
# This line is EXTREMELY important
.config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener")
.config("spark.extraListeners", "io.openlineage.spark.agent.OpenLineageSparkListener")
.config("spark.openlineage.transport.type", "http")
.config("spark.openlineage.transport.url", "http://localhost:5000/api/v1/lineage")
.config("spark.openlineage.namespace", "MyNamespace")
Expand Down

0 comments on commit dd1a930

Please sign in to comment.