Skip to content

Commit

Permalink
[Docs] Fix Spark example (#1789)
Browse files Browse the repository at this point in the history
  • Loading branch information
Gal Topper committed Mar 2, 2022
1 parent eccb108 commit 54000c1
Showing 1 changed file with 4 additions and 3 deletions.
7 changes: 4 additions & 3 deletions docs/feature-store/transformations.md
Expand Up @@ -294,18 +294,19 @@ def ingest_handler(context):
Ingestion invocation:
```python
from mlrun.datastore.sources import CSVSource
from mlrun.datastore.targets import ParquetTarget
from mlrun import code_to_function
import mlrun.feature_store as fstore

feature_set = fstore.FeatureSet("stock-quotes", entities=[fstore.Entity("ticker")], engine="spark")

source = CSVSource("mycsv", path="v3io:///projects/quotes.csv")

spark_service_name = "iguazio-spark-service" # As configured & shown in the Iguazio dashboard
spark_service_name = "spark" # As configured & shown in the Iguazio dashboard

fn = code_to_function(kind='remote-spark', name='func')

run_config=fs.RunConfig(local=False, function=fn, handler="ingest_handler")
run_config = fstore.RunConfig(local=False, function=fn, handler="ingest_handler")
run_config.with_secret('kubernetes', ['s3_access_key', 's3_secret_key'])
run_config.parameters = {
"s3_endpoint" : "s3.us-east-2.amazonaws.com",
Expand All @@ -317,6 +318,6 @@ target = ParquetTarget(
partitioned = False,
)

fstore.ingest(feature_set, source, targets=[target], run_config=config, spark_context=spark_service_name)
fstore.ingest(feature_set, source, targets=[target], run_config=run_config, spark_context=spark_service_name)
```

0 comments on commit 54000c1

Please sign in to comment.