Skip to content

Commit

Permalink
[Docs] Add mount_v3io_to_executor to with_igz_spark (#2090)
Browse files Browse the repository at this point in the history
  • Loading branch information
jillnogold committed Jul 6, 2022
1 parent a93abbf commit 13dff2b
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 3 deletions.
5 changes: 3 additions & 2 deletions docs/feature-store/feature-sets.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,10 +190,11 @@ To learn more about deploy_ingestion_service go to {py:class}`~mlrun.feature_sto
You can schedule an ingestion job for a feature set on an ongoing basis. The first scheduled job runs on all the data in the source and the subsequent jobs ingest only the deltas since the previous run (from the last timestamp of the previous run until `datetime.now`).
Example:

`cron_trigger = "* */1 * * *" #will run every hour
```cron_trigger = "* */1 * * *" #will run every hour
source = ParquetSource("myparquet", path=path, time_field="time", schedule=cron_trigger)
feature_set = fs.FeatureSet(name=name, entities=[fs.Entity("first_name")], timestamp_key="time",)
fs.ingest(feature_set, source, run_config=fs.RunConfig())`
fs.ingest(feature_set, source, run_config=fs.RunConfig())
```

The default value for the `overwrite` parameter in the ingest function for scheduled ingest is `False`, meaning that the target from the previous ingest is not deleted.
For the storey engine, the feature is currently implemented for ParquetSource only. (CsvSource will be supported in a future release). For Spark engine, other sources are also supported.
Expand Down
6 changes: 5 additions & 1 deletion docs/runtimes/spark-operator.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@
"# adds fuse, daemon & iguazio's jars support\n",
"sj.with_igz_spark() \n",
"\n",
"# Alternately, move volume_mounts to driver and executor-specific fields and leave \n",
"# v3io mounts out of executor mounts if mount_v3io_to_executor=False \n",
"# sj.with_igz_spark(mount_v3io_to_executor=False)\n",
"\n",
"# set spark driver volume mount\n",
"# sj.function.with_driver_host_path_volume(\"/host/path\", \"/mount/path\")\n",
"\n",
Expand Down Expand Up @@ -127,7 +131,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
Expand Down

0 comments on commit 13dff2b

Please sign in to comment.