-
Notifications
You must be signed in to change notification settings - Fork 394
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'guide/how-to' of https://github.com/iterative/dvc.org i…
…nto guide/how-to
- Loading branch information
Showing
6 changed files
with
75 additions
and
20 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
# Versioned storage | ||
|
||
What if we could **combine data and ML model versioning features with large file | ||
storage** solutions like traditional hard drives, NAS, or cloud services such as | ||
Amazon S3 and Google Drive? DVC brings together the best of both worlds by | ||
implementing easy synchronization between the data <abbr>cache</abbr> and | ||
on-premises or cloud storage for sharing. | ||
|
||
![](/img/model-versioning-diagram.png) _DVC's hybrid versioned storage_ | ||
|
||
> Note that [remote storage](/doc/command-reference/remote) is optional in DVC: | ||
> no server setup or special services are needed, just the `dvc` command-line | ||
> tool. |
50 changes: 50 additions & 0 deletions
50
content/docs/user-guide/how-to/add-dependency-or-output-to-stage.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
# Add Dependency or Output to Stage | ||
|
||
There are situations where we have executed a stage (either by writing | ||
`dvc.yaml` manually and using `dvc repro`, or with `dvc run`), but later notice | ||
that some of the dependencies, or the output files/directories it creates, which | ||
are already in the <abbr>workspace</abbr>, are missing from `dvc.yaml` (`deps` | ||
and `outs` field respectively). Follow the steps below to add existing files or | ||
directories as <abbr>dependency</abbr> or <abbr>outputs</abbr> to a stage | ||
without re-executing it again, which can be expensive/time-consuming, and is | ||
unnecessary. | ||
|
||
We start with an example `prepare`, which has a single dependency and output. To | ||
add a missing dependency `data/data.csv` and output `data/validate` to this | ||
stage, we can edit `dvc.yaml` like this: | ||
|
||
```git | ||
stages: | ||
prepare: | ||
cmd: python src/prepare.py | ||
deps: | ||
+ - data/data.csv | ||
- src/prepare.py | ||
outs: | ||
- data/train | ||
+ - data/validate | ||
``` | ||
|
||
> Note that you can also use `dvc run` with the `-f` and `--no-exec` options to | ||
> add another dependency/output to the stage: | ||
> | ||
> ```dvc | ||
> $ dvc run -f --no-exec \ | ||
> -n prepare \ | ||
> -d data/data.csv \ | ||
> -d src/prepare.py \ | ||
> -o data/train \ | ||
> -o data/validate \ | ||
> python src/prepare.py | ||
> ``` | ||
> | ||
> `-f` overwrites the stage in `dvc.yaml`, while `--no-exec` updates the stage | ||
> without executing it. | ||
Finally, we need to run `dvc commit` to save the newly specified dependency or | ||
output(s) to the <abbr>cache</abbr> (and to update the corresponding hash values | ||
in `dvc.lock`): | ||
```dvc | ||
$ dvc commit | ||
``` |
2 changes: 1 addition & 1 deletion
2
.../user-guide/how-to/add-output-to-stage.md → ...er-guide/how-to/add-outputs-to-a-stage.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters