index: experimental persistent data index #8827
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportBase: 93.13% // Head: 93.13% // Increases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## main #8827 +/- ##
=======================================
Coverage 93.13% 93.13%
=======================================
Files 455 453 -2
Lines 36644 36604 -40
Branches 5289 5287 -2
=======================================
- Hits 34127 34092 -35
+ Misses 2001 1998 -3
+ Partials 516 514 -2
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
efiop
force-pushed
the
fix-dvc-data-208
branch
from
2dc1a74
to
833c450
Compare
efiop
force-pushed
the
fix-dvc-data-208
branch
4 times, most recently
from
0021d29
to
7560919
Compare
efiop
force-pushed
the
fix-dvc-data-208
branch
17 times, most recently
from
914b469
to
e75a9d1
Compare
|
For the record: switched to a tree-based id (instead of git revs), which made it possible to use without git and for dirty changes. But that triggered a bit more bugs on the import side, where we assume index entries have |
efiop
added a commit
to efiop/dvc-data
that referenced
this pull request
Feb 7, 2023
This replaces `odb_map` and `remote_map` in `DataIndex`, and `fs` and `path` in `DataIndexEntry` incorporating everything into `Storage`, which describes where to get the data contents from no matter how they are stored (just as plain backup in a directory or in an object storage). `DataIndexEntry.fs/path` were very confusing, as it was not clear what they really represent and were often unecessarily used during different operations (for example `checkout` that mutates those instead of returning a new local index). This also removes unserializable `fs` instance from `DataIndexEntry`, making it much easier to work with after loading. The new `Storage.fs/path` concepts fit nicely into dvc's import functionality, by giving a clear way to declare where to get the data from. Related iterative/dvc#8827
efiop
added a commit
to efiop/dvc-data
that referenced
this pull request
Feb 7, 2023
This replaces `odb_map` and `remote_map` in `DataIndex`, and `fs` and `path` in `DataIndexEntry` incorporating everything into `Storage`, which describes where to get the data contents from no matter how they are stored (just as plain backup in a directory or in an object storage). `DataIndexEntry.fs/path` were very confusing, as it was not clear what they really represent and were often unecessarily used during different operations (for example `checkout` that mutates those instead of returning a new local index). This also removes unserializable `fs` instance from `DataIndexEntry`, making it much easier to work with after loading. The new `Storage.fs/path` concepts fit nicely into dvc's import functionality, by giving a clear way to declare where to get the data from. Related iterative/dvc#8827
efiop
added a commit
to iterative/dvc-data
that referenced
this pull request
Feb 7, 2023
This replaces `odb_map` and `remote_map` in `DataIndex`, and `fs` and `path` in `DataIndexEntry` incorporating everything into `Storage`, which describes where to get the data contents from no matter how they are stored (just as plain backup in a directory or in an object storage). `DataIndexEntry.fs/path` were very confusing, as it was not clear what they really represent and were often unecessarily used during different operations (for example `checkout` that mutates those instead of returning a new local index). This also removes unserializable `fs` instance from `DataIndexEntry`, making it much easier to work with after loading. The new `Storage.fs/path` concepts fit nicely into dvc's import functionality, by giving a clear way to declare where to get the data from. Related iterative/dvc#8827
efiop
force-pushed
the
fix-dvc-data-208
branch
6 times, most recently
from
17fb0f3
to
a926fae
Compare
|
One last significant problem to figure out is no-.dir outputs in cloud versioning, where we don't save |
efiop
force-pushed
the
fix-dvc-data-208
branch
3 times, most recently
from
3735388
to
b765b37
Compare
This was referenced Feb 9, 2023
efiop
force-pushed
the
fix-dvc-data-208
branch
3 times, most recently
from
e2a484d
to
b998e11
Compare
efiop
changed the title
efiop
force-pushed
the
fix-dvc-data-208
branch
2 times, most recently
from
7241147
to
6560412
Compare
efiop
added
enhancement
efiop
force-pushed
the
fix-dvc-data-208
branch
from
6560412
to
04e063d
Compare
efiop
force-pushed
the
fix-dvc-data-208
branch
from
04e063d
to
191f040
Compare
efiop
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Putting iterative/dvc-data#208 to use.
Note that this is
experimentalfor now and needs to be enabled withdvc config feature.data_index_cache true.For example, sequential run of
goes down from ~16sec to ~8sec (2x improvement)
and
goes down from ~11sec to ~1.5sec (7x improvement)