Skip to content

Conversation

@efiop
Copy link
Contributor

@efiop efiop commented Feb 20, 2023

Currently we use open, which is much slower. For example,

$ dvc get dvc-bench data/small/dataset

goes down from 110sec to 50sec (which is still too much).

Discovered as a part of #9098

@codecov
Copy link

codecov bot commented Feb 20, 2023

Codecov Report

Base: 93.02% // Head: 92.99% // Decreases project coverage by -0.03% ⚠️

Coverage data is based on head (464e904) compared to base (19038b3).
Patch coverage: 85.71% of modified lines in pull request are covered.

❗ Current head 464e904 differs from pull request most recent head 4fed069. Consider uploading reports for the commit 4fed069 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9057      +/-   ##
==========================================
- Coverage   93.02%   92.99%   -0.03%     
==========================================
  Files         457      457              
  Lines       36813    36827      +14     
  Branches     5338     5340       +2     
==========================================
+ Hits        34244    34246       +2     
- Misses       2050     2057       +7     
- Partials      519      524       +5     
Impacted Files Coverage Δ
dvc/fs/dvc.py 96.42% <85.71%> (-0.64%) ⬇️
tests/unit/test_analytics.py 95.71% <0.00%> (-4.29%) ⬇️
dvc/analytics.py 90.14% <0.00%> (-2.82%) ⬇️
dvc/repo/experiments/queue/celery.py 85.92% <0.00%> (-1.86%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Currently we use `open`, which is much slower. For example,

```
$ dvc get dvc-bench data/small/dataset
```

goes down from 110sec to 50sec.

Discovered as a part of https://github.com/iterative/dvc-bench/issues/419
@efiop efiop changed the title dvcfs: implement get_file [WIP] dvcfs: implement get_file Feb 20, 2023
@efiop efiop changed the title [WIP] dvcfs: implement get_file dvcfs: implement get_file Feb 20, 2023
@efiop efiop merged commit ea6e01d into treeverse:main Feb 20, 2023
@efiop
Copy link
Contributor Author

efiop commented Feb 20, 2023

There is a bit more so squeeze out in treeverse/dvc-objects#197

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant