Weekly Digest (19 April, 2020 - 26 April, 2020)

[weekly-digest]

Here's the Weekly Digest for iterative/dvc:

---

ISSUES

Last week 31 issues were created.
Of these, 14 issues have been closed and 17 issues are still open.

OPEN ISSUES

💚 #3682 progress: push and pull: single bytes-based bar, by casperdcl
💚 #3681 progress: append to desc, by casperdcl
💚 #3680 cache file name automatically changed, by lmxs1237
💚 #3679 import: \r\n line endings in DVC-file (Win), by jorgeorpinel
💚 #3678 add/run: improve "already tracked" message, by efiop
💚 #3676 [WIP] dvc: implement multistage dvcfile, by skshetry
💚 #3675 remote.ssh: suppress paramiko logging, by pmrowla
💚 #3673 get/import: check target path before downloading, by jorgeorpinel
💚 #3672 [WIP] remote: use string paths over PathInfo for performance reasons, by pmrowla
💚 #3670 config: rename config.local to config.private, by casperdcl
💚 #3668 In contrived circumstances, repro of an added file with hardlinks enabled results in a not-read-only file, by charlesbaynham
💚 #3666 move temporary files from .dvc to .dvc/tmp, by efiop
💚 #3662 dvc gc does not remove cache for uncommitted dvc-file, by hhe
💚 #3661 Pipeline show for all a project, by AmineBarrak
💚 #3656 Support dvcignore for external dependencies., by benjamintanweihao
💚 #3653 On Windows, dvc repro spends an awful lot of time in rwlock, by charlesbaynham
💚 #3652 SSH cannot establish connection, push(Windows), by Sunny-Day200

CLOSED ISSUES

❤️ #3677 git: is_tracked: use ls_files instead of walking the index, by efiop
❤️ #3674 flake8: fix ambiguous var names, by pmrowla
❤️ #3671 Restyle Convert all paths in config to posix-style, by restyled-io[bot]
❤️ #3669 dvc push and dvc pull doesn't work properly, by gurunath-p
❤️ #3667 Restyle [WIP] dvc: introduce local stage cache, by restyled-io[bot]
❤️ #3665 Convert all paths in config to posix-style, by charlesbaynham
❤️ #3664 Config: Cache dir is inconsistant on Unix and Windows, by charlesbaynham
❤️ #3663 output: use strings instead of PathInfo for performance reasons, by pmrowla
❤️ #3660 gc: don't forget to load remote index, by efiop
❤️ #3659 Restyle Fix repro duplicating effort while checking steps #3644, by restyled-io[bot]
❤️ #3658 metrics: rework metrics implementation, by efiop
❤️ #3655 refactor: extract stage loading outside of dvcfile, by skshetry
❤️ #3654 Secsh channel 10 open FAILED: open failed: Connect failed, by Sunny-Day200
❤️ #3650 metrics diff error message in --no-scm projects, by karajan1001

LIKED ISSUE

👍 #3650 metrics diff error message in --no-scm projects, by karajan1001
It received 👍 x1, 😄 x0, 🎉 x1 and ❤️ x1.

NOISY ISSUE

🔈 #3680 cache file name automatically changed, by lmxs1237
It received 7 comments.

---

PULL REQUESTS

Last week, 18 pull requests were created, updated or merged.

UPDATED PULL REQUEST

Last week, 7 pull requests were updated.
💛 #3676 [WIP] dvc: implement multistage dvcfile, by skshetry
💛 #3675 remote.ssh: suppress paramiko logging, by pmrowla
💛 #3672 [WIP] remote: use string paths over PathInfo for performance reasons, by pmrowla
💛 #3647 remote: add support for WebDAV, by shizacat
💛 #3603 [WIP] dvc: introduce local stage cache, by efiop
💛 #3585 get/import: clone using --rev, by casperdcl
💛 #3577 Metrics - plotting for multiple revisions initial, by pared

MERGED PULL REQUEST

Last week, 11 pull requests were merged.
💜 #3677 git: is_tracked: use ls_files instead of walking the index, by efiop
💜 #3674 flake8: fix ambiguous var names, by pmrowla
💜 #3665 Convert all paths in config to posix-style, by charlesbaynham
💜 #3663 output: use strings instead of PathInfo for performance reasons, by pmrowla
💜 #3660 gc: don't forget to load remote index, by efiop
💜 #3658 metrics: rework metrics implementation, by efiop
💜 #3655 refactor: extract stage loading outside of dvcfile, by skshetry
💜 #3650 metrics diff error message in --no-scm projects, by karajan1001
💜 #3645 Fix repro duplicating effort while checking steps #3644, by charlesbaynham
💜 #3634 remote: locally index list of checksums available on cloud remotes, by pmrowla
💜 #3594 progress: add main bars, by casperdcl

---

COMMITS

Last week there were 14 commits.
🛠️ [progress: add main bars (#3594) * progress: add main push/pull bar

Fixes #3452
Related #1840
Related #3565

• adjust desc

• fix merge

Co-authored-by: Ruslan Kuprieiev ruslan@iterative.ai
Co-authored-by: Ruslan Kuprieiev kupruser@gmail.com](b312895) by casperdcl
🛠️ [git: is_tracked: use ls_files instead of walking the index (#3677) Current implementation doesn't work with directories, which results in
hard-to-debug bugs like #3561. The bug that made us walk through index
manually is now fixed in gitpython, so we can simply use ls_files from
now on. Added some tests to prevent this from happening in the future.

Fixes #3561](e2981d2) by efiop
🛠️ pre-commit: use flake8 v3.7.9 by efiop
🛠️ flake8: freeze to 3.7.9 by efiop
🛠️ flake8: fix ambiguous var names (#3674) by pmrowla
🛠️ [Convert all paths in config to posix-style (#3665) * Convert all paths in config to posix-style

• Use PathInfo for consistancy, add unit test and change functional tests

• Exclude Windows test on Linux

• Don't patch Config for all the tests!

• Don't do the linux test on Windows either

• De-hackify the unit test, make the functional test more explicit

• tests: adjust tests

Co-authored-by: Charles Baynham charles.baynham@npl.co.uk
Co-authored-by: Ruslan Kuprieiev ruslan@iterative.ai](a8b7e33) by charlesbaynham
🛠️ [output: use strings instead of PathInfo for performance reasons (#3663) * output: don't use PathInfo when collecting used dir cache

• fix windows path handling](a2de48b) by pmrowla
  🛠️ pre-commit: make pre-commit hook verbose by efiop
  🛠️ [metrics: rework metrics implementation (metrics: rework metrics implementation #3658) Dropping support for all types except json and yaml. No more xpath
  support.

Fixes #3572](45944dd) by efiop
🛠️ [Fix repro duplicating effort while checking steps #3644 (#3645) * Only reproduce steps once

• Add test

• Fix linter nasties

• Got the order wrong

• Check for previous run

• Formatting

• Do the de-duplication in _reproduce_stages

Co-authored-by: Charles Baynham charles.baynham@npl.co.uk](52369bd) by charlesbaynham
🛠️ [metrics diff error message in --no-scm projects (#3650) * fix #3605

• changed exception raise position

• Updated error msg

• message changed

• metrics diff: output edits per
  per metrics diff error message in --no-scm projects #3650
  and metrics diff error message in --no-scm projects #3650 (review)

• Update tests/func/test_metrics.py

Co-Authored-By: Ruslan Kuprieiev kupruser@gmail.com

• NoSCM now had a default attribute getting method

NoSCM Object now raise NoSCMError when calling attributes which only in Git Object

• Joined lines

Mail change test

• tests: diff: use NoSCMError

Co-authored-by: myname myname@mymail.com
Co-authored-by: karajan1001 jah@mail.ustc.edu.cn
Co-authored-by: Jorge Orpinel jorge@orpinel.com
Co-authored-by: Ruslan Kuprieiev kupruser@gmail.com
Co-authored-by: Ruslan Kuprieiev ruslan@iterative.ai](254cf83) by karajan1001
🛠️ gc: don't forget to load remote index (#3660) Kudos @pmrowla by efiop
🛠️ [remote: locally index list of checksums available on cloud remotes (#3634) * remote: locally index list of checksums available on cloud remotes

• fix review issues

• use pytest fixtures
• use sha256 digest
• s/fd/fobj/
• check more specific IO errors

• remote: use .dir checksums to validate index

• command: add --drop-index option for push/pull/status

• tests: --drop-index unit tests

• remote: make index threadsafe

• remote: force re-index after gc

• only save/dump index once per command

• remote: add helper index functions update_all/replace_all

• remote: store index as sqlite3 database

• fix deepsource warnings

• rename --drop-index to --clear-index

• rename index.invalidate to index.clear

• functional tests for remote.index

• fix index not being updated on status -c

• remote: add index.intersection()

• remote: use index when making assumptions about remote .dir contents

• add missing func tests

• do not index standalone files

• cleanup index test fixture

• fix deepsource warnings

• update autocompletion scripts/cli help message for --clear-index

• tests: remove unused mocker

• push: only index successfully pushed dirs

• do not index files on partial push/upload

• remote: revert behavior to trust .dir files on remote

• skip unnecessary index check for dir contents if .dir file exists on
  the remote

• fix missing RemoteIndexNoop functions

• separate index validation from status dir exists query

• bugfix: include indexed checksums for all cache_exists cases

• review fix: cleanup debug messages

• tests: fix DS warning

• remove --clear-index

• remove unused index functions](b04964f) by pmrowla
  🛠️ [refactor: extract stage loading outside of dvcfile (refactor: extract stage loading outside of dvcfile #3655) * refactor: extract stage loading outside of dvcfile

• repo: simplify stage collection

• fix typing](63c466d) by skshetry

---

CONTRIBUTORS

Last week there were 6 contributors.
👤 casperdcl
👤 efiop
👤 pmrowla
👤 charlesbaynham
👤 karajan1001
👤 skshetry

---

STARGAZERS

Last week there were 39 stagazers.
⭐ Akshat4112
⭐ larsbijl
⭐ yunbow
⭐ sk2sat
⭐ DominikPott
⭐ jeremydouglass
⭐ crayonzx
⭐ xander163
⭐ henrylaynesa
⭐ sh
⭐ vvarp
⭐ zambujo
⭐ lynxnathan
⭐ transitive-bullshit
⭐ marcus-sa
⭐ nikitavoloboev
⭐ ahmednooor
⭐ luav
⭐ Skyblueballykid
⭐ huynhducduy
⭐ zarchary
⭐ a-canela
⭐ bkbncn
⭐ edumotya
⭐ doyley91
⭐ Runiko
⭐ justinaustin
⭐ omarabid59
⭐ satejsoman
⭐ drewfarris
⭐ kareemamrr
⭐ wbadart
⭐ hokiegeek2
⭐ RafalSkolasinski
⭐ arsedler9
⭐ daha
⭐ asciialex
⭐ srandoux
⭐ soycode
You all are the stars! 🌟

---

RELEASES

Last week there were no releases.

---

That's all for last week, please 👀 Watch and ⭐ Star the repository iterative/dvc to receive next weekly updates. 😃

You can also view all Weekly Digests by clicking here.

Your Weekly Digest bot. 📆