Weekly Digest (12 July, 2020 - 19 July, 2020)

[weekly-digest]

Here's the Weekly Digest for iterative/dvc:

---

ISSUES

Last week 40 issues were created.
Of these, 25 issues have been closed and 15 issues are still open.

OPEN ISSUES

💚 #4235 safe Dvcfile.dump(), by skshetry
💚 #4234 Problems with dvc push, by LarryThermo
💚 #4233 repro: update status message for dvc-added files, by sarthakforwet
💚 #4232 dvc move fails when cache dir is read-only, by florianspecker
💚 #4230 hdfs tests failing, by skshetry
💚 #4229 Special dir warning #3436, by aerubanov
💚 #4224 S3: Use key_id and key_secret directly, by farizrahman4u
💚 #4223 Make dvc pull --run-cache allow targets for a granular pull process, by maricatovictor
💚 #4220 bump azure-storage-blob version to 12.3.2, by dtrifiro
💚 #4218 dvc gc remove , by HaddocktheHorrible
💚 #4213 Allow pipeline to create pipeline, by MatthieuBizien
💚 #4211 metrics diff: accept multiple revisions (like plots diff), by jorgeorpinel
💚 #4210 Back-fill metrics, by jonilaserson
💚 #4206 Daemon fails when dvc is called via a wrapper script., by danfischetti
💚 #4194 TLDR for DVC [https://github.com/tldr-pages/tldr], by PuneethaPai

CLOSED ISSUES

❤️ #4231 Katacode - stages interactive tutorial, by yevgenypats
❤️ #4228 DAG with more structured dependencies per stage, by jonilaserson
❤️ #4226 Restyle S3: Use key_id and key_secret directly, by restyled-io[bot]
❤️ #4225 Restyle S3: Use key_id and key_secret directly, by restyled-io[bot]
❤️ #4222 snap: deploy: fix major channels not being pushed, by casperdcl
❤️ #4221 dvc: get rid of CleanTree, by efiop
❤️ #4219 Bump networkx, by faucct
❤️ #4217 Restyle dvc: get rid of WorkingTree, by restyled-io[bot]
❤️ #4216 dvc: get rid of WorkingTree, by efiop
❤️ #4215 tree: remove duplicated code, by efiop
❤️ #4214 dvc: remove tree-specific wrappers from cache, by efiop
❤️ #4212 dvc: split remotes from trees, by efiop
❤️ #4209 RepoDependency: never changed, by pared
❤️ #4208 deps/outs: get rid of remotes in favor of trees, by efiop
❤️ #4207 Restyle Optimize ignore performance, by restyled-io[bot]
❤️ #4205 dvc commit fails with files imported via dvc import, by Cortysus
❤️ #4204 list: support tracked file path as a target, by efiop
❤️ #4203 Repo cache?, by jorgeorpinel
❤️ #4202 list: granularity not working [qa], by jorgeorpinel
❤️ #4201 tests: http: fix typo, by efiop
❤️ #4200 tests: remove real SSH, by efiop
❤️ #4199 Experiments proof of concept, by pmrowla
❤️ #4198 Brew install/upgrade dvc overwrites current version of python3, by philipperemy
❤️ #4197 dvc add: .dvcignore example not working as expected [qa], by utkarshsingh99
❤️ #4196 remove: remove dvc.yaml and dvc.lock if they are empty, by nik123

LIKED ISSUE

👍 #4199 Experiments proof of concept, by pmrowla
It received 👍 x3, 😄 x0, 🎉 x4 and ❤️ x3.

NOISY ISSUE

🔈 #4205 dvc commit fails with files imported via dvc import, by Cortysus
It received 11 comments.

---

PULL REQUESTS

Last week, 23 pull requests were created, updated or merged.

UPDATED PULL REQUEST

Last week, 5 pull requests were updated.
💛 #4229 Special dir warning #3436, by aerubanov
💛 #4224 S3: Use key_id and key_secret directly, by farizrahman4u
💛 #4220 bump azure-storage-blob version to 12.3.2, by dtrifiro
💛 #4114 version: redesign output format for the dvc version command #3499, by sahilbhosale63
💛 #3585 get/import: --rev BRANCH: shallow clone, by casperdcl

MERGED PULL REQUEST

Last week, 18 pull requests were merged.
💜 #4222 snap: deploy: fix major channels not being pushed, by casperdcl
💜 #4221 dvc: get rid of CleanTree, by efiop
💜 #4219 Bump networkx, by faucct
💜 #4216 dvc: get rid of WorkingTree, by efiop
💜 #4215 tree: remove duplicated code, by efiop
💜 #4214 dvc: remove tree-specific wrappers from cache, by efiop
💜 #4212 dvc: split remotes from trees, by efiop
💜 #4209 RepoDependency: never changed, by pared
💜 #4208 deps/outs: get rid of remotes in favor of trees, by efiop
💜 #4204 list: support tracked file path as a target, by efiop
💜 #4201 tests: http: fix typo, by efiop
💜 #4200 tests: remove real SSH, by efiop
💜 #4199 Experiments proof of concept, by pmrowla
💜 #4196 remove: remove dvc.yaml and dvc.lock if they are empty, by nik123
💜 #4192 pull: target message, by dmpetrov
💜 #4190 config: use repo.tree, by efiop
💜 #4125 LocalRemoteTree: use repo tree as work_tree with local outputs, by pared
💜 #4120 Optimize ignore performance, by karajan1001

---

COMMITS

Last week there were 22 commits.
🛠️ [Experiments proof of concept (#4199) * experiments: add initial experiments.show

• only includes metrics + params (no code/data information)

• experiments: basic dvc experiments show functionality

• experiments: clone and run repro inside clone workspace

• repro: add -e/--experiment option

• experiments: hash experiments to identify duplicates

• experiments: update show

• experiments: checkout experiment after running repro

• experiments: add dvc experiments checkout

• experiments: add simple dvc experiments diff command

• just shows combined output from metrics diff and plots diff for
  now

• add very simple tests

• add some useful output messages

• experiments: suppress help messages while feature is in development

• experiment related commands will be accessible but hidden from the
  default command help messages

• add simple experiments command tests

• experiments: disable feature by default

• experiments only enabled in test environment (DVC_TEST) or when
  core.experiments config option is true

• use fetch on checkout missing revs](a56c8bb) by pmrowla
  🛠️ snap: deploy: fix major channels not being pushed (#4222) by casperdcl
  🛠️ dvc: bump to 1.1.11 by efiop
  🛠️ output: local: use repo tree if possible (#4125) by pared
  🛠️ RepoDependency: never changed (#4209) by pared
  🛠️ [dvc: get rid of CleanTree (dvc: get rid of CleanTree #4221) * dvcignore: move file/dir/parent checks into DvcIgnoreFilter

• dvc: get rid of CleanTree

CleanTree is a very awkward wrapper that spreads the context across
the codebase and results in unexpected behaviour when using it.

This PR starts moving dvcignore-related logic into the trees themselves
(it makes a lot of sense, kinda like state) so they could deal with
it however they like.

There are at least two temporary ugly parts about this PR:

1. dvcignore is used by individual trees and not packed into
   tree/base.py yet;

2. dvcignore_root argument. This one is caused by the dvcignore trying
   to collect everything topdown starting from the certain root dir. What
   it should do instead is for a certain path that is being checked look up
   the tree through the parents until it finds repo root (.dvc dir) and
   then stop. That would handle subrepos as well. At the same time we need
   to leverage existing dvcignore trie structure to cache those results.](625b208) by efiop
   🛠️ Bump networkx (#4219) To avoid conflict with https://github.com/aws-cloudformation/cfn-python-lint by faucct
   🛠️ [dvc: get rid of WorkingTree (dvc: get rid of WorkingTree #4216) Working tree is really just a regular local tree and should be
   used by outputs when trying to compute a hash for themselves.
   We didn't use it previously because local tree was embeded into
   the local remote class.

Related to #4050](33abc40) by efiop
🛠️ tree: remove duplicated code (#4215) Leftover from remote/tree separation. by efiop
🛠️ dvc: remove tree-specific wrappers from cache (#4214) This helps combat the remote/tree/cache confusion. by efiop
🛠️ [dvc: split remotes from trees (#4212) * dvc: remove tree-specific wrappers from remote

This helps combat the remote/tree/cache confusion.

• dvc: split remotes from trees

Related to #4050](76228a2) by efiop
🛠️ [deps/outs: get rid of remotes in favor of trees (#4208) Outputs and dependencies don't really care about the remotes, as they
don't do any pushing or pulling. What they really need are the trees,
so let's use them directly.

Part of #4050](bbb1c59) by efiop
🛠️ [Optimize ignore performance (#4120) * Remove Duplicate ignore Match

• Continue Optimize Dvcignore

fix #3869

• Add a new test with multi ignore files

• Solve merging two dvc files

• Solve Code Climate

• For Windows

• Complete addition of patterns. Add one test

• Systematic test

• Change request

• Change request

• Seperate path sepcification math

• Rename and add comment

• rename change_dirname to private

• Update dvc/pathspec_math.py

list comprehension

Co-authored-by: Alexander Schepanovski suor.web@gmail.com

• Change request

• Update dvc/ignore.py

Co-authored-by: karajan1001 jah@mail.ustc.edu.cn
Co-authored-by: Alexander Schepanovski suor.web@gmail.com
Co-authored-by: Ruslan Kuprieiev kupruser@gmail.com](58f6534) by karajan1001
🛠️ [list: support tracked file path as a target (#4204) * list: support tracked file path as a target

Fixes #4202

• erepo: add str to improve error readablity

Related to #4202](66def03) by efiop
🛠️ tests: http: fix typo (#4201) Fixes #4182 by efiop
🛠️ [tests: remove real SSH (#4200) * tests: remove real SSH

No longer needed, all tests are using mocked ssh already.

• travis: don't setup ssh](4c9245a) by efiop
  🛠️ readme: remove python versions note by efiop
  🛠️ [remove: remove dvc.yaml and dvc.lock if they are empty (remove: remove dvc.yaml and dvc.lock if they are empty #4196) * remove: remove dvc.yaml, dvc.lock if they are empty

• tests: remove: simplified dvc.yaml in test_remove_stage_removes_dvcfiles_if_no_stages_left

• tests: remove: minor fixes

• tests: remove: simplified test_remove_stage_removes_dvcfiles_if_no_stages_left

• remove: independent removal of dvc.lock and dvc.yaml](1c9588e) by nik123
  🛠️ dvc: bump to 1.1.10 by efiop
  🛠️ [pull: target message (pull: target message #4192) * pull: target message

• update fetch/status/etc help

Co-authored-by: Ruslan Kuprieiev ruslan@iterative.ai](59409be) by dmpetrov
🛠️ dvc: bump to 1.1.9 by efiop
🛠️ config: use repo.tree (#4190) Fixes #4188 by efiop

---

CONTRIBUTORS

Last week there were 8 contributors.
👤 pmrowla
👤 casperdcl
👤 efiop
👤 pared
👤 faucct
👤 karajan1001
👤 nik123
👤 dmpetrov

---

STARGAZERS

Last week there were 86 stagazers.
⭐ MarkTuddenham
⭐ craigrshenton
⭐ jvmncs
⭐ fxghqc
⭐ sp1thas
⭐ programminghumanity
⭐ knightelvis
⭐ Duchadian
⭐ craig-shenton
⭐ itsmeale
⭐ litteZe
⭐ zhangmx
⭐ TeisNP
⭐ apethani21
⭐ Randy-Soper
⭐ Natumsol
⭐ yanqinghao
⭐ sollde
⭐ xiguagua
⭐ drunkwretch
⭐ lfeakm
⭐ xhj
⭐ kebohiki
⭐ zxytim
⭐ yanyaoer
⭐ Anguliachao
⭐ sumtumn
⭐ buzai
⭐ robscc
⭐ ilias-ant
⭐ andreaschandra
⭐ lsanotes
⭐ galaxy001
⭐ DN-Web64
⭐ vagmcs
⭐ marcosgama
⭐ aankasman
⭐ dsatecnologia
⭐ syan4826
⭐ jeffhsu3
⭐ shan224
⭐ JoshuaTao
⭐ khalidperwira
⭐ kangvcar
⭐ uhhc
⭐ Svtter
⭐ maxmilia
⭐ philMarius
⭐ Emrys-Hong
⭐ spiros
⭐ shkey
⭐ dwelcaslu
⭐ monz
⭐ mfhbam
⭐ lucmos
⭐ vasulakkaraju
⭐ gad2103
⭐ Jamesits
⭐ gcgbarbosa
⭐ krzemienski
⭐ luxiaolei
⭐ moealmaw
⭐ Clement25
⭐ rbreunev
⭐ IvanYue
⭐ DACUS1995
⭐ BrunoGomesCoelho
⭐ marskar
⭐ almereyda
⭐ lucasvazq
⭐ robert35h
⭐ gabriellovate
⭐ scientistkev
⭐ JaeyeongYang
⭐ SolsticeDante
⭐ devender-yadav
⭐ victorgrubio
⭐ benwtrent
⭐ sabralod
⭐ wtyree
⭐ Hyun-June-Choi
⭐ vishweshs4
⭐ orrshilon
⭐ chapmanjacobd
⭐ Nazar96
⭐ abdulshawakri
You all are the stars! 🌟

---

RELEASES

Last week there were 3 releases.
🚀 1.1.11 1.1.11
🚀 1.1.10 1.1.10
🚀 1.1.9 1.1.9

---

That's all for last week, please 👀 Watch and ⭐ Star the repository iterative/dvc to receive next weekly updates. 😃

You can also view all Weekly Digests by clicking here.

Your Weekly Digest bot. 📆