pull: going completely crazy on large dataset

[shcheklein]

Version:

DVC version: 0.73.0
Python version: 3.6.8
Platform: Linux-4.15.0-1056-aws-x86_64-with-Ubuntu-18.04-bionic
Binary: False
Package: None
Filesystem type (cache directory): ('xfs', '/dev/nvme1n1')
Filesystem type (workspace): ('xfs', '/dev/nvme1n1')

Reproduce:

git clone git@github.com:iterative/dataset-registry-private.git
cd dataset-registry-private
dvc pull -j 100 ILSVRC.dvc

Output:

(sorry for the video in this format, was the fastest way to capture it while it was running)

https://www.dropbox.com/s/m821sgc1flk770o/dvc-pull-going-crazy.mov?dl=0

[Suor]

Should be fixed by combining pbars. @casperdcl

[casperdcl]

-j 100 would be the issue (nested bars scrolling off the screen). I should be able to come up with a temp patch for now. P.S. what happened with the old method? No scrolling but lots of flickering and very slow?

[Suor]

@casperdcl what's the point? Let's go for the actual fix, i.e. combining into a single pbar.

[casperdcl]

The point would be I can downgrade a p1 to p2/p3 today, but don't think I'll have time for a full fix soon.

[Suor]

@casperdcl p1 is not p0 (fix asap), p1 can wait. TBH, I see combining threads as top priority in pbars (maybe even UI as a whole) now since it's the highest value, so no point doing something else instead/in the meantime unless it makes dvc unusable.

[efiop]

Totally agree with @Suor .