dvc push -q is not completely quiet, shows file transfer tqdm progress bars

[WillNichols726]

Hey guys, love what you've done with DVC.

Had a quick bug that's causing me a little issue. When I use 'dvc push -q' I'm still seeing tqdm progress bars. Wouldn't be a huge issue, but I'm probably pushing 100K 250kb files. This is a local remote, so the transfer speeds are quick. I know in some of my other scripts where I use tqdm, if the iteration time is very small, the tqdm overhead of writing to std::out actually starts to contribute to performance.

dvc version: 0.83.0
os: Windows 10

[shcheklein]

@casperdcl could you take a look please?

[casperdcl]

@WillNichols726

I know in some of my other scripts where I use tqdm, if the iteration time is very small, the tqdm overhead of writing to std::out actually starts to contribute to performance

That should really not be the case as tqdm is designed to be negligible overhead in all cases - could you elaborate a bit more? I know this isn't actually the issue here, though.

In terms of this issue, the progress output should be suppressed completely by -q. Will take a look & fix soon.

[WillNichols726]

@casperdcl - Appreciate everything you've done with tqdm! Recently, I had something that was running at 200 iterations a second, when I set mininterval to 1 sec, jumped up to 300 iterations a sec. Didn't spend a lot of time determining root cause for the speedup, it could've been caching behavior or something else. I've also seen some minor speedups on other rapidly-iterating scripts when increasing mininterval. Not a huge issue. I'm typically running in Powershell or cmd on Windows 10.

[casperdcl]

Hmm interesting as the default mininterval=0.1 should be fine on most consoles.

No real measurable difference for me (it/s)... Note that it's under 200it/s because sleep isn't accurate.

$ python -c 'from time import sleep; from tqdm import trange
[sleep(1/200) for _ in trange(100)]'
100%|██████████| 100/100 [00:00<00:00, 192.77it/s]
$ python -c 'from time import sleep; from tqdm import trange
[sleep(1/200) for _ in trange(100, mininterval=1)]'
100%|█████████| 100/100 [00:00<00:00, 192.62it/s]

[casperdcl]

ah also (back to this issue) --quiet is actually currently ignored (i.e. not implemented) by push

[casperdcl]

nvm looks like this is a bug

[efiop]

@WillNichols726 0.85.0 is out with this fix, could you please give it a try and let us know if it fixed it for you? 🙂

[efiop]

@WillNichols726 Btw, I suppose you were using -q because pbars were jumping around and being annoying? 🙂 If so, just wanted to let you know that we are working on a fix, so stay tuned for that too 🙂

[WillNichols726]

@efiop - Yeah, was using -q to try and suppress the progress bars that were jumping around.