dvc push after dvc push is slow

[JohanMollevik]

Please provide information about your setup
DVC version(i.e. dvc --version), Platform and method of installation (pip, homebrew, pkg Mac, exe (Windows), DEB(Linux), RPM(Linux))

• debian stretch
• dvc installed from pip
  $ dvc --version
  0.71.0

I have a large dataset (#2512 ) and have been trying to debug performance to evaluate if dvc will work for this type of data.

I did one dvc push against azure taking 4 hours for 132 GB data in 2.5M files 1 .dvc file. That is ok assuming there was changes. I then immediately again ran dvc push and it is taking 4 hours again.

Why does dvc not compleat much faster on the second run. There should be no changes between the remote and local cache so I was expecting it to finish after a few minutes.

[efiop]

Hi @JohanMollevik !

I suppose it was "Collecting information from remote cache" stage that was talking a lot of time, right? If it was it, then it is caused by the fact that dvc needs to check each local file with the remote to make sure that all files from that data set are present, this is needed to provide per-file granularity for directories. It is a known optimization issue and we are solving it in #2147 . If your dataset is fairly static, I would consider adding it to dvc as an archive and unpacking it with something like dvc run -d data.tar.gz --outs-no-cache data tar -xvf data.tar.gz.

[JohanMollevik]

I do not think it said, the progress bar was only 80 chars wide so some data might have been hidden.
But, there was first a progress bar taking 4 minutes or so followed by one taking 4 hours.

[efiop]

@JohanMollevik Hm, we are in the middle of revisiting ui for pull/push/etc, so i might be missing something, but from the description it does seem to be the case. First pbar was collecting local status and the second one was collecting remote status. Please see my updated comment up above.

[JohanMollevik]

@efiop Yes that looks likely. The work on #2147 sounds promising, I'm looking forward to seeing that resolved.

[efiop]

@JohanMollevik I'll close this ticket in favor of #2147 . Please ping us if you have any follow up questions :)