dvc push doesn't recognise that files are missing in remote storage

[dldx]

For some reason, I was missing a number of files in my remote storage on GCS. When I run dvc pull, it fails with the following error:

ERROR: failed to download 'gs://redacted/dvc/redacted/00/642ad56326eb0b6caf3784810a49a0' to '../.dvc/cache/00/642ad56326eb0b6caf3784810a49a0' - 'NoneType' object has no attribute 'size'

In order to fix this, I decide to run dvc add and dvc push on a machine which still has these files. The commands run fine, and dvc push reports that everything is fine. However, in reality, it does not upload the missing files to the remote cache. In the end, I had to solve this by manually uploading these files to my remote storage in GCS.

Bug Report

Please provide information about your setup

Output of dvc version:

$ dvc version 
DVC version: 1.1.6
Python version: 3.7.6
Platform: Linux-4.9.0-12-amd64-x86_64-with-debian-9.12
Binary: False
Package: pip
Supported remotes: gs, hdfs, http, https, ssh
Cache: reflink - not supported, hardlink - supported, symlink - supported
Filesystem type (cache directory): ('ext4', '/dev/sda1')
Repo: dvc, git
Filesystem type (workspace): ('ext4', '/dev/sda1')

dvc_push_log.txt

[efiop]

@dldx Could you try removing .dvc/tmp/index and try again?

Did someone run garbage collection on your remote by any chance?

[dldx]

@efiop Hmm, nope, that didn't change anything... It is easy for me to replicate this. I just need to delete a file in the remote storage after I have pushed it.

I didn't run gc but someone else may have (I have warned everyone about running gc on the remote). I'm not really sure yet what caused it but I'm wondering if there are other missing files as well.

[efiop]

I just need to delete a file in the remote storage after I have pushed it.

@dldx That messes up with our index, hence why I've asked to remove .dvc/tmp/index to force it to check things manually. Could you remove it and check dvc status -c ?

[dldx]

Sorry for the confusion. I did remove the index, but it didn't trigger a check.

$ rm -rf ../.dvc/tmp/index        
$ dvc status -c PSScene4Band.dvc  
Data and pipelines are up to date.  
$ dvc push -R PSScene4Band
Everything is up to date.

No push triggered even though there are remote files missing.

[efiop]

@dldx Are you sure you've deleted a file that is used in PSScene4Band.dvc ?

[efiop]

@dldx Ah, sorry, I've missed that we actually trust the remotes in regards to the files in directories in order to make dvc operations faster. If your collegues are deleting stuff randomly from the cloud, you might consider making your remote untrusted with:

dvc remote myremote verify true

that will make it paranoid again.

[efiop]

@dldx But usually even during gc, we delete the .dir cache file first and then remove the actual files. Could you please double check that everyone is using the latest dvc version?

[efiop]

Thinking about it, we could consider throwing a warning if we assume that file is there but not able to pull it.

[efiop]

@dldx Any updates? 🙂

[raharth]

Sorry to reopen this, but I have a nearly identical problem. Though, in my case it is not caused by deleting files in the remote, but by adding a new remote to which I want to push.

I have tried both suggestions, removing tmp/index and setting verify to true, both is not solving the problem I only receive a "Everything is up to date", which is incorrect since the files do not appear in the remote storage.