pull: Unclear error message when remote support is not installed

[nielstenboom]

Bug Report

Description

If you've installed DVC via pip without the remote support you need (for example S3), the error message is very unclear.

Reproduce

1. Setup fresh repo & pip install dvc[s3]
2. Copy dataset.zip to repo
3. dvc init & dvc add dataset.zip
4. Setup remote & dvc push
5. Move to fresh python environment
6. pip install dvc
7. Clone repo
8. dvc pull -v

Expected

I would've expected an error message saying something like: "the remote type (s3) is not supported in your dvc installation".

Instead you get the following output:

$ dvc pull -v
2021-10-26 10:35:07,698 DEBUG: failed to pull cache for 'models'      
2021-10-26 10:35:07,699 DEBUG: failed to pull cache for 'data/evaluation'
2021-10-26 10:35:08,269 WARNING: No file hash info found for 'data/evaluation'. It won't be created.                                                                                                                                            
2021-10-26 10:35:08,304 WARNING: No file hash info found for 'models'. It won't be created.                                                                                                                                                     
2 files failed                                                                                                                                                                                                                                  
2021-10-26 10:35:08,326 ERROR: failed to pull data from the cloud - Checkout failed for following targets:
data/evaluation
models
Is your cache up to date?
<https://error.dvc.org/missing-files>

Took me a couple hours to figure it out I had to reinstall dvc with s3 support enabled 😅

Environment information

Output of dvc doctor:

$ dvc doctor

DVC version: 2.8.2 (pip)
---------------------------------
Platform: Python 3.9.7 on Linux-4.14.246-187.474.amzn2.x86_64-x86_64-with-glibc2.31
Supports:
        webhdfs (fsspec = 2021.10.1),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
Cache types: hardlink, symlink
Cache directory: nfs4 on xxxxxx.efs.eu-west-1.amazonaws.com:/data-pvc
Caches: local
Remotes: s3
Workspace directory: nfs4 on xxxxx.efs.eu-west-1.amazonaws.com:/data-pvc
Repo: dvc, git

[efiop]

@nielstenboom Hi. Are you sure that's full verbose log? It seems to be missing a bunch of info.

[nielstenboom]

Ah I'm sorry, I forgot a step to reproduce this error, set the cache directory to something else before pulling:

7.5 dvc cache dir /tmp
8. dvc pull -v

Results in:

dvc pull -v
2021-10-26 17:56:30,068 DEBUG: failed to pull cache for 'models'
2021-10-26 17:56:30,068 DEBUG: failed to pull cache for 'data/evaluation'
2021-10-26 17:56:30,134 WARNING: No file hash info found for 'models'. It won't be created.
2021-10-26 17:56:30,154 WARNING: No file hash info found for 'data/evaluation'. It won't be created.
2 files failed
2021-10-26 17:56:30,163 ERROR: failed to pull data from the cloud - Checkout failed for following targets:
models
data/evaluation
Is your cache up to date?
<https://error.dvc.org/missing-files>
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/dvc/command/data_sync.py", line 30, in run
    stats = self.repo.pull(
  File "/usr/local/lib/python3.9/site-packages/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/dvc/repo/pull.py", line 40, in pull
    stats = self.checkout(
  File "/usr/local/lib/python3.9/site-packages/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/dvc/repo/checkout.py", line 110, in checkout
    raise CheckoutError(stats["failed"], stats)
dvc.exceptions.CheckoutError: Checkout failed for following targets:
models
data/evaluation
Is your cache up to date?
<https://error.dvc.org/missing-files>
------------------------------------------------------------
2021-10-26 17:56:30,166 DEBUG: Analytics is enabled.
2021-10-26 17:56:30,180 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpbfgth088']'
2021-10-26 17:56:30,181 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpbfgth088']'

Without the cache command the expected error message does appear.

[efiop]

@nielstenboom The error in both logs indicates that your dvc files are missing version info, that usually happens when you forget to track dvc.lock with git.

If you don't install [s3] part during pip install, you'll get:

ERROR: failed to pull data from the cloud - URL 's3://' is supported but requires these missing dependencies: ['s3fs', 'boto3']. To install dvc with those dependencies, run:

        pip install 'dvc[s3]'

See <https://dvc.org/doc/install> for more info.

My guess is that maybe you've done git pull or something after switching the venv.

[nielstenboom]

@efiop thank you for your quick response!

I do believe this is a bug though, caused by changing the local cache setting. The following happens with a version of dvc that does not have s3 installed with it:

$ dvc pull
Everything is up to date.
ERROR: failed to pull data from the cloud - URL 's3://' is supported but requires these missing dependencies: ['s3fs', 'boto3']. To install dvc with those dependencies, run:

	pip install 'dvc[s3]'

See <https://dvc.org/doc/install> for more info.

$ dvc cache dir /tmp
$ dvc pull
WARNING: No file hash info found for 'data/evaluation'. It won't be created.
WARNING: No file hash info found for 'models'. It won't be created.
2 files failed
ERROR: failed to pull data from the cloud - Checkout failed for following targets:
data/evaluation
models
Is your cache up to date?
<https://error.dvc.org/missing-files>

The dvc cache dir /tmp command causes the error message to change, that does not seem like the intended behavior to me.

[efiop]

Hm, the everything is up to date is definitely a bug though. And I can indeed reproduce that warning as well, looking into it...

EDIT: when odb (aka cache dir) is not accessible, we fail to load the corresponding object and thus checkout throws those warnings that are clearly no longer relevant.

[nsorros]

I faced the same error and was able to reproduce it even without the cache change. My steps are

mkdir issue
cd issue
git init
dvc init
mkdir data
touch data/data.csv
dvc add data
dvc remote add remote s3://bucket
dvc push

then to replicate the problem

cd ..
mkdir test_issue
cd test_issue
git clone ../issue
python -m venv venv
source venv/bin/activate
pip install dvc
dvc pull

This seems to be related with catching a generic DvcException in output.py. I am starting a PR about it #7501