`dvc pull` with wrong S3 remote failed but user wasn't informed

[markf94]

System

My dvc version is 0.77.3, I installed & upgraded dvc through pip3 and I'm using Fedora 31.

Problem

I followed the 'Getting Started' guide and ran the following commands

$ dvc init
$ git add .dvc/* && git commit -m 'Initialized dvc'
$ dvc remote add -d s3remote https://s3.amazonaws.com/<bucket-name>
$ dvc add <some-file>
$ git add .gitignore <some-file>.dvc && git commit -m 'Added <some-file'
$ dvc push -r s3remote

The last command finished with Everything is up to date. which is weird since the S3 bucket was empty when checking it. I went on to delete <some-file> and run dvc pull -r s3remote which would restore <some-file> without throwing any errors. This was surprising since it did not get clear where dvc is storing my file backups and why my S3 bucket was still empty without dvc communicating any errors.

Only when I removed the .dvc/cache folder and ran dvc pull -r s3remote it would complain with the following:

ERROR: failed to download 'https://s3.amazonaws.com/<bucket-name>/31/69f7ce4ebb503afca037d35b7eb3a9' to '.dvc/cache/31/69f7ce4ebb503afca037d35b7eb3a9' - '301 Moved Permanently'

ERROR: failed to download 'https://s3.amazonaws.com/<bucket-name>/a3/04afb96060aad90176268345e10355' to '.dvc/cache/a3/04afb96060aad90176268345e10355' - '301 Moved Permanently'

ERROR: failed to pull data from the cloud - 2 files failed to download


Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

The fix

On my end, it was an easy fix. After consulting the documentation in more detail, I realized that I set up my S3 remote incorrectly using the https://s3.amazonaws.com/<bucket-name> URL to the bucket whereas I should have used the s3://<bucket-name> URL. Hence, running

$ dvc remote modify s3remote url s3://<bucket-name>

did the job.

It still remains weird why I was never warned about the remote being incorrectly set up and why dvc pull -r s3remote worked without any problems even though there were no files in the S3 bucket (it seems that it restored them from local cache as a fall-back) but it should have given me a headsup!

[efiop]

Hi @markf94 !

Thanks for reporting this issue! I can reproduce:

#!/bin/bash

set -e
set -x

rm -rf myrepo
mkdir myrepo
cd myrepo
git init
dvc init
git commit -m 'Initialized dvc'
dvc remote add -d s3remote https://s3.amazonaws.com/dvc-temp/ruslan
echo foo > foo
dvc add foo
git add .gitignore foo.dvc && git commit -m 'Added some-file'
dvc push

Same with dvc status -c. So there is definitely a bug, looking into it.

For the record, the sequential pull didn't complain because it saw that it already has everything it needs in the local cache, so it didn't have to download anything.

[fabiosantoscode]

Hey there, I picked this one up!

I found a likely culprit - the existence check for an HTTP/S remote is not exactly thorough.

[efiop]

For the record, #3169 (comment) . Reopenning.

[fabiosantoscode]

@efiop this seems to be fixed by my PR. When using your repro script, both dvc push and dvc status -c fail with the following error:

ERROR: failed to push data to the cloud - could not perform a HEAD request

The purpose of my PR was to generate this error in the face of redirects without a location header (which is something S3 does in this case).

[efiop]

@fabiosantoscode Unfortunately the main issues described in #3169 (comment) are not fixed and they need some research before we could close this ticket.