remote: getting a directory checksum requires uploading a file to an external cache

Version: 0.62.1

Description: The current approach to calculate a directory checksum is to collect its contents' checksum and then "upload" it and get the checksum provided by the remote:

https://github.com/iterative/dvc/blob/a091b67341b00fea8ef43906f59c88cb38f384ff/dvc/remote/base.py#L246-L256

For example, if it is an S3 remote, DVC will create a file containing an "index" of the directory with its checksums on the S3 cache and then will try to get the ETag of such file.

This is an easy way to check if the directory has changed. Since the "directory index" is already there, you are a head operation away from knowing if the directory changed or not. Otherwise, you'll need to recompute the file on the spot.

Right now we don't require setting up a cache for using external dependencies, but as mentioned above, this is needed for directories.

Let's discuss if there's another way around uploading files or just settle with that implementation and modify the code and docs accordingly.

Related: #1654

[shcheklein]

@MrOutis I don't understand the issue described, tbh. Could you provide please an example where it's clear what cache is being used, what remote is being used, where directory is located, etc.

Sure, @shcheklein . Take a look at the following script, it setups a mock S3 server with a directory which then is used as a dependency on a dvc run command:

#!/usr/bin/env bash

set -x

export AWS_ACCESS_KEY_ID='testing'
export AWS_SECRET_ACCESS_KEY='testing'
export AWS_SECURITY_TOKEN='testing'
export AWS_SESSION_TOKEN='testing'

moto_server s3 &> /dev/null &

# File structure:
#       dvc-temp
#       ├── data
#       │  ├── alice
#       │  ├── alpha
#       │  └── subdir
#       │     ├── 1
#       │     ├── 2
#       │     └── 3
#       ├── empty_dir
#       ├── empty_file
#       └── foo

python -c '
import boto3

session = boto3.session.Session()
s3 = session.client("s3", endpoint_url="http://localhost:5000")
s3.create_bucket(Bucket="dvc-temp")
s3.put_object(Bucket="dvc-temp", Key="cache/")
s3.put_object(Bucket="dvc-temp", Key="empty_dir/")
s3.put_object(Bucket="dvc-temp", Key="empty_file", Body=b"")
s3.put_object(Bucket="dvc-temp", Key="foo", Body=b"foo")
s3.put_object(Bucket="dvc-temp", Key="data/alice", Body=b"alice")
s3.put_object(Bucket="dvc-temp", Key="data/alpha", Body=b"alpha")
s3.put_object(Bucket="dvc-temp", Key="data/subdir/1", Body=b"1")
s3.put_object(Bucket="dvc-temp", Key="data/subdir/2", Body=b"2")
s3.put_object(Bucket="dvc-temp", Key="data/subdir/3", Body=b"3")
'

repository=$(mktemp -d)
cd $repository
dvc init --no-scm
dvc remote add s3 s3://dvc-temp
dvc remote modify s3 endpointurl http://localhost:5000

dvc remote add cache remote://s3/cache
dvc config cache.s3 cache

dvc run -d remote://s3/data 'echo something'

To use remote://s3/data as a dependency, DVC needs to create a file on the remote cache with the following content:

[{'etag': '6384e2b2184bcbf58eccf10ca7a6563c', 'relpath': 'alice'},
 {'etag': '2c1743a391305fbf367df8e4f069f9f9', 'relpath': 'alpha'},
 {'etag': 'c4ca4238a0b923820dcc509a6f75849b', 'relpath': 'subdir/1'},
 {'etag': 'c81e728d9d4c2f636f067f89cc14862c', 'relpath': 'subdir/2'},
 {'etag': 'eccbc87e4b5ce2fe28308fd9f2a7baf3', 'relpath': 'subdir/3'}]

Note: added newlines for readability but the file shouldn't contain newlines.

Such file would be located at remote://cache/af/134c33d65939675318e71363241eee.dir and this is what I called an index file before (probably wrongly named -- but it is a list of the files inside the directory and its checksums).

As I already stated, it is useful for checking the integrity of the directory.

The problem is that setting up a cache for working with external dependencies is not required by DVC.

As an example, you can do dvc import-url remote://s3/data/alice and it won't need a cache at all (because is a single file and doesn't run the _get_dir_info_checksum method).

Let's say that the following commands are equivalent:

$ dvc import-url remote://s3/data
$ dvc run -d remote://s3/data -o data/ 'aws s3 copy --recursive s3://dvc-temp/data data'

Because data/ is a directory, it would require a configured cache.

For external outputs (it doesn't matter if it is a directory or not), we throw an error when a cache is not set:

$ dvc run -o remote://s3/data/alice 'echo something'

ERROR: failed to run command - no cache location setup for 's3' outputs.

The discussion resolves around the behavior of creating such index file and requiring the user to set up a cache for dvc import-urling a directory.

Hope it is more understandable now, @shcheklein :)

[shcheklein]

Thanks, @MrOutis ! It's definitely helpful. Some questions that are still not exactly clear to me:

This is an easy way to perform an integrity check.

Is it an integrity check or a check to see that a remote dir has changed to rerun a stage if needed? If it's an integrity check, when do we run it?

head operation away from knowing if the directory changed or not

Even if create an index file on the remote cache, how does it help to see if the directory changed or not? We need to calculate that file every time, right?

Am I right that the question is not about creating an index file itself (we will need it anyway, right?) and more about if we need to cache it or just calculate when it's needed?

If so, then can you clarify again what are the benefits of caching it? I don't understand if caching this file saves us from recalculating the dir checksum every time we want to check if something has changed on not.

[efiop]

@MrOutis ^

[efiop]

Users are getting confused by this https://discordapp.com/channels/485586884165107732/563406153334128681/653666017368735782 🙁 Need to at least handle it gracefully, until we decide what to do with this.

[shcheklein]

@efiop @MrOutis count me in - I got confused once :) (even though I was aware about this ticket). My 2cs - I would say that the biggest problem is the UI here - that we left a user facing NPE .

@shcheklein:

Is it an integrity check or a check to see that a remote dir has changed to rerun a stage if needed? If it's an integrity check, when do we run it?

I used integrity check wrongly, I was talking about the remote directory changing as you pointed out.

If so, then can you clarify again what are the benefits of caching it? I don't understand if caching this file saves us from recalculating the dir checksum every time we want to check if something has changed on not.

@shcheklein , I'll come back to this one later, will need to think it better

[efiop]

Another user ran into this https://discordapp.com/channels/485586884165107732/485596304961962003/667036162472411144