dvc add --external fails on files created by gsutil compose

[dvdjlaw]

Bug Report

dvc add ... --external fails when the target is an object created by gsutil compose

Steps to reproduce:

# Create some dummy partitioned files
touch a.csv
touch b.csv

gsutil cp a.csv b.csv gs://$BUCKET
gsutil compose gs://$BUCKET/a.csv gs://$BUCKET/b.csv gs://$BUCKET/combined.csv

dvc add gs://$BUCKET/combined.csv --external

Please provide information about your setup

Output of dvc version:

$ dvc version
DVC version: 1.1.3
Python version: 3.7.6
Platform: Linux-4.9.0-12-amd64-x86_64-with-debian-buster-sid
Binary: False
Package: pip
Supported remotes: gs, hdfs, http, https
Repo: dvc, git
Filesystem type (workspace): ('ext4', '/dev/sda1')

Additional Information (if any):

If applicable, please also provide a --verbose output of the command, eg: dvc add --verbose.

2020-07-02 16:13:45,122 DEBUG: fetched: [(3,)]                                  
Adding...                                                                                                                         
2020-07-02 16:13:45,846 DEBUG: fetched: [(0,)]
2020-07-02 16:13:45,853 ERROR: unexpected error - argument should be a bytes-like object or ASCII string, not 'NoneType'
------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/dvc/main.py", line 53, in main
    ret = cmd.run()
  File "/opt/conda/lib/python3.7/site-packages/dvc/command/add.py", line 22, in run
    external=self.args.external,
  File "/opt/conda/lib/python3.7/site-packages/dvc/repo/__init__.py", line 36, in wrapper
    ret = f(repo, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/dvc/repo/scm_context.py", line 4, in run
    result = method(repo, *args, **kw)
  File "/opt/conda/lib/python3.7/site-packages/dvc/repo/add.py", line 90, in add
    stage.save()
  File "/opt/conda/lib/python3.7/site-packages/dvc/stage/__init__.py", line 380, in save
    self.save_outs()
  File "/opt/conda/lib/python3.7/site-packages/dvc/stage/__init__.py", line 391, in save_outs
    out.save()
  File "/opt/conda/lib/python3.7/site-packages/dvc/output/base.py", line 277, in save
    if not self.changed():
  File "/opt/conda/lib/python3.7/site-packages/dvc/output/base.py", line 219, in changed
    status = self.status()
  File "/opt/conda/lib/python3.7/site-packages/dvc/output/base.py", line 210, in status
    if self.changed_checksum():
  File "/opt/conda/lib/python3.7/site-packages/dvc/output/base.py", line 195, in changed_checksum
    return self.checksum != self.get_checksum()
  File "/opt/conda/lib/python3.7/site-packages/dvc/output/base.py", line 181, in get_checksum
    return self.remote.get_hash(self.path_info)
  File "/opt/conda/lib/python3.7/site-packages/dvc/remote/base.py", line 753, in get_hash
    return self.tree.get_hash(path_info, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/dvc/remote/base.py", line 299, in get_hash
    hash_ = tree.get_file_hash(path_info)
  File "/opt/conda/lib/python3.7/site-packages/dvc/remote/gs.py", line 193, in get_file_hash
    md5 = base64.b64decode(b64_md5)
  File "/opt/conda/lib/python3.7/base64.py", line 80, in b64decode
    s = _bytes_from_decode_data(s)
  File "/opt/conda/lib/python3.7/base64.py", line 46, in _bytes_from_decode_data
    "string, not %r" % s.__class__.__name__) from None
TypeError: argument should be a bytes-like object or ASCII string, not 'NoneType'
------------------------------------------------------------

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

[efiop]

Hi @haishiro ! Looks like gsutil compose doesn't set md5_hash for the blob it creates. Does it provide you with a way to do that?

[efiop]

@haishiro Also, --external is considered a pretty advanced scenario, could you share more details about your scenario, please? Usually we don't recommend it unless the user really knows what he is doing.

[dvdjlaw]

I don't think there are any options for setting md5_hash in gsutil compose

My use case:
The source dataset is in BigQuery and model training is using GCP's AI Platform Training. Due to the size of the dataset, I'm using BigQuery extract job to extract the data to CSV files in Google Cloud Storage. This results in partitioned CSV files which I'm combining using gsutil compose. I'd like to add the (combined) CSV to be tracked by DVC. Since the data is already in GCS and my remote will be GCS, I'd thought I should add it using --external since it doesn't make sense to download the dataset to my local directory just for DVC to cache it in GCS.

[efiop]

@haishiro Looks like it is not what you want. There are isolation issues to just start with. See #4151 , where we have the same discussion. What you probably want is for that command to copy from gs to gs remote without downloading full copy locally. There is no such feature in dvc right now, but you could do that through the regular workflow if you mount your gs storage through fuse and dvc add data as a local file.

[dvdjlaw]

Thanks, that makes sense. A "straight to remote" option would also be nice, especially since some of my collaborators would be using Windows and can't use fuse to mount.