dvc exp run: experiment metrics are not reported when metric files are on another device than training code

[AlexandreRozier]

Bug Report

Issue name

dvc exp run runs but does not store metrics.

Description

I'm running my training script on /dev/mapper/system-home and it outputs data (model checkpoints, metrics) in /data/.cache located on another partition (/dev/sdb1). /dev/sdb1 is a purposely large partition where we are supposed to store large files. Running dvc exp run works fine, but after completion dvc exp show does not show any metrics (aswell as dvc metrics show).

When outputting metrics to a folder on the same partition as the training script (/dev/mapper/system-home), dvc exp show works perfectly and shows metrics.

When using verbose mode, I get the following errors:

2022-06-08 17:49:08,234 DEBUG: [Errno 95] no more link types left to try out: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>: [Errno 18] Invalid cross
-device link
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hx/miniconda3/envs/origami/lib/python3.8/site-packages/dvc/fs/utils.py", line 28, in _link
    func(from_path, to_path)
  File "/home/hx/miniconda3/envs/origami/lib/python3.8/site-packages/dvc/fs/base.py", line 263, in reflink
    return self.fs.reflink(from_info, to_info)
  File "/home/hx/miniconda3/envs/origami/lib/python3.8/site-packages/dvc/fs/local.py", line 156, in reflink
    return System.reflink(path1, path2)
  File "/home/hx/miniconda3/envs/origami/lib/python3.8/site-packages/dvc/system.py", line 112, in reflink
    System._reflink_linux(source, link_name)
  File "/home/hx/miniconda3/envs/origami/lib/python3.8/site-packages/dvc/system.py", line 96, in _reflink_linux
    fcntl.ioctl(d.fileno(), FICLONE, s.fileno())
OSError: [Errno 18] Invalid cross-device link

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/hx/miniconda3/envs/origami/lib/python3.8/site-packages/dvc/fs/utils.py", line 69, in _try_links
    return _link(link, from_fs, from_path, to_fs, to_path)
  File "/home/hx/miniconda3/envs/origami/lib/python3.8/site-packages/dvc/fs/utils.py", line 32, in _link
    raise OSError(
OSError: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>

The above exception was the direct cause of the following exception:

Traceback (most recent call lastInvalid cross
-device link):
  File "/home/hx/miniconda3/envs/origami/lib/python3.8/site-packages/dvc/fs/utils.py", line 124, in _test_link
    _try_links([link], from_fs, from_file, to_fs, to_file)
  File "/home/hx/miniconda3/envs/origami/lib/python3.8/site-packages/dvc/fs/utils.py", line 77, in _try_links
    raise OSError(
OSError: [Errno 95] no more link types left to try out

The full traceback can be found here:
trace.Log

The Invalid cross-device link part seems to show that dvc cannot handle cross-devices operations.

Reproduce

1. Create a default project on partition /sda1/foo1 training and evaluating a model, writing metrics to another device /sdb1/foo2

# train.py on /sda1/foo1 
from dvclive import Live
live = Live( "/data/metrics") # /data mounted on /sdb1/foo2
for epoch in epochs:
    metrics = ...
    for metric_name, value in metrics.items():
          live.log(metric_name, value)
    live.next_step()

ex of /data/metrics.json:

{
    "step": 1,
    "loss": 0.7107148170471191,
    "directed_f1_weighed": 0.0,
    "undirected_f1_weighed": 0.0,
    "oriented_acc": 0.8346456692913385,
    "officical_f1_macro": 0.0
}

ex of /data/metrics/scalar/loss.tsv:

timestamp	step	loss
1654703111346	0	0.8031530231237411
1654703334339	1	0.7107148170471191

4. dvc exp show doesn't show any metrics column
   
   

Expected

dvc metrics show actually shows metrics columns.

Environment information

Python 3.8.13

Description: Ubuntu 20.04.3 LTS
Release: 20.04
dvclive 0.8.2

Output of dvc doctor:

$ dvc doctor
DVC version: 2.10.2 (pip)
---------------------------------
Platform: Python 3.8.13 on Linux-5.4.0-91-generic-x86_64-with-glibc2.17
Supports:
        hdfs (fsspec = 2022.5.0, pyarrow = 3.0.0),
        webhdfs (fsspec = 2022.5.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2022.5.0, boto3 = 1.21.21)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/system-home
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/mapper/system-home
Repo: dvc, git

Additional Information (if any):

I think the error comes from a missing support of cross-device copying (check https://stackoverflow.com/questions/42392600/oserror-errno-18-invalid-cross-device-link). Do you have any ideas ? Thanks for this nice piece of software 👍

[daavoo]

Hi @AlexandreRozier! Is that the full traceback? Are there any other exceptions logged above the traceback you shared?

[AlexandreRozier]

@daavoo Thanks for your answer, I added the full traceback in the issue description. You can also read it here:
trace.Log

[daavoo]

Hi @AlexandreRozier . It looks like the errors in the exp run -v traceback (the one you shared in previous comment) are just coming from #7865 so (I think) nothing to worry about. It looks like the experiment is completed successfully.

Could you share the output of dvc exp show --json -vv?

[dfossl]

Hi @AlexandreRozier

I also have had a similar issue, but all on the same machine, actually. When I run dvc exp show in the dvc/git root directory I get the metrics, however when I run it in a subdirectory I do not.

Rolling back to version 2.10.2 fixes this for me.

Hopefully this provides some additional context that can help narrow down the issue.

Let me know if you also want my logging information as well.

[AlexandreRozier]

@daavoo You might be right, the experiments are indeed completing successfully, but unfortunately they don't show up in dvc exp show. You'll find the output of dvc exp show --json -vv here :
dvc_exp_show_json.log
In particular, one can see that the metrics value is empty (line 1430)

@dfossl I'm also working on a single machine, but across 2 different partitions. And dvc exp show does not work at root directory level. But thanks for your feedback !

[pmrowla]

@AlexandreRozier is your metrics path inside your DVC repo? Or is it a completely external path?

From your sample code:

# train.py on /sda1/foo1 
from dvclive import Live
live = Live( "/data/metrics") # /data mounted on /sdb1/foo2

Where is your DVC project located (i.e. where is the .dvc directory)?

It sounds like /data/metrics is outside of the DVC project in which case DVC will not track it as an output - all output paths must be inside the DVC project root directory. Note that the path is what matters here, not the physical device a file is stored on.

So something like this would be allowed:

cd /foo
dvc init
mkdir bar
mount /dev/sdb2 bar
# write and track outputs in /foo/bar/... on /dev/sdb2

Since in this case, /foo/bar is inside the DVC repository located at /foo

[pmrowla]

But rather than nesting mount points like that, I think what you may be looking for is to configure dvc cache dir so that it resides on /dev/sdb1, and then configure your DVC repository (on /dev/sda1) to use symbolic links. Then you would set your metrics path so that it is inside the DVC repository. The end result would be that all of your output/metrics data would be stored in /dev/sdb1, and your repository would contain symlinks to those files.

see:
https://dvc.org/doc/command-reference/cache/dir#cache-dir
https://dvc.org/doc/user-guide/large-dataset-optimization

[AlexandreRozier]

@pmrowla Thanks, I had not understood that all output paths had to be in the dvc repo.
For the sake of clarity:

• The DVC project is located at /home/hx/ORIGAMI/tmp (on sda5)
• The metrics are stored in /data/... (on sdb1)
  Mounting /sdb1 somewhere in the DVC project is not possible since other users also work on this server, but I'll take a look at the symlinking approach you're advising.

In conclusion, it seems that my issue originates in the lack of support for output file paths outside of the DVC repo, and can be closed if that's a DVC design choice :)
Thanks again for your help.

[pmrowla]

Right, normally the output paths are validated in commands like dvc add or dvc stage add and DVC will error out if the output is outside the repo. We probably need to add similar checks in dvclive

cc @daavoo