exp: Standalone experiment running in `--temp` folder should be included in `dvc exp show` table

[aschuh-hf]

When executing an experiment using dvc exp run --temp, the running experiment is not included in the output of dvc exp list nor dvc exp show. Using --queue, on the other hand, shows the experiment as running. It would be great if also in case of dvc exp run --temp the experiment would be included in the output of dvc exp show. Similarly, it should be possible to follow metrics and other plots produced by DVCLive (e.g., using VS Code extension).

[karajan1001]

Hi @aschuh-hf , in which version of DVC and vscode-extention are you? I'm in the latest one and can get a proper exp table on VS Code extension. You can get the DVC version via dvc doctor

[aschuh-hf]

@karajan1001 Are you using dvc exp run --queue? In that case it does work. But not when using dvc exp run --temp.

EDIT: Nevermind, I've seen you use --temp in the screenshot just now. I may need to try the latest version (it hadn't been available in the DVC yum repo until yesterday it seems).

$ dvc doctor
DVC version: 2.31.0 (rpm)
---------------------------------
Platform: Python 3.8.3 on Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.14
Subprojects:

Supports:
        azure (adlfs = None, knack = 0.10.0, azure-identity = 1.11.0),
        gdrive (pydrive2 = 1.14.0),
        gs (gcsfs = None),
        hdfs (fsspec = None, pyarrow = 9.0.0),
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        oss (ossfs = 2021.8.0),
        s3 (s3fs = None, boto3 = 1.24.59),
        ssh (sshfs = 2022.6.0),
        webdav (webdav4 = 0.9.7),
        webdavs (webdav4 = 0.9.7),
        webhdfs (fsspec = None)
Cache types: hardlink, symlink
Cache directory: xfs on /dev/md124
Caches: local, s3
Remotes: s3, s3
Workspace directory: xfs on /dev/md124
Repo: dvc (subdir), git

[aschuh-hf]

@karajan1001 Which version of DVC were you using? Would 2.34.2 be sufficient? Having some troubles installing 2.35.

[karajan1001]

Would 2.34.2 be sufficient

I think 2.34.3 includes most of the recently merged exp related PRs.

[aschuh-hf]

I now have 2.35.1 installed (cf. treeverse/dvc-s3-repo#89 (comment)) and can try again with latest VS Code extension.

[aschuh-hf]

With the latest DVC version and VS Code Extension, I can now see the experiment executed with dvc exp run --temp in both the dvc exp show output and the VS Code Experiments tab. It took a few minutes after starting the experiment and a first checkpoint was made before it showed up. The VS Code Experiments table shows an entry for the checkpoints, but not a row with spinning wheel to indicate the running experiment (past the last checkpoint). Live plots of the running experiment are not showing.

The VS Code Experiments table seems to take quite some time to update (my Git repo is a larger mono-repo), but once I see it was executing dvc exp show -n 3 --json in the DVC Output window, the new checkpoints of the standalone experiment running in the --temp folder appear in the list / table.

I'm closing this issue and may open a new one for the DVC Live plots if I continue to observe these not showing.

UPDATE: The plots are also updating. It's just a bit slow for my repo, that's all.

[daavoo]

@aschuh-hf would be helpful if you could share a profile (i.e. https://github.com/iterative/dvc/wiki/Debugging,-Profiling-and-Benchmarking-DVC#generating-viztracer-data) of dvc exp show -n 3 --json / dvc plots diff to figure out what's taking so long

[aschuh-hf]

How should I share the ~150MB trace file?

Looking at the output, it is unfortunately not including a breakdown of all the steps in experiments.show() up to _collect_branches, i.e., where the majority of the time is actually spent.

[aschuh-hf]

This screenshot shows all the code that doesn't have a more detailed breakdown, but which consumes most of the time.

[aschuh-hf]

OK, I've increased the depth of the trace.

[daavoo]

OK, I've increased the depth of the trace.

cc @karajan1001 any ideas?