unexpected error at the end of the queued tasks running. #8615
Description
@karajan1001 I am still seeing an intermittent error produced by the CLI when running from the queue. This is what is causing the table to disappear in the extension. The errors look like this:
ERROR: unexpected error - b'574ca27a9bb5b541a99462436352022ac0992fdb'
Full verbose output of an error:
❯ dvc exp show -n 3 --json -v
2022-11-23 14:16:43,757 DEBUG: found refs for revs: {'829b01e5f2eac85bc95509fa1808dff08feabe6d': 'exp-b13b9', '72fbe2591b79c7c0d31b7026f6b127a97ddd1048': 'exp-c5083', '16a18b1afd9a75c66f4a53d542297ca97042b617': 'acc_v_loss', 'b0cbabb156a7bd0a79bae147c077cde5ad2a4293': 'fix-dvc-live', 'f2a325d64a3dcd475046111fbe911bde93aaab62': 'main', 'aaa47a9ce84153b4b93076615907e126ba95447a': 'plots'}
2022-11-23 14:16:43,796 DEBUG: Found active task f0d72998c4224a9fee5863e6fc011b54711085e6
2022-11-23 14:16:45,456 ERROR: unexpected error - b'e9bd4cbd4af2fb16d3a3f060e8050c050294a0ed'
------------------------------------------------------------
Traceback (most recent call last):
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dvc/cli/__init__.py", line 185, in main
ret = cmd.do_run()
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dvc/cli/command.py", line 22, in do_run
return self.run()
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dvc/commands/experiments/show.py", line 472, in run
all_experiments = self.repo.experiments.show(
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dvc/repo/experiments/__init__.py", line 483, in show
return show(self.repo, *args, **kwargs)
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dvc/repo/experiments/show.py", line 424, in show
running: Dict[str, Dict] = repo.experiments.get_running_exps(
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dvc/repo/experiments/__init__.py", line 462, in get_running_exps
result.update(queue.get_running_exps(fetch_refs))
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dvc/repo/experiments/queue/celery.py", line 435, in get_running_exps
fetch_running_exp_from_temp_dir(
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dvc/repo/experiments/queue/utils.py", line 48, in fetch_running_exp_from_temp_dir
for ref in executor.fetch_exps(
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dvc/repo/experiments/executor/base.py", line 367, in fetch_exps
dest_scm.fetch_refspecs(
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 289, in _backend_func
result = func(*args, **kwargs)
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 658, in fetch_refspecs
fetch_result = client.fetch(
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dulwich/client.py", line 1503, in fetch
refs = r.fetch(
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dulwich/repo.py", line 459, in fetch
target.object_store.add_pack_data(count, pack_data, progress)
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dulwich/object_store.py", line 182, in add_pack_data
write_pack_data(
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dulwich/pack.py", line 1803, in write_pack_data
for chunk in chunk_generator:
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dulwich/pack.py", line 1754, in _pack_data_chunks
for i, (type_num, object_id, delta_base, raw) in enumerate(records):
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dulwich/pack.py", line 1675, in <genexpr>
(
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dulwich/object_store.py", line 1114, in __iter__
yield self.store[sha], path
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dulwich/object_store.py", line 145, in __getitem__
type_num, uncomp = self.get_raw(sha)
File "/Users/mattseddon/projects/vscode-dvc/demo/.env/lib/python3.9/site-packages/dulwich/object_store.py", line 597, in get_raw
raise KeyError(hexsha)
KeyError: b'e9bd4cbd4af2fb16d3a3f060e8050c050294a0ed'
------------------------------------------------------------
2022-11-23 14:16:45,587 DEBUG: Removing '/Users/mattseddon/projects/vscode-dvc/.4AgqJFHRnpvNr66RTfeaUJ.tmp'
2022-11-23 14:16:45,587 DEBUG: Removing '/Users/mattseddon/projects/vscode-dvc/.4AgqJFHRnpvNr66RTfeaUJ.tmp'
2022-11-23 14:16:45,587 DEBUG: Removing '/Users/mattseddon/projects/vscode-dvc/.4AgqJFHRnpvNr66RTfeaUJ.tmp'
2022-11-23 14:16:45,587 DEBUG: Removing '/Users/mattseddon/projects/vscode-dvc/demo/.dvc/cache/.WqYMLqMneyGTcRH7ik7vQb.tmp'
2022-11-23 14:16:45,588 DEBUG: Version info for developers:
DVC version: 2.34.3
---------------------------------
Platform: Python 3.9.13 on macOS-13.0.1-arm64-arm-64bit
Subprojects:
dvc_data = 0.28.0
dvc_objects = 0.14.0
dvc_render = 0.0.14
dvc_task = 0.1.6
dvclive = 1.0
scmrepo = 0.1.3
Supports:
http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
s3 (s3fs = 2022.10.0, boto3 = 1.24.59)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-11-23 14:16:45,590 DEBUG: Analytics is enabled.
2022-11-23 14:16:45,653 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/sb/fqcw44jd19nfrhl_9lz_81d80000gn/T/tmphwziop3t']'
2022-11-23 14:16:45,654 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/sb/fqcw44jd19nfrhl_9lz_81d80000gn/T/tmphwziop3t']'
Originally posted by @mattseddon in treeverse/vscode-dvc#2756 (comment)