Running `exp queue status` is very slow

[behrica]

Running exp queue status is very slow.

I noticed that after submitting 200 run jobs the queue, the dvc exp queue status became very slow.
In the order of 5 minutes for giving a result.

dvc doctor:

DVC version: 2.36.0 (pip)
---------------------------------
Platform: Python 3.10.8 on Linux-6.0.11-arch1-1-x86_64-with-glibc2.36
Subprojects:
	dvc_data = 0.28.3
	dvc_objects = 0.14.0
	dvc_render = 0.0.14
	dvc_task = 0.1.6
	dvclive = 1.1.0
	scmrepo = 0.1.4
Supports:
	azure (adlfs = 2022.10.0, knack = 0.10.0, azure-identity = 1.11.0),
	http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb1
Caches: local
Remotes: azure, local
Workspace directory: ext4 on /dev/sdb1
Repo: dvc, git

[behrica]

I can imgine that this is hard to look at to you.
so maybe a bit more of information.

It seems to me that "search tasks" is very slow. If I run the dvc queue status with -v I see a rather slowy progressing sequence of

2022-12-20 18:16:57,134 DEBUG: Found done task 71046ebe10172885891d4bf4ee2c0322b2f4ab90
2022-12-20 18:16:58,738 DEBUG: Found done task 0d83d52ee66434f6ddadf39cc324fc8cb33253ef
2022-12-20 18:17:00,387 DEBUG: Found done task 95345375b06afeeca8171a5cc07e7547c6b533cc
2022-12-20 18:17:03,179 DEBUG: Found done task 71046ebe10172885891d4bf4ee2c0322b2f4ab90

One every second. Even after emptying the queue, I still have those.

Any idea, what I could do to speed up my experiment database ?

[behrica]

I think my problem is still related to "crashing tasks" resulting into a "bad state" of the task queue.
In this situation other dvc commands become very slow as well, like dvx exp show
The machine was rebooted several times, so there is nothing left running , but I still see "one running" tasks in the queue. I can neither stop nor kill it.

Is there a way to "wipe" certain folders / files on disk and remove all remains from queued tasks, so I can use my dvc repo again ?

[daavoo]

Is there a way to "wipe" certain folders / files on disk and remove all remains from queued tasks, so I can use my dvc repo again ?

@behrica You could force a "wipe" by removing .dvc/tmp/exps (https://dvc.org/doc/user-guide/project-structure/internal-files#internal-directories-and-files) if you are certain that there is no important information there

[dberenbaum]

@behrica Is it still an issue for you?

[behrica]

I have not tried it for a while.
Have you fixed something ?

It happens every time , when I queue a few thousand runs and need to kill processes or similar.

[behrica]

So it goes a bit into the question of how stable DVC is for queuing thousands of tasks and let it run for days.
I tried this 3 or 4 times and it always gave issues. I used version 2.36

[behrica]

Apart from stable, I could not recover from the crash as described before.