exp show: improve performance and stability of collection for vs code

[dberenbaum]

Per issues like #8762, exp show needs to collect faster and be more stable for vs code to work smoothly.

TODO

• cache collected exp data (exp show: cache collected experiments by git revision #9069)
• gc celery messages (Running exp queue status is very slow  #8676)
• investigate ref name performance (exp show: cache git refs per show() call #9077)

[karajan1001]

Current severe problems

• exp: Unable to resume checkpoint after using dvc exp run --queue #8622

hadn't look into it for now

• exp: Queuing experiment causes "Unable to acquire lock" error although no lock file exists at first #8621

Currently I had use a granular lock in experiments, during the running the exp will only lock its own experiments reference, but because we had a shared reference EXEC_CHECKPOINT. need to make each experiments use its own EXEC_CHECKPOINT

• queue: Queue status output displays incorrect number of workers #8614

Need a closer look, but I found another severe problem. Sometimes queue worker will not automatically shut down if something goes wrong inside it.

• exp apply: try to apply from other commits #8481

Hadn't look into it.

• exp queue: live metrics #8478

In VSCode extension I can see the live metrics, like in https://user-images.githubusercontent.com/6745454/212892100-ae2d032d-e23b-4b62-9207-28752a3de4c8.mp4

• exp show is slow: performance issue in dulwich #8477

Can solve fetch, but push to a non-bare Git repo is not supported for pygit2, maybe we can use fetch instead of push.

• queue: concurrent workers fail #8763

Related to #8477 push

• queue: checkpoint experiments hanging #8762

Related to #8477 fetch, a new had been created.

• Running exp queue status is very slow  #8676

Need to GC heartbeat messages, for now for every single task we create one heartbeat message per minute.

One potential problem. I still got into too many open files. Error even if I used git gc to clean the Git repository.