Gossip Block Processed delay increased in v0.34.0

[twoeths]

Describe the bug

Right now with v0.34.0 in hetzner-test1

Expected behavior

This is from v0.33.0

[twoeths]

one issue is BackfillSync query BlockArchive db a lot which takes 18% of cpu time

Backfill Sync also uses 3% of cpu time for blockToHeader() util

0207_hetzner_test1_backfill_sync.cpuprofile.zip

[twoeths]

the other issue is array.shift() in JobItem.push(), which takes 17% cpu time in another profile

[0209_hetzner_test1_processRpc.cpuprofile.zip]
(https://github.com/ChainSafe/lodestar/files/8030365/0209_hetzner_test1_processRpc.cpuprofile.zip)

@dapplion could we use a linked list in JobItemQueue?

[twoeths]

part of the issue is there are a lot of late gossip blocks recently in prater, even when I try with v0.33.0

mainnet gossip block processed delay is more stable in general, however our metric shows that it doubles the value I saw in the past, this is due to our node performance

[twoeths]

as with latest master, the metric is good so closing the issue

[contabo-17 - 30 validators]