[Bug] change implemented in v36 or v37 is causing slower subgraph indexing

[mindstyle85]

Bug report

Hi,

i am putting this here although i have been talking about this in the private DM group and in the software channel.
There was a change implemented between v36 or v37 that us causing significantly slower indexing compared to v35.

We have upgraded from v35 to v37 on the 8th which is exactly when issues began and falling back with a few heavier subgraphs, that were not problematic at all before this. The setup we have stays the same and the amount of subgraphs syncing/catching up has also not changed. I have even reassigned a few to see if it improves it for the particular one below but there has not been any significant change at all.

Observing across the board that indexing is slower in general, but the real problem is the heavy subgraphs that now can fall behind, like this screenshot below. This particular subgraph did not fall behind for months really (not sure why grafana doesnt show it), except at times where we were doing maintenance on the server.

Relevant log output

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

• Tick this box if this bug is caused by a regression found in the latest release.
• Tick this box if this bug is specific to the hosted service.
• I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux

[mindstyle85]

EDIT: i forgot to mention above, we upgraded to v37 on the 8th, skipped v36 and index the same set of subgraphs with only one additional after the upgrade

EDIT2: we also upgraded to v38 but no significant change to v38, same subgraphs are still problematic to catch up and are rather slowly falling behind

[mindstyle85]

here are some data sources for hardware usage, which suggest the hardware utilization is the same or even lower, while indexing on some subgraphs has become more difficult and cant keep on chain head (which didnt happen before v37)

note again that we skipped v36 and that at the moment this is the most i can get without going through a downgrade process which i am considering as well

[mindstyle85]

so we have downgraded to v36.1 as well and it looks like this one is the same, so whatever change in graph-node has made it slower, it was here (or 36.0)

[valamidev]

I can verify just made some measures, some stats might be help:

Version v0.35.1 -> v0.38.0 (same database, same machine, same datafeed (all RPC calls cached))

• small events processes 0.95p 2ms -> 0.95p 4ms
• medium events 0.95p 35ms -> 0.095p 45ms

There is some slowdown which start add 2ms here an there which can sum up.

[lutter]

Just for cut&pasteability, the subgraph above is QmZ5uwhnwsJXAQGYEF8qKPQ85iVhYAcVZcZAPfrF7ZNb9z

[mindstyle85]

this got resolved, we have been running a debug version which was obviously slower..

by the looks of it, there is no significant speed change on v38 compared to v35.1