Low speed of fetching block report and questions

[Index0011]

The Online env sometime fetches blocks extremely slow, it sometimes takes 10mins to fetch 2 blocks which are unable to fetch the latest block. Are there some known conditions that will lead to this? Do we have some methods to locate the problem? And we’d love to learn the best practice to improve our project if we have any. 

[jamesbayly]

If we added some kind of verbose logging that records benchmark times taken for each mapping function, would that help?

[h4x3rotab]

Yeah, that would be helpful. I think any "best practice" to help benchmark the mapper would be helpful as well. For example, maybe a recommended way to add timing tracing inside the mapping functions, or some performance tips for database access.

Btw, one thing we suspect the most was that we are using blockHandler() to trigger some db maintenance, but we don't actually use any block data in the handler. So the burden could be due to the unnecessary fetch of the block data (which is big on Khala / Phala). Do you think this could be investigated or improved?

Thanks for advice.

[jamesbayly]

There’s a new modulo filter that allows you to filter the block handler to every n blocks. We’re still doing optimisations on it but this should help a lot. Another approach that will work better today is to use an event handler on a regular occasional event, like EndEraOn 4/08/2022, at 5:13 AM, h4x3rotab ***@***.***> wrote: Yeah, that would be helpful. I think any "best practice" to help benchmark the mapper would be helpful as well. For example, maybe a recommended way to add timing tracing inside the mapping functions, or some performance tips for database access. Btw, one thing we suspect the most was that we are using blockHandler() to trigger some db maintenance, but we don't actually use any block data in the handler. So the burden could be due to the unnecessary fetch of the block data (which is big on Khala / Phala). Do you think this could be investigated or improved? Thanks for advice. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[h4x3rotab]

It would be nice to have a BlockHeaderHandler which doesn't fetch the block content (extrinsics), because the extrinsics may be very big and we don't actually need that. Alternatively, it can be some special handler only triggered once at a specific block number.

And I'd also like to point out that so far we just suspect the BlockHandler is the cause, but have no proper way to confirm that. So we'd love to learn if there's any best practice we can follow to profile the logic in our mapping functions.

[jiqiang90]

We made some performance improvement after v2.0.0 above. I will close this for now, happy to hear your feedback and reopen if necessary .