fast-sync uses too much memory when trying to sync a large number of blocks

[albrow]

I've observed an issue that comes up when starting Mesh for the first time after a while. One of our Mesh nodes was constantly restarting. The last message we see before restarting is "Some blocks have elapsed since last boot...". Here's a screenshot of the logs:

I used docker stats to see that memory usage was steadily climbing until the container is eventually forcibly killed by Docker.

I believe this is happening because the "fast-sync" feature in Mesh keeps all event logs in memory while trying to catch up to the latest block. We can solve this issue by either (1) processing the logs in batches instead of trying to keep them all in memory or (2) placing a cap on the maximum number of blocks that Mesh will attempt to fast-sync. If too many blocks have passed, we would be better off starting from scratch and revalidating all existing orders.

The workaround for now is to manually delete the db/ folder, causing Mesh to start from scratch instead of trying to use the fast-sync feature.