Rococo can't sync from scratch: node triggers memory leak

[bLd75]

When syncing Rococo from scratch, at certain chain height the node starts to use resources abnormally until it gets OOM.

I've been testing on several servers (with 32Gb RAM), result is always the same:

• from block ~1929500, messages like this one start to appear (may not be relevant)
  Allocating slots failed for lease period 134, with: Module(ModuleError { index: 31, error: [2, 0, 0, 0], message: Some("CannotUpgrade") })
• until block ~3018000, no problem node sync normally
• from block ~3018000, memory consumption starts increasing and disk IO spiking, other processes are becoming irresponsive but node still syncs
• 2 hours 30mn later (around block ~3638800), node get OOM and restarts
• from this point, the node will get OOM every 3 minutes and restart

[ggwpez]

Last time I tried syncing Kusama it also needed 64 GB or something like this. This is definitely not good.

[bLd75]

I've set a snapshot of today for whom may need it
https://partage.bldnodes.org/share/rococo-snapshot-20230825.tar.gz

[bkchr]

@koute can you please take a look?

[koute]

I'll look into this.

[koute]

@bLd75 Just for reference, can you paste full exact command-line invocation you're using and the exact commit on which you see this problem?

[bLd75]

Hi @koute I made some new tests and I just realized polkadot-v1.0.0 client manages to sync Rococo correctly.
I was using embedded parachain client on version polkadot-v0.9.43, this is the problematic version.
I'm closing the issue as latest version fixes it, but this would be to communicate with Rococo parachains (most of them don't use latest Polkadot client version) cc @ggwpez .

[ggwpez]

I will test it on a GCP to confirm that its fixed.

[ggwpez]

Yep syncing with 32GB works again in v1.0.0.