Excessive memory consumption when syncing a long way up to the `canonical head`

[mjfh]

Since PR #3191 the Nimbus EL has an annoying memory problem in the FC module as the syncer does not update base anymore while importing blocks. This happens at least when the syncer has to catch up a long way.

Previously, there was a kludge related to the syncer which used the forkChoice() function for base update.

Now base can only be updated if the CL triggers a forkChoiceUpdated which has no effect if the update is out of scope for the FC module, which in turn happens when syncing for an old or pristine database state.

In fact, this leads to a similar situation to when mainnet was unable to finalise transactions globally.

For the attached screenshot, I ran the syncer overnight (with turned off CL) and had the following memory usage in the morning

• 78.9GiB virtual (from metrics screen)
• 41.4GiB physical (from metrics screen)
• 22GiB extra swap space freed after stopping the process

As it seems, a big machine can handle the situation to an extend but the execution throughput decreases.

[KolbyML]

@mjfh is #3202 the PR you meant to link?

[mjfh]

Ops, was the wrong one -- lol
Thanks for noticing

[mjfh]

was somehow related to issue 3202 :)

[jangko]

fixed by #3204

[jangko]

Looks like the the problem not cured thoroughly. Need more investigation.

[advaita-saha]

Looks like the the problem not cured thoroughly. Need more investigation.

has the memory usage improved than before or exactly same as before ?

[jangko]

When syncing with hoodi using an empty database, initially everything looks ok, the base can move forward.
(apply #3237).

But when the gap is wider, the base stop moving. I'm not sure why CL suddenly request sync from head < 10K, then jump to > 200K+.

The problem is syncer downloading forward from known FC base(even though the segment request is reverse), but FC expecting the syncer to download backward from head.

Of course the FC expectation not satisfied by the syncer because the finalized hash(pendingFCU) not resolved into latestFinalizedNumber.

IIRC from discord discussion we agree on the syncer have two phase:

• Download headers from head to known base, and put it into cache. Probably also start another new session if CL request new target.
• Then download the block or body forward and import into FC.

That is what assume how the syncer works. But looks like not like that.

[mjfh]

When syncing with hoodi using an empty database, initially everything looks ok, the base can move forward. (apply #3237).

I observed the same in general although there was an outlier on hoodi when the CL was not fully in sync.

[mjfh]

[..]
IIRC from discord discussion we agree on the syncer have two phase:

* Download headers from head to known base, and put it into cache. Probably also start another new session if CL request new target.

* Then download the block or body forward and import into FC.

That is what assume how the syncer works. But looks like not like that.

That is exactly how it works apart from the fact that the CL cannot start a new syncer session while the current one is running.

[jangko]

Sync session 1
base=5324 head=5535 target=8539
download headers 8539..5536
resolved fin = 8468
download bodies 5536..8539

Sync session 2
base=8320 head=8539 target=9227
download headers 9227..8540
resolved fin = 9141
download bodies 8540..9227

Sync session 3
base=#8988 head=#9227 target=#9531
download headers 9531..9228
resolved fin = 9437
download bodies 9228..9531

Sync session 4
base=#9292 head=#9531 target=#259894
download headers 259894..9532
resolved fin = 9894    # <------------- ???????? 
download bodies 9532..... way past resolved fin, base is not moving anymore during this session lifetime.

EL=nimbus
CL=nimbus

Both FC and syncer expect CL give finalized hash near the target, not near the head.

The above sync sessions happen when I sync with hoodi. The question is, why CL send a finalized hash far from target? Considering this fact, the syncer cannot just ignore the finalized block if CL behave like this.