Synchrnoization breaks with WARN Ignoring already known beacon payload & L1 Info transaction differs

[astudnev]

System information

Network: mainnet
op-node version: v0.3.1
op-geth version: v0.3.0
OS & Version: Linux

Expected behaviour

After node is synced, it continues to follow the head

Actual behaviour

Sometimes it switches to 3-6 hours old block and re-sync again, which takes up to one hour

Steps to reproduce the behaviour

we run the node using snap sync method

This happens after some time ( 12-24 hours ) that the node issues the WARN Ignoring already known beacon payload and then according to the log below catches up in 6-3 hours ago

Backtrace

INFO [04-09|06:06:00.332] Imported new potential chain segment     number=20,889,037 hash=c2b538..9262ec blocks=1     txs=73      mgas=5.059   elapsed=101.019ms    mgasps=50.080  snapdiffs=3.34MiB    triedirty=1.66GiB
INFO [04-09|06:06:00.334] Chain head was updated                   number=20,889,037 hash=c2b538..9262ec root=d009d4..6215bd elapsed=1.108347ms
INFO [04-09|06:06:01.715] Imported new potential chain segment     number=20,889,038 hash=ba6322..01dec8 blocks=1     txs=65      mgas=4.247   elapsed=102.753ms    mgasps=41.334  snapdiffs=3.35MiB    triedirty=1.66GiB
INFO [04-09|06:06:01.721] Chain head was updated                   number=20,889,038 hash=ba6322..01dec8 root=89368f..2cd375 elapsed=2.989037ms
INFO [04-09|06:06:02.809] Imported new potential chain segment     number=20,889,039 hash=7b538b..7960e1 blocks=1     txs=58      mgas=3.481   elapsed=88.896ms     mgasps=39.156  snapdiffs=3.36MiB    triedirty=1.66GiB
INFO [04-09|06:06:02.811] Chain head was updated                   number=20,889,039 hash=7b538b..7960e1 root=e2ada6..cbe771 elapsed=1.060547ms
INFO [04-09|06:06:03.750] Imported new potential chain segment     number=20,889,040 hash=9e936e..d474cb blocks=1     txs=70      mgas=4.241   elapsed=94.359ms     mgasps=44.944  snapdiffs=3.37MiB    triedirty=1.66GiB
INFO [04-09|06:06:03.752] Chain head was updated                   number=20,889,040 hash=9e936e..d474cb root=bea9b1..b45b18 elapsed=1.12093ms
INFO [04-09|06:06:04.348] Imported new potential chain segment     number=20,889,041 hash=f73b7b..9f9f09 blocks=1     txs=68      mgas=4.274   elapsed=76.387ms     mgasps=55.948  snapdiffs=3.38MiB    triedirty=1.66GiB
INFO [04-09|06:06:04.350] Chain head was updated                   number=20,889,041 hash=f73b7b..9f9f09 root=4dde92..ae2b55 elapsed=1.264511ms
WARN [04-09|06:06:05.109] Ignoring already known beacon payload    number=20,888,677 hash=b424b9..1fcbae age=6m5s
INFO [04-09|06:06:13.181] Imported new potential chain segment     number=20,875,071 hash=233cc8..14bf6e blocks=1     txs=57      mgas=4.486   elapsed=493.257ms    mgasps=9.094   age=3h52m59s  snapdiffs=3.38MiB    triedirty=1.66GiB
INFO [04-09|06:06:13.460] Imported new potential chain segment     number=20,875,072 hash=626094..d21ec4 blocks=1     txs=75      mgas=5.670   elapsed=279.291ms    mgasps=20.302  age=3h52m58s  snapdiffs=3.38MiB    triedirty=1.66GiB
INFO [04-09|06:06:13.720] Imported new potential chain segment     number=20,875,073 hash=9579ba..8f8fbc blocks=1     txs=82      mgas=5.603   elapsed=260.161ms    mgasps=21.538  age=3h52m57s  snapdiffs=3.38MiB    triedirty=1.66GiB
INFO [04-09|06:06:13.950] Imported new potential chain segment     number=20,875,074 hash=2754fe..983773 blocks=1     txs=68      mgas=5.520   elapsed=230.028ms    mgasps=23.995  age=3h52m56s  snapdiffs=3.38MiB    triedirty=1.66GiB
INFO [04-09|06:06:14.155] Imported new potential chain segment     number=20,875,075 hash=b60dc1..38857d blocks=1     txs=69      mgas=4.340   elapsed=204.294ms    mgasps=21.244  age=3h52m56s  snapdiffs=3.38MiB    triedirty=1.66GiB
INFO [04-09|06:06:14.427] Imported new potential chain segment     number=20,875,076 hash=a14c2e..a1464c blocks=1     txs=84      mgas=6.317   elapsed=272.303ms    mgasps=23.199  age=3h52m55s  snapdiffs=3.38MiB    triedirty=1.66GiB
INFO [04-09|06:06:14.599] Imported new potential chain segment     number=20,875,077 hash=e4b6b2..82dff6 blocks=1     txs=70      mgas=4.793   elapsed=172.196ms    mgasps=27.832  age=3h52m54s  snapdiffs=3.38MiB    triedirty=1.66GiB
INFO [04-09|06:06:14.768] Imported new potential chain segment     number=20,875,078 hash=5cfaaa..af7314 blocks=1     txs=77      mgas=5.299   elapsed=168.884ms    mgasps=31.375  age=3h52m53s  snapdiffs=3.38MiB    triedirty=1.66GiB
INFO [04-09|06:06:15.014] Imported new potential chain segment     number=20,875,079 hash=07578f..b037ff blocks=1     txs=87      mgas=6.093   elapsed=245.987ms    mgasps=24.771  age=3h52m53s  snapdiffs=3.38MiB    triedirty=1.66GiB

at the same time op-node has one error ( L1 Info transaction differs ):

t=2024-04-09T06:06:05+0000 lvl=info msg="generated attributes in payload queue"  txs=38   timestamp=1,712,642,398
t=2024-04-09T06:06:05+0000 lvl=info msg="Sync progress"                          reason="reconciled with L1"                   l2_finalized=0xe1b3de5f4b940be045f81a823ce06f7448d2a93feaa745ae73eda472e1728775:20888634 l2_safe=0x22616ca237b8dcf0a31b7a79a9ea161a2e418950b32f04ae3e0e758ff7d9b6e1:20888675 l2_unsafe=0xf73b7b7cc87146476af7574af7c2c4c415d9eec6caa29abdbc1fd7eda69f9f09:20889041 l2_engineSyncTarget=0xf73b7b7cc87146476af7574af7c2c4c415d9eec6caa29abdbc1fd7eda69f9f09:20889041 l2_time=1,712,642,764 l1_derived=0x6f6584a166ac323136963f82f4d9a0fefe4983ae62ec7a114c047e32522f81c7:37701107
t=2024-04-09T06:06:05+0000 lvl=info msg="Found next batch"                       batch_timestamp=1,712,642,399 parent_hash=0x22616ca237b8dcf0a31b7a79a9ea161a2e418950b32f04ae3e0e758ff7d9b6e1 batch_epoch=0xa07bbd6a35d9805f99549943c4da3feac095dac9f21ea37ed5f0023f12fb3518:37701082 txs=54
t=2024-04-09T06:06:05+0000 lvl=info msg="generated attributes in payload queue"  txs=55   timestamp=1,712,642,399
t=2024-04-09T06:06:05+0000 lvl=info msg="Sync progress"                          reason="reconciled with L1"                   l2_finalized=0xe1b3de5f4b940be045f81a823ce06f7448d2a93feaa745ae73eda472e1728775:20888634 l2_safe=0xf9d2a9bfaa140b1241286bd2350a0bfaa7d351bf467ab2910918341c25bb752f:20888676 l2_unsafe=0xf73b7b7cc87146476af7574af7c2c4c415d9eec6caa29abdbc1fd7eda69f9f09:20889041 l2_engineSyncTarget=0xf73b7b7cc87146476af7574af7c2c4c415d9eec6caa29abdbc1fd7eda69f9f09:20889041 l2_time=1,712,642,764 l1_derived=0x6f6584a166ac323136963f82f4d9a0fefe4983ae62ec7a114c047e32522f81c7:37701107
t=2024-04-09T06:06:05+0000 lvl=info msg="Found next batch"                       batch_timestamp=1,712,642,400 parent_hash=0xf9d2a9bfaa140b1241286bd2350a0bfaa7d351bf467ab2910918341c25bb752f batch_epoch=0xa07bbd6a35d9805f99549943c4da3feac095dac9f21ea37ed5f0023f12fb3518:37701082 txs=46
t=2024-04-09T06:06:05+0000 lvl=info msg="generated attributes in payload queue"  txs=47   timestamp=1,712,642,400
t=2024-04-09T06:06:05+0000 lvl=eror msg="L1 Info transaction differs"            number=20,888,677 time=1,712,642,400 safe_l1_number=37,701,082 safe_l1_hash=0xa07bbd6a35d9805f99549943c4da3feac095dac9f21ea37ed5f0023f12fb3518 safe_l1_time=1,712,642,351 safe_seq_num=1 safe_l1_basefee=1,000,000,000 safe_batcher_add=0xa07bbd6a35d9805f99549943c4da3feac095dac9f21ea37ed5f0023f12fb3518 safe_gpo_scalar=0x00000000000000000000000000000000000000000000000000000000000a6fe0 safe_gpo_overhead=0x00000000000000000000000000000000000000000000000000000000000000bc unsafe_l1_number=37,701,082 unsafe_l1_hash=0xa07bbd6a35d9805f99549943c4da3feac095dac9f21ea37ed5f0023f12fb3518 unsafe_l1_time=1,712,642,351 unsafe_seq_num=1 unsafe_l1_basefee=3,000,000,000 unsafe_batcher_add=0xa07bbd6a35d9805f99549943c4da3feac095dac9f21ea37ed5f0023f12fb3518 unsafe_gpo_scalar=0x00000000000000000000000000000000000000000000000000000000000a6fe0 unsafe_gpo_overhead=0x00000000000000000000000000000000000000000000000000000000000000bc
t=2024-04-09T06:06:05+0000 lvl=warn msg="L2 reorg: existing unsafe block does not match derived attributes from L1" err="transaction 0 does not match. expected: 0x7ef90159a0b30bd5228b67a627e07d02439547269f704b0fb6efec4071af1203dd17e4e0a494deaddeaddeaddeaddeaddeaddeaddeaddead00019442000000000000000000000000000000000000158080830f424080b90104015d8eb900000000000000000000000000000000000000000000000000000000023f45da000000000000000000000000000000000000000000000000000000006614d92f00000000000000000000000000000000000000000000000000000000b2d05e00a07bbd6a35d9805f99549943c4da3feac095dac9f21ea37ed5f0023f12fb35180000000000000000000000000000000000000000000000000000000000000001000000000000000000000000ef8783382ef80ec23b66c43575a6103deca909c300000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0. got: 0x7ef90159a0b30bd5228b67a627e07d02439547269f704b0fb6efec4071af1203dd17e4e0a494deaddeaddeaddeaddeaddeaddeaddeaddead00019442000000000000000000000000000000000000158080830f424080b90104015d8eb900000000000000000000000000000000000000000000000000000000023f45da000000000000000000000000000000000000000000000000000000006614d92f000000000000000000000000000000000000000000000000000000003b9aca00a07bbd6a35d9805f99549943c4da3feac095dac9f21ea37ed5f0023f12fb35180000000000000000000000000000000000000000000000000000000000000001000000000000000000000000ef8783382ef80ec23b66c43575a6103deca909c300000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0" unsafe=0xf73b7b7cc87146476af7574af7c2c4c415d9eec6caa29abdbc1fd7eda69f9f09:20889041 safe=0xf9d2a9bfaa140b1241286bd2350a0bfaa7d351bf467ab2910918341c25bb752f:20888676
t=2024-04-09T06:06:05+0000 lvl=warn msg="engine is building block that reorgs previous unsafe head" onto=0xf9d2a9bfaa140b1241286bd2350a0bfaa7d351bf467ab2910918341c25bb752f:20888676 unsafe=0xf73b7b7cc87146476af7574af7c2c4c415d9eec6caa29abdbc1fd7eda69f9f09:20889041
t=2024-04-09T06:06:05+0000 lvl=info msg="Received signed execution payload from p2p" id=0x3b45ae15f1acae14a0937e986fa3208d2004f042af11406eeb9942081e339886:20889042 peer=16Uiu2HAm7Naptw4NgJWvLWHu3k6UYUXoKCbCZY6eK7KMCpW4PJ1H
t=2024-04-09T06:06:06+0000 lvl=info msg="Received signed execution payload from p2p" id=0xbe16f19d3efbbee380f7fb27550bb30ba0a54ea6dd02233d9cdf413d59618b85:20889043 peer=16Uiu2HAm7Naptw4NgJWvLWHu3k6UYUXoKCbCZY6eK7KMCpW4PJ1H
t=2024-04-09T06:06:07+0000 lvl=info msg="Received signed execution payload from p2p" id=0x4a6db86d890cb624cd51a37681d5e117f1fc9a0d3ba2bd5d192f45849ec0a5dc:20889044 peer=16Uiu2HAm7Naptw4NgJWvLWHu3k6UYUXoKCbCZY6eK7KMCpW4PJ1H
t=2024-04-09T06:06:08+0000 lvl=info msg="Received signed execution payload from p2p" id=0x33dc0d7ed4a55711f0a7f25867a5852e9547895d3a5f9b31b2cdec44de14ba44:20889045 peer=16Uiu2HAm65CuXyuykwMmQvParwJvJUmG9ehociAUsbuAYwDnMdCo
t=2024-04-09T06:06:09+0000 lvl=info msg="Received signed execution payload from p2p" id=0xcea151c7fd89aa9918993d6d6723599bc538ecea710001f0af918625d99bf4d3:20889046 peer=16Uiu2HAm7Naptw4NgJWvLWHu3k6UYUXoKCbCZY6eK7KMCpW4PJ1H

[astudnev]

note that the node is not re-starting, not issuing any errors.
Also we checked that it synced to mainnet correctly by matching block hashes ( they are visible in log as well )

[OmarMBNB]

Can you please send the op-node startup parameters?