zkevm-node crashing when serving traffic

[sjoshi10]

System information

zkEVM Node version: v0.0.3-RC36
OS & Version: Ubuntu 22.04.1 LTS
Network: Mainnet

Steps to reproduce the behaviour

We have zkevm-node running as rpc. It keeps crashing every 5-10 mins with following errors.

Logs

Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: fatal error: concurrent map writes
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: fatal error: concurrent map writes
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: goroutine 1023 [running]:
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*Storage).createFilter(0xc00012e188, {0x162fa98, 0x5}, {0x0?, 0x0}, 0xc00022dce0)
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]:         /tmp/zkevmnode-0.0.3-RC36/jsonrpc/storage.go:58 +0x24f
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*Storage).NewBlockFilter(0xc0009da480?, 0x5?)
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]:         /tmp/zkevmnode-0.0.3-RC36/jsonrpc/storage.go:43 +0x2c
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*EthEndpoints).newBlockFilter(0x0?, 0xc0009d83c0?)
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]:         /tmp/zkevmnode-0.0.3-RC36/jsonrpc/endpoints_eth.go:710 +0x28
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*EthEndpoints).Subscribe(0x5?, 0x0?, {0xc0003cfb30?, 0x4?}, 0x4?)
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]:         /tmp/zkevmnode-0.0.3-RC36/jsonrpc/endpoints_eth.go:898 +0x152
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: reflect.Value.call({0xc00049ccb0?, 0xc000012588?, 0x1392f80?}, {0x162e428, 0x4}, {0xc0002aa8a0, 0x4, 0x54c9e5?})
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]:         /usr/local/go/src/reflect/value.go:586 +0xb07
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: reflect.Value.Call({0xc00049ccb0?, 0xc000012588?, 0x10?}, {0xc0002aa8a0?, 0xc0000116e0?, 0xd?})
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]:         /usr/local/go/src/reflect/value.go:370 +0xbc
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*Handler).Handle(0xc000ac7df8?, {{{0xc0003cfa58, 0x3}, {0x13c9f60, 0xc0003cfa70}, {0x>
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]:         /tmp/zkevmnode-0.0.3-RC36/jsonrpc/handler.go:142 +0xc4b
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*Handler).HandleWs(0xc000ac7fb8?, {0xc00065bc00, 0x4c, 0x200}, 0xc00022dce0)
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]:         /tmp/zkevmnode-0.0.3-RC36/jsonrpc/handler.go:170 +0x27d
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*Server).handleWs.func3()
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]:         /tmp/zkevmnode-0.0.3-RC36/jsonrpc/server.go:375 +0x46
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: created by github.com/0xPolygonHermez/zkevm-node/jsonrpc.(*Server).handleWs
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]:         /tmp/zkevmnode-0.0.3-RC36/jsonrpc/server.go:374 +0x2d3
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: goroutine 1 [chan receive, 3 minutes]:
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: main.waitSignal({0x0, 0x0, 0x162ed04?})
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]:         /tmp/zkevmnode-0.0.3-RC36/cmd/run.go:291 +0x85
Apr 03 14:06:58 zkevm-node-vm zkevm-node[2014170]: main.start(0xc0000e5500)

[tclemos]

@sjoshi10 thanks for sharing it, looks like we have an issue with the way we handle the filters.

Do you have more details about the calls you are sending to the eth_createFilter endpoint?
If you are able to share the request bodies, it would be very helpful.

I'll try to reproduce the problem.