Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shutdown stuck with Interrupt signal #10192

Closed
MqllR opened this issue May 3, 2024 · 4 comments
Closed

Shutdown stuck with Interrupt signal #10192

MqllR opened this issue May 3, 2024 · 4 comments
Assignees
Milestone

Comments

@MqllR
Copy link

MqllR commented May 3, 2024

System information

Erigon version: v2.59.2

OS & Version: Linux

Erigon Command (with flags/config):

erigon --chain=sepolia --datadir=/data/erigon --db.size.limit=7697000000000 --db.pagesize=4KB --snap.stop --nat=stun --metrics=true --rpc.gascap=500000000 --maxpeers=50 --port=30303 --http --http.addr=0.0.0.0 --http.corsdomain=* --http.port=8545 --http.api=admin,eth,trace,net,engine,web3 --http.vhosts=* --authrpc.jwtsecret=/app/jwt-secret.hex --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --ws --pprof --ws.port=8546 --graphql --txpool.nolocals --txpool.accountslots=32 --txpool.globalslots=8192 --txpool.accountqueue=128 --txpool.globalqueue=2048 --txpool.lifetime=3h0m0s --metrics.addr=0.0.0.0 --metrics.port=9386

Consensus Layer: Teku 24.2.0

Chain/Network: Ethereum Sepolia

Expected behaviour

When sending an Interrupt signal, Erigon is supposed to gracefully shutdown.

Actual behaviour

Erigon gets stuck waiting the close the RPC service.

Logs:

[INFO] [05-02|18:22:51.645] [4/12 Execution] Completed on            block=5823152
[INFO] [05-02|18:22:59.237] RPC Daemon notified of new headers       from=5823140 to=5823152 amount=12 hash=0x6696fb67e014f78ecfda9fbfcc6b177f9bcebf75851d790b072ab1fb9e0041d6 header sending=72.431µs log sendin
[INFO] [05-02|18:22:59.237] head updated                             hash=0x6696fb67e014f78ecfda9fbfcc6b177f9bcebf75851d790b072ab1fb9e0041d6 number=5823152
[INFO] [05-02|18:23:02.381] [NewPayload] Handling new payload        height=5823153 hash=0x850a826f7d676aa841b5fad115126b34c95de601b7e3f9730e76cc3e79f24261
[INFO] [05-02|18:23:02.721] [updateForkchoice] Fork choice update: flushing in-memory state (built by previous newPayload)
[INFO] [05-02|18:23:03.404] RPC Daemon notified of new headers       from=5823152 to=5823153 amount=1 hash=0x850a826f7d676aa841b5fad115126b34c95de601b7e3f9730e76cc3e79f24261 header sending=25.876µs log sending
[INFO] [05-02|18:23:03.404] head updated                             hash=0x850a826f7d676aa841b5fad115126b34c95de601b7e3f9730e76cc3e79f24261 number=5823153
[INFO] [05-02|18:23:08.231] Got interrupt, shutting down...          sig=interrupt
[INFO] [05-02|18:23:08.231] Got interrupt, shutting down...          sig=interrupt
[INFO] [05-02|18:23:08.231] Got interrupt, shutting down...          sig=interrupt
[INFO] [05-02|18:23:08.231] Got interrupt, shutting down...          sig=interrupt
[INFO] [05-02|18:23:08.234] Got interrupt, shutting down...          sig=interrupt
[INFO] [05-02|18:23:08.231] Got interrupt, shutting down...          sig=interrupt
[INFO] [05-02|18:23:08.231] Got interrupt, shutting down...          sig=interrupt
[INFO] [05-02|18:23:08.231] Got interrupt, shutting down...          sig=interrupt
[INFO] [05-02|18:23:08.231] Got interrupt, shutting down...          sig=interrupt
[INFO] [05-02|18:23:08.234] Got interrupt, shutting down...
[INFO] [05-02|18:23:08.236] Exiting Engine...
[INFO] [05-02|18:23:08.237] Exiting...
[INFO] [05-02|18:23:08.237] RPC server shutting down
[INFO] [05-02|18:23:08.239] RPC server shutting down
[INFO] [05-02|18:23:08.242] Engine HTTP endpoint close               url=:8551
[INFO] [05-02|18:23:08.242] HTTP endpoint closed                     url=:8545
[INFO] [05-02|18:23:08.242] HTTP endpoint closed                     url=:8546
[INFO] [05-02|18:23:08.242] RPC server shutting down

List of running goroutine:

goroutine profile: total 70
20 @ 0x4603ce 0x42b7ed 0x42b3f2 0x13029c5 0x12fdc88 0x12fda1a 0x12f47af 0x493d61
#	0x13029c4	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).ensureBond+0xc4		github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:778
#	0x12fdc87	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).findnode+0x47		github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:370
#	0x12fda19	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).newLookup.func1+0x99	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:356
#	0x12f47ae	github.com/ledgerwatch/erigon/p2p/discover.(*lookup).query+0xae			github.com/ledgerwatch/erigon/p2p/discover/lookup.go:163

11 @ 0x4603ce 0x42b7ed 0x42b3f2 0x12f41c5 0x12f3fcd 0x12f3e85 0x12fd44f 0x12fca4f 0x131bf8d 0x131be8e 0x131bda5 0x493d61
#	0x12f41c4	github.com/ledgerwatch/erigon/p2p/discover.(*lookup).shutdown+0x2a4	github.com/ledgerwatch/erigon/p2p/discover/lookup.go:96
#	0x12f3fcc	github.com/ledgerwatch/erigon/p2p/discover.(*lookup).advance+0xac	github.com/ledgerwatch/erigon/p2p/discover/lookup.go:88
#	0x12f3e84	github.com/ledgerwatch/erigon/p2p/discover.(*lookup).run+0x24		github.com/ledgerwatch/erigon/p2p/discover/lookup.go:64
#	0x12fd44e	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).LookupPubkey+0xee	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:324
#	0x12fca4e	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).Resolve+0x12e	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:250
#	0x131bf8c	github.com/ledgerwatch/erigon/p2p.(*dialTask).resolve+0xac		github.com/ledgerwatch/erigon/p2p/dial.go:532
#	0x131be8d	github.com/ledgerwatch/erigon/p2p.(*dialTask).run+0x8d			github.com/ledgerwatch/erigon/p2p/dial.go:505
#	0x131bda4	github.com/ledgerwatch/erigon/p2p.(*dialScheduler).startDial.func1+0x44	github.com/ledgerwatch/erigon/p2p/dial.go:472

5 @ 0x4603ce 0x42b7ed 0x42b3f2 0x12fdf45 0x12fda1a 0x12f47af 0x493d61
#	0x12fdf44	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).findnode+0x304		github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:406
#	0x12fda19	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).newLookup.func1+0x99	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:356
#	0x12f47ae	github.com/ledgerwatch/erigon/p2p/discover.(*lookup).query+0xae			github.com/ledgerwatch/erigon/p2p/discover/lookup.go:163

4 @ 0x4603ce 0x42b7ed 0x42b3f2 0x12fd42a 0x12fca4f 0x131bf8d 0x131be8e 0x131bda5 0x493d61
#	0x12fd429	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).LookupPubkey+0xc9	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:322
#	0x12fca4e	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).Resolve+0x12e	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:250
#	0x131bf8c	github.com/ledgerwatch/erigon/p2p.(*dialTask).resolve+0xac		github.com/ledgerwatch/erigon/p2p/dial.go:532
#	0x131be8d	github.com/ledgerwatch/erigon/p2p.(*dialTask).run+0x8d			github.com/ledgerwatch/erigon/p2p/dial.go:505
#	0x131bda4	github.com/ledgerwatch/erigon/p2p.(*dialScheduler).startDial.func1+0x44	github.com/ledgerwatch/erigon/p2p/dial.go:472

2 @ 0x4603ce 0x42aa85 0x42a6d7 0x1329f1e 0x493d61
#	0x1329f1d	github.com/ledgerwatch/erigon/p2p.(*Server).runPeer+0x21d	github.com/ledgerwatch/erigon/p2p/server.go:1164

1 @ 0x430c89 0x4902c9 0x8fa7d3 0x493d61
#	0x4902c8	os/signal.signal_recv+0x28	runtime/sigqueue.go:152
#	0x8fa7d2	os/signal.loop+0x12		os/signal/signal_unix.go:23

1 @ 0x455111 0x48dafd 0xb83891 0xb836c5 0xb80166 0xbec9c8 0xbed4c5 0x72a4e9 0x72be02 0x72d1ce 0x7290b4 0x493d61
#	0x48dafc	runtime/pprof.runtime_goroutineProfileWithLabels+0x1c	runtime/mprof.go:844
#	0xb83890	runtime/pprof.writeRuntimeProfile+0xb0			runtime/pprof/pprof.go:734
#	0xb836c4	runtime/pprof.writeGoroutine+0x44			runtime/pprof/pprof.go:694
#	0xb80165	runtime/pprof.(*Profile).WriteTo+0x145			runtime/pprof/pprof.go:329
#	0xbec9c7	net/http/pprof.handler.ServeHTTP+0x4a7			net/http/pprof/pprof.go:267
#	0xbed4c4	net/http/pprof.Index+0xe4				net/http/pprof/pprof.go:384
#	0x72a4e8	net/http.HandlerFunc.ServeHTTP+0x28			net/http/server.go:2136
#	0x72be01	net/http.(*ServeMux).ServeHTTP+0x141			net/http/server.go:2514
#	0x72d1cd	net/http.serverHandler.ServeHTTP+0x8d			net/http/server.go:2938
#	0x7290b3	net/http.(*conn).serve+0x5f3				net/http/server.go:2009

1 @ 0x4603ce 0x42aa85 0x42a6d7 0x12ffafc 0x12ff417 0x493d61
#	0x12ffafb	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop.func3+0x1fb	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:612
#	0x12ff416	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop+0x3d6		github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:613

1 @ 0x4603ce 0x42b7ed 0x42b3f2 0x12f41c5 0x12f3fcd 0x12f3e85 0x12fd5ca 0x12f7cf9 0x493d61
#	0x12f41c4	github.com/ledgerwatch/erigon/p2p/discover.(*lookup).shutdown+0x2a4	github.com/ledgerwatch/erigon/p2p/discover/lookup.go:96
#	0x12f3fcc	github.com/ledgerwatch/erigon/p2p/discover.(*lookup).advance+0xac	github.com/ledgerwatch/erigon/p2p/discover/lookup.go:88
#	0x12f3e84	github.com/ledgerwatch/erigon/p2p/discover.(*lookup).run+0x24		github.com/ledgerwatch/erigon/p2p/discover/lookup.go:64
#	0x12fd5c9	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).lookupRandom+0x29	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:334
#	0x12f7cf8	github.com/ledgerwatch/erigon/p2p/discover.(*Table).doRefresh+0x98	github.com/ledgerwatch/erigon/p2p/discover/table.go:370

1 @ 0x4603ce 0x42b7ed 0x42b3f2 0x12fefcc 0x1303c9e 0x13025dc 0x130168f 0x493d61
#	0x12fefcb	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).handleReply+0x1ab	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:487
#	0x1303c9d	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).verifyPong+0x15d	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:907
#	0x13025db	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).handlePacket+0x33b	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:756
#	0x130168e	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).readLoop+0x1ae	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:713

1 @ 0x4603ce 0x42b7ed 0x42b3f2 0x1ebe2ab 0x493d61
#	0x1ebe2aa	github.com/ledgerwatch/erigon/turbo/debug.ListenSignals+0x28a	github.com/ledgerwatch/erigon/turbo/debug/signal.go:32

1 @ 0x4603ce 0x42b7ed 0x42b3f2 0x2117c8e 0x2117c7c 0x21303a5 0x212387f 0x7d6bbd 0x7d34fb 0x212fec5 0x212fe9d 0x45ff5b 0x493d61
#	0x2117c8d	github.com/ledgerwatch/erigon/node.(*Node).Wait+0x6d			github.com/ledgerwatch/erigon/node/node.go:271
#	0x2117c7b	github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).Serve+0x5b	github.com/ledgerwatch/erigon/turbo/node/node.go:34
#	0x21303a4	main.runErigon+0x404							github.com/ledgerwatch/erigon/cmd/erigon/main.go:75
#	0x212387e	github.com/ledgerwatch/erigon/turbo/app.MakeApp.func1+0x19e		github.com/ledgerwatch/erigon/turbo/app/make_app.go:52
#	0x7d6bbc	github.com/urfave/cli/v2.(*Command).Run+0x9dc				github.com/urfave/cli/v2@v2.27.1/command.go:279
#	0x7d34fa	github.com/urfave/cli/v2.(*App).RunContext+0x5da			github.com/urfave/cli/v2@v2.27.1/app.go:337
#	0x212fec4	github.com/urfave/cli/v2.(*App).Run+0x84				github.com/urfave/cli/v2@v2.27.1/app.go:311
#	0x212fe9c	main.main+0x5c								github.com/ledgerwatch/erigon/cmd/erigon/main.go:34
#	0x45ff5a	runtime.main+0x2ba							runtime/proc.go:267

1 @ 0x4603ce 0x42b7ed 0x42b412 0x129463c 0x493d61
#	0x129463b	github.com/ledgerwatch/erigon/consensus/ethash.(*meterArbiter).tick+0x7b	github.com/ledgerwatch/erigon/consensus/ethash/meter.go:143

1 @ 0x4603ce 0x42b7ed 0x42b412 0x1eb859f 0x493d61
#	0x1eb859e	gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun+0x3e	gopkg.in/natefinch/lumberjack.v2@v2.2.1/lumberjack.go:379

1 @ 0x4603ce 0x42b7ed 0x42b412 0xb64a25 0x1e8e2eb 0x1e8cfea 0xb647b2 0x493d61
#	0xb64a24	github.com/ledgerwatch/erigon-lib/direct.(*SubscribeLogsStreamS).Recv+0x24			github.com/ledgerwatch/erigon-lib@v1.0.0/direct/eth_backend_client.go:158
#	0x1e8e2ea	github.com/ledgerwatch/erigon/ethdb/privateapi.(*LogsFilterAggregator).subscribeLogs+0xaa	github.com/ledgerwatch/erigon/ethdb/privateapi/logsfilter.go:133
#	0x1e8cfe9	github.com/ledgerwatch/erigon/ethdb/privateapi.(*EthBackendServer).SubscribeLogs+0x29		github.com/ledgerwatch/erigon/ethdb/privateapi/ethbackend.go:256
#	0xb647b1	github.com/ledgerwatch/erigon-lib/direct.(*EthBackendClientDirect).SubscribeLogs.func1+0x91	github.com/ledgerwatch/erigon-lib@v1.0.0/direct/eth_backend_client.go:125

1 @ 0x4603ce 0x42b7ed 0x42b412 0xb64c45 0x1e93a3f 0x1c4e626 0x493d61
#	0xb64c44	github.com/ledgerwatch/erigon-lib/direct.(*SubscribeLogsStreamC).Recv+0x24			github.com/ledgerwatch/erigon-lib@v1.0.0/direct/eth_backend_client.go:185
#	0x1e93a3e	github.com/ledgerwatch/erigon/cmd/rpcdaemon/rpcservices.(*RemoteBackend).SubscribeLogs+0x1be	github.com/ledgerwatch/erigon/cmd/rpcdaemon/rpcservices/eth_backend.go:229
#	0x1c4e625	github.com/ledgerwatch/erigon/turbo/rpchelper.New.func2+0xe5					github.com/ledgerwatch/erigon/turbo/rpchelper/filters.go:101

1 @ 0x4603ce 0x458ad7 0x48df25 0x507387 0x50867a 0x508668 0x602145 0x614f05 0x722f37 0x493d61
#	0x48df24	internal/poll.runtime_pollWait+0x84		runtime/netpoll.go:343
#	0x507386	internal/poll.(*pollDesc).wait+0x26		internal/poll/fd_poll_runtime.go:84
#	0x508679	internal/poll.(*pollDesc).waitRead+0x279	internal/poll/fd_poll_runtime.go:89
#	0x508667	internal/poll.(*FD).Read+0x267			internal/poll/fd_unix.go:164
#	0x602144	net.(*netFD).Read+0x24				net/fd_posix.go:55
#	0x614f04	net.(*conn).Read+0x44				net/net.go:179
#	0x722f36	net/http.(*connReader).backgroundRead+0x36	net/http/server.go:683

1 @ 0x4603ce 0x458ad7 0x48df25 0x507387 0x50867a 0x508668 0x602145 0x614f05 0x72336b 0x6b1be3 0x6b1d13 0x72921c 0x493d61
#	0x48df24	internal/poll.runtime_pollWait+0x84		runtime/netpoll.go:343
#	0x507386	internal/poll.(*pollDesc).wait+0x26		internal/poll/fd_poll_runtime.go:84
#	0x508679	internal/poll.(*pollDesc).waitRead+0x279	internal/poll/fd_poll_runtime.go:89
#	0x508667	internal/poll.(*FD).Read+0x267			internal/poll/fd_unix.go:164
#	0x602144	net.(*netFD).Read+0x24				net/fd_posix.go:55
#	0x614f04	net.(*conn).Read+0x44				net/net.go:179
#	0x72336a	net/http.(*connReader).Read+0x14a		net/http/server.go:791
#	0x6b1be2	bufio.(*Reader).fill+0x102			bufio/bufio.go:113
#	0x6b1d12	bufio.(*Reader).Peek+0x52			bufio/bufio.go:151
#	0x72921b	net/http.(*conn).serve+0x75b			net/http/server.go:2044

1 @ 0x4603ce 0x458ad7 0x48df25 0x507387 0x50c86c 0x50c85a 0x604169 0x61f1de 0x61e070 0x1327f28 0x132568e 0x493d61
#	0x48df24	internal/poll.runtime_pollWait+0x84					runtime/netpoll.go:343
#	0x507386	internal/poll.(*pollDesc).wait+0x26					internal/poll/fd_poll_runtime.go:84
#	0x50c86b	internal/poll.(*pollDesc).waitRead+0x2ab				internal/poll/fd_poll_runtime.go:89
#	0x50c859	internal/poll.(*FD).Accept+0x299					internal/poll/fd_unix.go:611
#	0x604168	net.(*netFD).accept+0x28						net/fd_unix.go:172
#	0x61f1dd	net.(*TCPListener).accept+0x1d						net/tcpsock_posix.go:152
#	0x61e06f	net.(*TCPListener).Accept+0x2f						net/tcpsock.go:315
#	0x1327f27	github.com/ledgerwatch/erigon/p2p.(*Server).listenLoop+0x727		github.com/ledgerwatch/erigon/p2p/server.go:950
#	0x132568d	github.com/ledgerwatch/erigon/p2p.(*Server).setupListening.func2+0x6d	github.com/ledgerwatch/erigon/p2p/server.go:762

1 @ 0x4603ce 0x458ad7 0x48df25 0x507387 0x50c86c 0x50c85a 0x604169 0x61f1de 0x61e070 0x72d624 0x72d291 0x1ebdb05 0x1ebdb06 0x493d61
#	0x48df24	internal/poll.runtime_pollWait+0x84				runtime/netpoll.go:343
#	0x507386	internal/poll.(*pollDesc).wait+0x26				internal/poll/fd_poll_runtime.go:84
#	0x50c86b	internal/poll.(*pollDesc).waitRead+0x2ab			internal/poll/fd_poll_runtime.go:89
#	0x50c859	internal/poll.(*FD).Accept+0x299				internal/poll/fd_unix.go:611
#	0x604168	net.(*netFD).accept+0x28					net/fd_unix.go:172
#	0x61f1dd	net.(*TCPListener).accept+0x1d					net/tcpsock_posix.go:152
#	0x61e06f	net.(*TCPListener).Accept+0x2f					net/tcpsock.go:315
#	0x72d623	net/http.(*Server).Serve+0x363					net/http/server.go:3056
#	0x72d290	net/http.(*Server).ListenAndServe+0x70				net/http/server.go:2985
#	0x1ebdb04	net/http.ListenAndServe+0x64					net/http/server.go:3239
#	0x1ebdb05	github.com/ledgerwatch/erigon/turbo/debug.StartPProf.func1+0x65	github.com/ledgerwatch/erigon/turbo/debug/flags.go:235

1 @ 0x4603ce 0x458ad7 0x48df25 0x507387 0x50c86c 0x50c85a 0x604169 0x61f1de 0x61e070 0x72d624 0x72d291 0x8d7eed 0x493d61
#	0x48df24	internal/poll.runtime_pollWait+0x84				runtime/netpoll.go:343
#	0x507386	internal/poll.(*pollDesc).wait+0x26				internal/poll/fd_poll_runtime.go:84
#	0x50c86b	internal/poll.(*pollDesc).waitRead+0x2ab			internal/poll/fd_poll_runtime.go:89
#	0x50c859	internal/poll.(*FD).Accept+0x299				internal/poll/fd_unix.go:611
#	0x604168	net.(*netFD).accept+0x28					net/fd_unix.go:172
#	0x61f1dd	net.(*TCPListener).accept+0x1d					net/tcpsock_posix.go:152
#	0x61e06f	net.(*TCPListener).Accept+0x2f					net/tcpsock.go:315
#	0x72d623	net/http.(*Server).Serve+0x363					net/http/server.go:3056
#	0x72d290	net/http.(*Server).ListenAndServe+0x70				net/http/server.go:2985
#	0x8d7eec	github.com/ledgerwatch/erigon-lib/metrics.Setup.func1+0x2c	github.com/ledgerwatch/erigon-lib@v1.0.0/metrics/setup.go:28

1 @ 0x4603ce 0x458ad7 0x48df25 0x507387 0x603a53 0x603a37 0x603a54 0x61a4f2 0x61a054 0x60eb51 0x61ed49 0x61ebe5 0x61ebe6 0x5f82f5 0x5f7c28 0x5f72e5 0x5f6d3e 0x1318d54 0x131c3bb 0x131be65 0x131bda5 0x493d61
#	0x48df24	internal/poll.runtime_pollWait+0x84					runtime/netpoll.go:343
#	0x507386	internal/poll.(*pollDesc).wait+0x26					internal/poll/fd_poll_runtime.go:84
#	0x603a52	internal/poll.(*pollDesc).waitWrite+0x732				internal/poll/fd_poll_runtime.go:93
#	0x603a36	internal/poll.(*FD).WaitWrite+0x716					internal/poll/fd_unix.go:683
#	0x603a53	net.(*netFD).connect+0x733						net/fd_unix.go:141
#	0x61a4f1	net.(*netFD).dial+0x3b1							net/sock_posix.go:151
#	0x61a053	net.socket+0x293							net/sock_posix.go:70
#	0x60eb50	net.internetSocket+0xf0							net/ipsock_posix.go:142
#	0x61ed48	net.(*sysDialer).doDialTCPProto+0xe8					net/tcpsock_posix.go:78
#	0x61ebe4	net.(*sysDialer).doDialTCP+0x64						net/tcpsock_posix.go:68
#	0x61ebe5	net.(*sysDialer).dialTCP+0x65						net/tcpsock_posix.go:64
#	0x5f82f4	net.(*sysDialer).dialSingle+0x254					net/dial.go:649
#	0x5f7c27	net.(*sysDialer).dialSerial+0x247					net/dial.go:614
#	0x5f72e4	net.(*sysDialer).dialParallel+0x3e4					net/dial.go:515
#	0x5f6d3d	net.(*Dialer).DialContext+0x6dd						net/dial.go:506
#	0x1318d53	github.com/ledgerwatch/erigon/p2p.tcpDialer.Dial+0x53			github.com/ledgerwatch/erigon/p2p/dial.go:68
#	0x131c3ba	github.com/ledgerwatch/erigon/p2p.(*dialTask).dial+0x5a			github.com/ledgerwatch/erigon/p2p/dial.go:551
#	0x131be64	github.com/ledgerwatch/erigon/p2p.(*dialTask).run+0x64			github.com/ledgerwatch/erigon/p2p/dial.go:501
#	0x131bda4	github.com/ledgerwatch/erigon/p2p.(*dialScheduler).startDial.func1+0x44	github.com/ledgerwatch/erigon/p2p/dial.go:472

1 @ 0x4603ce 0x470985 0x12e004d 0x131a973 0x493d61
#	0x12e004c	github.com/ledgerwatch/erigon/p2p/enode.(*FairMix).Next+0x14c		github.com/ledgerwatch/erigon/p2p/enode/iter.go:219
#	0x131a972	github.com/ledgerwatch/erigon/p2p.(*dialScheduler).readNodes+0x132	github.com/ledgerwatch/erigon/p2p/dial.go:335

1 @ 0x4603ce 0x470985 0x12f67ff 0x493d61
#	0x12f67fe	github.com/ledgerwatch/erigon/p2p/discover.(*Table).loop+0x35e	github.com/ledgerwatch/erigon/p2p/discover/table.go:270

1 @ 0x4603ce 0x470985 0x13199af 0x493d61
#	0x13199ae	github.com/ledgerwatch/erigon/p2p.(*dialScheduler).loop+0x2ee	github.com/ledgerwatch/erigon/p2p/dial.go:251

1 @ 0x4603ce 0x470985 0x603efa 0x493d61
#	0x603ef9	net.(*netFD).connect.func2+0x79	net/fd_unix.go:118

1 @ 0x4603ce 0x470985 0xbb87ab 0xbb8370 0x20bce08 0x20bcdf8 0x20bfe4f 0x20bf156 0x20bebc5 0x20bdaa5 0x20bd838 0x12e072c 0x493d61
#	0xbb87aa	golang.org/x/time/rate.(*Limiter).wait+0x3ea					golang.org/x/time@v0.5.0/rate/rate.go:285
#	0xbb836f	golang.org/x/time/rate.(*Limiter).WaitN+0x4f					golang.org/x/time@v0.5.0/rate/rate.go:248
#	0x20bce07	golang.org/x/time/rate.(*Limiter).Wait+0x67					golang.org/x/time@v0.5.0/rate/rate.go:233
#	0x20bcdf7	github.com/ledgerwatch/erigon/p2p/dnsdisc.(*Client).resolveEntry+0x57		github.com/ledgerwatch/erigon/p2p/dnsdisc/client.go:175
#	0x20bfe4e	github.com/ledgerwatch/erigon/p2p/dnsdisc.(*subtreeSync).resolveNext+0x4e	github.com/ledgerwatch/erigon/p2p/dnsdisc/sync.go:261
#	0x20bf155	github.com/ledgerwatch/erigon/p2p/dnsdisc.(*clientTree).syncNextRandomENR+0x75	github.com/ledgerwatch/erigon/p2p/dnsdisc/sync.go:136
#	0x20bebc4	github.com/ledgerwatch/erigon/p2p/dnsdisc.(*clientTree).syncRandom+0x244	github.com/ledgerwatch/erigon/p2p/dnsdisc/sync.go:97
#	0x20bdaa4	github.com/ledgerwatch/erigon/p2p/dnsdisc.(*randomIterator).nextNode+0x44	github.com/ledgerwatch/erigon/p2p/dnsdisc/client.go:284
#	0x20bd837	github.com/ledgerwatch/erigon/p2p/dnsdisc.(*randomIterator).Next+0x17		github.com/ledgerwatch/erigon/p2p/dnsdisc/client.go:263
#	0x12e072b	github.com/ledgerwatch/erigon/p2p/enode.(*FairMix).runSource+0xcb		github.com/ledgerwatch/erigon/p2p/enode/iter.go:282

1 @ 0x4603ce 0x4719b8 0x47198f 0x48fb65 0x4b2ec8 0x12fc8cc 0x4b191f 0x12fc85c 0x12fc82f 0x1326c4a 0x493d61
#	0x48fb64	sync.runtime_Semacquire+0x24						runtime/sema.go:62
#	0x4b2ec7	sync.(*WaitGroup).Wait+0x47						sync/waitgroup.go:116
#	0x12fc8cb	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).Close.func1+0x4b	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:226
#	0x4b191e	sync.(*Once).doSlow+0xbe						sync/once.go:74
#	0x12fc85b	sync.(*Once).Do+0x3b							sync/once.go:65
#	0x12fc82e	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).Close+0xe		github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:223
#	0x1326c49	github.com/ledgerwatch/erigon/p2p.(*Server).run+0x13a9			github.com/ledgerwatch/erigon/p2p/server.go:883

1 @ 0x4603ce 0x4719b8 0x47198f 0x48fb65 0x4b2ec8 0x1323019 0x2113ec5 0x2113eb0 0x1ec8612 0x1ec8185 0x493d61
#	0x48fb64	sync.runtime_Semacquire+0x24						runtime/sema.go:62
#	0x4b2ec7	sync.(*WaitGroup).Wait+0x47						sync/waitgroup.go:116
#	0x1323018	github.com/ledgerwatch/erigon/p2p.(*Server).Stop+0x118			github.com/ledgerwatch/erigon/p2p/server.go:453
#	0x2113ec4	github.com/ledgerwatch/erigon/p2p/sentry.(*GrpcServer).Close+0x224	github.com/ledgerwatch/erigon/p2p/sentry/sentry_grpc_server.go:1217
#	0x2113eaf	github.com/ledgerwatch/erigon/eth.(*Ethereum).Stop+0x20f		github.com/ledgerwatch/erigon/eth/backend.go:1421
#	0x1ec8611	github.com/ledgerwatch/erigon/node.(*Node).stopServices+0xb1		github.com/ledgerwatch/erigon/node/node.go:216
#	0x1ec8184	github.com/ledgerwatch/erigon/node.(*Node).Close+0x184			github.com/ledgerwatch/erigon/node/node.go:159

1 @ 0x4603ce 0x4719b8 0x47198f 0x48fc25 0x4b15bd 0x1300cf5 0x1300cae 0x1300806 0x493d61
#	0x48fc24	sync.runtime_SemacquireMutex+0x24					runtime/sema.go:77
#	0x4b15bc	sync.(*Mutex).lockSlow+0x15c						sync/mutex.go:171
#	0x1300cf4	sync.(*Mutex).Lock+0x74							sync/mutex.go:90
#	0x1300cad	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop.func1.1+0x2d	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:518
#	0x1300805	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop.func1+0x205	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:581

1 @ 0x4603ce 0x4719b8 0x47198f 0x48fc25 0x4b15bd 0x13455ec 0x13455b4 0x493d61
#	0x48fc24	sync.runtime_SemacquireMutex+0x24									runtime/sema.go:77
#	0x4b15bc	sync.(*Mutex).lockSlow+0x15c										sync/mutex.go:171
#	0x13455eb	sync.(*Mutex).Lock+0x26b										sync/mutex.go:90
#	0x13455b3	github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runBlockExecutionListener.func1+0x233	github.com/ledgerwatch/erigon-lib@v1.0.0/diagnostics/block_execution.go:28

1 @ 0x4603ce 0x4719b8 0x47198f 0x48fc25 0x4b15bd 0x1347dce 0x1347d9c 0x493d61
#	0x48fc24	sync.runtime_SemacquireMutex+0x24										runtime/sema.go:77
#	0x4b15bc	sync.(*Mutex).lockSlow+0x15c											sync/mutex.go:171
#	0x1347dcd	sync.(*Mutex).Lock+0x24d											sync/mutex.go:90
#	0x1347d9b	github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSegmentDownloadingListener.func1+0x21b	github.com/ledgerwatch/erigon-lib@v1.0.0/diagnostics/snapshots.go:69

1 @ 0x4603ce 0x4719b8 0x47198f 0x48fc25 0x4b15bd 0x1348f0f 0x1348ed0 0x493d61
#	0x48fc24	sync.runtime_SemacquireMutex+0x24										runtime/sema.go:77
#	0x4b15bc	sync.(*Mutex).lockSlow+0x15c											sync/mutex.go:171
#	0x1348f0e	sync.(*Mutex).Lock+0x22e											sync/mutex.go:90
#	0x1348ecf	github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSnapshotFilesListListener.func1+0x1ef	github.com/ledgerwatch/erigon-lib@v1.0.0/diagnostics/snapshots.go:179

Steps to reproduce the behaviour

I haven't been able to reproduce locally as it seems to require a fully synced chain (or after passing some steps of the ETL workflow).

But with a synced chain: kill -INT <erigon_pid>

Sending a Term signal works well.

@battlmonstr
Copy link
Contributor

battlmonstr commented Jun 4, 2024

Thanks for reporting this. We'll work on a fix.

Analysis

It is a deadlock in v4_udp.go
Thread A waits on mutex.Lock() in resetTimeout() called after reading listUpdate channel.
Thread B waits on listUpdate <- plist.PushBack(p) called after locking mutex.Lock().
Apparently listUpdate is full and thread B can't proceed writing while holding the mutex.
The listUpdate is not getting drained because the only reader is stuck on the mutex.

Thread A:

1 @ 0x4603ce 0x4719b8 0x47198f 0x48fc25 0x4b15bd 0x1300cf5 0x1300cae 0x1300806 0x493d61
#	0x48fc24	sync.runtime_SemacquireMutex+0x24					runtime/sema.go:77
#	0x4b15bc	sync.(*Mutex).lockSlow+0x15c						sync/mutex.go:171
#	0x1300cf4	sync.(*Mutex).Lock+0x74							sync/mutex.go:90
#	0x1300cad	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop.func1.1+0x2d	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:518
#	0x1300805	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop.func1+0x205	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:581

Thread B:

1 @ 0x4603ce 0x42aa85 0x42a6d7 0x12ffafc 0x12ff417 0x493d61
#	0x12ffafb	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop.func3+0x1fb	github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:612
#	0x12ff416	github.com/ledgerwatch/erigon/p2p/discover.(*UDPv4).loop+0x3d6		github.com/ledgerwatch/erigon/p2p/discover/v4_udp.go:613

@mh0lt
Copy link
Contributor

mh0lt commented Jun 5, 2024

@MqllR it would be great if you can test this and confirm it fixes your problem - in which case we can merge it into the next release candidate.

@mh0lt mh0lt added this to the 2.60.2-fixes milestone Jun 5, 2024
@MqllR
Copy link
Author

MqllR commented Jun 5, 2024

Sure ! I'll run some tests and will keep you posted.

@MqllR
Copy link
Author

MqllR commented Jun 5, 2024

I tried to kill Erigon multiple times with an interrupt signal and haven't been able to reproduce. I guess it's fixed ! Thanks for taking care of it @mh0lt !

[INFO] [06-05|15:51:58.351] Build info                               git_branch=fix-potential-p2p-shutdown-hangup git_tag=v2.60.1-1-g4db78c73b9-dirty git_commit=4db78c73b9f563114f0a53e863dc97f5797aa686
...
[INFO] [06-05|15:51:56.332] Got interrupt, shutting down...
[INFO] [06-05|15:51:56.332] Got interrupt, shutting down...          sig=interrupt
[INFO] [06-05|15:51:56.332] Exiting Engine...
[INFO] [06-05|15:51:56.332] RPC server shutting down
[INFO] [06-05|15:51:56.332] RPC server shutting down
[INFO] [06-05|15:51:56.332] Engine HTTP endpoint close               url=:8551
[INFO] [06-05|15:51:56.332] Exiting...
[INFO] [06-05|15:51:56.333] HTTP endpoint closed                     url=:8545
[INFO] [06-05|15:51:56.333] HTTP endpoint closed                     url=:8546
[INFO] [06-05|15:51:56.333] RPC server shutting down

@mh0lt mh0lt closed this as completed Jun 7, 2024
AskAlexSharov pushed a commit that referenced this issue Jun 10, 2024
This is a fix for: 

#10192

This fixes is a deadlock in v4_udp.go where 
* Thread A waits on mutex.Lock() in resetTimeout() called after reading
listUpdate channel.
* Thread B waits on listUpdate <- plist.PushBack(p) called after locking
mutex.Lock()
  
This fix decouples the list operations which need locking from the
channel operations which don't by storing the changes in local
variables. These updates are used for resetting a timeout - which is not
order dependent.
yperbasis pushed a commit that referenced this issue Jun 21, 2024
This is a fix for: 

#10192

This fixes is a deadlock in v4_udp.go where 
* Thread A waits on mutex.Lock() in resetTimeout() called after reading
listUpdate channel.
* Thread B waits on listUpdate <- plist.PushBack(p) called after locking
mutex.Lock()
  
This fix decouples the list operations which need locking from the
channel operations which don't by storing the changes in local
variables. These updates are used for resetting a timeout - which is not
order dependent.
blxdyx added a commit to node-real/bsc-erigon that referenced this issue Jun 27, 2024
* fix Consensus specification tests CI (erigontech#10391) (erigontech#10396)

Cherry-pick:
erigontech@bc5fa6f

Need this to get PR CI green for v2.60.1 patches, e.g. -
erigontech#10390

Co-authored-by: Andrew Ashikhmin <34320705+yperbasis@users.noreply.github.com>

* rpc/handler: do not append null to stream when json may be valid (erigontech#10390)

Cherry-pick:
erigontech@4d1c954
Relates to: erigontech#10376

* Fixed Bor Log appearing on Ethereum Mainnet (erigontech#10405) (erigontech#10420)

Cherry-pick:
erigontech@be889f6

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* fix gas price not right problem (erigontech#10456)

Cherry pick PR erigontech#10451 into the release branch

Co-authored-by: mars <marshalys@gmail.com>

* eth_estimateGas: default feeCap to base fee (erigontech#10499)

Copy PR erigontech#10495 into the release branch

* Add flag for bor waypoint types (erigontech#10501)

Cherry pick PR erigontech#10281 into the release branch

Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>

* try to fix 'method handler crashed' for debug_traceCall of erigontech#9090 (erigontech#10502)

Cherry pick PR erigontech#10401 into the release branch

Co-authored-by: mars <marshalys@gmail.com>

* diagnostics: cherry pick speedtest disable (erigontech#10509)

Cherry pick PR erigontech#10449 into the release branch

* Enable DNS p2p discovery on holesky (erigontech#10507)

Cherry pick PR erigontech#10460 into the release branch

Co-authored-by: Willian Mitsuda <wmitsuda@gmail.com>

* fix eth_call 'method handler crashed' error when tx has set maxFeePerBlobGas (erigontech#10506)

Cherry pick PR erigontech#10452 into the release branch

Co-authored-by: mars <marshalys@gmail.com>

* e2: remove overlapped files only after merge (erigontech#10487)

Otherwise: if start after `kill -9` in the middle of merge - may remove
small files of 1 type of file, but leave small files of another type of
files (which merge was not finished) - and leave node in un-mergable
state: erigontech#10485

---------

Co-authored-by: awskii <awskii@users.noreply.github.com>

* add flag checking for pruning waypoints (erigontech#10508)

Cherry pick PR erigontech#10468 into the release branch

Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>

* p2p/sentry: sentry doesn't start with ErrNoHead (erigontech#10454) (erigontech#10523)

cherry-pick erigontech#10494 to
release/2.60

* add lock to purgeMilestoneIDsList (erigontech#10524)

Cherry pick PR erigontech#10493 into the release branch

Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>

* polygon/heimdall: fix checkpoint json marshalling (erigontech#10530)

Fixes a recent regression causing unwinds due to checkpoints having zero
root hash:
```
[WARN] [05-18|23:58:54.662] [bor] Root hash mismatch while whitelisting checkpoint expected=ac1c57270479250af3ce8eee90075cd8b2ba1bac55353105e063d9a4c87c743e got=0000000000000000000000000000000000000000000000000000000000000000
[WARN] [05-18|23:58:54.662] [bor] Rewinding chain due to checkpoint root hash mismatch number=57125727
```

Note this has already been fixed on Erigon 3 branch but as part of a
non-related PR -
https://github.com/ledgerwatch/erigon/pull/10124/files#diff-47d4532f399f2d6a45e6f19944a45c80bac573b4d1b5cb51485d0254229d1b16

* Fix capacity for immediate appends (erigontech#10539)

Cherry pick PR erigontech#10528 into the release branch

Co-authored-by: Shoham Chakraborty <shhmchk@gmail.com>

* core/vm: set tracer-observable value of a delegatecall to match parent value (erigontech#10370)

requested by erigontech#9549

port of ethereum/go-ethereum#26632

* params: version 2.60.1 (erigontech#10555)

* blobGasPrice should be marshalled as hex (erigontech#10571)

Cherry pick PR erigontech#10551 into the release branch

* Caplin: Fixed reforwarding of Bls Execution changes (erigontech#10577)

Cherry pick PR erigontech#10546 into the release branch

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* Caplin: Proper "Normalization" of length of ForkVersions to 8 hex characters (erigontech#10578)

Cherry pick PR erigontech#10512 into the release branch

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* Caplin: Update BlobSidecars Beacon API endpoint to the latest specs (erigontech#10580)

Cherry pick PR erigontech#10576 into the release branch

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* bor blocks retire: infinity loop fix (erigontech#10596)

Problem: `+1` was added to maxBlockNum instead of minBlockNum
for: erigontech#10554

* txpool: EIP-3860 should only apply to create transactions (erigontech#10609)

This fixes Issue erigontech#10607

* qa-tests: update 2.60.x test workflows from main (erigontech#10627)

* Fix potential p2p shutdown hangup (erigontech#10626)

This is a fix for: 

erigontech#10192

This fixes is a deadlock in v4_udp.go where 
* Thread A waits on mutex.Lock() in resetTimeout() called after reading
listUpdate channel.
* Thread B waits on listUpdate <- plist.PushBack(p) called after locking
mutex.Lock()
  
This fix decouples the list operations which need locking from the
channel operations which don't by storing the changes in local
variables. These updates are used for resetting a timeout - which is not
order dependent.

* downloader: Number of DNS requests seem excessive (erigontech#5145) (erigontech#10739)

cherry-pick erigontech#10693 to release

* rpc: Fix incorrect txfeecap (erigontech#10643)

Cherry pick PR erigontech#10636 to
Erigon 2

* downloader: don't block erigon startup if devs deploy new hashes (of same files) (erigontech#10761)

* skip hidden files when list files with given extension  (erigontech#10654)

for erigontech#10644

* qa-tests: backport to release/2.60 improvements made to e3 github action workflows (erigontech#10778)

This PR backports improvements that we added to the E3 tests: recording
runner name and db version used for testing on MongoDB database.

* e2: more snaps (all networks) (erigontech#10794)

* e2: configurable hashers amount (erigontech#10785)

* Revert "e2: configurable hashers amount" (erigontech#10834)

* diagnostics: move E3 changes to E2 (erigontech#10806)

Merged all the work done from main branch to keep diagnostics up to
date.

* Downloader: fix staticpeers flag (erigontech#10798)

Cherry pick erigontech#10792

* Fix NewPayload Validation during header download (erigontech#10837)

Cherry pick PR erigontech#10093 into the release branch

Co-authored-by: Minhyuk Kim <kimminhyuk1004@gmail.com>

* e2: mainnet blob 9.3M (erigontech#10842)

* Fix gas fee calculation for debug calls (erigontech#10880)

Cherry pick PR erigontech#10825 into the release branch

Co-authored-by: Minhyuk Kim <kimminhyuk1004@gmail.com>

* Revert "eth_estimateGas: default feeCap to base fee (erigontech#10499)" (erigontech#10904)

This reverts PR erigontech#10499. See
erigontech#10495 (comment)
and PR erigontech#10901

* params: version 2.60.2 (erigontech#10905)

* upstream v2.60.2

* fix ci lint

---------

Co-authored-by: milen <94537774+taratorio@users.noreply.github.com>
Co-authored-by: Andrew Ashikhmin <34320705+yperbasis@users.noreply.github.com>
Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>
Co-authored-by: mars <marshalys@gmail.com>
Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
Co-authored-by: Dmytro <vovk.dimon@gmail.com>
Co-authored-by: Willian Mitsuda <wmitsuda@gmail.com>
Co-authored-by: awskii <awskii@users.noreply.github.com>
Co-authored-by: battlmonstr <battlmonstr@users.noreply.github.com>
Co-authored-by: Shoham Chakraborty <shhmchk@gmail.com>
Co-authored-by: Michelangelo Riccobene <michelangelo.riccobene@gmail.com>
Co-authored-by: Minhyuk Kim <kimminhyuk1004@gmail.com>
MakarovSg pushed a commit to chainstack/bsc-erigon that referenced this issue Jul 2, 2024
* fix Consensus specification tests CI (erigontech#10391) (erigontech#10396)

Cherry-pick:
erigontech@bc5fa6f

Need this to get PR CI green for v2.60.1 patches, e.g. -
erigontech#10390

Co-authored-by: Andrew Ashikhmin <34320705+yperbasis@users.noreply.github.com>

* rpc/handler: do not append null to stream when json may be valid (erigontech#10390)

Cherry-pick:
erigontech@4d1c954
Relates to: erigontech#10376

* Fixed Bor Log appearing on Ethereum Mainnet (erigontech#10405) (erigontech#10420)

Cherry-pick:
erigontech@be889f6

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* fix gas price not right problem (erigontech#10456)

Cherry pick PR erigontech#10451 into the release branch

Co-authored-by: mars <marshalys@gmail.com>

* eth_estimateGas: default feeCap to base fee (erigontech#10499)

Copy PR erigontech#10495 into the release branch

* Add flag for bor waypoint types (erigontech#10501)

Cherry pick PR erigontech#10281 into the release branch

Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>

* try to fix 'method handler crashed' for debug_traceCall of erigontech#9090 (erigontech#10502)

Cherry pick PR erigontech#10401 into the release branch

Co-authored-by: mars <marshalys@gmail.com>

* diagnostics: cherry pick speedtest disable (erigontech#10509)

Cherry pick PR erigontech#10449 into the release branch

* Enable DNS p2p discovery on holesky (erigontech#10507)

Cherry pick PR erigontech#10460 into the release branch

Co-authored-by: Willian Mitsuda <wmitsuda@gmail.com>

* fix eth_call 'method handler crashed' error when tx has set maxFeePerBlobGas (erigontech#10506)

Cherry pick PR erigontech#10452 into the release branch

Co-authored-by: mars <marshalys@gmail.com>

* e2: remove overlapped files only after merge (erigontech#10487)

Otherwise: if start after `kill -9` in the middle of merge - may remove
small files of 1 type of file, but leave small files of another type of
files (which merge was not finished) - and leave node in un-mergable
state: erigontech#10485

---------

Co-authored-by: awskii <awskii@users.noreply.github.com>

* add flag checking for pruning waypoints (erigontech#10508)

Cherry pick PR erigontech#10468 into the release branch

Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>

* p2p/sentry: sentry doesn't start with ErrNoHead (erigontech#10454) (erigontech#10523)

cherry-pick erigontech#10494 to
release/2.60

* add lock to purgeMilestoneIDsList (erigontech#10524)

Cherry pick PR erigontech#10493 into the release branch

Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>

* polygon/heimdall: fix checkpoint json marshalling (erigontech#10530)

Fixes a recent regression causing unwinds due to checkpoints having zero
root hash:
```
[WARN] [05-18|23:58:54.662] [bor] Root hash mismatch while whitelisting checkpoint expected=ac1c57270479250af3ce8eee90075cd8b2ba1bac55353105e063d9a4c87c743e got=0000000000000000000000000000000000000000000000000000000000000000
[WARN] [05-18|23:58:54.662] [bor] Rewinding chain due to checkpoint root hash mismatch number=57125727
```

Note this has already been fixed on Erigon 3 branch but as part of a
non-related PR -
https://github.com/ledgerwatch/erigon/pull/10124/files#diff-47d4532f399f2d6a45e6f19944a45c80bac573b4d1b5cb51485d0254229d1b16

* Fix capacity for immediate appends (erigontech#10539)

Cherry pick PR erigontech#10528 into the release branch

Co-authored-by: Shoham Chakraborty <shhmchk@gmail.com>

* core/vm: set tracer-observable value of a delegatecall to match parent value (erigontech#10370)

requested by erigontech#9549

port of ethereum/go-ethereum#26632

* params: version 2.60.1 (erigontech#10555)

* blobGasPrice should be marshalled as hex (erigontech#10571)

Cherry pick PR erigontech#10551 into the release branch

* Caplin: Fixed reforwarding of Bls Execution changes (erigontech#10577)

Cherry pick PR erigontech#10546 into the release branch

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* Caplin: Proper "Normalization" of length of ForkVersions to 8 hex characters (erigontech#10578)

Cherry pick PR erigontech#10512 into the release branch

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* Caplin: Update BlobSidecars Beacon API endpoint to the latest specs (erigontech#10580)

Cherry pick PR erigontech#10576 into the release branch

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* bor blocks retire: infinity loop fix (erigontech#10596)

Problem: `+1` was added to maxBlockNum instead of minBlockNum
for: erigontech#10554

* txpool: EIP-3860 should only apply to create transactions (erigontech#10609)

This fixes Issue erigontech#10607

* qa-tests: update 2.60.x test workflows from main (erigontech#10627)

* Fix potential p2p shutdown hangup (erigontech#10626)

This is a fix for: 

erigontech#10192

This fixes is a deadlock in v4_udp.go where 
* Thread A waits on mutex.Lock() in resetTimeout() called after reading
listUpdate channel.
* Thread B waits on listUpdate <- plist.PushBack(p) called after locking
mutex.Lock()
  
This fix decouples the list operations which need locking from the
channel operations which don't by storing the changes in local
variables. These updates are used for resetting a timeout - which is not
order dependent.

* downloader: Number of DNS requests seem excessive (erigontech#5145) (erigontech#10739)

cherry-pick erigontech#10693 to release

* rpc: Fix incorrect txfeecap (erigontech#10643)

Cherry pick PR erigontech#10636 to
Erigon 2

* downloader: don't block erigon startup if devs deploy new hashes (of same files) (erigontech#10761)

* skip hidden files when list files with given extension  (erigontech#10654)

for erigontech#10644

* qa-tests: backport to release/2.60 improvements made to e3 github action workflows (erigontech#10778)

This PR backports improvements that we added to the E3 tests: recording
runner name and db version used for testing on MongoDB database.

* e2: more snaps (all networks) (erigontech#10794)

* e2: configurable hashers amount (erigontech#10785)

* Revert "e2: configurable hashers amount" (erigontech#10834)

* diagnostics: move E3 changes to E2 (erigontech#10806)

Merged all the work done from main branch to keep diagnostics up to
date.

* Downloader: fix staticpeers flag (erigontech#10798)

Cherry pick erigontech#10792

* Fix NewPayload Validation during header download (erigontech#10837)

Cherry pick PR erigontech#10093 into the release branch

Co-authored-by: Minhyuk Kim <kimminhyuk1004@gmail.com>

* e2: mainnet blob 9.3M (erigontech#10842)

* Fix gas fee calculation for debug calls (erigontech#10880)

Cherry pick PR erigontech#10825 into the release branch

Co-authored-by: Minhyuk Kim <kimminhyuk1004@gmail.com>

* Revert "eth_estimateGas: default feeCap to base fee (erigontech#10499)" (erigontech#10904)

This reverts PR erigontech#10499. See
erigontech#10495 (comment)
and PR erigontech#10901

* params: version 2.60.2 (erigontech#10905)

* upstream v2.60.2

* fix ci lint

---------

Co-authored-by: milen <94537774+taratorio@users.noreply.github.com>
Co-authored-by: Andrew Ashikhmin <34320705+yperbasis@users.noreply.github.com>
Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>
Co-authored-by: mars <marshalys@gmail.com>
Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
Co-authored-by: Dmytro <vovk.dimon@gmail.com>
Co-authored-by: Willian Mitsuda <wmitsuda@gmail.com>
Co-authored-by: awskii <awskii@users.noreply.github.com>
Co-authored-by: battlmonstr <battlmonstr@users.noreply.github.com>
Co-authored-by: Shoham Chakraborty <shhmchk@gmail.com>
Co-authored-by: Michelangelo Riccobene <michelangelo.riccobene@gmail.com>
Co-authored-by: Minhyuk Kim <kimminhyuk1004@gmail.com>
MakarovSg pushed a commit to chainstack/bsc-erigon that referenced this issue Jul 19, 2024
* fix Consensus specification tests CI (erigontech#10391) (erigontech#10396)

Cherry-pick:
erigontech@bc5fa6f

Need this to get PR CI green for v2.60.1 patches, e.g. -
erigontech#10390

Co-authored-by: Andrew Ashikhmin <34320705+yperbasis@users.noreply.github.com>

* rpc/handler: do not append null to stream when json may be valid (erigontech#10390)

Cherry-pick:
erigontech@4d1c954
Relates to: erigontech#10376

* Fixed Bor Log appearing on Ethereum Mainnet (erigontech#10405) (erigontech#10420)

Cherry-pick:
erigontech@be889f6

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* fix gas price not right problem (erigontech#10456)

Cherry pick PR erigontech#10451 into the release branch

Co-authored-by: mars <marshalys@gmail.com>

* eth_estimateGas: default feeCap to base fee (erigontech#10499)

Copy PR erigontech#10495 into the release branch

* Add flag for bor waypoint types (erigontech#10501)

Cherry pick PR erigontech#10281 into the release branch

Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>

* try to fix 'method handler crashed' for debug_traceCall of erigontech#9090 (erigontech#10502)

Cherry pick PR erigontech#10401 into the release branch

Co-authored-by: mars <marshalys@gmail.com>

* diagnostics: cherry pick speedtest disable (erigontech#10509)

Cherry pick PR erigontech#10449 into the release branch

* Enable DNS p2p discovery on holesky (erigontech#10507)

Cherry pick PR erigontech#10460 into the release branch

Co-authored-by: Willian Mitsuda <wmitsuda@gmail.com>

* fix eth_call 'method handler crashed' error when tx has set maxFeePerBlobGas (erigontech#10506)

Cherry pick PR erigontech#10452 into the release branch

Co-authored-by: mars <marshalys@gmail.com>

* e2: remove overlapped files only after merge (erigontech#10487)

Otherwise: if start after `kill -9` in the middle of merge - may remove
small files of 1 type of file, but leave small files of another type of
files (which merge was not finished) - and leave node in un-mergable
state: erigontech#10485

---------

Co-authored-by: awskii <awskii@users.noreply.github.com>

* add flag checking for pruning waypoints (erigontech#10508)

Cherry pick PR erigontech#10468 into the release branch

Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>

* p2p/sentry: sentry doesn't start with ErrNoHead (erigontech#10454) (erigontech#10523)

cherry-pick erigontech#10494 to
release/2.60

* add lock to purgeMilestoneIDsList (erigontech#10524)

Cherry pick PR erigontech#10493 into the release branch

Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>

* polygon/heimdall: fix checkpoint json marshalling (erigontech#10530)

Fixes a recent regression causing unwinds due to checkpoints having zero
root hash:
```
[WARN] [05-18|23:58:54.662] [bor] Root hash mismatch while whitelisting checkpoint expected=ac1c57270479250af3ce8eee90075cd8b2ba1bac55353105e063d9a4c87c743e got=0000000000000000000000000000000000000000000000000000000000000000
[WARN] [05-18|23:58:54.662] [bor] Rewinding chain due to checkpoint root hash mismatch number=57125727
```

Note this has already been fixed on Erigon 3 branch but as part of a
non-related PR -
https://github.com/ledgerwatch/erigon/pull/10124/files#diff-47d4532f399f2d6a45e6f19944a45c80bac573b4d1b5cb51485d0254229d1b16

* Fix capacity for immediate appends (erigontech#10539)

Cherry pick PR erigontech#10528 into the release branch

Co-authored-by: Shoham Chakraborty <shhmchk@gmail.com>

* core/vm: set tracer-observable value of a delegatecall to match parent value (erigontech#10370)

requested by erigontech#9549

port of ethereum/go-ethereum#26632

* params: version 2.60.1 (erigontech#10555)

* blobGasPrice should be marshalled as hex (erigontech#10571)

Cherry pick PR erigontech#10551 into the release branch

* Caplin: Fixed reforwarding of Bls Execution changes (erigontech#10577)

Cherry pick PR erigontech#10546 into the release branch

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* Caplin: Proper "Normalization" of length of ForkVersions to 8 hex characters (erigontech#10578)

Cherry pick PR erigontech#10512 into the release branch

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* Caplin: Update BlobSidecars Beacon API endpoint to the latest specs (erigontech#10580)

Cherry pick PR erigontech#10576 into the release branch

Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>

* bor blocks retire: infinity loop fix (erigontech#10596)

Problem: `+1` was added to maxBlockNum instead of minBlockNum
for: erigontech#10554

* txpool: EIP-3860 should only apply to create transactions (erigontech#10609)

This fixes Issue erigontech#10607

* qa-tests: update 2.60.x test workflows from main (erigontech#10627)

* Fix potential p2p shutdown hangup (erigontech#10626)

This is a fix for: 

erigontech#10192

This fixes is a deadlock in v4_udp.go where 
* Thread A waits on mutex.Lock() in resetTimeout() called after reading
listUpdate channel.
* Thread B waits on listUpdate <- plist.PushBack(p) called after locking
mutex.Lock()
  
This fix decouples the list operations which need locking from the
channel operations which don't by storing the changes in local
variables. These updates are used for resetting a timeout - which is not
order dependent.

* downloader: Number of DNS requests seem excessive (erigontech#5145) (erigontech#10739)

cherry-pick erigontech#10693 to release

* rpc: Fix incorrect txfeecap (erigontech#10643)

Cherry pick PR erigontech#10636 to
Erigon 2

* downloader: don't block erigon startup if devs deploy new hashes (of same files) (erigontech#10761)

* skip hidden files when list files with given extension  (erigontech#10654)

for erigontech#10644

* qa-tests: backport to release/2.60 improvements made to e3 github action workflows (erigontech#10778)

This PR backports improvements that we added to the E3 tests: recording
runner name and db version used for testing on MongoDB database.

* e2: more snaps (all networks) (erigontech#10794)

* e2: configurable hashers amount (erigontech#10785)

* Revert "e2: configurable hashers amount" (erigontech#10834)

* diagnostics: move E3 changes to E2 (erigontech#10806)

Merged all the work done from main branch to keep diagnostics up to
date.

* Downloader: fix staticpeers flag (erigontech#10798)

Cherry pick erigontech#10792

* Fix NewPayload Validation during header download (erigontech#10837)

Cherry pick PR erigontech#10093 into the release branch

Co-authored-by: Minhyuk Kim <kimminhyuk1004@gmail.com>

* e2: mainnet blob 9.3M (erigontech#10842)

* Fix gas fee calculation for debug calls (erigontech#10880)

Cherry pick PR erigontech#10825 into the release branch

Co-authored-by: Minhyuk Kim <kimminhyuk1004@gmail.com>

* Revert "eth_estimateGas: default feeCap to base fee (erigontech#10499)" (erigontech#10904)

This reverts PR erigontech#10499. See
erigontech#10495 (comment)
and PR erigontech#10901

* params: version 2.60.2 (erigontech#10905)

* upstream v2.60.2

* fix ci lint

---------

Co-authored-by: milen <94537774+taratorio@users.noreply.github.com>
Co-authored-by: Andrew Ashikhmin <34320705+yperbasis@users.noreply.github.com>
Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>
Co-authored-by: mars <marshalys@gmail.com>
Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>
Co-authored-by: Dmytro <vovk.dimon@gmail.com>
Co-authored-by: Willian Mitsuda <wmitsuda@gmail.com>
Co-authored-by: awskii <awskii@users.noreply.github.com>
Co-authored-by: battlmonstr <battlmonstr@users.noreply.github.com>
Co-authored-by: Shoham Chakraborty <shhmchk@gmail.com>
Co-authored-by: Michelangelo Riccobene <michelangelo.riccobene@gmail.com>
Co-authored-by: Minhyuk Kim <kimminhyuk1004@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants