Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Send failed" error more than once per second #1817

Closed
awrelll opened this issue Jun 26, 2018 · 5 comments
Closed

"Send failed" error more than once per second #1817

awrelll opened this issue Jun 26, 2018 · 5 comments
Labels
C:p2p Component: P2P pkg T:bug Type Bug (Confirmed) T:perf Type: Performance
Milestone
Wishlist

Comments

@awrelll
Copy link

awrelll commented Jun 26, 2018

This error is spamming the logs.
While grepping my logs, I found this error :
Using 0.19.0-c671181 on Ubuntu 16.04 + go version go1.10 linux/amd64

E[06-25|11:10:57.703] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0C10001802202804
E[06-25|11:10:57.739] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0C10001802201004
E[06-25|11:10:57.871] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0C10001802203404
E[06-25|11:10:57.895] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0C10001802200804
E[06-25|11:10:57.895] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0C10001802203204
E[06-25|11:10:58.278] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0C10001802200604
E[06-25|11:10:58.396] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=C96A6FAB09000000000006DF0D100018032000280004
E[06-25|11:10:58.737] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=C96A6FAB09000000000006DF0D100018042000280004
E[06-25|11:10:58.739] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801202004
E[06-25|11:10:58.740] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801203E04
E[06-25|11:10:58.763] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801203004
E[06-25|11:10:58.850] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801202604
E[06-25|11:10:58.854] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801200404
E[06-25|11:10:58.861] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801204C04
E[06-25|11:10:58.862] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801204004
E[06-25|11:10:58.863] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801200004
E[06-25|11:10:58.864] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801201E04
E[06-25|11:10:58.864] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801204804
E[06-25|11:10:58.865] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801202A04
E[06-25|11:10:58.872] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801202204
E[06-25|11:10:58.889] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801204604
E[06-25|11:10:58.901] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801203A04
E[06-25|11:10:58.910] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801201404
E[06-25|11:10:58.966] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801204204
E[06-25|11:10:58.990] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801202E04
E[06-25|11:10:58.990] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801200A04
E[06-25|11:10:58.991] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801200E04
E[06-25|11:10:58.995] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801203804
E[06-25|11:10:58.998] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801201604
E[06-25|11:10:59.002] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801204404
E[06-25|11:10:59.028] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801202C04
E[06-25|11:10:59.044] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801201C04
E[06-25|11:10:59.068] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801203604
E[06-25|11:10:59.068] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801200204
E[06-25|11:10:59.082] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=C96A6FAB09000000000006DF0D100018062000280004
E[06-25|11:10:59.083] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802203E04
E[06-25|11:10:59.084] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801200C04
E[06-25|11:10:59.085] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801200804
E[06-25|11:10:59.109] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801202404
E[06-25|11:10:59.115] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801201804
E[06-25|11:10:59.116] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801201204
E[06-25|11:10:59.117] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802203A04
E[06-25|11:10:59.155] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801203C04
E[06-25|11:10:59.171] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802204804
E[06-25|11:10:59.181] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802200004
E[06-25|11:10:59.188] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802204C04
E[06-25|11:10:59.192] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801201A04
E[06-25|11:10:59.192] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802203004
E[06-25|11:10:59.202] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801204A04
E[06-25|11:10:59.202] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801202804
E[06-25|11:10:59.258] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802200404
E[06-25|11:10:59.272] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802202004
E[06-25|11:10:59.283] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802201E04
E[06-25|11:10:59.286] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802204004
E[06-25|11:10:59.300] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802202604
E[06-25|11:10:59.312] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802201604
E[06-25|11:10:59.316] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802204204
E[06-25|11:10:59.318] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801203204
E[06-25|11:10:59.340] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802200204
E[06-25|11:10:59.340] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802202A04
E[06-25|11:10:59.342] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802201404
E[06-25|11:10:59.350] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801203404
E[06-25|11:10:59.358] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802200A04
E[06-25|11:10:59.368] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802203804
E[06-25|11:10:59.368] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001801201004
E[06-25|11:10:59.375] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802204604
E[06-25|11:10:59.389] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802202204
E[06-25|11:10:59.391] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802200C04
E[06-25|11:10:59.394] Send failed                                  module=p2p peer=172.104.177.239:51428 channel=32 conn=MConn{172.104.177.239:51428} msgBytes=1919B3D309000000000006DF0D10001802201204
@ValarDragon ValarDragon mentioned this issue Jun 28, 2018
Closed
@ebuchman ebuchman added the C:p2p Component: P2P pkg label Jun 28, 2018
@xla xla added the T:enhancement Type: Enhancement label Jun 28, 2018
@xla xla added this to the launch milestone Jun 28, 2018
@ebuchman ebuchman modified the milestones: v1.0, launch Sep 21, 2018
@ebuchman ebuchman added T:bug Type Bug (Confirmed) T:perf Type: Performance and removed T:enhancement Type: Enhancement labels Sep 23, 2018
@ebuchman ebuchman modified the milestones: launch, v1.0 Sep 23, 2018
@jaekwon
Copy link
Contributor

jaekwon commented Nov 11, 2018

tendermint/go-amino’s aminoscan tool now lets you type --concrete-name as an option…
Channel is 32, which is the consensus StateChannel (0x20).

bash go-amino$ ./aminoscan -concrete-name tendermint/HasVote
Disamb bytes: 324AD5
Prefix bytes: 1919B3D5

So those are HasVote messages… it’s expected to fail often, because HasVote messages are sent opportunistically.

mconnection.Send() returns false… the caller should log an Error if it wants to, instead of logging it from mconnection.Send(), imo. Or it could be Info/Debug level logging so it doesn’t clog up output. I propose that we turn the "Send failed ..." log messages to Info.

@jaekwon
Copy link
Contributor

jaekwon commented Nov 11, 2018

Another note: We really shouldn't be creating new goroutines for every send in Broadcast for things like this... I don't know if there's an existing issue to refactor Broadcast to be more mindful of goroutine resource consumption (it's not like a Service or peer connection where there are some limited number of them; Broadcast gets called N times where N can be large), but another option may be to implement BroadcastTry and call TrySend instead, not in separate go-routines, and see if we can migrate each Broadcast calls to BroadcastTry call.

@ebuchman
Copy link
Contributor

There is: #805

@mdyring mdyring mentioned this issue Nov 13, 2018
Closed
27 tasks
@ebuchman ebuchman mentioned this issue Nov 15, 2018
Merged
4 tasks
@ebuchman
Copy link
Contributor

Changed log level to debug in #2857

@awrelll
Copy link
Author

awrelll commented Nov 16, 2018

Awesome !!! 🍺

iKapitonau pushed a commit to scrtlabs/tendermint that referenced this issue Jul 10, 2024
@mergify @sergio-mena
Small improvements in tendermint#1806 not present in tendermint#1750 (t…
1575d38 
…endermint#1808) (tendermint#1817)

(cherry picked from commit 149ac1e)

Co-authored-by: Sergio Mena <sergio@informal.systems>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C:p2p Component: P2P pkg T:bug Type Bug (Confirmed) T:perf Type: Performance
Projects
None yet
Milestone
Wishlist
Development

No branches or pull requests
4 participants
@xla @jaekwon @ebuchman @awrelll