Groom Tendermint Logs #5912

alexanderbez · 2021-01-15T14:46:38Z

Summary

I introduced a PR into the Cosmos-SDK that changed the underlying logger and logging behavior -- cosmos/cosmos-sdk#8072. Please see the PR, specifically log_level section for the rationale, which I believe is strongly justified.

Problem Definition

The problem that was introduced via cosmos/cosmos-sdk#8072 is that it exposed how verbose and superfluous many of Tendermint's logs are. The logs now are very very verbose, many of which are completely useless and uninformative or should be moved down a level (i.e. DEBUG) at the very least.

Because of the now exposed verbosity of Tendermint's log, it will becomes difficult for node operators to grok logs and make sense of them for useful diagnostic info. Of course, operators should be sending their logs to a logging service that provides rich query functionality, but still...

Proposal

Go through all of Tendermint's logs (a global search for log.Info(...)) and evaluate (1) does this log make sense/should we have it? And (2), if so, does this log make sense at INFO or should it be downgraded to DEBUG?

I do not think it should take that much effort, but the review might be a bit lengthy. Perhaps instead of doing this in one fell swoop, we evaluate the noisy logs in Gaia and see what can be removed/downgraded and then incrementally go from there.

Groom/Investigate

consensus
p2p/pex
mempool

/cc @marbar3778 @alessio

For Admin Use

Not duplicate issue
Appropriate labels applied
Appropriate contributors tagged
Contributor assigned/self-assigned

The text was updated successfully, but these errors were encountered:

tac0turtle · 2021-01-15T15:34:52Z

especially things like:

:34PM INF Received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"0260D92AFBCEE2652E6CF5B49F2C463676A0F9969C9183DEAA07992008FF
0924","parts":{"hash":"30B0E53B9DCE9306B6B70481C49CD82F01BE54B2EE8D5479C2C2E5F99926
E7F2","total":1}},"height":292,"pol_round":-1,"round":0,"signature":"efi1fWKUY8xViXKaycbimGjhsSdloMsBYKllmLRdg3+Q
OxK+Aym0gnmq2UEuDHndi0ghh6GPs1n2+gQCd2lDAw==","timestamp":"2021-01-15T15:34:13.107859Z"}
4:34PM ERR Error with msg err="error part set invalid proof" 
height=292 module=consensus msg={"Height":292,"Part":
{"bytes":"0A8A030A02080B12046D656D6518A402220B08BCE086800610FBD28C792
A480A202F145052D865DD76A195642B0C6A7CF106D09D48BA2ACDC26A630A09079C55
8A1224080112204B41EE103E471CEB5E9103760F856DE6A76FB4030388633E2629DD1
5087269DB32203EC639AABE990B9EC546A9B6B57F48537E5557FEDD32EDEA883B45A4962D3A3F3A20E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B85542203380F954CDF03953427B99B6B3475DFA4EF789651D0CBD9281AD7920ECF0779B4A203380F954CDF03953427B99B6B3475DFA4EF789651D0CBD9281AD7920ECF0779B5220048091BC7DDC283F77BFBF91D73C44DA58C3DF8A9CBC867405D8B7F3DAADA22F5A20C31E6398856A14DDD5490B59C8494977A9BDB0D8A359938BD1067723ABA4B9236220E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B8556A20041A7AB164DAD80FAEBE8123D134B9A16DBB7FBAB853E28E9F8AB38348E5AAEE721490BFC1CEFFC7142DB5200A2409010707688BAE5512001ACB2F0ABD020ABA020A
6E080210F401220212002A0B08D1DE86800610D08FF257321490BFC1CEFFC7142DB52
00A2409010707688BAE554240B0B8276356BF54193C9E015525A043927DB9682708CE
451E12BEEE7730869E50B720E428CBA8BA0F27411DAC5F18C7F3B32CEFC41B905F006
FC027305E61960612B501080210F40122480A2055EBCF9B3F9CA689E674898BBEB48F
2FA88CC507959AF8745F5B9B9164B6C13A1224080112207866773A8EA3A19163A5E7B
3365FDC53114472C63B96269D3DD93A7CC4167C682A0C08D0DE86800610F4FB9CC502
321490BFC1CEFFC7142DB5200A2409010707688BAE5542400D543997E89ED60C76B2C
35B7FB968D8FC2AFA8C523539255ACE0F8B865B526F2119AD8146595F01671927B249
C52FC11319EED256147C31DFC2924F4BABC60D186420642A0C08D0DE86800610F4FB9
CC5020ABE020ABB020A6F080210F801220212002A0C0889DF86800610B89695810232
1490BFC1CEFFC7142DB5200A2409010707688BAE554240BD5AC116A0BE1BD3D0F824B
CF17334654C63B26FE8B9622DE5A44CA967F631F5DFBA86ABCE677C166D1C12B3C370
F9B36D4ADD661E84CC2C83D828EB6DD2950012B501080210F80122480A20B2B6D78DC
90868DB59A91ACF5F9036E57CE80C9579C89163F6075D56BE20E725122408011220AB
31A3AECB3AC44BA31E29C89F3D3119F0BCD83E3D1BC3DDA13C9F87047393072A0C08E
4DE86800610B2DDE3D902321490BFC1CEFFC7142DB5200A2409010707688BAE554240
26372D14C19A0925043698FFFF44DB326C217787EA9E23B05ED5BFDCD6511BCECCC8D
4FB7096E09D93FA6DAF2FBCD69CB46789FE17483FB06766F8B2FF46D103186420642A
0C08E4DE86800610B2DDE3D9020ABD020ABA020A6E080210F901220212002A0B088AD
F8680061090FA8009321490BFC1CEFFC7142DB5200A2409010707688BAE554240926C
86A8BD7E0A4741667E07AC77DC8C6E6015D98F7F4FD2112618E8713DF5E603703684A
396C055C7C7856E5DB982145BCD96E5052944BD7EA924E449FFDC0312B501080210F9
0122480A20AFED9C0E205D3E34024806A9345090502AE0B3B77F0A4DEF0BF12AF4C1F
CFCDC1224080112208E6B831C82F3E064EDB32AFD953F2E104DD29226A25C24000434
FDFB3BE186C42A0C08E9DE86800610949FDFDE02321490BFC1CEFFC7142DB5200A240
9010707688BAE554240CAEA5408C07EBFCB8D0D3E1538BF9E9BD153DEA474240FD5E1
7FFB2C01018374D9C74960D0405DBB9F0F62842E98BEAF76D1DFF56276FF289EB4233
4D3992E01186420642A0C08E9DE86800610949FDFDE020ABE020ABB020A6F080210FA
01220212002A0C088ADF86800610D882918802321490BFC1CEFFC7142DB5200A24090
10707688BAE5542408EA1ECA805B19BBADEA24079B85329C85D8DE7AF31401D09CEDA
4EB7975754FEDA7239BE63C19B8BD051713FEC371B9B18DFD89FDC20678944D3E2E13
46BBC0112B501080210FA0122480A20E55DC47F843917E5C7062ACEA9781624D1FAE6
51A71168A700D6366BEF19A8961224080112200FFB0FE78CFF4F3B078EA9A04F337C3
FD5C781DA91A902967EBC247DBCAF28582A0C08EEDE86800610EC8EC4E402321490BF
C1CEFFC7142DB5200A2409010707688BAE554240CC7657F4E1696FDF9451BC2E1A807
110A25AE91FF16877D40C5BFF02904E6058F5A394BF70FA50728F5E2A0CB7B0785657
9FDFD948CBED2E8429EA9994E75E05186420642A0C08EEDE86800610EC8EC4E4020AB
D020ABA020A6E080210FB01220212002A0B088BDF86800610F0D7C81B321490BFC1CE
FFC7142DB5200A2409010707688BAE5542408FFC74B1A77CBF57203D342B5ABF1F3D4
DF80728FC2E9583E9D841A7D0157BEF414A67276515DCD7701B230858622BA9CB45B8
5D139DDD02698EE00FE59B8D0312B501080210FB0122480A201763A0E037839B40168
F5CAB93DE4B5896C32A5E71497F2AC7CADC026FC988D1122408011220CDB28D37D6F7
A29B0927DA7E619485BB1C462B6E7F20198B14C6F5231E7A57862A0C08F3DE8680061
0C9F4B6E902321490BFC1CEFFC7142DB5200A2409010707688BAE554240C598FF30CD
4F74CDAD25C0FE660DE445542962982A17FDFCB77D6B2E2D6FFD8682D64A466C0F86D
919220151D761E9E42E0BCE0A77DCA4F69634BDA7C2E5E20D186420642A0C08F3DE86
800610C9F4B6E9020ABD020ABA020A6E080210FD01220212002A0B088CDF86800610D097EF65321490BFC1CEFFC7142DB5200A2409010707688BAE554240C94F69725A545E90210DA21202D533210413D452C881A798173749593D3A660CA1719CB17D69D56E45AE3E4BF6A3E7578725282EFD7CB521E0F34E7C39B1DF0712B501080210FD0122480A202C2B2071F8488C8B936F580AC2E8DF975F717EBF8A4ACE874BBA32F6A8F2C1FF122408011220C75332A2ACD58CF58F4D32F83AF77C7B587568AAAA66DF37855B487D095AE3902A0C08FDDE868006108BFDC5F202321490BFC1CEFFC7142DB5200A2409010707688BAE55424087F5223F3A34227B3D5091D787B636A82851897029975C9D9DFF041325D792224E12502B241A84B9C0DBA7A319D01F3BEF7920184942BB4828DA28458B9C1700186420642A0C08FDDE868006108BFDC5F2020ABE020ABB020A6F080210FF01220212002A0C088DDF86800610B0E7C1E501321490BFC1CEFFC7142DB5200A2409010707688BAE554240265292F2B2785D72EBCA6226274A0821BD6336C2F38A4C2BBBDEDB6E96C09771E43CB466C468C47DE861E14C5B3EE3CE210BE888222A7FF2BD5D950ED70E3C0312B501080210FF0122480A203B85B39288E703A69A21F0B98513864DA82B1277D498CF5CCFBEEA6F23B530841224080112204D2E1475336A03B84B5A7B454579D9E79E51B0B0547AD2F3E5D3E26F83979E272A0C0887DF86800610D796C6FF02321490BFC1CEFFC7142DB5200A2409010707688BAE5542404C8CB926F4FB77CC60310D69F42BAEDEC1B5424F377848D2A07CA3AA62F78FB0CC479E1BE8F87552BAF3CA9F0B7FA2C8F0E97D27AAEE42233F4B59C2B653230D186420642A0C0887DF86800610D796C6FF020ABD020ABA020A6E0802108002220212002A0B088EDF86800610D8FC9138321490BFC1CEFFC7142DB5200A2409010707688BAE5542405A3921A8130FD67DF01C945C487734177905C7ED88D699D7D36CAA94B7045925B16D18344EAADBF3681DA6AAB35ECED118CF9F7D50F15BBDF371DEBD33158E0D12B501080210800222480A20C41C79F050B0F3D5B3E1287BFEAC16F60E8C29902F9DA7A6109CB8678ABCC89812240801122080E5AF2AC17395408176D663C349CD7D70C21D0533975D77DC87D0AF6C60F1182A0C088CDF86800610FCE5908803321490BFC1CEFFC7142DB5200A2409010707688BAE554240B7EFD89EDEEB52418F958FC9A7AFAB6107D5D30313D42754C753E45755B16BF3C2F2AEA20C66D66621CD5BEF6E2F68A031E74D2C096E005D1B74091473CD8E0F186420642A0C088CDF86800610FCE59088030ABE020ABB020A6F0802108302220212002A0C089CDF86800610E8E1D8A901321490BFC1CEFFC7142DB5200A2409010707688BAE554240ADFB44C3F0D0F85517F264D881BC2480476BDC49277C2148C0CEF962B601645AA1FB638F2EFF0552F6D7CA2FF10D467ECE4C15A2749ACA3A79CD943140A88F0212B501080210830222480A20B38001FDED8D4566768768BF846CC18CC0BFFC79910DD2B873231C8E330914B11224080112202AAB49412EC90D1DD9068F21EAB6818FA3A6B6A1B0DF45E1E366F14226F0DB7D2A0C089BDF86800610CF9DC89C03321490BFC1CEFFC7142DB5200A2409010707688BAE5542405170334F7040B1725B72A9614217F6306D194A04ACBFA424361DDCA575A6479F6C89E42945FB896760EB523918F2169891A0B9E1FDE1B8BEC03D3442D35F7402186420642A0C0897DF86800610C8CEC5AE010ABE020ABB020A6F0802108402220212002A0C08A1DF86800610C0D0D8B101321490BFC1CEFFC7142DB5200A2409010707688BAE554240B3FD5CD1A051B2684C625BF62900B7F359A6101802D882312F9362164DB14A98DA170CC3A3D9E29C204A5CE3CC0C89801C2E053476127CAD5D98590992EF9B0312B501080210840222480A2026A296DBEB5290695BBA232373B8500EF17026320B755F09A31D930F97B07727122408011220BC9304ED9BC8E591982CD135E873AA4D947DBCBE57AFFD45626A0CD39F9C17F82A0C08A0DF8680061096A3B4A203321490BFC1CEFFC7142DB5200A2409010707688BAE554240238ED6783BF4D3F05F88D5835E19968B42EF8C3AB0543F2B124FBF3ED60D9FE61682D2E1FEC7FB1D274B8CC8BBC0603D9A756414F86AA06DA94A4A10B282AD0E186420642A0C08A0DF8680061096A3B4A2030ABE020ABB020A6F0802108502220212002A0C08A6DF86800610C8E0B3B201321490BFC1CEFFC7142DB5200A2409010707688BAE554240839EEF8B3752993022D0B78051514C728356376E2329499D860F12B813ADC538BF8F1A41D636E571F1D48C0F5472E0AE0FE276C07DEE4959B31EC0DF4F21290F12B501080210850222480A2064DD3E096C6D0BB81FE275E6AE3D3F413270712266603656B9B6E6E34307B6F6122408011220ADAA92BD8B6A473FF0992AB4A81CE8189859775735B97AFF773BF45FAFE3B3AD2A0C08A5DF86800610E18C98A803321490BFC1CEFFC7142DB5200A2409010707688BAE554240E9D9149F4AA6734A1E0F7D53A099026EB0F6A6F6A3A6FB0C96DED7EF6E03CD415E7A677EC69FBAA50A6126198DC043D02CDF2031C83D37ACED8EEFBAB47ACC09186420642A0C08A5DF86800610E18C98A8030ABD020ABA020A6E0802108802220212002A0B08C1E186800610E8A2D235321490BFC1CEFFC7142DB5200A2409010707688BAE554240E170FCC8AD710E2532F2CB4DD6091DE165F916E264AC909643F4DDD6812E17F69D25A8F36E6CE692F0C12F9EE4AB0C293DA3E672CABC16DE2D97A656C093810412B501080210880222480A2023861C179D76081C19E22C375930930DE86D743874D303642BF368621E735997122408011220945C8FC1453BB9530BB50DAD747122BD9B050403A219D6ED9867CD3E9E0C829D2A0C08B4DF86800610B4EBF1B803321490BFC1CEFFC7142DB5200A2409010707688BAE554240D80926674F7F63250DBC58988F722FA2EAA00E05386EE486BAFA41AFA447B484AD81CEE3272503489833AB0CA32593D0A2C981486FFEE0F1CE91140E6A2A7903186420642A0C08B4DF86800610B4EBF1B8030ABE020ABB020A6F0802108902220212002A0C08C1E186800610F09FFBA602321490BFC1CEFFC7142DB5200A2409010707688BAE5542406E93B9AEBA48BD64B5A3937D6329F7AC1FEBE8F745D74F242F443CD6A7A41CDB7BC0B2356A2416CB5F1E0AFF826C290501543B775AE3C78DAD7EBD5C1B0B930512B501080210890222480A2059EE1F9B5554E348689AAED843A591A1916EE8BA3EF4C05670A12AA416CD062B122408011220E8418BC2404DBA231A5C77DC5ECE79AF820CECC9821EDEE9C37B8EF72199AC562A0C08B9DF86800610C3A99BBE03321490BFC1CEFFC7142DB5200A2409010707688BAE5542400DBAF03293C0BC845E532537A6B12CAEAE95E98A7E259B2589337DB9726CF06683B1255DCCDF173C91E9C45901CECA8A8704BC9AA83215739EE7DFC3A791E503186420642A0C08B4DF86800610B4EBF1B8030ABE020ABB020A6F0802108A02220212002A0C08C2E18680061080BE98F901321490BFC1CEFFC7142DB5200A2409010707688BAE554240024C569A2C64809789958D8E587AE1E1EB1977EDA54DC98214DB3532997A4A7865137CF17BC9F036E00B3A1A48E26D474A17597993AC56A3573A643B3C0E7D0712B5010802108A0222480A200620F8CE44DAEA7CDC1D6A35B7456D20D698847572D5B2E96FBC94AD685E3714122408011220E60EA474173AB036AC5CE69B530E8C8C98B054F9D334DA75F9B1D3E863E3A6C82A0C08BEDF86800610B78CC7C303321490BFC1CEFFC7142DB5200A2409010707688BAE5542406D243608192D7C67023F33B33415DA00210FD361B547F1451E8F3E929D1619B71FF56D9F1CEBC0406E485149714807C90395F42C7A28E62E2CB192A279C9FB04186420642A0C08B9DF86800610C3A99BBE030ABD020ABA020A6E0802108B02220212002A0B08C3E186800610B8B7C371321490BFC1CEFFC7142DB5200A2409010707688BAE554240E2BAE8C0CD7C6BA02FDFDD7242BAC779468A51FB1B957450CC9B241F161DD256E08B3B4397826AAC87DAF93D2F19ACAEC10D025F649C587DF8D25E9EB73CA60912B5010802108B0222480A20FA57DA64FEA84EE3BBDA45C8C838052C0AE5928940F258BD4EF6C219DBE161951224080112205B18452016AE511CC222736C290D290E4E01D390C98D1F916E2DEE4386956FAC2A0C08C3DF86800610E9B5F6C803321490BFC1CEFFC7142DB5200A2409010707688BAE5542400786432E8D9CD344C356E23A35D7849C574382D491BD691552356A4B69A486288656E8286F4061F8C75505EE2312324969D671B8BA55994DABFEC9945425BD0D186420642A0C08C3DF86800610E9B5F6C8030ABE020ABB020A6F0802108C02220212002A0C08C3E186800610C0DA80E602321490BFC1CEFFC7142DB5200A2409010707688BAE554240A12AD973A5768673C3B04A2B74CE929FF5AAD15D649F2D11D1A86DB688C0EA7DB5BAA14EEEC6DBA6282BEC589F650D9632683B6C44A2DD33DB46F0157B6DDE0712B5010802108C0222480A202522D01247AA05E89568E021B2A59757BBC05E7B057F773CBC66B6F171CBC0C212240801122051C27E558373C522B432DE00536D559A9A6509BDF01B179C7D085A375B5E8D6A2A0C08C8DF8680061094ADA6CE03321490BFC1CEFFC7142DB5200A2409010707688BAE55424048EBC81E7A4ACCB3E5D8C885BB8653C24FB4C95F8EE4EFD31420A7BB3BA2B7B60E66CC4F11B68F63B25A83F661D80ED3444DADBDCCB3ECFCE22EB8E320BE0A04186420642A0C08C3DF86800610E9B5F6C8030ABE020ABB020A6F0802108D02220212002A0C08C4E186800610D8E0BC8E01321490BFC1CEFFC7142DB5200A2409010707688BAE554240352852643C209E971C9C197DCA7B987E8C2AB39C70A8011608B462674912084335F12F151D41B9597044F057604F5BCA70B9C627C63566A2900FFAEE80D9AF0D12B5010802108D0222480A204DD23485E8A9930A68FAEF244866FD2E1CA2B2E0FE1BB9A224C913579AE030F7122408011220DC91DDEABD407AEAD3E10918AC8F3AE1281096271CCDB451681BAA7544B90B092A0C08CDDF86800610B69A91D403321490BFC1CEFFC7142DB5200A2409010707688BAE554240B25C367BCF07986A0CB4FB0AB0C22FEFC7FA0C13A079CDA0FC002C6EFF521EF5618045468F41F1ABF70028D48473B0D6E58340A7ADDE6C2F77C8B07938329F0A186420642A0C08C8DF8680061094ADA6CE030ABE020ABB020A6F0802108E02220212002A0C08C4E186800610C8A6E7AF03321490BFC1CEFFC7142DB5200A2409010707688BAE554240E278C21AFA272F440D5C81ADD75D419E340CC2ED52132A3A2FB70200CEB354E61CB4E700B5A1960E97863ED1C4C732F8AFD37B6A00483B19444F2E512980350B12B5010802108E0222480A200180FD6DE426E393EA92745197410F23F9946AAB34489C919A0BB9C04C7C8BE2122408011220E82B98729C87252955FBF5CA6FF8C02733405A6A86042A2359630A1310199E072A0C08D2DF86800610B183F5DA03321490BFC1CEFFC7142DB5200A2409010707688BAE55424054219C489F68CDD3762A693B95D8BBA52BB0504C8121B4FFC0EB3A74046487A5A9958D2295E830734D6A51CAB5CC53E7116939D8F90678B00E8A11BE1B29390C186420642A0C08D2DF86800610B183F5DA030ABD020ABA020A6F0802108F02220212002A0C08C5E186800610C09685E101321490BFC1CEFFC7142DB5200A2409010707688BAE55424072A8C48A3CC7DFE7B359AEB6A4ADA44335365F44318BE8A2BC3225ECE84DBE116CC7C9B732147AA7A6138D05E8E9B385E8CC4D3E3F26657C3D30869F803B0C0212B4010802108F0222480A209198DE51A04B6798E5021FB77FFEDA6EBC51DEB794C3D4A6512B6F38753ADE61122408011220376E82EE614983A3276D0E7BFDFE511FEA9D31A50461CA633E8F4D617DA5B8522A0B08D8DF86800610F9C1A603321490BFC1CEFFC7142DB5200A2409010707688BAE5542400C04443A0BAFF451824E9D3ACF587020CA83AB9D40DF1E1A34FDC472CAD52C7D7D6467001C2E7E7A7F3163D4DCCBB9ED3D7220DB7D0BDB5F4D317E9930BBA30E186420642A0C08D2DF86800610B183F5DA0322B60108A3021A480A202F145052D865DD76A195642B0C6A7CF106D09D48BA2ACDC26A630A09079C558A1224080112204B41EE103E471CEB5E9103760F856DE6A76FB4030388633E2629DD15087269DB22670802121490BFC1CEFFC7142DB5200A2409010707688BAE551A0B08BCE086800610FBD28C792240C152D76767504B2728FD760C145FF52419624555009792592D7E7BF7AC9515D312F26164175B7F49A8359213E3C6D1A7A2E35932F4EF2F7164C24E4402C84305","index":0,"proof":{"aunts":[],"index":0,"leaf_hash":"MLDlO53Okwa2twSBxJzYLwG+VLLujVR5wsLl+Zkm5/I=","total":1}},"Round":0} peer= round=0

Executed a local network using simapp and looked for logs that seemed superfluous. This isn't by any means an exhaustive grooming, but should drastically help legibility of logs. ref: #5912

tac0turtle · 2021-02-02T14:14:13Z

Should groom some logs outputted while syncing.

2:11PM INF Executed block height=2788 invalidTxs=0 module=state validTxs=0
2:11PM INF commit synced commit=436F6D6D697449447B5B313437203220313234203139342031373620323438203631203133342031343620313337203235332031303020323338203232382032333120333420333220323033203234352031333820312038372031313420313437203233392031353720333120323436203138392031383120313337203136315D3A4145347D
2:11PM INF Committed state appHash=93027CC2B0F83D869289FD64EEE4E72220CBF58A01577293EF9D1FF6BDB589A1 height=2788 module=state txs=0
2:11PM INF Indexed block height=2788 module=txindex

The above are logs for a block but when a node is syncing these values are not useful. It would be nice to see the highest height synced every second, if an error arises it will be displayed via an error log.

Logging ends up looking like this

Screen.Recording.2021-02-02.at.3.17.48.PM.mov

alexanderbez · 2021-02-09T14:13:12Z

Those logs are across multiple modules. It might be difficult to do that. Why is this bad? I don't think this is necessarily a bad thing.

tac0turtle · 2021-02-09T14:28:06Z

Those logs are across multiple modules. It might be difficult to do that. Why is this bad? I don't think this is necessarily a bad thing.

Not necessarily bad, just annoying and doesn't provide any value to the node operator.

It could use the catchup field to see if its suppose to print all the logs or not.

alexanderbez · 2021-02-09T14:38:47Z

It could use the catchup field to see if its suppose to print all the logs or not.

Neat! What's that? Could you point me to how/where that works?

tac0turtle · 2021-02-09T14:45:42Z

defined: https://github.com/tendermint/tendermint/blob/master/consensus/reactor.go#L47

used: https://github.com/tendermint/tendermint/blob/master/rpc/core/status.go#L64

It's currently used in RPC to tell node operators if they are syncing or not. It would be easier if we had start phases

alexanderbez · 2021-02-09T15:44:13Z

I'm confused. What does that have to do with logging and how does that translate across multiple reactors in terms of how they log?

cyborgshead · 2021-02-17T03:29:25Z

Please, merge this. It's very hard to develop, test and make operations with all this non-configurable logs in console.

I'm using locally latest release patched to avoid all logs but it's not solution. Node operators were completely stunned when watched this.

alexanderbez · 2021-02-17T03:57:31Z

Stunned how? What logs do you believe should be downgraded or removed? Recall, previously logging had zero application logs -- it was all Tendermint. Now, all logs are present, it's just that Tendermint logs are naturally more verbose and this issue is more or less a meta-issue to investigate what can be removed or level-downgraded.

Please post same example logs for us to look into.

Executed a local network using simapp and looked for logs that seemed superfluous. This isn't by any means an exhaustive grooming, but should drastically help legibility of logs. ref: #5912

ref: #5912

ref: #5912 (cherry picked from commit 8a3637a) # Conflicts: # consensus/state.go

alexanderbez · 2021-02-19T16:51:04Z

Next up, we need to groom logs in connection.go (MConnection) and all the transports.

I think once we groom these, we should cut an 0.34.x release for the SDK to use @tessr . Logs should be pretty minimal then.

cmwaters · 2021-02-19T17:38:45Z

Do we need to change any of the logs in the mempool? I remember them being a bit noisy in the e2e tests but maybe they've already been modified.

gaia · 2021-02-19T18:35:12Z

Wanted to chime into why I'd also like to see less logging

In my case I ship logs to a remote service, where it combs thru the logs looking for trouble and alerts via pushover when it finds strings that indicate trouble. So having a few MB per day is fine. But when it grows to nearly a hundreds of MBs per day, the remote service costs increase drastically.

For example, this is a log line, which is surely not intended for humans

bHVlIGFuZCBlbmhhbmNlIGl0cyBsaXF1aWRpdHlAj/TbAUoMCJGH3PwFEK3CjcwBUksKOwoSMjAwMDAwMDAwMDAwMDAwMDAwEhIzMDAwMDAwMDAwMDAwMDAwMDAaETEwMDAwMDAwMDAwMDAwMDAwEgwI9vXj7wUQvsK2yQNaATEKlgMKFSG0n3radRUMzLeKm0uK9L2lCuYHjBL8Ago0Y29zbW9zdmFsb3BlcjFrajBoNGtuNHo1eHZlZHUybmQ5YzRhOWE1NTl3dnB1dnUwaDZxbhJDCh0vY29zbW9zLmNyeXB0by5lZDI1NTE5LlB1YktleRIiCiBmKGaMLG8hmKcPPWFNTr3Tbi40gpbI6RlUTZDlJvCXnyADKgw2MTMyOTkxNDI4OTYyHjYxMzM2MDQ3ODg4MDg1MjM2MTcwNDM1NTM3Mjc0NDprCg1DcnlwdGl1bSBMYWJzEhA1QTMwOUI1Q0ExODlEOEIzGhNodHRwczovL2NyeXB0aXVtLmNoKjNTZWN1cmUgYW5kIGF2YWlsYWJsZSB2YWxpZGF0aW9uIGZyb20gdGhlIFN3aXNzIEFscHNAs4v/AUoMCI+K3v4FEL/4y88BUkwKPAoSMTEwMDAwMDAwMDAwMDAwMDAwEhMxMDAwMDAwMDAwMDAwMDAwMDAwGhExMDAwMDAwMDAwMDAwMDAwMBIMCJO37eQFEK/ruYcBWgExCtkEChUhtOLuIyuXCVFlfwad1uLpuzkTeq8SvwQKNGNvc21vc3ZhbG9wZXIxa24zd3VnZXRqdXk0emV0bHE2d2FkY2hmaHZ1M3g3NDBhZTZ6NngSQwodL2Nvc21vcy5jcnlwdG8uZWQyNTUxOS5QdWJLZXkSIgogweH0gbAWGno4QnZ52bH53W9UNxR1rxyvWfLOIsRKrx8gAyoNMTQ0NzcxMDc0MDg4NTIfMTQ0NzcxMDc0MDg4NTAwMDAwMDAwMDAwMDAwMDAwMDq+AgoJSHVvYmlQb29sEhAyMzUzNkM1QkRFM0VCOTQ5GhpodHRwczovL3d3dy5odW9iaXBvb2wuY29tLyqCAkh1b2JpIFBvb2wgaXMgYSBzdWItYnJhbmQgb2YgSHVvYmkgR3JvdXAsIHdoaWNoIGlzIGFuIGltcG9ydGFudCBwYXJ0IG9mIHRoZSBnbG9iYWwgZWNvbG9naWNhbCBzdHJhdGVneSBvZiBIdW9iaS5IdW9iaSBQb29sIGhhcyBiZWNvbWUgb25lIG9mIHRoZSBsYXJnZXN0IFBPUyBjb21tdW5pdGllcyBpbiB0aGUgQXNpYS0gUGFjaWZpYyByZWdpb24sIHRoZSBsZWFkaW5nIFBPVyBwb29sIGFuZCBub2RlcyBvZiBhIG51bWJlciBvZiBwdWJsaWMgY2hhaW5zLkoAUkoKOgoRNDAwMDAwMDAwMDAwMDAwMDASEjIwMDAwMDAwMDAwMDAwMDAwMBoRMTAwMDAwMDAwMDAwMDAwMDASDAi4qdT0BRDzvrPbAloBMQrpBAoVIbVr5sgqANNjh/WrGvUBv/tV2Es5Es8ECjRjb3Ntb3N2YWxvcGVyMWs0NDdkanAycXJmazhwbDQ0dmQwMnFkbGxkMmFzamVlcGp4c2VwEkMKHS9jb3Ntb3MuY3J5cHRvLmVkMjU1MTkuUHViS2V5EiIKIEiMVvwhUqoDqcKaIhpwgnyykwPtBQdXbJblbVikxK+mGAEgASoJMTIzNTQ1NDAxMhsxMjM1NDU0MDEwMDAwMDAwMDAwMDAwMDAwMDA6xwIKGU5lcHR1bmUgU3Rha2UgKFRTWFY6REFTSCkSEDEyQTk1NkU2NUNCNzE4ODIaF2h0dHBzOi8vbmVwdHVuZWRhc2guY29tKv4BTmVwdHVuZSBEYXNoIChUU1hWOkRBU0gpIGlzIGEgcHVibGljIGNvbXBhbnkgbGlzdGVkIG9uIHRoZSBUb3JvbnRvIFN0b2NrIEV4Y2hhbmdlIHRoYXQgd29ya3Mgd2l0aCBpbnN0aXR1dGlvbmFsIGludmVzdG9ycyBsb29raW5nIGZvciByZWd1bGF0ZWQgYWNjZXNzIHRvIGNyeXB0b2N1cnJlbmN5IHN0YWtpbmcuIE91ciB2YWxpZGF0b3IgcnVucyBvbiBvdXIgb3duIGhhcmR3YXJlIGFuZCBIU00gaW4gYSB0b3AgQ2FuYWRpYW4gZGF0YWNlbnRlci5KDAif9vDuBRDsv5SZAVJLCjsKEjEwMDAwMDAwMDAwMDAwMDAwMBISMjAwMDAwMDAwMDAwMDAwMDAwGhExMDAwMDAwMDAwMDAwMDAwMBIMCLTSv+cFEMv9nacDWgExCtMCChUhtXawMeQuqwyt/SDPl5R0rsIIVMkSuQIKNGNvc21vc3ZhbG9wZXIxazRtdHF2MHk5NjRzZXQwYXlyOGUwOXI1NG1wcXM0eGZ3MHlzZXcSQwodL2Nvc21vcy5jcnlwdG8uZWQyNTUxOS5QdWJLZXkSIgogZZrsMjUnACNTocjqfdaT/GrjcAdVQYVAyj1Rgm/5y14gAyoJNTk2ODYzMDI1Mhs1OTY5ODI0MTEyNDEwOTA1ODk4NTY2NjI5Nzc6LwoX4oCcS0lORy5ETElOT0RFUy5DT03igJ0aFGh0dHBzOi8vZGxpbm9kZXMuY29tQPmFpwJKDAipjIOBBhDHjbzLA1JLCjwKEjEwMDAwMDAwMDAwMDAwMDAwMBISMjAwMDAwMDAwMDAwMDAwMDAwGhIxMDAwMDAwMDAwMDAwMDAwMDASCwjl/4P7BRCzu9UhWgExCpsCChUhtefHNDMjNcWwBoh+uJtb7a8R4KsSgQIKNGNvc21vc3ZhbG9wZXIxa2hudXdkcG55djZ1dHZxeDNwbHQzeDZtYWtoM3JjOXR2MHd2djkSQwodL2Nvc21vcy5jcnlwdG8uZWQyNTUxOS5QdWJLZXkSIgogQRBrvDNfyXBcbWCdi+RCg93Fv4j3e1akmzmwQGM+A48gASoHMTAwMDAwMDIZMTAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDoMCgpjb3Ntb3Mtc3RhSgBSSwo7ChIxMDAwMDAwMDAwMDAwMDAwMDASEjIwMDAwMDAwMDAwMDAwMDAwMBoRMTAwMDAwMDAwMDAwMDAwMDASDAjJ77ntBRCqiLmFA1oBMQrvAgoVIbY5/OZC3PuD/yJIUF5+0qISSi+JEtUCCjRjb3Ntb3N2YWxvcGVyMWtjdWxlZWp6bW5hYzhsZXpmcGc5dWxrajVnZnk1dHVmOWV6Mjl3EkMKHS9jb3Ntb3MuY3J5cHRvLmVkMjU1MTkuUHViS2V5EiIKIFvOxojRGaXcMtXCQ1q4vKgIlw2ubcwNSVWbu3ZjHIdeGAEgASoHMzAwMjMxMTIZMzAwMjMxMTAwMDAwMDAwMDAwMDAwMDAwMDpRCgVBVE9NSRIQODIyMzZBQTJFOTc5RDNEORoXaHR0cHM6Ly9hdG9taWkubW9kb28uYXQqHUV2ZXIgSmFpbGVkIFZhbGlkYXRvciAtIEFUT01JSgwIrsqr6AUQn7vH9wFSTAo8ChE1MDAwMDAwMDAwMDAwMDAwMBITMTAwMDAwMDAwMDAwMDAwMDAwMBoSNTAwMDAwMDAwMDAwMDAwMDAwEgwIvbqq5gUQ37/huAJaATEK8gIKFSG6T9KRGiQcuR/oxf4/a3jM5Km33RLYAgo0Y29zbW9zdmFsb3BlcjFoZjhhOXlnNnlzd3RqOGxnY2hscjc2bWNlbmoybmQ3YTBlNnowehJDCh0vY29zbW9zLmNyeXB0by5lZDI1NTE5LlB1YktleRIiCiBBlQmQibzPd3CP9KN5BlwZSdBpGirQEo/nq7p5gT3xxiABKgYyNjcxMDEyGDI2NzEyNjY4NzUxMTQyNzQ0NTM0NzA3NTpVCg9CdW1ibGViZWVfX/CfkJ0SEDY2QTJEN0M0NkQyN0RGQzYaGWh0dHA6Ly9kYWh1YW5nZmVuZy5vbmxpbmUqFUNvc21vcyB0byBldmVyeXRoaW5nIUDb9ypKDAi77+LyBRCtru/aA1JLCjsKEjEzMDAwMDAwMDAwMDAwMDAwMBISMjAwMDAwMDAwMDAwMDAwMDAwGhExMDAwMDAwMDAwMDAwMDAwMBIMCO+MxvAFEIS23IUDWgExCvkDChUhuyDUviv+nUDNLEsog/A3k2rkuVcS3wMKNGNvc21vc3ZhbG9wZXIxaHZzZGYwM3RsNnc1cG5mdmZ2NWc4dXBoamQ0d2Z3Mmg0Z3ZubDcSQwodL2Nvc21vcy5jcnlwdG8uZWQyNTUxOS5QdWJLZXkSIgogp/UWIafZcXxHEVOnT67ih02jGaT4RbuDcnDQm7Y7VB0gAyoMMTAwNTQ0MDc4Njg5Mh4xMDA1NTQxMzM3NjQ4NjU0Mzk3MDA1OTMwMDk2OTM61AEKCUJpMjMgTGFicxIQRUIzNDcwOTQ5QjNFODlFMhoQaHR0cHM6Ly9iaTIzLmNvbSqiAUJpMjMgTGFicyBpcyBhIHRydXN0ZWQgUE9TIGluZnJhc3RydWN0dXJlIHByb3ZpZGVyIGFuZCB2YWxpZGF0b3IgdG8gY29tZm9ydGFibHkgc3Rha2UgeW91ciBjb2lucyBhbmQgZWFybiByZXdhcmRzIHdpdGggQ2VsbyxDb3Ntb3MsSVJJU25ldCxUZXJyYSxLQVZBLFNvbGFuYSxEdW5lLkoMCOyZ2+oFEJPsjM8CUkoKOwoSNTAwMDAwMDAwMDAwMDAwMDAwEhI1MDAwMDAwMDAwMDAwMDAwMDAaETEwMDAwMDAwMDAwMDAwMDAwEgsI6NKh8wUQvt6QQFoBMgr1AgoVIbu5VwaEr5qpr3GC20NkiATBzsc2EtsCCjRjb3Ntb3N2YWxvcGVyMWh3dTR3cDV5NDdkMm50bTNzdGQ1eGV5Z3FucXVhM2VrY3B6cDB2EkMKHS9jb3Ntb3MuY3J5cHRvLmVkMjU1MTkuUHViS2V5EiIKIKKnRytT0vuxxz2U8X1Bglvnjhl2YTi0uQ67qE1D2ZwKGAEgASoHMzk5OTYwMDIZNDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDpVCghGaXNoS2luZypJTGV0J3MgZGVjZW50cmFsaXplZCBvdXIgd29ybGQuIEEgaG9uZXN0IHRoaXJkIHBhcnR5IHZhbGlkYXRvciBmcm9tIFRhaXdhbkDN4gFKDAjuwr/wBRDerfb0AlJKCjoKETUwMDAwMDAwMDAwMDAwMDAwEhIyMDAwMDAwMDAwMDAwMDAwMDAaETEwMDAwMDAwMDAwMDAwMDAwEgwIyIL85gUQz86VlgJaATEKxQMKFSG8sL0D0wwgGCItQwVowak5YB38KRKrAwo0Y29zbW9zdmFsb3BlcjFoamN0NnE3bnBzc3BzZzNkZ3Z6azNzZGY4OXNwbWxwZmRuNm05ZBJDCh0vY29zbW9zLmNyeXB0by5lZDI1NTE5LlB1YktleRIiCiCf1tb2Ga/BLCb8Ccm0PBk0YD7ZFKhCx+gwWxzZGtvS9iADKg0zMDk2NTMyMDA1NjIxMh8zMDk2NTMyMDA1NjIxMDAwMDAwMDAwMDAwMDAwMDAwOqoBCgdGaWdtZW50EhBFNUYyNzRCODcwQkRBMDFEGhJodHRwczovL2ZpZ21lbnQuaW8qeU1ha2VycyBvZiBIdWJibGUgYW5kIENhbmFkYeKAmXMgbGFyZ2VzdCBDb3Ntb3MgdmFsaWRhdG9yLCBGaWdtZW50IGlzIHRoZSBlYXNpZXN0IGFuZCBtb3N0IHNlY3VyZSB3YXkgdG8gc3Rha2UgeW91ciBBdG9tcy5KAFJKCjoKETkwMDAwMDAwMDAwMDAwMDAwEhIzMDAwMDAwMDAwMDAwMDAwMDAaETEwMDAwMDAwMDAwMDAwMDAwEgwI8o2p7wUQs6PuygJaATEK5QMKFSG+F9T0DMo803Wn7lqYq6BUj+8MeRLLAwo0Y29zbW9zdmFsb3BlcjFoY3RhZmFxdmVnN2R4YWQ4YWVkZjMyYXEyajg3N3JyZWMyZjVhdRJDCh0vY29zbW9zLmNyeXB0by5lZDI1NTE5LlB1YktleRIiCiA5gR02TQY72yiFxlcDf+bbxusvbR6ooM4Aav9dwLUsSCABKgcxMDAwMDAwMhkxMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwOtQBCglCaTIzIExhYnMSEEVCMzQ3MDk0OUIzRTg5RTIaEGh0dHBzOi8vYmkyMy5jb20qogFCaTIzIExhYnMgaXMgYSB0cnVzdGVkIFBPUyBpbmZyYXN0cnVjdHVyZSBwcm92aWRlciBhbmQgdmFsaWRhdG9yIHRvIGNvbWZvcnRhYmx5IHN0YWtlIHlvdXIgY29pbnMgYW5kIGVhcm4gcmV3YXJkcyB3aXRoIENlbG8sQ29zbW9zLElSSVNuZXQsVGVycmEsS0FWQSxTb2xhbmEsRHVuZS5Ai7otSgwIzuP08gUQw4GI7gJSPAosCgEwEhMxMDAwMDAwMDAwMDAwMDAwMDAwGhIxMDAwMDAwMDAwMDAwMDAwMDASDAjtl5/tBRDi/dzYAloBMQqtAgoVIcA+WFuvN9Q7I0jDHzDWMYonpJEQEpMCCjRjb3Ntb3N2YWxvcGVyMWNxbDlza2EweGwycmtnNmdjdjBucDQzMzNnbjZmeWdzNTVhc3JzEkMKHS9jb3Ntb3MuY3J5cHRvLmVkMjU1MTkuUHViS2V5EiIKICvUsYOg3PvSUdetZNkf8gqBniEHHp06dTswnBGRT6NeGAEgASoJMTkwNjA5ODQ2MhsxOTA2Mjg5MDM4MDc4MDUwMDY2NTcyNDU2Mjg6DgoMQXRvbSBTYW5kbGVyQJFOSgwI2rW38AUQ/66D2wFSRgo8ChIyNTAwMDAwMDAwMDAwMDAwMDASEzEwMDAwMDAwMDAwMDAwMDAwMDAaETEwMDAwMDAwMDAwMDAwMDAwEgYI8Jam5AVaATEK4gIKFSHAtApMkHrRwIFbWUvdw+vIUfMBDhLIAgo0Y29zbW9zdmFsb3BlcjFjejZxNW55czB0Z3VwcTJtdDk5YW1zbHRlcGdseHFnd2ZzNmZ1bRJDCh0vY29zbW9zLmNyeXB0by5lZDI1NTE5LlB1YktleRIiCiBkP9VdNjDiVLFWhhi0ZzzwNqk5VuGRqsuOGHZN8a/eOxgBIAEqCzExNTcyNzQ4NTYxMh0xMTU3Mjc0ODU2MTAwMDAwMDAwMDAwMDAwMDAwMDo3CgdtZWxhbmdlEhBENDg2NTk1RDVFQjgwNzhDGhpodHRwczovL21lbGFuZ2VzdGFraW5nLmNvbUCdk54CSgwI9tXCgAYQr8memQJSTAo8ChExMDAwMDAwMDAwMDAwMDAwMBITMTAwMDAwMDAwMDAwMDAwMDAwMBoSMTAwMDAwMDAwMDAwMDAwMDAwEgwIvaen+AUQqZb6uANaATEKmAMKFSHAwbjQpxaBsk1Wtj5Wi9wEKgaCYxL+Ago0Y29zbW9zdmFsb3BlcjFjcnFtMzU5OHo2cW15bjJra2NsOWR6N3VxczRxZHFucjZzOGpkbhJDCh0vY29zbW9zLmNyeXB0by5lZDI1NTE5LlB1YktleRIiCiBb0hMbiecv1iQXOJEQynNzWME4t+Z2I2aFsGS6ZiYI7yADKgwxNjI0MDI5MDgzOTIyHjE2MjQwMjkwODM5MjAwMDAwMDAwMDAwMDAwMDAwMDqAAQoMQmlzb24gVHJhaWxzEhBBMjk2NTU2RkY2MDMxOTdDGg5iaXNvbnRyYWlscy5jbypOQmlzb24gVHJhaWxzIGlzIHRoZSBlYXNpZXN0IHdheSB0byBydW4gaW5mcmFzdHJ1Y3R1cmUgb24gbXVsdGlwbGUgYmxvY2tjaGFpbnMuSgBSSQo6ChE4MDAwMDAwMDAwMDAwMDAwMBISMjAwMDAwMDAwMDAwMDAwMDAwGhExMDAwMDAwMDAwMDAwMDAwMBILCK/h5+0FEMeAzQ5aATEK5QIKFSHA0qOyi

alexanderbez · 2021-02-19T20:07:50Z

Do we need to change any of the logs in the mempool? I remember them being a bit noisy in the e2e tests but maybe they've already been modified.

Yes! I've updated the issue body.

alexanderbez · 2021-02-19T20:09:36Z

@gaia that doesn't look like a log, but rather some sort of encoding. Can you past the full log in pastebin please? It's probably something from consensus which we've already addressed and is slated for 0.34.x

gaia · 2021-02-19T20:49:42Z

@gaia that doesn't look like a log, but rather some sort of encoding. Can you past the full log in pastebin please? It's probably something from consensus which we've already addressed and is slated for 0.34.x

yep, i looked for it in the raw logs, can't find it. this is how they should up at the external log facility

it would be SUPER to be able to have logs NOT be colored (for exporting to 3rd party collectors)

alexanderbez · 2021-02-19T23:01:11Z

yep, i looked for it in the raw logs, can't find it. this is how they should up at the external log facility

Not much we can do unless you have the actual log.

it would be SUPER to be able to have logs NOT be colored (for exporting to 3rd party collectors)

format: text -> human readable and meant to be colored
format: json -> for external logging facilities

You should be using json format always when sending logs to an external log facility.

gaia · 2021-02-20T16:49:55Z

thx alex.

format: text -> human readable and meant to be colored

i have log_format = "plain". should I change it to text ?

RyanHendricks · 2021-02-21T04:34:36Z

i have log_format = "plain". should I change it to text ?

If that doesn't work a temporary solution is the use the --log_format=json when running the start command. Separate issue being tracked here it seems: cosmos/cosmos-sdk#8132 (comment)

Another log that would be great to move from error to info is error adding vote which appears to originate here:

tendermint/consensus/state.go

Line 36 in 27eb10a

ErrAddingVote = errors.New("error adding vote")

tendermint/consensus/state.go

Line 1979 in 27eb10a

return added, ErrAddingVote

At least in the context of the following error message:

ERR Error with msg err="error adding vote" height=5232540 module=consensus msg={"Vote":{"block_id":{"hash":"1110EB8E8697C0CC049671468D4608BA85CFC64C178EBAF05F0394A3C078570E","parts":{"hash":"4285FAD7C6B04EE1B5337D047784842D0D2A3CA82AC613FA414671860CD8A3CE","total":1}},"height":5232540,"round":0,"signature":"R8gIYQfBulaam7Er8zcvVyZR9CoTRTFu8fBzb7FyQTrPHJ0jowJCUN12WnsQUTc3bfuheTlaWLZP+4ox7/RZDw==","timestamp":"2021-02-21T04:17:39.118953465Z","type":1,"validator_address":"D540AB022088612AC74B287D076DBFBC4A377A2E","validator_index":59}} peer=a0aca8fb801c69653a290bd44872e8457f8b0982 round=0

I cross-referenced the pubkey for that validator between the tendermint validator set and cosmos-sdk validators and turns out that it belongs to "moniker": "jackzampolin" whom I believe is running a threshold validator.

tendermint/consensus/state.go

Line 1976 in 27eb10a

// 3) tmkms use with multiple validators connecting to a single tmkms instance

Bad peers might warrant an error message but in this particular case I think it would be better surfaced info or debug otherwise we will see an error every block for every validator of similar nature.

alexanderbez · 2021-02-22T01:22:54Z

thx alex.

format: text -> human readable and meant to be colored

i have log_format = "plain". should I change it to text ?

No, if you're sending logs to a log facility, use log_format = "json". This way these log consumers can parse and digest the logs while allowing you to leverage their structured nature to perform rich queries 😄

alexanderbez · 2021-02-22T01:25:38Z

Thanks @RyanHendricks, sure, I can move this to info. Thanks for the rationale as well 👍

done in #6152

ref: #5912

ref: #5912 (cherry picked from commit c21f4c6) # Conflicts: # state/execution.go

tac0turtle · 2021-06-07T07:47:42Z

This is mainly completed, The last step would be an audit of what interfaces we include in logging, and ideally removing any of them.

alexanderbez added the T:observability Type: Observability label Jan 15, 2021

alexanderbez mentioned this issue Jan 18, 2021

consensus: Groom Logs #5917

Merged

tessr added this to Untriaged 🥚 in Tendermint Core Project Board via automation Jan 20, 2021

tessr moved this from Untriaged 🥚 to To Do 📝 in Tendermint Core Project Board Jan 20, 2021

tessr added the T:jank Type Jank! Non-urgent but still high-impact fixes. label Jan 20, 2021

liamsi mentioned this issue Jan 25, 2021

Decide what to do about missing logging flags for the start sub-command celestiaorg/celestia-app#9

Closed

alexanderbez mentioned this issue Feb 15, 2021

gaiacli: Is it possible to provide logging configuration? cosmos/gaia#474

Closed

tac0turtle mentioned this issue Feb 17, 2021

consensus: Groom Logs (#5917) #6121

Merged

alexanderbez mentioned this issue Feb 18, 2021

consensus: more log grooming #6140

Merged

mergify bot pushed a commit that referenced this issue Feb 18, 2021

consensus: more log grooming (#6140)

8a3637a

ref: #5912

mergify bot pushed a commit that referenced this issue Feb 18, 2021

consensus: more log grooming (#6140)

1033573

ref: #5912 (cherry picked from commit 8a3637a) # Conflicts: # consensus/state.go

alexanderbez mentioned this issue Feb 19, 2021

state executor: groom logs #6152

Merged

mergify bot pushed a commit that referenced this issue Feb 23, 2021

state executor: groom logs (#6152)

c21f4c6

ref: #5912

mergify bot pushed a commit that referenced this issue Feb 23, 2021

state executor: groom logs (#6152)

c441fd7

ref: #5912 (cherry picked from commit c21f4c6) # Conflicts: # state/execution.go

clevinson mentioned this issue Feb 24, 2021

feat: file watcher for cosmovisor cosmos/cosmos-sdk#8590

Merged

9 tasks

tac0turtle closed this as completed Jun 28, 2021

Tendermint Core Project Board automation moved this from To Do 📝 to Done 🎉 Jun 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Groom Tendermint Logs #5912

Groom Tendermint Logs #5912

alexanderbez commented Jan 15, 2021 •

edited

tac0turtle commented Jan 15, 2021 •

edited

tac0turtle commented Feb 2, 2021 •

edited

alexanderbez commented Feb 9, 2021

tac0turtle commented Feb 9, 2021

alexanderbez commented Feb 9, 2021 •

edited

tac0turtle commented Feb 9, 2021 •

edited

alexanderbez commented Feb 9, 2021

cyborgshead commented Feb 17, 2021

alexanderbez commented Feb 17, 2021

alexanderbez commented Feb 19, 2021 •

edited

cmwaters commented Feb 19, 2021

gaia commented Feb 19, 2021

alexanderbez commented Feb 19, 2021

alexanderbez commented Feb 19, 2021

gaia commented Feb 19, 2021

alexanderbez commented Feb 19, 2021

gaia commented Feb 20, 2021

RyanHendricks commented Feb 21, 2021

alexanderbez commented Feb 22, 2021 •

edited

alexanderbez commented Feb 22, 2021 •

edited

tac0turtle commented Jun 7, 2021

Groom Tendermint Logs #5912

Groom Tendermint Logs #5912

Comments

alexanderbez commented Jan 15, 2021 • edited

Summary

Problem Definition

Proposal

Groom/Investigate

For Admin Use

tac0turtle commented Jan 15, 2021 • edited

tac0turtle commented Feb 2, 2021 • edited

alexanderbez commented Feb 9, 2021

tac0turtle commented Feb 9, 2021

alexanderbez commented Feb 9, 2021 • edited

tac0turtle commented Feb 9, 2021 • edited

alexanderbez commented Feb 9, 2021

cyborgshead commented Feb 17, 2021

alexanderbez commented Feb 17, 2021

alexanderbez commented Feb 19, 2021 • edited

cmwaters commented Feb 19, 2021

gaia commented Feb 19, 2021

alexanderbez commented Feb 19, 2021

alexanderbez commented Feb 19, 2021

gaia commented Feb 19, 2021

alexanderbez commented Feb 19, 2021

gaia commented Feb 20, 2021

RyanHendricks commented Feb 21, 2021

alexanderbez commented Feb 22, 2021 • edited

alexanderbez commented Feb 22, 2021 • edited

tac0turtle commented Jun 7, 2021

alexanderbez commented Jan 15, 2021 •

edited

tac0turtle commented Jan 15, 2021 •

edited

tac0turtle commented Feb 2, 2021 •

edited

alexanderbez commented Feb 9, 2021 •

edited

tac0turtle commented Feb 9, 2021 •

edited

alexanderbez commented Feb 19, 2021 •

edited

alexanderbez commented Feb 22, 2021 •

edited

alexanderbez commented Feb 22, 2021 •

edited