machine-readable peer discovery logs #327

whilei · 2017-08-01T19:06:08Z

Preliminary implementation of writing machine-readable logs to mlog files; peer discovery logs are the first API.

Find initial working spec here: https://gist.github.com/whilei/2acddb54d169881e1bfb5651bd16f234

Still builds from, and includes unmerged commits from #324. Unique changes for this PR begin with 670b285.

may be worth thinking about more flexible and beautiful ways to display information... 'monitor', 'attach', etc, along with termui, graphs... output can be 'machiney'-- line-by-line, or human-y -- pictures of buffalo and whatnot.

solution: use a go func connected to the node's Ethereum to grab and calculate relevant logs rel ethereumproject#127 rel ethereumproject#318 --- TODO: fix or remove: - downloading-from peers, eg 'this var/x/x peers' - shows mode 'FullSync' before downloader initializes, even if it will be 'FastSync'

solution: comment code and include small explanation

solution: remove exploratory code that is not useless

solution: move to Warn from Error

solution: update it and simplify

solution: - remove dead code, old notes - remove units from --log-pace= argument, just use seconds as value (simpler is better) - if mining, show STATUS MINE with minerthreads and hashrate (TODO: more?) - gofmt Rel ethereumproject#127, ethereumproject#318

solution: improve legibility

solution: relegate to debug

solution: share chan in main between shutdown and closing status log goroutine

solution: implement and glog SetAlsoToStdErr exported function and use in case of --log-dir

results in missing any logs after shutdown processes in FS logs solution: defer log Flush after shutdown services also decrease flushInterval from 30 -> 5 seconds (rate at which log buffer is written to FS)

solution: rename flag and flagvar and status log function also adds logstatus flag to correct category for usage output

solution: refactor to use '--log-status='sync=60'' syntax, which can eventually support other features besides 'sync', eg. mine, net, disk

…to cmd.go solution: cut and paste and update imports

solution: error if neither fastblock nor fullblock match desired head number fixes ethereumproject#325

solution: don't share os-interrupt channel as a global. Notify library documentation clearly says Notify events can be called multiple times and notification will be copied, so there's no harm in listening for the same event on mutliple channels. This pattern avoids a global and encourages per-feature handling of shutdown scenarios.

…g-pace' wording solution: fix

solution: implement initial spec with tentative mlog-dir and filename defaults peer 'handle' func sends mlogs to file rel ethereumproject#317

solution: remove import

solution: refactor mlog from p2p to discover package - establish mlog label and detail structs Resolves ethereumproject#317

…tation solution: create vars for all available lines and comment them. not even sure if concision is a word Rel ethereumproject#317

…s by them solution: refactor details, implementing function to set details with variadic vals

solution: use type switch filtering in send/handle functions instead of dispersed mlog calls 'farther down the food chain'

whilei · 2017-08-03T16:09:58Z

Here's just a copy-and-paste example of what the logs look like:

2017/08/03 09:31:59 [discover] PING SEND TO 209.222.30.133:30388 127
2017/08/03 09:31:59 [discover] FINDNODE SEND TO 149.202.184.138:30303 171
2017/08/03 09:31:59 [discover] FINDNODE SEND TO 46.5.207.98:30303 171
2017/08/03 09:31:59 [discover] PONG HANDLE FROM 192.99.194.116:57955 285d57553cd31781018e7d055f297197c70d9d70f84eeeae4b6ab7f67d63c60b0ffabcdabb39f3bc1a405afe057155a89b39a291f91a06dfa3b6ee3867266186 149
2017/08/03 09:31:59 [discover] PONG HANDLE FROM 209.222.30.133:30388 c65ca649729d72749dc3c90c206bb6bff76930f3411da0005f06271989185f0124aa52daa88e2e3d921f2d2838f8d3b40ed0d693cc172e90fda4461316edcb99 149
2017/08/03 09:32:04 [discover] FINDNODE SEND TO 62.45.159.216:30303 171
2017/08/03 09:32:04 [discover] FINDNODE SEND TO 178.134.252.152:30303 171
2017/08/03 09:32:04 [discover] FINDNODE SEND TO 31.31.78.100:30303 171
2017/08/03 09:32:04 [discover] NEIGHBORS HANDLE FROM 62.45.159.216:30303 3090a9103ba95cc45114efd410630eb9f95ac9f207bd7cc46d1a201649547390a16556ab21c7affae69173b0568fae4419e4dd01a60fe182dc56f446c1f54801 1057
2017/08/03 09:32:04 [discover] NEIGHBORS HANDLE FROM 62.45.159.216:30303 3090a9103ba95cc45114efd410630eb9f95ac9f207bd7cc46d1a201649547390a16556ab21c7affae69173b0568fae4419e4dd01a60fe182dc56f446c1f54801 425
2017/08/03 09:32:04 [discover] PING SEND TO 188.106.30.87:30303 127
2017/08/03 09:32:05 [discover] PING SEND TO 104.131.49.217:30303 127
2017/08/03 09:32:05 [discover] NEIGHBORS HANDLE FROM 178.134.252.152:30303 4f9a08798c269172d578cbf78100420387bea6714ad63d1dde1d872254b73beefb29cb072db188ae392f7789afd80eb28c921868852775bd1ade7e1cb0e68393 1057
2017/08/03 09:32:05 [discover] NEIGHBORS HANDLE FROM 178.134.252.152:30303 4f9a08798c269172d578cbf78100420387bea6714ad63d1dde1d872254b73beefb29cb072db188ae392f7789afd80eb28c921868852775bd1ade7e1cb0e68393 425

solution: use type switch (faster)

whilei · 2017-08-09T13:04:38Z

Closing in deference to #328, which includes and extends these changes.

whilei added 21 commits July 27, 2017 17:52

sketches for using API for logging... may be overkill

8effd93

may be worth thinking about more flexible and beautiful ways to display information... 'monitor', 'attach', etc, along with termui, graphs... output can be 'machiney'-- line-by-line, or human-y -- pictures of buffalo and whatnot.

💾

7abe6ca

problem: no reason to force RPC

8323600

solution: comment code and include small explanation

problem: no reason to return eth from StartNode

0fd6e5f

solution: remove exploratory code that is not useless

problem: warning log should use logger.Warn

80f7222

solution: move to Warn from Error

problem: log-pace flag usage antiquated

6f530fe

solution: update it and simplify

problem: minor variable renaming and comments

cdafd37

solution: improve legibility

problem: logging genesis account balances on Core log-level is too high

5e5f52c

solution: relegate to debug

problem: can share interrupt chan

2990308

solution: share chan in main between shutdown and closing status log goroutine

problem: logging to dir should also output to stderr

861d49b

solution: implement and glog SetAlsoToStdErr exported function and use in case of --log-dir

problem: log io flush never called after interrupt

05dca17

results in missing any logs after shutdown processes in FS logs solution: defer log Flush after shutdown services also decrease flushInterval from 30 -> 5 seconds (rate at which log buffer is written to FS)

problem: rename '--log-pace' to '--log-status'

027a8d0

solution: rename flag and flagvar and status log function also adds logstatus flag to correct category for usage output

problem: want --log-status to be extensible for other interfaces

a06f511

solution: refactor to use '--log-status='sync=60'' syntax, which can eventually support other features besides 'sync', eg. mine, net, disk

problem: (nonfunctional,refactor) should move cmd-oriented functions …

492c7ca

…to cmd.go solution: cut and paste and update imports

solution: run gofmt

a117f2f

problem: rollback command logging confirmation should check fastblock

af4d7e4

solution: error if neither fastblock nor fullblock match desired head number fixes ethereumproject#325

problem: (log syntax) log-status interval initial log still shows 'lo…

82ea60b

…g-pace' wording solution: fix

problem: begin proof-of-concept sketch for mlog

670b285

solution: implement initial spec with tentative mlog-dir and filename defaults peer 'handle' func sends mlogs to file rel ethereumproject#317

whilei mentioned this pull request Aug 1, 2017

[WIP]: standardize peer discovery log syntax #326

Closed

whilei added 5 commits August 1, 2017 17:31

problem: package sync imported and not used

762ae7a

solution: remove import

problem: implement peer discovery mlogging

892fe50

solution: refactor mlog from p2p to discover package - establish mlog label and detail structs Resolves ethereumproject#317

problem: refactor discovery mlogging for better concision and documen…

ab7669b

…tation solution: create vars for all available lines and comment them. not even sure if concision is a word Rel ethereumproject#317

problem: should use pattern to self-document mlog specs and set value…

da65759

…s by them solution: refactor details, implementing function to set details with variadic vals

problem: refactor discover mlogs for symmetry and simplicity

1a97d43

solution: use type switch filtering in send/handle functions instead of dispersed mlog calls 'farther down the food chain'

whilei mentioned this pull request Aug 3, 2017

Feature: machine-readable logging #328

Merged

3 tasks

whilei changed the title ~~[WIP]: machine-readable peer discovery logs~~ machine-readable peer discovery logs Aug 4, 2017

problem: can use type switch instead of sprintf

00fc0a4

solution: use type switch (faster)

whilei requested a review from splix August 7, 2017 14:43

whilei closed this Aug 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

machine-readable peer discovery logs #327

machine-readable peer discovery logs #327

whilei commented Aug 1, 2017 •

edited

whilei commented Aug 3, 2017

whilei commented Aug 9, 2017

machine-readable peer discovery logs #327

machine-readable peer discovery logs #327

Conversation

whilei commented Aug 1, 2017 • edited

whilei commented Aug 3, 2017

whilei commented Aug 9, 2017

whilei commented Aug 1, 2017 •

edited