[Merged by Bors] - Logging via the HTTP API #4074

AgeManning · 2023-03-13T04:43:44Z

This PR adds the ability to read the Lighthouse logs from the HTTP API for both the BN and the VC.

This is done in such a way to as minimize any kind of performance hit by adding this feature.

The current design creates a tokio broadcast channel and mixes is into a form of slog drain that combines with our main global logger drain, only if the http api is enabled.

The drain gets the logs, checks the log level and drops them if they are below INFO. If they are INFO or higher, it sends them via a broadcast channel only if there are users subscribed to the HTTP API channel. If not, it drops the logs.

If there are more than one subscriber, the channel clones the log records and converts them to json in their independent HTTP API tasks.

Workaround format_args! with closure

paulhauner · 2023-03-21T06:03:55Z

It looks like this one didn't make it, sorry Age :(

book/src/api-lighthouse.md

makru86

not sure if I am right or not.

AgeManning · 2023-03-27T00:41:23Z

@atokama - Thanks for the review. I'll change it to be consistent :).

paulhauner

Nice!

paulhauner · 2023-05-17T08:58:46Z

bors r+

This PR adds the ability to read the Lighthouse logs from the HTTP API for both the BN and the VC. This is done in such a way to as minimize any kind of performance hit by adding this feature. The current design creates a tokio broadcast channel and mixes is into a form of slog drain that combines with our main global logger drain, only if the http api is enabled. The drain gets the logs, checks the log level and drops them if they are below INFO. If they are INFO or higher, it sends them via a broadcast channel only if there are users subscribed to the HTTP API channel. If not, it drops the logs. If there are more than one subscriber, the channel clones the log records and converts them to json in their independent HTTP API tasks. Co-authored-by: Michael Sproul <micsproul@gmail.com>

bors · 2023-05-17T12:17:53Z

Timed out.

michaelsproul · 2023-05-17T12:24:08Z

bors retry

This PR adds the ability to read the Lighthouse logs from the HTTP API for both the BN and the VC. This is done in such a way to as minimize any kind of performance hit by adding this feature. The current design creates a tokio broadcast channel and mixes is into a form of slog drain that combines with our main global logger drain, only if the http api is enabled. The drain gets the logs, checks the log level and drops them if they are below INFO. If they are INFO or higher, it sends them via a broadcast channel only if there are users subscribed to the HTTP API channel. If not, it drops the logs. If there are more than one subscriber, the channel clones the log records and converts them to json in their independent HTTP API tasks. Co-authored-by: Michael Sproul <micsproul@gmail.com>

bors · 2023-05-17T15:37:11Z

Timed out.

paulhauner · 2023-05-17T23:11:02Z

It seems like the eth1-simulator-ubuntu is consistently timing out: https://github.com/sigp/lighthouse/actions/runs/5003142484/jobs/8964003366

I haven't looked into it so I'm not sure why.

Squashed commit of the following: commit 16771dc Author: Age Manning <Age@AgeManning.com> Date: Wed May 17 15:27:47 2023 +1000 fmt commit 1deeafe Merge: 9f5ca3d b29bb2e Author: Age Manning <Age@AgeManning.com> Date: Wed May 17 15:26:13 2023 +1000 Merge latest unstable commit 9f5ca3d Author: Age Manning <Age@AgeManning.com> Date: Wed May 17 15:24:06 2023 +1000 Enable SSE logs only on GUI flag and misc reviewers comments commit 5eeca70 Merge: 3efdbe2 d64be0d Author: Age Manning <Age@AgeManning.com> Date: Tue May 9 10:08:05 2023 +1000 Merge latest unstable commit 3efdbe2 Author: Age Manning <Age@AgeManning.com> Date: Mon May 8 17:13:12 2023 +1000 fmt commit 202aad8 Author: Age Manning <Age@AgeManning.com> Date: Wed May 3 16:30:19 2023 +1000 Remove api token requirement for sse logs commit 5411d5a Author: Age Manning <Age@AgeManning.com> Date: Wed May 3 15:57:59 2023 +1000 Remove double json encoding commit 29fb985 Author: Age Manning <Age@AgeManning.com> Date: Wed May 3 14:32:31 2023 +1000 fmt commit 3e10b26 Merge: 37ec9b7 826e748 Author: Age Manning <Age@AgeManning.com> Date: Wed May 3 14:30:41 2023 +1000 Merge latest unstable commit 37ec9b7 Author: Age Manning <Age@AgeManning.com> Date: Tue Mar 14 16:25:45 2023 +1100 Revert old clippy change commit b3d0167 Author: Age Manning <Age@AgeManning.com> Date: Tue Mar 14 14:45:31 2023 +1100 Appease clippy commit 1df3191 Author: Age Manning <Age@AgeManning.com> Date: Tue Mar 14 14:08:28 2023 +1100 Fix tests commit fc7645e Author: Age Manning <Age@AgeManning.com> Date: Tue Mar 14 13:43:52 2023 +1100 Fix logging tests via tokio upgrade commit 4988a15 Merge: 6cf9338 90cef1d Author: Age Manning <Age@AgeManning.com> Date: Mon Mar 13 15:47:26 2023 +1100 Merge latest unstable commit 6cf9338 Merge: 5deabe0 3ad77fb Author: Age Manning <Age@AgeManning.com> Date: Mon Mar 13 15:39:51 2023 +1100 Merge latest unstable commit 5deabe0 Author: Age Manning <Age@AgeManning.com> Date: Mon Mar 13 15:39:45 2023 +1100 fmt commit 5f24f65 Author: Age Manning <Age@AgeManning.com> Date: Mon Mar 13 15:38:36 2023 +1100 A working version commit 4da3351 Author: Age Manning <Age@AgeManning.com> Date: Thu Mar 9 13:44:22 2023 +1100 Tokio broadcast version commit e34aab2 Merge: cd6edbc 9d821ce Author: Age Manning <Age@AgeManning.com> Date: Thu Mar 9 13:17:38 2023 +1100 Merge pull request #3 from michaelsproul/sse-logs Workaround format_args! with closure commit 9d821ce Author: Michael Sproul <micsproul@gmail.com> Date: Thu Mar 9 11:57:51 2023 +1100 Workaround format_args! with closure commit cd6edbc Author: Age Manning <Age@AgeManning.com> Date: Thu Mar 9 11:18:02 2023 +1100 Getting closer commit e9eb455 Author: Age Manning <Age@AgeManning.com> Date: Wed Mar 8 16:38:58 2023 +1100 Crossbeam version commit b688cfb Author: Age Manning <Age@AgeManning.com> Date: Tue Mar 7 14:56:17 2023 +1100 More temp commits commit 228b930 Merge: 035ca37 17d9a62 Author: Age Manning <Age@AgeManning.com> Date: Wed Mar 1 13:05:45 2023 +1100 Merge latest unstable commit 035ca37 Author: Age Manning <Age@AgeManning.com> Date: Wed Feb 8 20:24:39 2023 +1100 Temp progress

paulhauner · 2023-05-22T05:56:52Z

eth1-simulator-ubuntu passed 🎉

bors r+

This PR adds the ability to read the Lighthouse logs from the HTTP API for both the BN and the VC. This is done in such a way to as minimize any kind of performance hit by adding this feature. The current design creates a tokio broadcast channel and mixes is into a form of slog drain that combines with our main global logger drain, only if the http api is enabled. The drain gets the logs, checks the log level and drops them if they are below INFO. If they are INFO or higher, it sends them via a broadcast channel only if there are users subscribed to the HTTP API channel. If not, it drops the logs. If there are more than one subscriber, the channel clones the log records and converts them to json in their independent HTTP API tasks. Co-authored-by: Michael Sproul <micsproul@gmail.com>

bors · 2023-05-22T08:47:11Z

Pull request successfully merged into unstable.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

This PR adds the ability to read the Lighthouse logs from the HTTP API for both the BN and the VC. This is done in such a way to as minimize any kind of performance hit by adding this feature. The current design creates a tokio broadcast channel and mixes is into a form of slog drain that combines with our main global logger drain, only if the http api is enabled. The drain gets the logs, checks the log level and drops them if they are below INFO. If they are INFO or higher, it sends them via a broadcast channel only if there are users subscribed to the HTTP API channel. If not, it drops the logs. If there are more than one subscriber, the channel clones the log records and converts them to json in their independent HTTP API tasks. Co-authored-by: Michael Sproul <micsproul@gmail.com>

AgeManning and others added 12 commits February 8, 2023 20:24

Temp progress

035ca37

Merge latest unstable

228b930

More temp commits

b688cfb

Crossbeam version

e9eb455

Getting closer

cd6edbc

Workaround format_args! with closure

9d821ce

Merge pull request #3 from michaelsproul/sse-logs

e34aab2

Workaround format_args! with closure

Tokio broadcast version

4da3351

A working version

5f24f65

fmt

5deabe0

Merge latest unstable

6cf9338

Merge latest unstable

4988a15

AgeManning added ready-for-review The code is ready for review v4.0.0 Mainnet Capella release expected late March 2023 labels Mar 13, 2023

AgeManning added 2 commits March 14, 2023 13:43

Fix logging tests via tokio upgrade

fc7645e

Fix tests

1df3191

AgeManning requested review from michaelsproul and paulhauner March 14, 2023 03:09

Appease clippy

b3d0167

paulhauner self-assigned this Mar 14, 2023

Revert old clippy change

37ec9b7

paulhauner added v4.1.0 Post-Capella minor release and removed v4.0.0 Mainnet Capella release expected late March 2023 labels Mar 21, 2023

makru86 reviewed Mar 24, 2023

View reviewed changes

book/src/api-lighthouse.md Show resolved Hide resolved

makru86 reviewed Mar 24, 2023

View reviewed changes

paulhauner added v4.2.0 Q2 2023 and removed v4.1.0 Post-Capella minor release labels Apr 13, 2023

Merge latest unstable

3e10b26

Enable SSE logs only on GUI flag and misc reviewers comments

9f5ca3d

AgeManning added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels May 17, 2023

AgeManning added 2 commits May 17, 2023 15:26

Merge latest unstable

1deeafe

fmt

16771dc

paulhauner approved these changes May 17, 2023

View reviewed changes

paulhauner added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels May 17, 2023

paulhauner added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-merge This PR is ready to merge. labels May 18, 2023

paulhauner mentioned this pull request May 19, 2023

[Merged by Bors] - v4.2.0 #4309

Closed

Update cargo lock

8c76cbc

bors bot changed the title ~~Logging via the HTTP API~~ [Merged by Bors] - Logging via the HTTP API May 22, 2023

bors bot closed this May 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Logging via the HTTP API #4074

[Merged by Bors] - Logging via the HTTP API #4074

AgeManning commented Mar 13, 2023

paulhauner commented Mar 21, 2023

makru86 left a comment

AgeManning commented Mar 27, 2023

paulhauner left a comment

paulhauner commented May 17, 2023

bors bot commented May 17, 2023

michaelsproul commented May 17, 2023

bors bot commented May 17, 2023

paulhauner commented May 17, 2023

paulhauner commented May 22, 2023

bors bot commented May 22, 2023

[Merged by Bors] - Logging via the HTTP API #4074

[Merged by Bors] - Logging via the HTTP API #4074

Conversation

AgeManning commented Mar 13, 2023

paulhauner commented Mar 21, 2023

makru86 left a comment

Choose a reason for hiding this comment

AgeManning commented Mar 27, 2023

paulhauner left a comment

Choose a reason for hiding this comment

paulhauner commented May 17, 2023

bors bot commented May 17, 2023

michaelsproul commented May 17, 2023

bors bot commented May 17, 2023

paulhauner commented May 17, 2023

paulhauner commented May 22, 2023

bors bot commented May 22, 2023