feat(debug-page): Proof of concept

[jakmeier]

A very basic display website hosted on /debug showing some live-data about the node.

For now, it just shows a (very poorly visualized) list of all running generator tasks with their id, age, and number of pokes so far.

Each task gets a DebugPageTaskHandle by calling register_task. The handle can be used to update the task's representation on the debug page. Tasks currently render their info eagerly in their main loop. The debug page itself then only collects all the info sent by tasks previously.

Rendering all tasks on-demand might be better for performance. But I didn't want to clutter users of a DebugPageTaskHandle (e.g. struct TripleGenerator) with too much code. This can be optimized later with different types of handles that render in different ways.

I have chosen https://maud.lambda.xyz/ for its html! macro to render information into HTML right inside Rust. I'm not sure if this is a good approach. But it gets us started and can be improved upon.

The debug page is entirely disabled unless the node is compiled with --features=debug-page.

When using it in component tests, it will host each node in :8200/debug, :8201/debug, ...

To make register_task easily accessible, without passing around a reference to the registry, I'm using some global state. It's split by account id to make it work with component tests. See comments in code for more details.

[jakmeier]

As I wrote above, it's very poorly visualized 😅

Here is an example:

[jakmeier]

@ppca @ChaoticTempest @volovyks this is what I have so far in terms of a debug page.

Take a look if you are curious.

We can already merge this in if you want to start using and improving it. Otherwise, I will clean it up some more next week and we can merge it afterwards.

[volovyks]

Nice addition to our development flow!
Looks good enough to be merged.

P.S. Why do you use fork? It makes testing changes harder for reviewers.

[ChaoticTempest]

Looks pretty good! I do wonder if we can pipe tokio-console output to this as well so we can get raw task statistics.

This is currently only being ran from one node right? Being able to display multiple nodes would be an interesting to see and how we can potentially crossweave info. Like the service can see that for the same protocol ID, it failed to posit or generate for whatever reason.

[jakmeier]

P.S. Why do you use fork? It makes testing changes harder for reviewers.

@volovyks No killer reason. But I have clear thoughts on why I prefer forks:

• Not needing write access in my workflow to submit a PR is nice. It allows for better repository protection, plus it means I can use the same workflow for all projects I contribute to on GitHub.
• Not using forks pollutes the main repo. Especially when developers are not cleaning up their branches regularly. git branch -a | grep origin | wc -l tells me I'm tracking 244 branches on origin and it's only growing. Not a big issue since our current number of developers is small. But it doesn't scale well. Not only does it make it harder to find my own branches, it also uses more bandwidth for a git fetch, which I actually care about when working on the go.
• Everyone using forks also ensures our CI works for PRs from external contributors. This is more or less important, depending on how important being "truly" open-source is to a project. It's completely okay to have source code licences under an open-source license without being particularly welcoming towards external contributors. But if you want to encourage external contributions, I find it a good idea to make no technical difference in workflows for internal and external contributors.

That's why I default to the fork workflow on GitHub.

It makes testing changes harder for reviewers.

Yeah since forks are a GitHub construct, the vanilla git isn't greatly integrated.

Have you tried gh cli? If you install it, you can use the commands on the top right of each PR to checkout the code. In this case, gh pr checkout 553.

You can also fetch PRs with plain github commands though.

git fetch origin pull/553/head:pr-553
git checkout pr-553

But personally, I run git remote add ${username} ${url} for people I work together frequently and then I can pull directly from their remote.

All this said, if this slows you down, I can easily change my workflow for this project. I seem to be the only person using forks, so I should probably adapt rather than the rest of the team. :P

[jakmeier]

Looks pretty good! I do wonder if we can pipe tokio-console output to this as well so we can get raw task statistics.

Yeah might be a good idea. Have you successfully use tokio-console on this project already?

This is currently only being ran from one node right? Being able to display multiple nodes would be an interesting to see and how we can potentially crossweave info. Like the service can see that for the same protocol ID, it failed to posit or generate for whatever reason.

Hm, what you describe sounds like it should be a separate service. We will also need that, at some point. For example for an explorer. But here I'm not trying to build a system monitoring debug page, just a node monitoring debug page.

[jakmeier]

Fixed clippy warning, I guess we can merge it as is and I'll send improvements separately.

[volovyks]

@jakmeier on using forks:

Not needing write access in my workflow to submit a PR is nice. It allows for better repository protection, plus it means I can use the same workflow for all projects I contribute to on GitHub.

No pressure, it is not hard for me to adjust to your flow. Native Git commands seams the best option. Now we prefix branchse with our names (like serhii/<feature-name>). Prefixing them with repo is not that big of a change.

Not using forks pollutes the main repo. Especially when developers are not cleaning up their branches regularly. git branch -a | grep origin | wc -l tells me I'm tracking 244 branches on origin and it's only growing. Not a big issue since our current number of developers is small. But it doesn't scale well. Not only does it make it harder to find my own branches, it also uses more bandwidth for a git fetch, which I actually care about when working on the go.

Valid point. Our current setup removes the branch after PR is merged. But we have a ton of stale branches. I've removed mine. @ChaoticTempest, @ppca, @auto-mausx , and @DavidM-D, please remove yours.

What cought my utension is security. I do not mind using this approach in the team, but we need to discuss it.