Design Spec for a Sharding Visualization Front-End

[rauljordan]

The goal of this potential project is to create a web front-end that visualizes a sharded Ethereum blockchain network. Could be built as an extension to ethstats or as its own, independent interface.

Requirements / Acceptance Criteria

• Ability to inspect transaction load on n number of shards
• Ability to visualize cross-shard interactions
• Ability to see number of nodes and distribution of nodes across shards
• See collations happening in each period for each shard
• Ability to inspect the size of canonical shard chains

Cool to Have

• Some way of visualizing the actual p2p network topology

Alternative

Create a command line tool to visualize a sharded network. Can be baked into a sharding client implementation and could be a combination of ASCII art and text to visualize the different information pertaining to each shard.

This is a long, multi-step project, so at this point we are only looking for a design spec with possible visuals of how the front-end would look or how ethstats can be extended to accommodate these requirements.

The necessary reading to understand the context of the requirements and basic terms around sharding can be found in our project's README. Many sections, however, are deprecated while a minimal sharding protocol is still being researched.

We see this as a good first bounty for Prysmatic Labs.

[gitcoinbot]

This issue now has a funding of 0.31 ETH (159.32 USD @ $513.94/ETH) attached to it.

• If you would like to work on this issue you can claim it here.
• If you've completed this issue and want to claim the bounty you can do so here
• Questions? Get help on the Gitcoin Slack
• $4963.8 more Funded OSS Work Available at: https://gitcoin.co/explorer

[vs77bb]

Hi @justpixel! Could you please comment with your plan of action for this ticket / any clarifying questions you might have? Excited to see your interest.

[justpixel]

@vs77bb @rauljordan I think the first step is to build a wireframe. Do you use any IM where we can chat?
Thanks.

[prestonvanloon]

@justpixel join us on gitter

https://gitter.im/prysmaticlabs/geth-sharding

[gitcoinbot]

@justpixel are you still working on this issue?

• warning 1 (3 days)
• [] warning 2 (6 days)
• auto removal (10 days)

[justpixel]

@rauljordan @prestonvanloon
I've made this wireframe with a dashboard of widgets, each widget representing a requirement:

• Transaction load / shard - Ability to inspect transaction load on n number of shards
• Cross-shard interactions - Ability to visualize cross-shard interactions
• Number of nodes / shards - Ability to see number of nodes and distribution of nodes across shards
• Collations / shard - See collations happening in each period for each shard
• Canonical shard chains - Ability to inspect the size of canonical shard chains

Is this a good starting point?

[jamesray1]

This looks good! I think what would also be good to see is a geographical distribution of nodes e.g. like https://www.ethernodes.org, where you can select by a shard, or it's colour-coded by different shards, and brightness could be how dense an area is. And as mentioned above, "Some way of visualizing the actual p2p network topology" would be nice, but that might be tricky until the p2p sharding protocol is more well-developed.

[prestonvanloon]

@justpixel Can you elaborate on why this is a good design?
I'm curious how the data will be represented in a meaningful way.

Transaction load / shard: How is this quantified? Is the average fullness of last N blocks for each shard? There will be around 100 shards, possibly more in the future. While these widgets look nice in the wireframe you have presented, have you considered how this will look for all shards?

If you are looking for some inspiration or examples of existing dashboards/charts:

• https://etherscan.io/charts
• https://amberdata.io/
• https://ethstats.net/
• https://www.etherchain.org/charts

@jamesray1 has some great feedback. VIsualizing the p2p sharded network is a really nice thing to have.

If you are looking for general feedback on the aesthetics of your widgets wireframe; I think these color choices are nice.

[terencechain]

@justpixel What does the color represents for Collations / shard widget?

[jamesray1]

Oh, another note on the colours. Pastel colours are less distracting than saturated colours.

Suppression of the eye’s scanning movements can fixate a strongly colored image on the retina. This temporarily reduces the sensitivity of the photoreceptors, leading to afterimages... For this reason, it’s unwise to use large areas of bright color in a display.

p. 21, http://www.cs.tufts.edu/comp/177/docs/ColorInGraphics.pdf

I agree that representing 100 shards or more at once may be difficult. Being able to visually represent a single shard may be easier; although cross-shard transactions complicate matters, but we don't need to worry about that until phase 4.

[justpixel]

@prestonvanloon @jamesray1 I'll be away for two weeks, so if it's anybody that wants to continue the work from here feel free to jump in.

[ConnorChristie]

I took a stab at a design for information at the period level per each shard and I came up with this timeline concept that allows the shard to be selected and the period.

Not sure whether there will ever be more than 5 periods but if so then that could probably switch to a dropdown as well.

I am unsure of the bubbles on top, they would most likely go underneath as some sort of short stats for that block and shard. As for the details under the period, that could be expanded to the whole screen and show graphs but I am unsure about the specifics of this whole sharding thing so I didn't know what to put in there...

[jamesray1]

Nice! The expectation is that the number of shards will be constant at around 100 initially, although they may grow over time. So being able to not just pick a block and see how many shards it has, but see a range of blocks where there was X number of shards, then the next range where the number of shards increased, and so on. But I suggest not implementing that visualization until we have a better idea of shard numbers and dynamics. For each shard you could have a visualization of a sparse binary Merkle tree, and maybe stats like ETH Gas station (although that could be done by an external provider, like ETH Gas Station) e.g. on gas burners and a tx pool.

Also see this, it has a visualization of network topology across shards and within shards: https://twitter.com/VitalikButerin/status/991021062811930624.

[ConnorChristie]

Here is an alternative mockup (more adminish), this one has much more detail and shows an overview of each shard's statistics. Another page will probably come soon after this detailing the information around a specific shard.

[ConnorChristie]

Here are two different graphs that I believe can fit the role of displaying cross-shard interactions, let me know which one seems better!

Here is a link to the mockups if you'd like to interact with them directly: https://connorchristie.github.io/geth-shard-design/

[rauljordan]

Really great mockup @ConnorChristie ! I like the graph on the left for cross-shard tx's as it is easier for me to see the links between shards. I imagine the graph on the right would get pretty hairy the more shards are added and it could become difficult to visualize the edges. Thanks for putting this together, this is awesome.

[terencechain]

Great job @ConnorChristie. I prefer the left one as it's more manageable when shards get large. Do you have a mockup to display the stats of the individual shard? Everything else looks really good!

[nisdas]

It looks really good @ConnorChristie , I like how you have all the shard metrics on display. It looks really intuitive for the user. Also the one on the left looks better for cross-shard visualization. Great Job!

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work has been started.

1. @ConnorChristie

has committed to working on this project to be completed 11 months, 2 weeks from now.

[rauljordan]

Final Requirements for Closing the Bounty

Hi @ConnorChristie and all, as a final requirement to close this bounty, we'd like you to answer the following user stories in order to understand how your design offers efficient solutions to them.

1. Bob wants is thinking about deploying a new contract that he expects to have high traffic. Bob wants a shard with mostly non-full collations so he can take advantage of lower gas fees. How can your design accomplish this?

2. Alice is a user of Cryptokitties. She wants to explore the cross-shard transactions in the shard the Cryptokitties contract lives in. What are some ways she can achieve this?

3. Bob is interested in seeing the performance of the Sharding Manager Contract's functions on the main chain. He should be able to visualize the load on this contract via the shard dashboard and see which shards have the most function calls to this contract.

4. Alice wants to check how much storage it would require to download canonical shard chains. How can she visualize how big the state of the shards is? Keep in mind each collation is 1 Mb in size.

5. Bob is interested in exploring the security of the sharded system against attacks. He wants to see how much ETH is staked into the Sharding Manager Contract on the main chain and how much of this ETH is staked by notaries participating on a particular shard. What could be a good way to visualize this?

[ConnorChristie]

I have just put some more work into the mockups to allow these use cases to be performed! Much of the inspiration came from etherscan so if it looks kinda like etherscan, that's why. I however, tried to add more graphs :)

I have included a few screenshots below but I would encourage you to interact with the live version here: https://connorchristie.github.io/geth-shard-design/

Answers to the user stories above:

1. The shard listing can have an option to sort by "Transactions / Blocks" for each shard which defines the proportion of txns that shard processed in the last n blocks.

2. She can figure out which shard the contract lives in by going to its address page. Then she may click on the shard and see a list of all collations for that shard. Each collation displays how many cross-shard transactions it has. (Also by looking at the graph on the shard page)

3. Not entirely sure how to show that. Could they just look at the contract address and see all inbound transactions from the different shards? Ideas would be appreciated for this!

4. She can look at the total number of collations for a shard (on the shards listing page) and translate that number directly into the number of Mb that shard takes up.

5. He may navigate to the shard detail page and see the current collators for that shard. Each collator in the list has their stake listed.

It should be noted that collators are now called notaries along with the collations possibly being labeled by the period number. The notary pool and publisher should also be distinguished in the design, ideally on the shard detail page.

Collation detail page. Unsure of whether to include cross-shard transactions in the main list since you want the details about which shard the receipt is going to and the transaction created for that receipt on the other shard.

Since a transaction can contain a receipt and send it as txn to another shard, I tried to symbolize that here with the main transaction labeling which shard and address it is going to and then include the supplementary txn in the "Cross-Shard Transactions" list.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work for 0.31 ETH (251.56 USD @ $811.47/ETH) has been submitted by:

1. @ConnorChristie

@vs77bb please take a look at the submitted work:

• (Link Not Provided) by @ConnorChristie

---

• Learn more on the Gitcoin Issue Details page
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
  $9,983.26 more funded OSS Work available on the Gitcoin Issue Explorer

[jamesray1]

She can look at the total number of collations for a shard (on the shards listing page) and translate that number directly into the number of Mb that shard takes up.

That would be an upper bound on the MB, assuming all collations are full.

[rauljordan]

Hi @ConnorChristie, we really like your responses to the questions and the work you put together. Great job! We are happy to now close this bounty and award the ETH to you.

[ConnorChristie]

@rauljordan If there are any other use case workflows you would like me to consider, I would be happy to adapt this design to them! I have also been thinking about adding a graphic regarding the actual network topology with node connectivity among the different shards.

At first glance of this issue, it didn't seem like you were necessarily looking for a user facing design but rather a more developer and internal focused design. Because of this, let me know if there are any more internal tools you'd like me to add.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

The funding of 0.31 ETH (227.49 USD @ $733.85/ETH) attached to this issue has been approved & issued to @ConnorChristie.

• Learn more at on the Gitcoin Issue Details page
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $8,626.81 more funded OSS Work available on the Gitcoin Issue Explorer