Output Of Local Example Different Than README + Help Understanding Values

[bonedaddy]

I ran the benchmark locally, and the returned output is slightly different than what is displayed in the README. Obviously the values themselves will be different due to being tested on different machines, etc.. However some values are filled in for my result, and they are missing in the readme example. Primarily in my run, the bitswap nodes all have some value for BLOCKSRECV vs the readme which only has 1 node with value for BLOCKSRECV.

Is there some sort of documentation within the codebase explaining how the values are calculated? Thanks.

My example:

# Bandwidth
+-------------------+----------------------+---------+----------+---------+---------+
|       QUERY       |         NODE         | TOTALIN | TOTALOUT | RATEIN  | RATEOUT |
+-------------------+----------------------+---------+----------+---------+---------+
| (not 'neighbors') | bp6qj8t85vcoqnd7imu0 | 442 MB  |  226 kB  | 73 MB/s | 37 kB/s |
+-------------------+----------------------+---------+----------+---------+---------+
|         -         | bp6qj8t85vcoqnd7imug | 1.3 kB  |  203 MB  |  0 B/s  | 36 MB/s |
+                   +----------------------+         +----------+         +---------+
|                   | bp6qj8t85vcoqnd7imv0 |         |  252 MB  |         | 37 MB/s |
+-------------------+----------------------+---------+----------+---------+---------+
|                            TOTAL         | 442 MB  |  455 MB  | 73 MB/s | 73 MB/s |
+-------------------+----------------------+---------+----------+---------+---------+

# Bitswap
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
|       QUERY       |         NODE         | BLOCKSRECV | BLOCKSSENT | DUPBLOCKS | DATARECV | DATASENT | DUPDATA |
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
| (not 'neighbors') | bp6qj8t85vcoqnd7imu0 |   1,887    |     0      |     5     |  486 MB  |   0 B    | 531 kB  |
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
|         -         | bp6qj8t85vcoqnd7imug |   1,227    |    873     |    28     |  313 MB  |  225 MB  | 4.2 MB  |
+                   +----------------------+------------+------------+-----------+----------+----------+---------+
|                   | bp6qj8t85vcoqnd7imv0 |   1,199    |   1,075    |     0     |  309 MB  |  274 MB  |   0 B   |
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+

Now compare this with the one from the readme:

# Bandwidth
+-------------------+----------------------+---------+----------+----------+----------+
|       QUERY       |         NODE         | TOTALIN | TOTALOUT |  RATEIN  | RATEOUT  |
+-------------------+----------------------+---------+----------+----------+----------+
| (not 'neighbors') | bp3ept7ic6vdctur3dag | 397 MB  |  204 kB  | 106 MB/s | 54 kB/s  |
+-------------------+----------------------+---------+----------+----------+----------+
|         -         | bp3eptfic6vdctur3db0 |  96 kB  |  188 MB  | 26 kB/s  | 52 MB/s  |
+                   +----------------------+---------+----------+----------+----------+
|                   | bp3eptvic6vdctur3dbg | 109 kB  |  212 MB  | 28 kB/s  | 55 MB/s  |
+-------------------+----------------------+---------+----------+----------+----------+
|                            TOTAL         | 397 MB  |  400 MB  | 106 MB/s | 107 MB/s |
+-------------------+----------------------+---------+----------+----------+----------+

# Bitswap
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
|       QUERY       |         NODE         | BLOCKSRECV | BLOCKSSENT | DUPBLOCKS | DATARECV | DATASENT | DUPDATA |
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
| (not 'neighbors') | bp3ept7ic6vdctur3dag |   1,866    |     0      |     2     |  481 MB  |   0 B    | 263 kB  |
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
|         -         | bp3eptfic6vdctur3db0 |     0      |    888     |     0     |   0 B    |  228 MB  |   0 B   |
+                   +----------------------+            +------------+           +          +----------+         +
|                   | bp3eptvic6vdctur3dbg |            |    978     |           |          |  252 MB  |         |
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+
|                            TOTAL         |   1,866    |   1,866    |     2     |  481 MB  |  481 MB  | 263 kB  |
+-------------------+----------------------+------------+------------+-----------+----------+----------+---------+

[hinshun]

This is just a rendering feature from the table writer: https://github.com/olekukonko/tablewriter
Rows that have the same value have the cells merged. i.e. in the README's benchmark the neighbor nodes both received 0 blocks and 0 duplicate blocks, whereas in your run the neighbor nodes received a few duplicate blocks.

[bonedaddy]

Ah okay, makes sense. Thanks.

[hinshun]

@bonedaddy Btw, I added an additional section on benchmarking the quic libp2p transport: https://github.com/Netflix/p2plab#live-updating-the-cluster. More to come later.

[bonedaddy]

@hinshun Thanks! Another question I have is determining what kind of results are better than others. For example in the bitswap tests do we want BLOCKSRECV to be equal across all nodes, and as low as possible? For the bandwidth tests do we want the fastest mb/s as possible?

Thanks.

[hinshun]

It depends on your scenario and what you care about. In this specific example, you probably want:

• Minimize total time for all (not 'neighbor') nodes to receive the DAG
• Minimize total DUPBLOCKS, aka minimize BLOCKSRECV
• Minimize BLOCKSRECV in neighbor nodes (they have the full DAG so they shouldn't need any blocks)
• Balance the BLOCKSSENT in neighbor nodes (so load is distributed)

That said, the bitswap implementation should perform well generally. So if a change in bitswap improves one scenario but greatly slows down in another, then it may not be acceptable. In practice, you probably want to write a suite of scenarios, assign a weight to each scenario, and then compute a final score.

[bonedaddy]

Makes sense, thanks.