Unexpected network overhead: 1800%

[nanw1103]

Version information:

go-ipfs version: 0.4.13-
Repo version: 6
System version: amd64/windows
Golang version: go1.9.2

Type:

Enhancement

Severity:

Medium

Description:

My experiment scenario:
Setup 1: A private swarm with 30 IPFS nodes (bootstrap contains only the nodes in the network itself), in an isolated private network (LAN).
Setup 2: File A, size 10 MB, full random content without duplicated block at 256KB.

Step 1: Snapshot the current bandwidth usage of total network, by "ipfs stats bw" on all nodes, and sum the bytes-in.
Step 2: "ipfs add File_A" on node1
Step 3: "ipfs get File_A_hash" on all nodes
Step 4: The same as Step 1, snapshot the bandwidth usage of the network.
Step 5: Calculate delta between Step 4 and Step 1 to know the total network bandwidth cost.

Theoretically the ideal network transfer should be (30 - 1) * 10 = 290 MB. The actual result from Step 5 is 5339MB, which is about 1800%.

P2P for sure will have some overhead, however 1800% seems too high.

[nanw1103]

Further investigation to know whether it's an issue with the "ipfs stats bw" or the swap implementation: checked OS network card statistics. It matches (approximately) the report from "ipfs stats bw". So it seems some optimization possibilities in the block swap implementation.

[Stebalien]

Thanks for the detailed report!

Unfortunately, this is a known issue. You're probably seeing thisbecause some peers are downloading faster than others. When the lagging peers ask the network for the pieces of the file, they'll get it from everyone (not good). We're working on fixing this by using something we call bitswap sessions, see: #3786

You can actually check this by looking at the "dup blocks received" field of ipfs bitswap stat on each peer.

This could also be other traffic but that's unlikely. You can check that by looking at, e.g.:

> ipfs stats bw -t /bitswap/1.1.08

(and comparing it to other protocols)

---

Also note, ipfs stats bw is off by a factor of 5: #4476 (should be fixed next release).

---

To try to keep these conversations in one place, I'm going to close this in favor of #4588.

[nanw1103]

Close as Stebalien's comment.