-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
IPFS Performance #5226
Comments
@davinci26 Hi, this looks like a cool test to run! What are the axes in your graph? |
@ajbouh Oooops rookie mistake. Sorry about that, the X-axis is the number of IPFS nodes and the Y-axis is the average time required to download the file measured in minutes. The error bars are the standard deviation of the time. I can also link you the script I am using to perform the calculations if you want to reproduce the graph |
Do you mind providing an outline of the steps your benchmark takes? Is this essentially a graph of the time it takes for the Nth ipfs node to get the same file? Are the nodes all completely connected before for the first fetch? Or is node connection time also a part of the measured file fetch time? |
My simulation is as follows:
So each point in the graph is average time required to distribute the file to all nodes, and the X-axis is the number of nodes that try to get the file. The nodes are brand new, so I assume that they are not connected before. I am not sure about the answer to the second question. I am just timing the command |
It sounds like you might be measuring multiple things: time to be able to download any file, and then time to download the 10MB test file. In my experience, ipfs dameon takes different amounts of time finish starting up. I wonder what would happen if you "warmed up" each daemon by downloading a tiny 1-byte file, and then downloaded the 10MB file. I'd be interested in knowing what the time for each of these numbers is: time to boot and download the tiniest file, time to download the 10MB file. I'd also be interested in knowing the other statistics for measurements, like median, 90th percentile, etc. |
Ill try to do the above, and ill post here my findings!! |
Awesome, @davinci26. Looking forward to it! |
Also note, bitswap is currently very dumb when it comes to asking for chunks. It'll ask everyone so more nodes having the file can actually slow things down. |
@Stebalien Thanks for the info. I changed my approach and forked ipfs-network-test, hoping that I would be able to better results. Some observations that I made so far (I will post again when I finalize my simulation):
My end goal is to compare the IPFS distribution model against the Client-Server model for distributing medium size files. From the results, I have so far I don't expect IPFS to outperform the Client-Server model, but I expect to push the system to the limit and possibly discover some useful findings which I will post here. |
We're working on reducing the memory usage. This will involve:
|
@davinci26 great investigation! A few questions:
|
@ajbouh thanks!!
|
Thanks for explaining all that. I am not familiar with the low level
details of bbitswap.
Is the list of blocks that are sent randomized at all? Seems that if there
are enough other peers advertising the same blocks, we can request that
they send them in a random order.
After we get blocks from enough peers, we can prioritize the peers that
seem to be fastest and ask them to the blocks we need most/next.
Perhaps @whyrusleeping or @Stebalien have thoughts here?
The fancy move here would be to use forward error codes so that we can
collect *any* number of blocks from *all* peers and (with high probability)
they each would help fill in the parts of the file we're still missing.
On Jul 29, 2018 08:49, "davinci26" <notifications@github.com> wrote:
@ajbouh <https://github.com/ajbouh> thanks!!
-
What do you mean by "due to duplicate blocks"
As I understand the way bitswap works right now is as follows (maybe
someone can confirm this):
Alice publishes to network a want list of blocks. Then other users
participating in the exchange start transmitting data (bytes) to Alice.
When the transmission finishes Alices calculates the Hash of the data
received and check if it matches any of the blocks in her want list. The
problem is that as the number of users increases then multiple users send
Alice the same block. As a result, Alice spends her bandwidth to download
the same block multiple times causing the average time required to download
the file to increase. I understand that this not a trivial problem to fix.
-
Did you try a much smaller file?
I think the trend would be similar although I havent really simulated it
rigorously. For a smaller file, the number of blocks would be less and as a
result, you would have fewer duplicate blocks. The issue will remain but it
will be less visible.
-
Did you try waiting until the daemon had fully booted and fully
connected to other peers?
All nodes all fully loaded and connected and the time it takes to
connect and boot is not included in the graphs above. The simulation code
uses the IPTB framework and can be found here
<https://github.com/davinci26/iptb>. The function distCmd implements my
simulation and all the calculations.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#5226 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcnQQUc5y0ohsG1cO87smGIsE9Y6UVks5uLdmJgaJpZM4VPSHp>
.
|
@davinci26 very nice graphs, could you share how you made them? I would love to be able to use them as a way to benchmark improvements. On the duplicate blocks issue, We've actually laid the groundwork for fixing this issue with 'bitswap sessions', where we track which peers are likely to have a given piece of data based on information about who sends us other related pieces. The next step is to more precisely limit which peers we send those wants to, and track our hit rate (higher hit rate == more certainty, can send one want per peer, lower hit rate == maybe we should send wants to multiple peers still), response times, bandwidth estimators, and so on. We can implement a pretty simple 'want scheduler' (or better named thing) that will likely make a very noticeable impact. |
@whyrusleeping Just add one more step(have and request message as in bittorrent) between want and block sending. And do not need so many crutches. |
@whyrusleeping thanks for the bitswap explanation. I produced the data with my forked version of IPTB to make the file distribution simulation I can prepare a pull request to IPTB to add the graphs and the simulation to it. |
@ivan386 you could do that, but doing so adds an extra round trip to every request, and results in a non-trivial overhead for smaller blocks. There is room for some amount of 'have' style messages though, we could definitely add them for other related blocks as part of the message, or even more interestingly, return a bitfield along with each block that indicates which of that blocks children we have (at some non-zero expense to the side sending the blocks). @davinci26 thats really cool, i'd love to see what changes are needed in iptb to support that. We're actively working on improvements to it, this is the sort of thing we want :) |
Amazing work! but i didn't find the 'distCmd' function in the fork of IPTBhere, would you please point how to implement the simulation and get the data? |
Hope the above helps. |
@whyrusleeping |
The situation has significantly improved but there's still some work to be done. Sessions don't currently rank peers within the session to try to pick the fastest one. |
Could we get the tests that generated these graphs re-run? |
@whyrusleeping I have the code for the graphs in this fork of IPTB. This code is more than 6 months old. When I developed it I tried to make a PR for IPTB to add it. I agree with the IPTB folks that IPTB is not the best place for this code. With that said I still believe that there is value in these graphs/simulation for perf monitoring purposes. Feel free to include me in any discussion if you want to add this into your codebase. I would be happy to help :) |
Hey Folks - I believe more up-to-date metrics for this are here: #6782 (comment). We've made a lot of improvements to Bitswap over the past 6 months that are rolling out in our next release -- addressing this exact "many seeders/leachers" performance challenge, so closing this issue to redirect the conversation/evaluation there. =] |
Hey y'all,
I was testing IPFS performance using a script that implements the following logic:
I am using Go-IPFS deamon and I use and the python IPFS API to do the GET
============================= EDIT Updated Figure =============================
The X-axis is the number of IPFS nodes and the Y-axis is the average time required to download the file measured in minutes. The error bars are the standard deviation of the time
In terms of scalability I was expecting the time to be increasing logarithmic as more people request the file. The file has more seeders and more leechers.
Is this the expected performance of the system? Am I doing something wrong causing the performance of the system to deteriorate?
If this kind of results are something that interests the community, please let me know and I will provide you with the information.
The text was updated successfully, but these errors were encountered: