IPNS publishing performance #3
Comments
|
Running IPFS v .15 on Debian based system When I try to publish it takes over ~2 minutes I have tried with no change the following
Example: Publish takes ~ 2 minutes Example: Request of IPNS content takes 2 mins However another user trying this had no problems |
|
@darkdrgn2k you are starting the |
|
Could be a second or an hour does not have impact on it. I tried a day later with same results. Even after the first successfull try second one still takes as long too. |
|
@flyingzumwalt shall we link this into a ipfs repo? |
|
Hey @benhylau, Just now seeing this issue. To get good latencies, you will want to use the ipns over pubsub. There are some docs here: https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#ipns-pubsub Also see the You will need to have this feature enabled on both the publishers and the resolvers (your gateway nodes, ideally) and they will need to be connected in some sort of topology (to receive pubsub messages from a publisher, you have to be connected through some graph of subscribed nodes). Both @lgierth and @flyingzumwalt are out on vacation right now, so you can ping me if you need anything until they are back. |
|
@whyrusleeping we looked at this a while ago but it does not seem to have any effect. |
|
@darkdrgn2k ah, yeah. Under the hood it publishes to both pubsub and the slow DHT. subscribers should receive updates very quickly (less than a second ideally). We should probably add an option to publish only to pubsub, but doing that prevents non-pubsub users from resolving the content (the DHT is a fallback). If you just want to test out the pubsub stuff working quickly, you could add a |
|
How are remove clients subscribed? Is it just enabling it in the client I would assume if we are asking for the data from a gateway they need to do it a well? And ipfs. Io probobly doesn't run experimental flags? Also anothe user is able to publish via regular ipns in a fraction of the time What contributes to ipns speed? |
|
Yeah, if the client has the feature enabled, they will use it. The first resolve will still be slow, as thats when the subscription starts, and since it doesnt have any value at the beginning of the subscription it has to fall back to the DHT, but subsequent updated lookups will be fast. If you are asking for data from the gateway, the gateway will need this feature enabled. And yes, our ipfs.io gateways do not run experimental flags, though you might be able to ask @lgierth nicely when he comes back ;) |
Standard (non-pubsub) IPNS is a simple DHT 'Put' operation. The main reason it is slow is that connectivity across the network is rather poor right now (too many clients behind NATs). The way that our DHT (Kademlia) works, is that it selects the K (we use 20, as suggested by the paper) closest peers by XOR metric to the key you are publishing to, and sends them the value. With how many badly NATed clients we're seeing, the majority of these dials are timing out, causing the 'Put' operation to be slow. To address this, we are working on a few different things, but the two main ones are; first: an option for clients to auto-detect if they are behind a bad NAT, and not participate in the DHT if so. Second: better relay logic, so we can ask other peers to relay connection for us to these peers that are NATed. |
|
Thanks for the explanation. If I understand correctly currently it is pure luck if IPNS will be fast or slow. If i have a closed network (CJDNS) that has no access to the "Internet". Would the CJDNS nodes be populated with unreachable Internet IP addresses as well? And as such still cause the IPNS delays we have on the regular network. |
Yes, as things stand now, its pretty much just luck that determines DHT publishing speed. If you ran ipfs entirely on a cjdns network (with no egress to the normal internet), then you wouldnt have a problem. The current issues are only happening because so many users are running ipfs nodes at home without any NAT traversal or port forwarding. |
|
IPNS over PUBSUB seems to be the way to go after all. We timeout the publish so it doesn't spend time doing DHT First grab of the IPNS is still a bit slow. But I think it is only the "first" person pulling from that gateway. |
|
@darkdrgn2k before we close this one:
Thanks for the help @whyrusleeping :) |
|
@benhylau Not a problem :) Glad to hear things are working, please please please let me know how it goes or if anything goes wrong. I'll be watching the stream when its up! |
|
I have been running the stream for several hours now work well Only issue so far is that the ipns sets a cache of 1 min creating a "stall" of new segments. |
|
@darkdrgn2k ah, interesting. That happens only on new streams, right? things run smoothly once it gets started? The one minute timeout is a server-side namesys cache. I think we should actually not be using that cache if we're using the pubsub routing (as the pubsub router effectively maintains its own cache). I'm gonna call this one a bug and file it for you. |
|
Spinning up a production rig for the stream we found some issues with pubsub name. Background We publish the m3u8 text file every ~ 15 seconds (after every new chunk is created and added to ipfs) We add them as such We found that IPNS would at times become stunned for a little while before updating the list. Other Times it would seem to work decently. Few things to possibly consider
After switching to a plain http hosed version of the m3u8 file things seem to stream much better. |
|
First we need to move the IPNS publishing into a separate process. I am doing it manually to observe the results, using
A publish-resolve run looks like this: Let's see across multiple nodes: If we put in those 15s timeouts in both publish and resolve: I think the way forward is to publish less frequently, and more aggressively resolve, such that most of the time resolutions by the player are fast-returning. A few things will trip up the player:
So we should aim for minimal downtime and have playlist always available in < 1s. |
|
Sometimes we'd run into this bug where on one node you resolve: And on another node (in this case the mirror): It is not a transient thing, |
|
@benhylau very interesting... I wonder if its a connectivity thing. If a pubsub ipns node gets disconnected from the swarm of the publisher, it won't be able to resolve until they get connected again (its not receiving any updates). Could you verify with |
|
@whyrusleeping we unfortunately have to remove IPNS dependency for v1.0 which is the tag we intend to use for Our Networks. Perhaps we can have a sprint with @lgierth to debug these IPNS issues we are experiencing :) |
|
@benhylau i'm pretty sure I know the issue now. The pubsub code doesnt add any connection manager tags, so ipfs doesnt know to keep connections to pubsub topic friends alive, meaning they will randomly be closed. I'm gonna try and hack that in this week and we can see how it helps. |
|
This will re-enable IPNS for testing purposes: https://github.com/tomeshnet/ipfs-live-streaming/tree/enable-ipns |
IPNS publishing takes more than a minute to propagate which is currently a blocker.
The text was updated successfully, but these errors were encountered: