cache ipns entries to speed things up a little #1887

whyrusleeping · 2015-10-22T23:48:08Z

This adds a quick minute long cache to ipns entry resolution.
Currently, ipns is pretty much unusable for webpages as each sub-request must re-resolve the entry.

License: MIT
Signed-off-by: Jeromy jeromyj@gmail.com

ion1 · 2015-10-22T23:51:12Z

It would be good to reflect that in a Cache-Control max-age header for /ipns addresses on the gateway.

#1818

kyledrake · 2015-10-23T21:49:37Z

👍

whyrusleeping · 2015-10-24T07:12:51Z

After this merges i encourage people to really start using ipns, and reporting any issues to me. I'm pretty confident in it (post this changeset), but thats not to say it won't have any issues.

ion1 · 2015-10-25T21:34:07Z

I did an experiment with the current master, publishing a new object every few seconds and polling the IPNS address on the public gateway. It seemed to work impressively quickly at times while being slow to publish and resolve at other times.

I have been thinking of doing an experiment with live video streaming in this way. Unfortunately, this change might affect the current update rate negatively.

On the other hand, if I were to publish a blog using IPNS, I might be fine with nodes having a stale version of the record for up to an hour.

I realize future pubsub support will cover the live streaming use case, but having the ability to do ipfs name publish --ttl=5s for the streaming experiment and ipfs name publish --ttl=1h for the blog example would let the publisher make the choice between low latency and greater caching (in this example, choosing to have a cache time an order of magnitude greater than the one minute).

Don’t let this delay merging this PR, but I hope IPNS records will gain a TTL field (like in DNS records) in the future.

ghost · 2015-10-26T21:07:19Z

core/core.go

+		return 0, err
+	}
+
+	ct := cfg.Ipns.ResolveCacheTime


Should we have this in the default config too?

we can, but we dont need to. It assumes that an empty value means the default value of one minute.

i thought for the sake of documentation ;)

added to the default config

jbenet · 2015-10-28T00:22:33Z

namesys/routing.go

+		eol, ok := checkEOL(entry)
+		if ok && eol.Before(cacheTil) {
+			cacheTil = eol
+		}


this stuff could get easily wrong, and cache time errors could behavior drastically. i'd propose feeding entry into r.cacheSet and put the computation in one place, calculated always from the original data, instead of intermediate values.

jbenet · 2015-10-28T00:25:26Z

i'd support the addition of the --ttl flag for name publish, as a means to experiment for now (mark it experimental)

jbenet · 2015-10-28T00:25:33Z

👍 to this PR in general.

whyrusleeping · 2015-10-28T04:08:39Z

@ion1 @jbenet i have a --nocache option that will ignore the cached value when requesting a record. I will also add the --ttl, but wiring that down is a little more intrusive.

whyrusleeping · 2015-10-28T04:16:06Z

er, wait. that option is already there in the form of --lifetime

whyrusleeping · 2015-10-28T05:51:52Z

@jbenet do you want --lifetime renamed to --ttl? or are we good to merge?

jbenet · 2015-10-28T08:13:16Z

@jbenet do you want --lifetime renamed to --ttl? or are we good to merge?

~~ah, no, good to merge.~~ not merge yet, actually

jbenet · 2015-10-28T08:15:03Z

no, dont merge yet. still CRing

jbenet · 2015-10-28T08:16:26Z

namesys/routing.go

+	cacheTil := time.Now().Add(r.cachelife)
+	eol, ok := checkEOL(rec)
+	if ok && eol.Before(cacheTil) {
+		cacheTil = eol


um... this isn't used anywhere.

this is an interesting case of go's unused variable error. Its used, then assigned to, then that assigned value is never used.

ghost · 2015-10-28T15:05:33Z

I'm a bit worried this will hide speed problems enough that they don't get fixed...
Also, it makes the "IPNS livestream" idea kind of problematic, since you'll have a minute long delay on new updates thanks to the cache.

whyrusleeping · 2015-10-28T15:14:39Z

@gamemanj read the comment chain.

whyrusleeping · 2015-10-28T15:30:09Z

@jbenet actually used that variable.

ion1 · 2015-10-28T15:37:54Z

publish --lifetime=48hours --ttl=1hour would be perfectly sensible:

The record is considered invalid 48 hours after the publish time.
You don’t need to ask the network for possible updates to the record for 1 hour after having received a copy, i.e. I’m okay with others keeping a stale version of my record for up to 1 hour.

The Cache-Control/max-age header is determined by --ttl (the semantics match perfectly), not --lifetime.

This adds a quick minute long cache to ipns entry resolution.

This is the time --ttl would control.

“No cache” should be a property of the record (i.e. --ttl=5s when you are livestreaming for example).

whyrusleeping · 2015-10-28T17:35:28Z

@ion1 @gamemanj I also want to point out that live streaming will be done best by pubsub, ipns polling for streaming is equatable to setting your dns records to each frame of the video, one at a time. Not to say that you shouldnt try it, and that ipns should be slow, but thats just not its intended purpose.

whyrusleeping · 2015-11-01T02:12:02Z

changed the value back to 16, and changed the cache to an lru, with a configurable size.

No other DHT parameters changed here, weird things started happening when i tweaked K, so i'm not going to bother until later.

ion1 · 2015-11-01T14:42:05Z

Nice. Is the cache lifetime limit useful in addition to the LRU policy though? How about letting popular records with long TTLs be cached as long as their TTL value allows?

You could get the TTL value from the record, fall back to a default of one minute, compute time.Now().Add(ttl), limit that according to the record EOL and use that as the cache expiry for that record. I could then return the cache entry’s expiry.Sub(time.Now()) as the current TTL in #1921.

ion1 · 2015-11-01T15:19:37Z

Also, the LRU cache implementation does its own locking.

cryptix · 2015-11-01T16:34:12Z

core/commands/publish.go

+				return
+			}
+
+			ctx = context.WithValue(ctx, "ipns-publish-ttl", d)


👍 we should enhance the context much more often.

This would allow us to get a trace of where it failed in timeouts instead of a blank "deadline exceeded".

ion1 · 2015-11-01T17:41:24Z

@whyrusleeping, please see my commits in https://github.com/ion1/go-ipfs/commits/feat/ipns-cache

The first commit has some fixes for things including the redundant locking and lru.New failing given a zero size.

The second commit removes cachelife with the assumption that the LRU policy along with respecting the TTL (and EOL) fields is enough.

cryptix · 2015-11-01T17:53:54Z

@whyrusleeping I also fixed your tests.. :) Qmduc5TqSk5hWnuvBCDHQMXnPALPZ131Jz9wbU9SYTbjyB

License: MIT Signed-off-by: Johan Kiviniemi <devel@johan.kiviniemi.name>

whyrusleeping · 2015-11-02T18:30:57Z

@jbenet this is RFM i beleive. @ion1 @cryptix can i get some thumbs ups?

cryptix · 2015-11-02T19:28:54Z

👍 used it already

ion1 · 2015-11-02T20:19:44Z

ion1@f531ad3

I refactored the EOL computation out of cacheSet. I’m going to need its result for the Cache-Control PR, it would be a bit hairy to pull the value out of cacheSet.

ion1 · 2015-11-02T20:41:20Z

ion1@bd84894

A bit better: return both the absolute eol and the relative ttl and use ttl > 0 to decide whether the result should be cached. This will also make it nicer for the Cache-Control PR.

ion1 · 2015-11-02T20:47:50Z

<whyrusleeping> ion: okay. Just go ahead and apply that in your PR once 1887 merges

Alright.

👍, especially after squashing so one commit doesn’t add cachelife just for another to remove it.

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

jbenet · 2015-11-02T23:11:01Z

this LGTM

cache ipns entries to speed things up a little

jbenet · 2015-11-02T23:11:33Z

@whyrusleeping @ion1 @cryptix damn i merged this without realizing-- there's no tests. can we write some?

whyrusleeping · 2015-11-02T23:40:42Z

@jbenet sure thing! #1934

jbenet added the status/in-progress In progress label Oct 22, 2015

whyrusleeping force-pushed the feat/ipns-cache branch from 47a0685 to b0e2090 Compare Oct 22, 2015

RichardLitt mentioned this pull request Oct 23, 2015

Sprint Oct 19th ipfs/team-mgmt#40

Closed

35 tasks

whyrusleeping force-pushed the feat/ipns-cache branch from 8fa19cf to 0c59c63 Compare Oct 24, 2015

ghost reviewed Oct 26, 2015
View changes

whyrusleeping added RFCR and removed status/in-progress In progress labels Oct 27, 2015

jbenet reviewed Oct 28, 2015
View changes

whyrusleeping force-pushed the feat/ipns-cache branch 2 times, most recently from 1bc21f0 to 5db2437 Compare Oct 28, 2015

jbenet reviewed Oct 28, 2015
View changes

whyrusleeping force-pushed the feat/ipns-cache branch from 5db2437 to d93eee0 Compare Oct 28, 2015

whyrusleeping force-pushed the feat/ipns-cache branch from f9352b3 to 26d7561 Compare Nov 1, 2015

cryptix reviewed Nov 1, 2015
View changes

whyrusleeping force-pushed the feat/ipns-cache branch from 26d7561 to 4342b8e Compare Nov 1, 2015

ion1 added a commit to ion1/go-ipfs that referenced this issue Nov 1, 2015

Some fixes to ipfs#1887

95fd983

License: MIT Signed-off-by: Johan Kiviniemi <devel@johan.kiviniemi.name>

whyrusleeping force-pushed the feat/ipns-cache branch 2 times, most recently from 1bd03c0 to 9957772 Compare Nov 1, 2015

cryptix mentioned this pull request Nov 2, 2015

Sprint Oct 26 ipfs/team-mgmt#41

Closed

47 tasks

cache ipns entries to speed things up a little

c7fb7ce

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

whyrusleeping force-pushed the feat/ipns-cache branch from 9957772 to c7fb7ce Compare Nov 2, 2015

jbenet added a commit that referenced this issue Nov 2, 2015

Merge pull request #1887 from ipfs/feat/ipns-cache

d673275

cache ipns entries to speed things up a little

jbenet merged commit d673275 into master Nov 2, 2015
3 checks passed

jbenet deleted the feat/ipns-cache branch Nov 2, 2015

whyrusleeping mentioned this pull request Nov 2, 2015

Tests for ipns-caching #1934

Open

cache ipns entries to speed things up a little #1887

cache ipns entries to speed things up a little #1887

whyrusleeping commented Oct 22, 2015

ion1 commented Oct 22, 2015

kyledrake commented Oct 23, 2015

whyrusleeping commented Oct 24, 2015

ion1 commented Oct 25, 2015

ghost Oct 26, 2015

whyrusleeping Oct 26, 2015

ghost Oct 26, 2015

whyrusleeping Oct 28, 2015

jbenet Oct 28, 2015

jbenet commented Oct 28, 2015

jbenet commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

jbenet commented Oct 28, 2015

jbenet commented Oct 28, 2015

jbenet Oct 28, 2015

whyrusleeping Oct 28, 2015

ghost commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

ion1 commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

whyrusleeping commented Nov 1, 2015

ion1 commented Nov 1, 2015

ion1 commented Nov 1, 2015

cryptix Nov 1, 2015

ion1 commented Nov 1, 2015

cryptix commented Nov 1, 2015

whyrusleeping commented Nov 2, 2015

cryptix commented Nov 2, 2015

ion1 commented Nov 2, 2015

ion1 commented Nov 2, 2015

ion1 commented Nov 2, 2015

jbenet commented Nov 2, 2015

jbenet commented Nov 2, 2015

whyrusleeping commented Nov 2, 2015

cache ipns entries to speed things up a little #1887

cache ipns entries to speed things up a little #1887

Conversation

whyrusleeping commented Oct 22, 2015

ion1 commented Oct 22, 2015

kyledrake commented Oct 23, 2015

whyrusleeping commented Oct 24, 2015

ion1 commented Oct 25, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbenet commented Oct 28, 2015

jbenet commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

jbenet commented Oct 28, 2015

jbenet commented Oct 28, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

ion1 commented Oct 28, 2015

whyrusleeping commented Oct 28, 2015

whyrusleeping commented Nov 1, 2015

ion1 commented Nov 1, 2015

ion1 commented Nov 1, 2015

Choose a reason for hiding this comment

ion1 commented Nov 1, 2015

cryptix commented Nov 1, 2015

whyrusleeping commented Nov 2, 2015

cryptix commented Nov 2, 2015

ion1 commented Nov 2, 2015

ion1 commented Nov 2, 2015

ion1 commented Nov 2, 2015

jbenet commented Nov 2, 2015

jbenet commented Nov 2, 2015

whyrusleeping commented Nov 2, 2015