[WIP] Implement ipfs name follow #4462

vyzo · 2017-12-07T11:54:48Z

Implements ipfs follow in a minimally non intrusive way; #4435

TODO:

UX improvements (per @lgierth)
Implement persistence for follows (per @magik6k)
tests

magik6k · 2017-12-07T14:49:41Z

core/commands/follow.go

+	OK bool
+}
+
+var IpnsFollowCmd = &cmds.Command{


This should really go to core/commands/name.go or core/commands/name/follow.go (along with the other ipns related commands). Imo core/commands should only contain the top-level commands.

duly noted, but there are some dependencies on common helpers in the package; plus i would rather leave command refactoring out of this PR.

magik6k · 2017-12-07T14:54:49Z

core/commands/follow.go

+
+		res.SetOutput(&ipnsFollowResult{true})
+	},
+	Arguments: []cmdkit.Argument{


Most commands order these like that: Helptext/Arguments/Options/Run/Type/Marshalers

ok, will reorder.

magik6k · 2017-12-07T14:57:41Z

namecache/namecache.go

+)
+
+const (
+	followInterval = 60 * time.Minute


We might want to have this in config

We may also want to make this a flag ipfs follow add --refresh-interval X.

The configurable flag is a good idea.

Added --refresh-interval parameter to add.

magik6k · 2017-12-07T15:00:38Z

namecache/namecache.go

+	defer nc.mx.Unlock()
+
+	if _, ok := nc.follows[name]; ok {
+		return


Might want to return an error here

magik6k · 2017-12-07T15:01:34Z

namecache/namecache.go

+	defer nc.mx.Unlock()
+
+	cancel, ok := nc.follows[name]
+	if ok {


Same as above, If user somehow makes a type there would be no indication

I would invert this check. if !ok { return err }

ok, prettier this way

magik6k · 2017-12-07T15:05:31Z

namecache/namecache.go

+	return follows
+}
+
+func (nc *nameCache) followName(ctx context.Context, name string, pinit bool) {


Having this as a single worker resolving one name each followInterval/len(nc.follows) will ensure no activity spikes when many names are followed.

I can coalesce in a single goroutine, more resource economical.

I am not sure there is significant benefit by having a single goroutine, plus it's a lot more complicated to write if we want to ensure immediate initial resolution upon follow.
The pins will be serialized by the pin lock.

How strongly do you feel about using a single goroutine?

It was more of a suggestion, though it's still something we may want to do in the feature

magik6k · 2017-12-07T15:08:10Z

namecache/namecache.go

+		name = "/ipns/" + name
+	}
+
+	p, err := nc.nsys.Resolve(ctx, name)


Is there a built-in timeout in nsys.Resolve? If not we might want to wrap the ctx with some.

Probably not, i'll add a timeout here.

magik6k · 2017-12-07T15:12:00Z

namecache/namecache.go

+		return
+	}
+
+	err = nc.pinning.Pin(ctx, n, true)


Should call pinning.Update when the content wasn't pinned before previous resolve here.

Oh and take the pinning lock somewhere around here - defer n.Blockstore.PinLock().Unlock()

Oh yeah, the lock -- good point.
Should Update be called instead of Pin?

Actually seems like a more complicated logic is needed:

In first resolution, call Pin.

In subsequent resolutions:

Do nothing if the name resolves to the same node

Call Update if the pin has changed

I also wonder whether we should unpin when we cancel a follow.

I also wonder whether we should unpin when we cancel a follow.

I'd say yes. Also call Pin instead of Update when the hash was pinned before the follow, so we don't unintentionally unpin it.

vyzo · 2017-12-08T10:44:05Z

@magik6k updated per review.
The most significant change is in the pinning logic. I am still keeping one goroutine per follow, depending on how strong you feel about it.

TBD: make follow interval a configuration option.

whyrusleeping · 2017-12-08T20:32:57Z

core/commands/follow.go

+)
+
+type ipnsFollowResult struct {
+	OK bool


maybe instead of a boolean, we just have a State field thats a string, since you convert it into a string later anyways

We generally use SetError to indicate that the command has failed anyways. We should probably do something like what we do in ipfs pin add: {"Follows": [CIDs]}.

Not sure it adds much though -- in reality it's just a void command that can error.

Actually, I think it's best to just use an empty struct for void and just output ok.
As @Stebalien points out, we indicate errors with SetError anyway.

Hrm, can't have it be just void. I will make it a string as why suggested.

whyrusleeping · 2017-12-08T20:37:51Z

namecache/namecache.go

+	nc.mx.Lock()
+	defer nc.mx.Unlock()
+
+	follows := make([]string, 0)


could do:

follows := make([]string, 0, len(nc.follows)) ``

whyrusleeping · 2017-12-08T21:09:44Z

@vyzo I think its probably worth modifying the namesys interface to return TTLs from resolve calls so we can use them here.

vyzo · 2017-12-09T09:09:17Z

Ok, make sense.
I will simply add an extended resolve method to the namesys api instead of changing the old one (that is too much work).

frrist · 2017-12-11T20:05:52Z

core/commands/follow.go

+	if output.OK {
+		state = "ok"
+	} else {
+		state = "wtf"


nit - Is that actually what we want to return to the user?

certainly no, placeholder.

frrist

lgtm

Stebalien

First pass.

This PR ignores all of the design discussion from #4435. Why?

Stebalien · 2017-12-11T21:52:47Z

core/commands/follow.go

+)
+
+type ipnsFollowResult struct {
+	OK bool


We generally use SetError to indicate that the command has failed anyways. We should probably do something like what we do in ipfs pin add: {"Follows": [CIDs]}.

Stebalien · 2017-12-11T21:57:04Z

core/commands/follow.go

+	},
+	Subcommands: map[string]*cmds.Command{
+		"add":    ipnsFollowAddCmd,
+		"pin":    ipnsFollowPinCmd,


I'd collapse pin and add to add --pin or something like that.

Good point.

Stebalien · 2017-12-11T21:58:10Z

core/commands/follow.go

+			return
+		}
+
+		err = n.Namecache.Follow(req.Arguments()[0], false)


Should allow multiple arguments.

Stebalien · 2017-12-11T22:00:55Z

namecache/namecache.go

+)
+
+const (
+	followInterval = 60 * time.Minute


We may also want to make this a flag ipfs follow add --refresh-interval X.

vyzo · 2017-12-12T10:03:43Z

@Stebalien

This PR ignores all of the design discussion from #4435. Why?

Not true. It has been discussed in slack a bit, as the initial design would require very invasive changes to implement. So as a first pass, I opted for the simplest possible implementation in terms of invasiveness
and get there incrementally.

Stebalien · 2017-12-12T17:45:15Z

It has been discussed in slack a bit

Policy: If it didn't happen on GitHub, it didn't happen. Those decisions need to go on the tracking issue.

So as a first pass, I opted for the simplest possible implementation in terms of invasiveness
and get there incrementally.

Then we:

Need some form of experimental flag (default to off). Otherwise, users will come to rely on these commands. We're adding features to the API that we intend to take away.
Need issues to track what needs to be done, probably in that tracking issue (maybe with multiple linked issues).

Stebalien · 2017-12-12T18:09:40Z

namecache/namecache.go

+	nc.mx.Lock()
+	defer nc.mx.Unlock()
+
+	follows := make([]string, len(nc.follows))[:0]


make([]string, 0, len(nc.follows)). Yes, technically this works but it's weird.

Stebalien · 2017-12-12T18:12:25Z

namecache/namecache.go

+func (nc *nameCache) resolve(ctx context.Context, name string) (path.Path, error) {
+	log.Debugf("resolving %s", name)
+
+	if !strings.HasPrefix(name, "/ipns/") {


Can we do this at a higher layer?

Probably, but does it matter?

Sanitize data as soon as it enters the program. I also advocate moving this into the command's Run function above.

The Run function is perhaps inappropriate, it doesn't cover direct uses of the API.
If we are to enforce the /ipns/ prefix then it should be done at the Namecache interface.

Still, it will have to be done in 3 2 places instead of 1.

I moved it to the Namecache interface methods (Add and Cancel).

Stebalien · 2017-12-12T18:18:39Z

Otherwise..., LGTM. Thanks for implementing this (although I do worry that we're accumulating too many half-finished features).

vyzo · 2017-12-12T18:44:34Z

Need some form of experimental flag (default to off). Otherwise, users will come to rely on these commands. We're adding features to the API that we intend to take away.

Well, not necessarilly take away but potentially change behaviour. So agreed on making an experimental flag for the namecache.

vyzo · 2017-12-12T18:47:00Z

Need issues to track what needs to be done, probably in that tracking issue (maybe with multiple linked issues).

Sure, will open a bunch once we are ready to merge.

vyzo · 2017-12-13T08:38:04Z

Added experimental flag --enable-follow-experiment.

vyzo · 2017-12-13T08:40:13Z

@Stebalien
A few things that are part of the design discussion in #4435 but not implemented by this PR:

Offline resolution and resolution of expired names
TTL-based refresh interval
Persistence of follows

keks

Otherwise LGTM

keks · 2017-12-13T09:57:32Z

core/commands/follow.go

+	"strings"
+	"time"
+
+	cmds "github.com/ipfs/go-ipfs/commands"


Is there a reason you use go-ipfs/commands instead of go-ipfs-cmds? That is a legacy package.

it's a subcommand of name, which is an old style command and needs references to its children.

Fair enough. I think once the commands refactor has settled a bit we should update the parent command to use the new cmds lib instead though. For now I think it's okay because that process is still ongoing.

keks · 2017-12-13T10:43:11Z

namecache/namecache.go

+func (nc *nameCache) resolve(ctx context.Context, name string) (path.Path, error) {
+	log.Debugf("resolving %s", name)
+
+	if !strings.HasPrefix(name, "/ipns/") {


Sanitize data as soon as it enters the program. I also advocate moving this into the command's Run function above.

ghost · 2017-12-18T00:10:17Z

This looks pretty good to me 👍 Thanks a lot :)

Minor issue: should we call the cancel command rm instead?
- It's technically correct in that it cancels an ongoing repeated operation, but the opposite of add that's used in other places is rm.
UX issue: I ran ipfs name follow add /ipns/ipfs.io/docs/examples to try this out, then noticed that it wasn't fetching anything. I then ran the command again with --pin, and got an "already following" error.
1. It would be fine for additional invocations of the same command to override the previous invocations.
2. I'm thinking we should fetch the nodes regardless of --pin, i.e. do the equivalent of ipfs refs -r.

ghost · 2017-12-18T00:11:17Z

By the way, it's great that this works with fully-qualified paths a la /ipns/ipfs.io/docs/examples, and not just with plain PeerIDs :)

vyzo · 2017-12-18T08:11:37Z

Minor issue: should we call the cancel command rm instead?
It's technically correct in that it cancels an ongoing repeated operation, but the opposite of add that's used in other places is rm.

If we call it rm in other places, then it makes sense to call it rm here too.

vyzo · 2017-12-18T08:18:22Z

It would be fine for additional invocations of the same command to override the previous invocations.

Agreed, just a little tricky to implement. Having to manually cancel and re-add is not great UX.

I'm thinking we should fetch the nodes regardless of --pin, i.e. do the equivalent of ipfs refs -r

~~So you want to traverse the object for references and resolve those too?~~
Maybe we should -- fetching regardless of pin seems like desirable behaviour for a cache.

eingenito · 2019-01-15T17:28:46Z

@vyzo sorry - I see that it was @magik6k that picked it back up. Please ignore.

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

Use update when moving a pin, unpin on cancel when owning a pin License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

- coalesces pin into add --pin - add and cancel accept multiple names License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

Also adds --refresh-interval to follow command. License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>

magik6k · 2019-02-06T02:56:30Z

namecache/namecache.go

+		// just iterate over all nodes
+		walker := ipld.NewWalker(ctx, ipld.NewNavigableIPLDNode(toFetch, nc.dag))
+		if err := walker.Iterate(func(node ipld.NavigableNode) error {
+			return nil


It would be nice to add cids of nodes we iterate to some set and skip children we have already seen, but I'm not sure what's the best way to do this.

cc @schomatis

It would be nice to add cids of nodes we iterate to some set and skip children we have already seen, but I'm not sure what's the best way to do this.

Solution at the Visitor level.

The simplest way would be for the caller to keep that set and use it in the Visitor, in there we could check if the CID of the current child the Walker is pointing to has already been visited and if so, skip it, something like:

// For every child of the current `node`: childCid := node.Links()[walker.ActiveChildIndex()] if _, ok := visited[childCid]; ok { walker.NextChild(); // Skip this child, already visited. } else { break; // out of the `for`, so when this call to `Visit` ends the `Walker` // will iterate into this child we haven't visited. } // If we have skipped all of the children the `Walker` will just continue // the iteration going up to the current `node`'s parent.

The downside of this approach is that NavigableIPLDNode still prefetches many children in advance, so if we didn't skip all of the children, if we request at least one, NavigableIPLDNode will also start prefetching many more after that one (that we may end up actually skipping).

Solution at the NavigableIPLDNode level.

In that case NavigableIPLDNode should be extended to handle this scenario (which would actually be a very nice feature). (Note here that Walker has no idea what an IPLD node is, it only operates on the very abstract notion of a DAG of any kind of node that implements the NavigableNode interface; the motivation behind this decoupling was to allow the possibility of using the Walker for other types of DAG like the dependency graphs that Gx iterates over, although at the moment IPLD is the only type of node that the Walker knows).

The way I think this could be implemented (but this is just an untested idea) would be to modify the preload function to, instead of prefetching a contiguous batch of preloadSize nodes through GetNodes, iterate the childCIDs one at a time and skip the ones already visited, calling each prefetch separately with GetNode (singular).

How should the map of visited nodes be handled in that case? Every NavigableIPLDNode is independent of its parent and children, there's no central authority, the way they communicate with each other is through the context.Context that was used in NewWalker, that context will be passed to every FetchChild issued by the Walker when going down the DAG. (Maybe this is too coupled and the NavigableIPLDNode should be extended to have its own context now, that gets passed from parent to child.) So the map could be stored there, and there could be a check that if there's no map of visited nodes, then just prefetch everything as preload does now. At the moment the Walker doesn't visit the DAG in parallel so this map would be safe (but it should be documented that in future refactorings it may need to be protected from multiple accesses).

/cc @hsanjuan who is interested in something very similar, so maybe you two could join forces here.

(@hsanjuan in your case, inside the Visitor you could artificially add the CIDs of the nodes you want to skip entirely in this map, which would result in the same effect.)

magik6k · 2019-02-06T02:57:17Z

Still WIP, Pushed the prefetch variant

remon-nashid · 2021-01-24T01:41:54Z

This feature could open IPNS to many more uses case. I wonder if the community is still interested in this implementation, or if there are newer alternatives to pinning ipns names.

ghost assigned vyzo Dec 7, 2017

ghost added the status/in-progress In progress label Dec 7, 2017

vyzo added the topic/ipns Topic ipns label Dec 7, 2017

magik6k requested changes Dec 7, 2017

View reviewed changes

whyrusleeping reviewed Dec 8, 2017

View reviewed changes

whyrusleeping requested review from Stebalien and daviddias December 8, 2017 21:09

whyrusleeping requested a review from frrist December 9, 2017 17:43

frrist reviewed Dec 11, 2017

View reviewed changes

frrist approved these changes Dec 11, 2017

View reviewed changes

Stebalien requested changes Dec 11, 2017

View reviewed changes

Stebalien requested changes Dec 12, 2017

View reviewed changes

Stebalien requested a review from keks December 12, 2017 18:15

keks reviewed Dec 13, 2017

View reviewed changes

magik6k mentioned this pull request Feb 1, 2018

Usage - How to setup/init a repository? ipfs-shipyard/git-remote-ipld#6

Open

vyzo and others added 19 commits February 5, 2019 20:28

namecache: ipfs name follow

2969f79

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: obey conventional order in commands/follow

ceaa752

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: return errors in the Follow/Unfollow interface

b8125ad

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: add timeout to namesys resolution

ca22001

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: rework pinning logic

2586c5f

Use update when moving a pin, unpin on cancel when owning a pin License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: add debug log on pin updates

e76e4c5

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: initialize blockstore

795d9e8

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: fix logging stray word

33fd9e0

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: code cosmetics

8958b3d

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: wtf is just not a good unexpected error response

dfc90e8

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: make ipnsFollowResult a string

0bb07b5

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: rework follow subcommand

ec9b120

- coalesces pin into add --pin - add and cancel accept multiple names License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: make followInterval a follow parameter

143ef0d

Also adds --refresh-interval to follow command. License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: avoid slice construction weirdness

4054585

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: make feature experimental with --enable-follow-experiment

ec7116a

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: normalize name at namecache interface boundary

36a1eb6

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: post-rebase updates

96648f2

License: MIT Signed-off-by: vyzo <vyzo@hackzen.org>

namecache: update name follow to new cmd lib

7d6de7e

License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>

namecache: basic persistent version

4f56358

License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>

magik6k force-pushed the feat/ipns-follow branch from 9fbd159 to 4f56358 Compare February 5, 2019 19:29

magik6k added 2 commits February 6, 2019 02:15

namecache: remove pin for now

0bfcf7e

License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>

namecache: prefetch option

930196a

License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>

magik6k reviewed Feb 6, 2019

View reviewed changes

npfoss mentioned this pull request Jul 23, 2019

IPNS advanced tooling ipfs/js-ipfs#2292

Closed

7 tasks

Stebalien unassigned vyzo and magik6k Apr 22, 2021

[WIP] Implement ipfs name follow #4462

Are you sure you want to change the base?

[WIP] Implement ipfs name follow #4462

Conversation

vyzo commented Dec 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magik6k Dec 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magik6k Dec 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyzo commented Dec 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whyrusleeping commented Dec 8, 2017

vyzo commented Dec 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frrist left a comment

Choose a reason for hiding this comment

Stebalien left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyzo commented Dec 12, 2017

Stebalien commented Dec 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyzo Dec 13, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stebalien commented Dec 12, 2017

vyzo commented Dec 12, 2017

vyzo commented Dec 12, 2017

vyzo commented Dec 13, 2017

vyzo commented Dec 13, 2017

keks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Dec 18, 2017

vyzo commented Dec 7, 2017 •

edited

Loading

magik6k Dec 7, 2017 •

edited

Loading

magik6k Dec 7, 2017 •

edited

Loading

vyzo Dec 13, 2017 •

edited

Loading

vyzo commented Dec 18, 2017 •

edited

Loading

eingenito commented Jan 15, 2019 •

edited

Loading

remon-nashid commented Jan 24, 2021 •

edited

Loading