Add kinds 10 and 11 to prevent race conditions when updating contact lists #349
Conversation
|
One potential implementation difficulty is that if someone follows/unfollows a single user multiple times, when streaming the events back in out of order they'll have to keep track of when each pubkey's follow status was last updated. |
|
Interesting. This looks useful for more than what you describe. |
|
I like to think of the ContactList sync problem fundamentally being that you create an event that doesn't account for past events you didn't know about. This can be solved as suggested here with "diff" type events. But most software that deals with merging things that have gone out of sync (git, palm OS sync, AvantGo's thing) has over time been done by storing different states, and computing differences from a common ancestor, and not storing the differences themselves. The reasons are subtle I think and have to do with missing diffs just like you might miss contact lists. In any case, all we are missing from doing that kind of diff algorithm is the link back to the previous contact list. If every contact list had a link back to the previous contact list that it thought it was replacing, we could find a common ancestor, do the diff algorithms, and merge without needing kinds 10 and 11, and I think it would avoid future issues with diffs themselves going missing. Using the motivating example of this PR: User starts coracle and adds a new person to follow. Coracle generates a ContactList event B with just 1 person, pointing back to NULL (somehow indicating it has no prior). Then Coracle receives a contact list A pointing back to NULL that was created earlier. This triggers it to create a merged contact list C which it then pushes out, and that merged list points back to both A and B. This handles deletions just as well as insertions. DIFF(NULL,A) may have both deletions and insertions. Same with DIFF(NULL,B). Then you apply all those deletions and insertions. In the event of a collision, you just pick one, probably the latest one. If you have a ContactList and didn't have the previous one, you don't know the full set of diffs to apply, but at least you know they are missing, whereas with diff events you don't. |
|
Having thought a bit more on this, I'm not sure about my idea. It could be better or worse and I can't decide. You would have to keep old ContactLists to find a common ancestor so it couldn't be replaceable meaning we'd need a new kind. And I'm not actually sure what the downsides of using patches (kind 10/11 as suggested) are, I just have a belief that the industry learned not to do that over time, but my understanding is not nuanced enough to really make any determination without doing some research. |
|
in astral I just created a warning every time someone creates their first follow to confirm that they don't actually have any follows. my gut feeling about this pr is that its is trying to solve an edge case that is only an issue bc of our lack of targeted relay strategy. as we adopt @mikedilger 's proposal for relay metadata lists this should really cease to be an issue. generally it feels wrong and overly complex to have 3 different kinds just for your follows list. without fixing the relay strategy first, we are likely to have the same issues we see with kind 3 with kind 10 and 11 as well. what in this proposal makes kind 10 and 11 more reliably retrievable than kind 3? and once relay strategy is more targeted, will kind 3 really be so unreliable to find that we still need kind 10 and 11? if we do move forward on this pr, can we have more recommendations around when kind 3, 10, and 11 should actually be created? should all three be created with every contact list update? if only 10 and 11 should be updated regularly, when should we be updating kind 3? and if kind 3 isn't updated every time, is this really backwards compatible for older clients? when kind 10 is initially created should it contain all existing follows from kind 3? |
|
I am in favor of killing the idea of a contact list altogether. I know this breaks current implementations and makes it heavier to assemble a user's home feed, but here's the thing: the list shouldn't even exist. It's not how people think about their follows. The current spec forces a consensus (order of events) that is not required. This is equivalent to Nostr not needing a blockchain: the consensus of an order of events is not required for a social protocol. Think about contacts as likes (kind 7). If you like a user, you are following it. If you delete the like, you stop following. Same for relay lists. Follows is a simple type filter with authors=me and my followers is a similar type filter, but with p == me. Then we also kill the aberration of a sub-10000 replaceable event. |
|
at first glance, I really like this idea @vitorpamplona . making an individual follow event for each person you follow seems pretty straightforward and eliminates the possibility of wiping existing contact lists. |
|
As I work more with it, I think Nostr is basically an event-sourcing database with domain objects implicitly layered on top. In this PR I started by trying to define new "edit operations" on our implicit "contact list" domain object because the current ones don't represent what the user wants to do. But we should be thinking about what the user wants to do and modeling those operations instead. I think @mikedilger's proposal is more in line with "editing implicit domain objects directly", and @vitorpamplona's observation gets at the heart of the issue (although I think there are use cases for saying "forget everything I said before, THIS is my contact list", so I don't see a reason to deprecate kind 3's). I looked briefly into whether CRDTs would help with this problem, and I don't think they're really relevant, except to say that a list with independent set, add, and remove operations is a CRDT (CRDTs are usually more complex because they deal with order or dependency of elements). The problem we currently have is that we're trying to map add/remove onto the set operation, resulting in apparent conflicts. Edit: another way to think of this problem is by re-framing "kind" as an operation rather than a data type. Kind 3 is not a "contacts list", it is a "set contacts list operation". Some relevant conversation also exists on the NIP 65 discussion at #218 (comment) and following. |
The real problem is the thinking that there is a single contact list shared among the multiple relays a user has access to. You are assuming there is only ONE list to add/remove to. Which is incorrect. The majority of active users have outdated lists in old relays, either on purpose (because in that relay, they follow more people) or not. Any implementation of edit operators must consider diverging lists. |
|
@vitorpamplona no need to kill contact lists, they are useful for building a web of trust, they should be shared as a courtesy to others so they can be used as input against spam or other things. Seperate from that It's up to the clients to decide how they implement their following feed, using whatever is coming from the relays as the source of truth is probably not a good idea. |
A list is not required to build the web of trust. Individual events to demonstrate follows work just as well as an ordered list. My point is simply that the following list doesn't need to be ordered, or packed together in the same event. |
|
I like Vitor's idea, which is more-or-less what this PR proposes. We'll have to live with both approaches, but eventually one may win against the other. Is there a difference between deleting a "follow" event and publishing an "unfollow" event though? What is better? |
|
To delete an event, you must know its id. For lots of reasons, you might not have that id, but especially if the follow came from a kind 3 |
|
Wait, kind 10002 has follows too? |
From a UI perspective, you have to know that the user is following another to hit unfollow. You will have to have the event. |
Sorry, I get confused between follow lists and relay lists, ignore that
If the follow came from a kind 3, you don't want to delete the entire list and re-create it. More broadly, I think event deletions are a mistake, because they introduce a dependency graph between events. They also don't represent a user's intention in many cases. |
|
It occurred to me this morning that another use case this supports is allowing clients to show notifications when you are followed/unfollowed by someone. Currently, since 3's are replaceable, there's no way to do a diff without keeping a cache of kind 3 for all users. |
|
@staab can you give an example of how a kind 10 event would look like? If i understand this PR right, then clients would have to gather the latest kind3, and all kind 10 and 11 events that are older than the kind 3 in order to build a list, right? If I follow 2-5 keys per day, that would add up quite quickly, no? |
|
I'm thinking something along the lines of:
All 10 and 11 events that are newer, but I expect that's what you meant. Pulling a few thousand events doesn't seem that bad to me, especially since the alternative is pulling a single event with thousands of tags instead. |
|
Practically speaking this proposal's performance and kind-3s alone are quite the same. For reference, Amethyst takes 5-10 seconds to parse 4000 kind-3s of your followers (to do the follower list/count). Processing 4000 individual reaction events (kind 7) to do the same work (follower list/count) happens in less than 0.5 seconds. |
|
Thanks for clarifying. I definitely agree that we need a less-destructive approach for follower-lists. Now I agree that the overhead of an event is quite slime, so having multiple events instead of a single one with a long tag-array does not matter performance wise, but I am worried about the connection. Sometimes realys are slow and because of the nature of this approach I would have to wait on EOSE before beginning to construct a follower list, as otherwise we would be introducing a new race condition between kind 10 and 11 events. Additionally clients would need to verify the signature verfication of every single event. |
|
edit: I may have mixed follow/follower, I think it is fixed now
Now I think kind 3 CL isn't that bad xD. Maybe @staab idea should be just a client-side logic between CL edit moments. As an addition, we could keep kind 3 CL and just add follow events with expiration 3 days ahead with the sole purpose of notifying like "someone has followed you". Or maybe it's not worth it. |
|
That list seems pretty unfair tbh, most of those downsides on my proposal are marginal performance differences, general criticisms of nostr itself (relays may have incomplete set of events), or restatements of how kind 3 is already broken and the proposal doesn't automatically fix kind 3. Whereas the importance of the race condition is understated, it's one of the most common and annoying problems people experience with nostr. |
|
Additional point, after loading up all contact lists, if your client just wants to listen to contact lists updates (to keep things up to date) the amount of events is significant and they are all super large events. If we switch to a lighter model, clients could keep listening and just receiving the new follows/unfollows as opposed to an entire list over and over again. Keep in mind Battery life and networking use (most people will use NOSTR in their phones, on mobile data plans). |
|
It occurs to me a difficulty with this is counting followers, since if these aren't replaceable events, there can be duplicates. A "d" tag would probably fix this. |
|
While there are these other reasons to not have monolithic full contact list events (and those battles need to continue to be fought), I've never been convinced that the current state of affairs actually creates race condition problems of any real significance. I think if you do the following you should be good (and I must admit gossip isn't doing this currently):
|
|
@staab I wonder if it should just be labels. You can label people you follow. |
jb55
left a comment
jb55
left a comment
There was a problem hiding this comment.
event sourcing is the way. this is simple and is easy to pull incrementally.
I suspect it would be still be buggy in the context of other clients updating kind3 without knowledge of 10,11s ?
|
I have a similar proposal for kind-scoped follows, which can also be applied to non-kind-scoped follows. The idea is to create a follow event of kind 967 whenever you follow someone (or multiple people) and delete that event when you unfollow them. The reason I prefer this over the deltas approach is that deltas require reconstructing the full state to determine the correct current state (not easy in nostr). The other approach allows for operating with a partial state. |
mikedilger
left a comment
mikedilger
left a comment
There was a problem hiding this comment.
This proposal seems to describe an operation-based conflict-free replicated data type representing an LWW-Element-Set (see https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type). After taking time out to actually think about this issue, I think this is the right solution, modulo a small detail.
It could be that instead of two kinds 10 and 11, there is only a single kind, and instead two kinds of tags: "add" and "remove". I don't think they need to be searchable tags.
But that isn't terribly important.
|
So if I add someone and remove and add and remove a thousand times that means people will have to download a thousand events just in order to figure out if I am following that person or not? And then a relay cannot purge old events anymore because that will destroy the state computation? So it has to keep many thousands of events on my behalf? |
|
The only practical way to use this approach is probably to use it only when you're not sure you have the most recent kind 3, need to do a conflict-free follow/unfollow for some reason, or want to reduce the number of times you're writing a truly massive follow list. In other words, these events should probably be temporary and expire after a few days/weeks, and be periodically collected and merged into a kind 3 list to get a "snapshot". Once that's done, previous events can be dropped. I'm not sure the complexity is worth it, but I'd be interested to hear @mikedilger elaborate on why he thinks it's the best solution. |
|
The notion of a "general follow/unfollow" no longer holds true in today's Clients. Each client cultivates its own unique context and use case, leading to distinct "follow" relationships. The sooner we agree on this, the faster we can move away from this "single follow-list per key" architecture. For instance, the people I choose to follow on Olas might align with specific interests (e.g. family members) that are entirely different from those I follow on Amethyst, Habla (e.g. work-related writers), or 0xChat (e.g. local friends), where the focus could be on a different community or content type. Similarly, my follows on Primal are shaped by its own approach to content rendering even though it uses the same kinds as Amethyst and Coracle. As a result, follow lists are no longer universal. They are inherently tied to the context and purpose of the client being used. This differentiation emphasizes the need to rethink how we conceptualize follow mechanics in Nostr, recognizing that follows are increasingly context-dependent rather than key-bound. |
The cost of this is fragmenting the social graph, which is one of the biggest selling points of nostr. I don't disagree there are different reasons you might follow someone, but being overly monolithic with follows and fixing it with lists is a better default than being over granular with follows and having to start over with every new client. |
The social graph is still the same. It's just more appropriately labeled. It's indeed more costly for devs but it is also better for UX. We are stuck in the kind 1 mindset where the whole of Nostr is just kind1s. That does not work anymore. It only worked because Nostr was just kind 1s until last year. As soon as we started adding other event kinds, follow lists immediately lost all the sense of it. I favor having all CONTACTS (not follows) in separate events, like on #761. When the user installs a new client, the user can pick which labels he/she wants to import from. We just need to specify the context used for the |
If you wanted to do that... yes. To prevent this from happening incidentally, people would publish kind-3's that aggregate the prior change set now and then.
It could purge. Embrace the chaos! |
I agree with the goals here, and in theory I like the relationship status NIP very much, but I am afraid in practice fetching a big list will be so much better. If we are going to standardize different "contexts" for follow lists already, why not make each of these a different NIP-51 list? |
Even Jon doesn't defend his own proposal, so I'm alone in this?
That is true if you are talking about what you see in your feed. But without it, gossip couldn't do it's friends-of-friends calculation.
I may have misjudged this PR as an implementation of LWW-Element-Set. I think that thing has two sets, an add set and a remove set, and each entry has a timestamp (when added or removed). A dumb leap of logic made me think each event could be thought of as a line in the set, as each event is timestamped... but I think each event would have to communicate both entire sets else it becomes an operations-based CRDT instead of a state-based one (which needs reliable messaging that we don't have). So I was wrong. |
That's how we do it today, but:
|
For example, most relays nowadays limit stored events query results to 500 events by default, and I think that is a pretty generous limit. But for most users who follow more than that this will introduce unnecessary complications when fetching lists of follows. Not to mention all the extra signature verifications and event JSON overhead, but these I think are ok to disregard. |
|
Forget this one. I pushed a new PR. |
Working around these limits is better than the current straight-up rejection of big lists. Most clients have to go around the 500 events already simply to get all reactions/zaps/boots/replies of popular kind1 notes. |
all these things are very optional and it's fine if you miss some of them or decide to implement complex logic to fetch everything later. Contact lists are fundamental to the operation of most clients.
Rejection of big lists is a good measure, relays should not be expected to store a ton of garbage from reckless people, so it's good that clients help users keep smaller lists or instruct them to use more tolerant relays instead of some public default free relay. Also I have seen lists go much above 500 items before they are rejected. |
|
Relays will always store all the garbage users want. If they dont store as one list, we will break it down into multiple events which might be worse for relays. But the "garbage" WILL be there. |
This is so incredibly dumb and short-sighted. If you try to force relays and remove their choice you will just eventually exclude those the smaller ones, the community relays, the more idealistic, lower-budget people, the cheap relays that would provide a good service and add to the decentralization -- and you'll leave the ecosystem to be dominated only by the megacorps or whoever has other secondary interests and may accept that storing garbage is a reasonable cost for them. If relays prefer lists lists or not I don't know, this wasn't even the topic, but the way you're approaching this question is certainly wrong. |
This is @mikedilger's new PR, for the record: #1630 It agains chooses to ignore the concerns I raised here. @pablof7z's approach in #1605 with deletes is much saner than these attempts to create a consistent chain of events that assume a globally always-accessible state in Nostr, although lists are still probably better. |
Most clients I have looked into have coded ways around many different kinds of relay limits. They are doing the same thing, just via other means. Some Nostr libraries provide those things for free. My point is not about how dumb or smart this is. I am just saying that it IS happening right now and will continue to happen because users want it. It's the same debate about binary images on relays. Relays kept rejecting NIP-95, so users just added base64 directly on Kind 1s. Done. Now base64 images are everywhere. And if relays start filtering by content, they will change the content signature to go around it. It's a dumb cat and mice race. Some relay limits work. Others just make clients go around to do the same things via other means. The ones people can go around are dumb and just make Nostr more difficult to code without solving the problem for the relay operator. List size limits are one of them. In practice, many of these limits don't achieve what relay operators want to achieve and are just making things harder. |
What kind of rationale is that? It is happening now therefore it will always happen? I am making a statement about how I see future developments and consequences of actions, you're just describing what is happening, predicting that it will continue to happen and defending it as the best possible thing just because you saw it happening today.
I agree, but relays will eventually find ways to ban everything as real attackers come, so it's pointless to keep pushing them now by acting as an attacker that fakes as a normal client, it's confusing for everybody. But I also agree that relays should actually remove some of the arbitrary limits they have, like number of open subscriptions, number of items in a query etc -- unless of course they decide to enforce those limits more consistently without allowing trivial bypasses that just complicate client code. |
I am a very pragmatic person. I don't care what you or I think is going to happen. What our preferred solution is. I only care about things that DO happen. Wishful thinking is not for me. I am also not defending anything. My personal opinion is irrelevant in the face of reality.
My point in bringing that up is that your statement is misaligned with what I see in reality. You are asking clients to play nice and not abuse relays out there. Which is the complete opposite of what we have seen for the past 2 years. It's wishful thinking. Users want 10,000 people follow lists. Corporates want 1M+ people follow lists. Saying things like: "Rejection of big lists is a good measure" or assuming big lists are a "ton of garbage" or that these users are "reckless people" doesn't work in your favor. No client today is "helping users keep smaller lists". Absolutely NO ONE. In fact, it is common for those users to abandon clients that even suggest that they should reduce their follow lists. I know because I tried multiple times. No user is happy when the client is trying to control how many follows they can have. And they are right. |
|
OK, this discussion is pointless. But let me just say that you are living in a big contradiction because not caring about the future is not "pragmatism" and if you really only cared about things that "DO happen" you would never DO anything, because in order to DO anything you have to make a plan, even if it's a plan for the next 2 minutes, and a plan involves thinking about the future and thinking about what you will DO -- and anything you haven't DONE yet hasn't yet happened, therefore it is outside the realm of what you just said you can conceive. But I don't expect you to understand this. |
|
I agree. It's pointless. |
13 participants
A very common experience on Nostr is that of "losing follows" due to race conditions when sending kind
3events. Earlier this week someone signed in to Coracle, their contact list failed to fully sync before they followed someone, and they ended up deleting all their follows. The only solution a client can implement to avoid this currently is to disallow certain functionality if a person's account isn't fully synced, but of course, you don't know what you don't have, it's always possible a user hasn't yet followed anyone.This change is backwards compatible, and simply introduces two new event kinds to solve the above problem. There's no reason to remove kind
3, since there are valid use cases when a user might want to definitively say, "this is my contacts list".The same problem exists for relay lists to a less severe degree (because relay list cardinality is lower), and for #183, but I figured we'd have the conversation here first.
Sources:
nostr:nevent1qyt8wumn8ghj7etyv4hzumn0wd68ytnvv9hxgtcpz4mhxue69uhhyetvv9ujuerpd46hxtnfduhszythwden5te0dehhxarj9emkjmn99uq3kamnwvaz7tmjv4kxz7fwdaexzmn8v4cxjmrv9ejx2a30qy2hwumn8ghj7un9d3shjtn4w3ux7tnrdakj7qghwaehxw309aex2mrp0yhxummnw3ezu6twvehj7qgewaehxw309aex2mrp0yhxummnw3exzarf9e3k7mf0qyt8wumn8ghj7un9d3shjtnddaehgu3wwp6kytcqyr9yfkqe2yuky5lyl6wgexn6s6wjc9erntn9zp20j5cqdlk6qywsgnfgvxn)