We need a way to request direct replies to a note

[arthurfranca]

EDIT: Hello client authors, please skip to this comment to read the tl;dr of the last version of this proposal.

Currently, if we use the filter { kinds: [1], #e: ['note id'] }, it won't bring only the direct replies to the note, making it hard (impossible?) to lazily (e.g. infinite scroll) build a thread, specially the ones with lots of engagement.

This is because of:

• NIP-08 (Handling Mentions) supporting clients can add extra e tags mentioned in content
• NIP-10 supporting clients can add many e tags (1 root, maybe 1 reply and maybe one or more mentions)

There is no way to fetch just the direct replies (that would be the NIP-10 with the root marker OR the NIP-10 with the reply marker; it depends).

Currently, nostr expects clients to fetch all thread at once (which will probably get limited by relays when the thread is big).
It gets really difficult when you use limit on the request filter as it will sort by created_at desc, so it may retrieve latest non-direct replies or mentions.
E.g.: { kinds: [1], #e: ['root note id'], limit: 10 } may bring 2 mentions and 8 replies not directed to the root note

So i think we need to add a tag denoting direct reply. Maybe an R tag (just one max per event).

What do you think of adding it so that a client can use the following filter?
{ kinds: [1], #R: ['note id or long form content address'], limit: 10 }

And this should supersede NIP-10 reply marker.

[staab]

I like this solution a lot. Replies are the one universal concept in social media applications, event tags are far too broad to support them efficiently.

[barkyq]

I like this too. In general, would be nice to have efficient ways to build reply trees.

Use-case 1: Start at a leaf note and move to the root note. Use the IDs field in the query filter, taking the ID from the R tag.

Use-case 2: Start at root note and fetch the child notes (building the reply tree from top down). With this proposal, can use R query tag. Anticipated improvement in bandwidth + specificity of the query. Still will probably return a fair amount spam if spammers include the R tag linking to popular messages. Limiting only one R tag per note could mitigate that a bit though.

[fiatjaf]

In principle this is fine, but this is introducing a huge breaking change that also possibly makes it more inefficient to download a full thread and get live updates from it.

And the goal is just to solve the problem of "thread too big"? What is a thread too big? A thousand replies? Clients are downloading much more than a thousand events routinely on feeds today, for reactions, contact lists and whatnot. How many threads have a thousand replies? Where are these big threads? Are these replies all in the same relay (in the future they probably would not be)?

If a thread has hundreds of replies then I think there are other problems that must be solved before you introduce a way to do these fine-grained queries, because no one is going to read these huge threads which are likely to be full of spam and valueless comments, so clients must already be mindful of querying for events in a thread only in relays that provide good signal (and other techniques).

[barkyq]

I guess the size is relative to the relays max # of events returned per query.

If everyone in the thread agrees to tag the "root", then any query which includes the root tag will return every note in the thread. I suppose one could paginate using since and until.

Theoretically, I think that it would be nice to have some way of asking for the "direct children" of a note (query the R tag). I also think it is nice to be able to ask for all notes which refer to a given note (query the e tag)... These two ways can hopefully coexist. Thus I disagree that it would make it more inefficient to download a full thread, since the proposal would not replace the current way of doing things.

I think that, if a note N is on Relay X, the client should ideally be able to recreate the surrounding "thread reply tree" near N without having to go to other relays. At least, I think this should be a worthwhile goal. I agree that if Relay X is a super spammy relay, then it may be difficult to achieve this, even with @arthurfranca's proposal.

---

I suppose it would be inelegant to include the same data in an R tag and in an e tag. But duplicating the data would be the least breaking change. There could be something like R tag is an integer pointer to the e tags index, and the relay should use the corresponding e tag data when inserting the R tag into an index. Since this is not "generic tag query" behavior, perhaps one would want to switch away from a single letter. Maybe something like dd (for direct descendant). For example:

{
  ...,
  "tags":[["e","aaa..."],["p","ccc..."],["e","bbb..."],["dd","2"]],
  ...
}

This note would be a direct descendant of bbb.... The filter:

{
  "#dd": ["bbb..."],
}

would match the above note.

For reducing amount of logic, the dd tag MUST come after the parent e tag in the list of tags. The dd tag contents MUST be a base 10 integer encoded as a string without leading 0s. The integer MUST be the 0-indexed position of an e tag in the tag array. Any dd tag after the first dd tag SHOULD be ignored by clients and relays implementing this.

Seems like something like this could slowly replace reply markers, without breaking anything too badly.

[barkyq]

It is late here. and perhaps my idea is ridiculous. will leave it up for posterity.

EDIT (added later):

1. I do like that the extra data would be very lightweight for relays/clients which do not implement this...
2. Seems a bit overkill since websocket compression would already compress repeated information quite well.
3. Basically just moving the reply marker to its own tag, modulo the case where there is only the root marker. So seems a bit silly to add a new tag.
4. (tongue-in-cheek) If we go with dd, then it would natural to call this NIP-221 since 0xdd=221.

[aaafrancaaa]

OP here, not main acc
@fiatjaf main goals are:

1. reduce bandwidth usage to lower relay hosting costs
2. increase page load speed
3. reduce cumulative layout shift (notes moving up an down while replies come at any order from different branches. All clients i've tested are like that)

2 and 3 are good to ux and ranking better at search engine results.

It is common for a user to read just some comments of a thread and go back to the feed. Very inneficient to request all thread for every user at every client reading a thread.

There could be something like R tag is an integer pointer to the e tags index, and the relay should use the corresponding e tag data when inserting the R tag into an index

@barkyq as you describe it, it seems the relay would have to store it either way so to index the r tag if i got it right. And the two char tag wouldn't get indexed by relays.

[barkyq]

Relays are able to index tags with more than 1 character, it's just not part of the generic tag queries NIP. My thought was, if the desired query behavior does not follow the generic tag query format, it should not use 1 character.

[aaafrancaaa]

Ok got it barkyq. I don't know what the best approach is but i think we do need this feature.

[barkyq]

Well I suppose there are (at least) three options.

Option 1: [["e","aaa..."], ["e","bbb..."], ["R","bbb..."]]
Option 2: [["e","aaa..."], ["e","bbb..."], ["dd","1"]]
Option 3: [["e","aaa..."], ["e","bbb..."], ["R","1"]]

Option 1 is nice because it leverages existing generic query tag behavior. In this sense, relays do not need to change anything to support this. A bit inelegant because of repeated info.
Option 2 is nice because it has a smaller event size. It is opt-in by the relay since it does not use generic query tag format.
Option 3 is bad since relays which support generic tag query but not this NIP will index these events as having useless R tag content.

The bandwidth downside to Option 1 is mitigated if compression is used.

[fiatjaf]

It is common for a user to read just some comments of a thread and go back to the feed.

They can already fetch the parent note.

[fiatjaf]

1. reduce bandwidth usage to lower relay hosting costs
2. increase page load speed
3. reduce cumulative layout shift (notes moving up an down while replies come at any order from different branches. All clients i've tested are like that)

1. How?
2. This is not obvious at all. Doing a bunch of queries is probably much slower than a single query -- but of course it depends on many other implementation details and thread size.
3. Web clients are horrible and there is ultimately no way to fix this. Yet it is very possible to make them better and remove these issues without breaking the current standard. Also I am pretty sure that just changing the standard wouldn't automatically make web clients better, that would still require the same amount of work to make the UIs not suck.

[aaafrancaaa]

@fiatjaf for instance, if we consider we can start as low as loading 5 direct children of the main note (depends on note bubble height), here it goes:

1. { kinds: [1], #R: ['note id or long form content address'], limit: 5 } (if multiple relays, req the same for each one, merge and show at UI just the 5 latest replies)
2. Optionally one would want to show a "continue thread" link below each children that has its own children. So for each 5 items we do { kinds: [1], #R: ['note id'], limit: 1 }, limiting to 1 item just to know if there is more or not.
3. User scrolls to bottom
4. Repeat 1 (add until: xxxx), 2 and 3

The inner branches use same algorithm, but triggered by user click instead of scrolling.

It is lighter bandwidth-wise and potentially faster if we consider main content comes from step 1.

[barkyq]

Right now most threads are pretty small so its not a big deal. I do think that this "feature" is a fairly fundamental one though. Clients should be able to say where they want their event to be positioned in the reply tree, in a way that relays can understand, and in a way which other clients can query.

I don't see why there is an emphasis on "breaking the current standard." Seems like this would be an opt-in feature which would not break the old way of doing things. Sure, if a client wanted to implement this, it would require a few new lines of code, but probably they already have that logic with deciding where to add the reply marker to.

Also, sidebar, there is not really a well-adopted "current standard." Half the people use reply and root and the other half just add a bunch of unmarked e tags.

There is certainly some discussion to be had whether this actually achieves anything beyond what can already be achieved using since, until, limit, and #e. Perhaps not. I think the lazy-loading use case described by @aaafrancaaa is nice.

In general, I think there is merit to having collections of events organized in a tree (e.g., posts in a github issues style thing, where each repo is an event, each issue event sets its parent to the repo event, each post sets its parent to the issue event).

[arthurfranca]

In general, I think there is merit to having collections of events organized in a tree (e.g., posts in a github issues style thing, where each repo is an event, each issue event sets its parent to the repo event, each post sets its parent to the issue event).

Yes @barkyq and tag name should be P for Parent event or something like that instead of R. R for direct Reply was kind of an inverted name, cause i was naming similar to NIP-10 reply marker. A tag to indicate the parent note/long form content(/any event) of which the current note is direct descendant.

The NIP-10 is marked as a draft. I can try to edit it, keeping the current way of doing things as it may be preferred by clients, just adding the P tag.

But it seems @fiatjaf isn't convinced.

[fiatjaf]

I like the idea, if it was like that since the beginning it would have probably been great. I am not convinced it's worth changing. If other client developers like it enthusiastically as @staab does, including Damus, we can for sure modify NIP-10 to include this yet new way of replying.

Although r for "root" and p for "direct parent" makes more sense to me -- so exactly the opposite of what you suggested.

[mikedilger]

Also, sidebar, there is not really a well-adopted "current standard." Half the people use reply and root and the other half just add a bunch of unmarked e tags.

Yes, and this suggestion adds a third way. Adding new ways doesn't eliminate the old ways. https://xkcd.com/927/

If everyone were doing marked 'e' tags as recommended in NIP-10, then this suggestion could be implemented entirely by relays, right? And we wouldn't need a new tag, except we would need a new way to specify in the filter which events we were seeking.

[barkyq]

If everyone were doing marked 'e' tags as recommended in NIP-10, then this suggestion could be implemented entirely by relays, right? And we wouldn't need a new tag, except we would need a new way to specify in the filter which events we were seeking.

Yes, except that an e tag cannot be marked reply and root simultaneously.

Although I guess this could be handled (in a bit of a complicated way) by checking if there is only "root" and then assuming that is also supposed to be the "reply".

[barkyq]

I am partial to the addition of tags like ["dd", "3"], where "3" points to the index 3 entry in the tag array.

But I am certainly biased since it was my own idea..

One advantage is that multiple of these pointer tags could point to the same e tag. One disadvantage is new code for both relays and clients (and the XKCD comic's observation). Also having trouble thinking if this type of pointer tag would have any other use cases.

[arthurfranca]

Yes, and this suggestion adds a third way. Adding new ways doesn't eliminate the old ways. https://xkcd.com/927/

At least NIP-10 is a draft, maybe because we are at a time clients are trying things and seeing what works best or maybe we are just lucky lol cause as a draft we theoretically have the possibility to together decide how new and up-to-date social clients should work to interoperate and deprecate the old ways.

If everyone were doing marked 'e' tags as recommended in NIP-10, then this suggestion could be implemented entirely by relays, right? ...

I don't think adding relay logic and complexity, like the "dd" tag @barkyq suggests, just for an use case is better than adding a tag. But of course it is open for discussion.

The way of building all thread at once using the root marker works well (with caveats already listed) when the client is viewing the thread from the beginning. But when e.g. a twitter-like client is showing a view starting from a reply X deep in the tree it doesn't work, cause X isn't marked as root (but yeah, client could still load all thread just to show the slice). Yet, a tag/marker for root is good even when lazy loading, cause it enables showing something like this:

root
... "load more" ...
X
direct replies to X (plus "continue thread" for each one that has children)
... "load more" or infinite scroll ...

We have to think of 3 things: what is best for relays (bandwidth, disk space, simplicity), clients (lazy loading possibility? loading everything at once if it wishes? both or just one way?) and users (ux, notifications – NPT-10 lowercase p tag, discoverability by seo). Just client point of view isn't enough.

I don't see disqus, discourse, fb, instagram, twitter, telegram, whatsapp, reddit (all?) loading all thead at once. But if we still want to support the 2 ways of fetching a thread with minimum change, we could kill reply marker and add the P tag. The only "problem" is when P and e-root (or R tag as @fiatjaf said; uppercase cause r is already taken meaning reference?) have the same value we can't keep just the e-root/R as before because of lazy loading way and can't keep just the P tag because of the load-all-thread-at-once way. Not that bad the two tags having same value in this case (when at first level of replies) i guess.

If we use the R tag for root, we could kill markers, keeping e just for mentions (NIP-08 inline or reposts). R tag would allow long-form content address as value beside event id. Also it would make the load-all-thread-at-once way better by not fetching events that just mention the root when in fact one wants events that have a specific root (filter by R instead of the broader e – same that happens when using P instead of e).

Speaking of markers, now there is #293 pr for p markers which would reintroduce markers. But that should be discussed there.

[arthurfranca]

tl;dr

I've come up with an even better and simple way of connecting events as replies, needed for more advanced queries that would fight spam as a side effect.

---

First, an introduction to explain why it is needed. Yesterday i was paying attention to how hamstr.to ("Hamstr is a twitter-style Nostr web client") load replies compared to twitter and noticed it differs in an important way among other things.

Twitter tries to show the user the most interesting threads (MIT) on a post page first. I think MITs are the conversations/mini-threads that include OP replies or user (the one who is viewing the page) follows' replies – a series of linked replies (on twitter, these are linked by vertical lines between avatars).

MIT's also appear on Instagram (they highlight responses from OP) and on Youtube (they highlight replies liked by OP). MITs are always on top of other replies.

MITs have a great side effect of pushing down least intereresting conversations, including the damned spam, cause below MITs, considering that limit desc sort things on nostr, will be the latest direct descendant replies, which may include spam the relay wasn't able to block. (I'm considering the lazy loading way of loading threads instead of all at once that could sort however one wants).

If we consider MITs are desired, we need a way to fetch them without loading all replies at once, specially when considering spam could make threads huge and all other caveats already listed before. Loading all thread at once is just not best practice.

On twitter, when user clicks on a reply X, a page loads with MITs considering X as the OP of the branch.
One example of a 2 levels MIC would be Y replying to X then X replying to Y.
This is an important example that shows that sometimes we want MITs that don't consider the root as OP

Current way and my previous proposal aren't enough to request MITs. Recently, I was finally able to come up with a solution that would allow queries such as: show me X-OP direct replies to X OR X-OP/user friends replies to X's direct descendants. Also, show me X's direct descendant replies that were liked by X-OP. Examples later below.

---

New tags:

The branch b tags are responsible for telling what branch of the tree a reply is part of. Each b tag value is the id or address (if replaceable event) of the parent event, grandparent and so on up until reaching the root one. For instance, a reply event D should copy the replied to event C's b tags and then append C event id or address. A root event doesn't have b tags.

The level l tag is responsible for telling at what level in the tree a reply is. A reply event b tag count should be used as l value. A root event l tag has value '0' to allow requesting just root events.

An event can not be a reply to more than one event. (Or else it would mess with the l tag)

A relay may limit events to a certain max level (5 maybe?) <- Clients just need to not append a b tag if already at level 5.

Reaction events (NIP-25) must copy b and l tags from the reacted to event.

Example of a reply to 'ghi...' event:

  {
    ...,
    tags: [
      ['b', 'abc...'] // root event id or address
      ['b', 'def...']
      ['b', 'ghi...'] // replied to event id or address
      ['l', '3']
    ],
  }

Example load-all-thread-at-once filter

  {
    "kinds": [1],
    "#b": ['first X\'s b tag occurence value'], // root event id or address
  }

Example parent filter

  {
    "ids": ['last X\'s b tag occurence value'],
    "kinds": [1],
    "limit": 1
  }

Example direct descendant filter

  {
    "kinds": [1],
    "#b": ['X id or address']
    "#l": ['3'], // considering X is on level 2
    "limit": 5
  }

Example MIT filter

  {
    "authors": [X pubkey, user contact 1, user contact 2...],
    "kinds": [1],
    "#b": ['X id or address']
    "#l": ['3', '4'], // considering X is on level 2
    "limit": 5
  }

Example MIT filter by X-OP reaction

  {
    "authors": [X pubkey],
    "kinds": [7],
    "#b": ['X id or address']
    "#l": ['3'], // considering X is on level 2
    "limit": 5
  }

[arthurfranca]

Please people mention client authors you know so that they see this issue.
@jb55 <- Damus (only know this one)

[staab]

I like this in the abstract, the level tags are pretty interesting. This is pretty similar to how e tags were originally specified (which ended up being unreliable because order wasn't specified?). At this point I'm with @fiatjaf because of the compatibility issue. Because a client can't rely on other clients implementing a new tag scheme, they can't rely on the tag scheme. Asking for l=3 will drop everything published by non-conforming clients. Maybe it's not time for the protocol to ossify yet, but that's the feeling I'm starting to get.

An alternative solution to this problem could be to solve it on the relays' side with a new filter, for example ["REQ", "23974", [{"marks": [["<event-id>", "reply"]]}]]. I personally tend to think marks were a mistake (in the vein of d tags, although those are better specified), but they are pretty well supported, and would solve the immediate children problem.

[arthurfranca]

@staab thank you for your feedback as a client author.

Looking at what you said at issue #319, it would make it possible to re-use e tags instead of adding another one. Then clients would just need to change a little (adding all ancestors as e tags instead of just root and reply ones) and add the level/depth l tag. Markers could be there unchanged just for backward compatiblity. What do you think?

Edit: Either way, some e mentions aren't that bad and could be just filtered out client-side.

[mikedilger]

I think this issue is addressing a real need, but not one that comes up very often. So I support the effort. But I haven't taken the time to read this PR or the comments and I have no idea how to solve this. I'm just taking enough time to make this comment since you called for client authors.

[arthurfranca]

@mikedilger thank you for keeping an eye on this.

The tl;dr is have a way to 1) request replies to an event (not necessarily root one) n levels down on the tree of replies 2) request event ancestors n levels up on the tree.

The changes would be viable only if most client authors, like you, would be willing to add some tags when creating a kind 1 event. The least disruptive way (keeping backward-compatiblity while adding a minimum set of changes) I could come up with would be something like this example of a reply event:

{
    tags: [
      ['e', 'abc...', '<relay-url>', 'root'] // root event id
      ['e', 'def...', '<relay-url>'], // NEW - another ancestor, just like 'root' and 'reply' are ancestors of this event too ('ghi' is a reply to 'def')
      ['e', 'xyz...', '<relay-url>', 'mention'], // whatever mentions
      ['e', 'ghi...', '<relay-url>', 'reply'] // replied to event id
      ['l', '3'] // NEW - zero-based level/depth (in practice, just add up 1 to what was 'ghi' `l` tag OR count the non-mention e tags)
    ],
}

So just keeping all ancestors around inside e tags (not just root and reply) and adding an l tag to count the level/depth.

Edit: added the missing '<relay-url>' part.

[arthurfranca]

[...] then querying for tags of event C no longer only returns direct replies but also replies of replies. the same problem we have with root events currently.

You make a good point. It does can be a breaking change for clients that currently do this (Are you currently relying on this? As you said, the problem already exists for root tags so it seems clients are loading all replies at once and then filtering client-side the slice they want, so maybe won't be a problem)

can you explain how querying for events at an arbitrary level makes it an interesting thread? comment, like, and zap counts definitely indicate whether a thread is interesting, but not level. [...]

It wouldn't be an arbitrary level. See how Twitter shows a branch A-B-C and A-X-Y at top if C and Y are from the same author than A? (It is an interesting thread/slice cause the original poster replied to it) So you request for replies 2 levels down A's level that has A's pubkey. This is one of many examples unlocked by the branch (where A is ancestor) and level/depth (n levels below A) tags.

Likes alone may be spammed, so that's why I also added that reaction events should copy all ancestor tags and the level tag from replied to event. (But @vitorpamplona don't like it because it makes reactions bigger). Knowing from what branch and level a zap is would be also good, but i haven't read zap NIP.

no I don't think so. you cannot query a relay by 'e' tag position, nor 'e' tag marker. only by 'e' tag's second element, the event ID. when querying by 'e' tag, any event that has an 'e' tag (regardless of position or marker) will be returned.

I think markers were a mistake because of what you said. I think not being able to query by relay url isn't a big deal, but many times you want just the events with a specific marker but you end up getting events with a different one. Ideally, e tags shoud be just for one use case such as for mentions.

actually it would address this, as long as this full path leveraged the 'b' tag instead of 'e' tags. which is what I thought @vitorpamplona was suggesting.

Yes you are right. I was assuming a new way of requesting direct descendant could be used (one level down, so request with event current's l value + 1). You are right, checking the reply marker and using a tag different from e would work. Although not for the Twitter example I gave you.

Any upgrade is a win. If you think allowing more use cases with the l tag isn't worth it I'm ok. Not my call as I'm not a client author yet. I just think it would be a small addition not that hard to support.

[arthurfranca]

I would support a branch 'b' tag to denote the thread branch a event is replying as to @vitorpamplona suggested with some stipulations: [...]

@monlovesmango So if I wasn't able to change your mind regarding l tag with my previous comment, an example reply event you suggest would be:

1. With markers (not sure if you want to keep e-root for backward compatibility for a while or not so i removed it):

{
    /* ..., */
    tags: [
      ['b', 'abc...', '<relay-url>', 'root'], // root ancestor id (no particular order)
      ['b', 'def...', '<relay-url>', 'ancestor'], // itermediate ancestor id (optional - no particular order, use created_at for sorting)
      ['e', 'xyz...', '<relay-url>', 'mention'], // mention id (no particular order)
      ['e', 'ghi...', '<relay-url>', 'reply'] // parent ancestor id (no particular order)
      ['b', 'ghi...', '<relay-url>', 'reply'] // parent ancestor id (no particular order)
    ]
}

2. With ordering, no markers

{
    /* ..., */
    tags: [
      ['b', 'abc...', '<relay-url>'], // root ancestor id (must be first b tag)
      ['b', 'def...', '<relay-url>'], // itermediate ancestor id (optional - below root and above the direct parent one)
      ['e', 'xyz...', '<relay-url>', 'mention'], // mention id
      ['e', 'ghi...', '<relay-url>', 'reply'], // parent ancestor id
      ['b', 'ghi...', '<relay-url>] // parent ancestor id (last b tag)
    ]
}

While @vitorpamplona (if still not buying the l tag need), which said "Current e tags are enough for that" and "I am in favor of differentiating "branch-path" and "citation"/"mention" e tags" wants:

{
    /* ..., */
    tags: [
      ['e', 'abc...', '<relay-url>', 'root'], // root ancestor id (no particular order)
      ['e', 'def...', '<relay-url>', 'ancestor'], // itermediate ancestor id (mandatory - no particular order, use created_at for sorting)
      ['e', 'xyz...', '<relay-url>', 'mention'], // mention id (no particular order)
      ['e', 'ghi...', '<relay-url>', 'reply'] (no particular order)
    ]
}

Sorry if i misunderstood. Now I ask you:

• How can the three ways converge?
• And have I convinced both of you of the importance of the l tag for future use cases and its easy of implementation?

[monlovesmango]

I like option 1 a lot. this would also enable doing thread-like queries for non-root events, which I think would be really convenient. the last 'b' tag with 'reply' marker should also be marked as optional. agree that 'e'-'root' tag should no longer be needed.

I know I suggested ordering, but would prefer to use markers over ordering as nip10 deprecated using ordering and want to stay consistent.

I am against option 3 with only 'e' tags as this will now make finding direct replies inefficient for non-root events as well.

Sorry, I am still not buying that 'l' tag is useful/needed. But would be nice to hear what other client devs think.

Thanks for pushing this conversation forward @arthurfranca !

[vitorpamplona]

To be clear, I don't have a use for l or for a filter for immediate replies on kind 1. Those who do should have a bigger say in this than me.

What about an additional filter attribute/format for any non-index-1 tag element:

  {
    "kinds": [1, 30023],
    "#e": [ 'abc...' ]
    "#e:3": [ 'reply' ]
    "limit": 5
  }

#e is assumed to be #e:1. You could filter by relay using #e:2 and by marker using #e:3.

All #e* tags are required to pass the tag filter together, so the last example reads:

Give me the latest 5 events 
where kind in [1, 30023] 
  AND at least one tag where ( tag[0] == "e" AND tag[1] in ['abc...'] AND tag[3] in ['reply'] )

You could download only kind 1s that directly cite a user:

  {
    "kinds": [1],
    "#p": [ 'abc...' ]
    "#p:3": [ 'mention' ]
  }

The same could happen for p, where #p is assumed to be #p:1 and #p:2 is a relay filter for people, and #p:3 is a nickname filter.

Then all tags should be indexed. You could even do a search by delegation token using #delegation:3 on NIP-26,

  {
    "kinds": [1],
    "#delegation:3": [ "6f44d7f...e5f524"]
   ]
  }

or do a search for nudity reports using #e:2 in kind 1984.

  {
    "kinds": [1984],
    "#e:2": [ "nudity" ]
  }

What about a search for image sizes on NIP-58?

  {
    "kinds": [30009],
    "#image:2": [ "1024x1024" ]
  }

The format could be #<tag name>:<array index>

I assume this could be beneficial for other event kinds that have more information on them.

[monlovesmango]

relay devs would have to weigh in on that, as it means that more than just the first two elements of a tag need to be indexed in the relay db. if people are ok going down this route then we can probably get by with just 'e' tags.

personally I do see the appeal of having these kinds of filters, however I worry more about bloating the relay db. indexing the 3rd element provides significantly less value than indexing the 2nd element, but they both have the same cost. also there are no essential use cases that need anything but the first 2 elements of a tag for querying, and all of the potential use cases you highlighted above are bit weird and made up (except the first one that this new type of filter would solely be created for).

[arthurfranca]

@monlovesmango – I like option 1 a lot. this would also enable doing thread-like queries for non-root events [...]
@vitorpamplona – I am in favor of the practice of keeping all reply ids of the current branch of the thread (full path from leaf to root). Amethyst re-assembles the branch even if they are not there, but it's nice (faster) when they do. Current e tags are enough for that.

@monlovesmango @vitorpamplona I see we can settle that both of you liked having all ancestor ids listed, very useful for walking up the tree faster among other uses. We effectively moved away from having a parent tag. Though we still have to solve the problem of immediate replies for root events.

@staab – I like this solution a lot. Replies are the one universal concept in social media applications, event tags are far too broad to support them efficiently.
@barkyq – I like this too. In general, would be nice to have efficient ways to build reply trees.
@mikedilger – I think this issue is addressing a real need, but not one that comes up very often [...]
@monlovesmango – In my mind the only scenario in which querying for direct replies is problematic currently is for replies to root events

Although @staab, @monlovesmango, @mikedilger, @barkyq and I consider this part important, @vitorpamplona doesn't care much for his client as he said "I don't have a use for l or for a filter for immediate replies on kind 1". Yet, @vitorpamplona suggests a way of solving it: "an additional filter attribute/format for any non-index-1 tag element", which is ingenious, but would certainly have to be a new NIP (instead of NIP-10 edit, keeping an eye on breaking changes as @fiatjaf is worried), with a risk of less adoption cause it would also depend on relay operators. As @monlovesmango says, "relay devs would have to weigh in on that [...]".

So, as far as I know, for root immediate replies we have this novel filter (needs new relay functionality)...

{
    "kinds": [1],
    "#e": [ 'root id' ]
    "#e:3": [ 'reply' ]
    "limit": 5
}

... versus this filter (need the l tag your guys don't like, but would solve it among other uses)

{
    "kinds": [1, 30023],
    "#b": [ 'root id' ], /* I used #b instead of #e cause of @monlovesmango*/
    "#l": [ '1' ]
    "limit": 5
}

[vitorpamplona]

I think the main question is:

Are markers good? Are markers a good design pattern for new NIPs moving forward? Do we want more e tags with a classifier or not?

If so, we should have extended filters for them.

If not, we should expect, and work through all event kinds, to avoid the use of generic tags like e and p in favor of kind-specific tags, such as branch, reply, root as tag names for kind 1.

[vitorpamplona]

Here are my 2 cents:

e and p are very useful generalizations for the event/user graph. e and p are, in my mind, fundamental parts of the protocol and not a Kind 1 element. When relays and clients see e and p tags they can generally assume the next element is the hex and index it. There is no need to create base-graph-named indexes for each event kind out there. New event kinds can reuse indexing if they simply follow this naming convention.

d and a tags messed things around a bit, but they are still generalizable and indexable, independent of event kind. As long as the d tag stays a pub-key-based unique id and the a tag remains a reference to that id, it should be fine.

r, t, and g tags are already more domain (event kind) specific.

By moving to a b tag, we would lose that automatic indexing element and opt for a kind-1-only tag, disabling such node connections if a relay only processes e tags.

IMHO, It would be better to keep using e with markers so that the event graph is maintained. Once that is established, it's natural to suggest a filter by a marker (indexed or not). If we always assume such queries to not be indexed, then it's natural to suggest a filter by any "column" in tags.

By having everything filterable, there is less of a need to create new tag names when simple extensions to the graph are required by domain-specific needs of each event kind out there.

For instance, I would love to have the same direct mention/citation marker for r, t, and g tags. When things are directly cited in a message, they take a whole different meaning at the client level. I don't think we should be making mentioned-r, mentioned-t, and mentioned-g tags.

[arthurfranca]

When relays and clients see e and p tags they can generally assume the next element is the hex and index it. New event kinds can reuse indexing if they simply follow this naming convention.

I think relays aren't applying special treatment depending if the tag is e or z. Well, atleast up until now it seems care was taken so that a tag with same char has the same possible array of value types no matter what event kind. But I wouldn't count on that as searchable tags only have 1 char, so a little limited set.

---

Today the problem with markers is that relays won't filter by them. As a result, clients must fetch a possibly much bigger number of events than would otherwise be needed.

So, as things are today, markers maybe were not the best idea, because the data inside an e marker is important, not secondary. e started as meaning "reply event" because nostr was at its infancy. But now it can mean "root event", "reply event", "mention event" and the possible meanings (markers) may keep expanding, and these aren't filterable when requesting.

Currently, using different 1-char tags is the way to differentiate things and so using e for different contexts doesn't work very well. But it is not the end of the world while there are only 3 meanings. Cause root and reply can be merged with the concept of ancestors (if using what I already proposed) and then filtering out cliet-side the mentions, that are just a few in a thread (and commit to not adding new markers).
Edit: Considering the proposed solution for the specific use cases on the current open issue, markers are not a big problem. Perhaps it will become a problem if new meanings for event references on text notes arise.

Expanding all tag values (not just the value at position 1) to be searchable may rule out some relay databases, decreasing db diversity. Perhaps it is easier to squeeze the current nostr-protocol further, as it already demonstrates it is capable of enabling many use cases without change. I really don't know.

[arthurfranca]

I'm going to illustrate this issue with screenshots to try pushing this forward. Hopefully I will undoubtedly demonstrate how treating replies as a tree (branch and level/depth) will allow nostr clients to present themselves as real contenders –think it as real adoption.

First, see below how Youtube push an old comment to the top just because the OP liked it. It will go all the way up above any spam! Great, uh?

Example nostr filter:

  {
    "authors": [OP pubkey],
    "kinds": [7], /* reaction */
    "#e": ['root event id']
    "#l": ['1'],
    "limit": 5
  }

It works with events other than the root one too, just change #e and #l.
Note: it would need reaction to copy all b (or e if we reuse them instead) and l tags from the reacted to event

[arthurfranca]

Now we can see how Instagram push direct replies to the top if it was replied by the OP. All the way above any spam! Great again!

Example nostr filter:

 {
    "authors": [OP pubkey],
    "kinds": [1],
    "#e": ['root event id']
    "#l": ['2'],
    "limit": 5
  }

Easy peasy! (But not with current NIP-10 👎 )

[arthurfranca]

Now featuring @fiatjaf himself! You can see the same pattern on Twitter! How the direct reply is shown at the top of the thread, beacause it was replied by fiatjaf, the OP:

[arthurfranca]

Above screenshots show how we could load most "interesting" replies at the top, while keeping spam and least interesting replies at the bottom.

All apps shown also benefit from lazy-loading the thread instead of loading everything at once. It is just not a best practice requesting all thread at once.

All this would be possible with the changes I'm proposing. Changes to kind 1 creation that aren't complex at all.

[fiatjaf]

I personally think these screenshots show some of the worst aspects of these centralized social platforms. The UX is optimized for useless pumping of engagement and awful for actual conversations, it's a lab rat. So far Nostr clients have been delivering an experience that is far superior to the annoyances depicted above.

[arthurfranca]

I personally think these screenshots show some of the worst aspects of these centralized social platforms. The UX is optimized for useless pumping of engagement and awful for actual conversations, it's a lab rat. [...]

Omg, I consider it is totally the opposite ^-^. I understand this is debatable. Also, you created Nostr so you get to decide what gets merged.

A person reading a post by Alice will want to see comments that Alice replied to, because she started the coversation. If I'm reading Alice's blog, I want to know her opinion. I want to follow conversations in which she is involved. Of course not exclusively, but it has a higher probablity of being more important than a Random guy's comment that could be spam or a sub thread going totally out of topic. I also consider it a good way to push down spam.

Likes are usually evil, but if we promote to the top the comments liked by your friends, I guess now they are not anymore. In the example I gave, it was a like from OP, but could be from your friends.

If we are talking about a client as Twitter that limits char count, I also want to read the thread of Alice replying to herself multiple times first.

[...] So far Nostr clients have been delivering an experience that is far superior to the annoyances depicted above.

If using paid relays, paid with satoshis, that regular Joes can't use to join conversation. Haven't heard of any relay that fights spam by using another technique yet. What I presented at least may allow clients to hide spam below comments that client at least consider on topic.

Lets not forget the cell phone plan spending GB quota faster than it could be by downloading entire threads from multiple relays. We need lazy-loading possibility in my opinion.

[arthurfranca]

Just as a side note: thinking about it, there is no strong reason to need a branch b tag.

It works ok with regular e and/or a, cause as subscription filters will be relative to known event at hand, we know if it is a text note or a long-form content. I will edit the screenshot filters above accordingly.

In a nutshell, a new reply event just needs to:

1. copy all e and a from replied to event that aren't mentions (removing reply marker if present), so to keep all ancestor list – instead of just the root as current clients do
2. append the replied to event's id/address as e/a tag with root or reply marker – as already is done by current clients
3. count the resulting e and a entries that are not mentions and add as an l tag ([l]evel or capital D for [D]epth)

But yeah, if no developer other than me gets the importance of this, this will unfortunately never see the light of the day.

[arthurfranca]

@vitorpamplona I've downloaded Amethyst and noticed it splits feed into two tabs: "New topics" and "Threads" (don't know exact text as mine is in pt-BR).

In the future, if clients adopted the level/depth tag, including even on root events (['l', '0']), you would be able to fetch each tab content separately, just when user touches it. It would be more efficient.

It would allow you to get only root events with the filter
{ kinds: [1], authors: [...], #l: ['0'], limit: 20 }

And also the other tab with the following filter
{ kinds: [1], authors: [...], #l: ['1', '2', '3', '4'], limit: 20 }

[vitorpamplona]

The issue is that I never expect ALL clients to adopt this mode. Thus, to avoid UX inconsistency with posts that should have been there but are not, I most likely would lever use this as a filter. I would download everything and "fix stuff up" when they arrive.

[arthurfranca]

Yes many clients will want to support all the known patterns of replying to an event to not miss events when fetching them. Atleast for a transition period. For example, I don't expect the "deprecated" section of NIP-10 to be supported forever by clients, specially new ones.

I wanted to point this out to show the usefulness of the proposed changes, among other uses already mentioned.

This issue is aiming to reduce number of events retrieved on some event subscriptions, while unlocking a few new use cases. If it makes it into a NIP, I expect that eventually up-to-date or new clients will adopt.

[monlovesmango]

again, just to reiterate, the only posts that you currently cannot request direct replies from is the root event. with the current model you can already request direct replies of any event that is not the root event. your new proposed strategy would break this ability for clients that don't adopt l tag, and add a ton of metadata to replies that are deeper in the thread.

I am STRONGLY AGAINST adding a whole list of e tags for every single ancestor.

However, I think you have changed my mind about the level l tag. for the root event (which is the ONLY event we currently cannot request direct replies for) I actually do agree the level l could solve this and add a lot more flexibility to querying replies.

If we keep NIP-10 as is and just add the l tag to denote the thread depth, this would:

• enable all the examples you posted above
• be 100% backward compatible (wouldn't break the ability to query direct replies of non-root events for clients that don't adopt this standard)
• allow much finer control over how much data is requested for a thread
• not bloat the metadata of replies deep within a thread

what do you think? if you still think we need all ancestors listed as e tags in an event please provide an example use case of when that's needed.

[arthurfranca]

@monlovesmango considering the following reply tree with three brances (R A B C D, R W X Y Z and R W X N):
(root) R - A - B - C - D
             \_ W _ X _ Y _ Z
                           \_ N

If i'm reading the R event view, I can get the root direct replies by subscribing to { '#e': [R], '#l': ['1'] } (returns A and W).
If i'm reading the W event view, I CAN'T get the W direct replies filtering by { '#e': [R], '#l': ['2'] } (returns B and X instead of just X).

That's why we need to have all ancestors, so that in the second example we can instead use { '#e': [W], '#l': ['2'] } (returning X instead of nothing), cause X would have W (and R) as an e tag.

We can safely get X direct replies with { '#e': [X], '#l': ['3'] } (Y and N)

Considering the screenshot examples, we can also get non-immediate descendants deeper in a branch to unlock said use cases.

Just to make it clear, Y for example would have the following e tags: R, W and X. So just the ancestors of same branch, not bloated.

[monlovesmango]

in current state, you CAN get the W direct replies filtering by { '#e': [W] }. however with your proposal you wouldn't be able to. I acknowledge that mentions will be mixed in as well, and am open to discussions on how to adapt the mention functionality to fix this.

Considering the screenshot examples, we can also get non-immediate descendants deeper in a branch to unlock said use cases.

I don't think this is useful enough to warrant listing all ancestors as e tags. even your screenshots only needed this for the root event, not decedents. as you get deeper in a thread the usefulness of getting non-immediate replies drops off considerably.

[arthurfranca]

in current state, you CAN get the W direct replies filtering by { '#e': [W] }. however with your proposal you wouldn't be able to.

I was took by surprise when you first said that some clients are fetching direct replies of non-root events, because if I were to build a nostr client I would use just one strategy, if possible, no matter if loading a root note view or a non-root one.

So currently the only strategy in common would be using { '#e': ['root-event-id'] }, even for non-root event view loading.

So my goal when starting this gh issue was to present an alternative strategy, shareable by root and non-root views, that would allow for fetching just direct replies, while letting clients still use the previous strategy if they wish.

It could be done with a capital P, for "parent", tag addition always present inside non-root events (at the start of the discussion it was a capital R). This would allow the missing { '#P': ['root-event-id'] } for getting root event direct replies. Or could be done by making it possible to fetch by the e tag which has the reply marker (if also adding the e-marker in cases where today there is just the e-root), which I think is the hardest change.

The "all ancestors plus the l tag" approach seems to me a step further, as:

• all ancestors there would make loading them faster than requesting one by one as is today (could introduce a new tag instead of reusing e as we discussed before)
• the l tag would allow showing at top direct replies that the OP replied to (also here), no matter if the OP of a root or non-root note view, along with the same benefit of a simple "parent" tag
• if copying these tags to reactions, it would allow showing direct replies liked by OP, also no matter if the OP of a root or non-root note view

So as you can see, the "all ancestors plus the l tag" approach would enable the birth of different clients (I think higher quality clients, pushing spam to the bottom, but some people may not agree).

Though at this point, I'm skeptical that any of these solutions will get picked. The majority of client authors seem to be comfortable with the current way of handling event replies as things are already coded. Unfortunately just you @monlovesmango and me kept interest in this discussion.

[arthurfranca]

I'm closing to keep repo lean as this was a lost battle =)