Generalise event lifetime preferences

[hoytech]

Background

Implementing NIP-33 and NIP-40 for my relay (strfry) caused me to think a bit about user-specified event lifetime preferences. We can roughly classify these into the following categories:

• Replaceable events, including parameterised replaceable (NIP-16, NIP-33): These indicate to a relay that an event can be removed when another more recent qualifying event is received
• Ephemeral events (NIP-16): These events should be broadcast to all connected users with a matching filter, but then deleted (or not stored in the first place)
• Expiring events (NIP-40): These events should be deleted when a specifed timestamp is reached

Note that if a relay supports NIP-09 deletions then all of the above could in principle be implemented with delete events:

• Replaceable events could be deleted immediately before posting a replacing event
• Ephemeral events could be deleted immediately after posting
• Expiring events could be deleted at their expiration time

By using deletions, any of these categories of event lifetimes could be implemented for events of any kind.

Problems

Replaceable, parameterised replaceable, and ephemeral events are specified to only apply to events within ranges of kinds. To me this seems sub-optimal:

• Protocols can't group their kinds together if they need different event lifetimes for different kinds. For instance, NIP-28 (public chat) reserves the range of kinds 40-49 but specified kind 41 to be replaceable, before NIP-16 was specified, I assume. With NIP-16, a non-contiguous kind in the 10000-19999 range would need to be reserved.
• Kinds can't compose with event lifetimes. A new protocol might like to have some messages of the same kind have different event lifetimes. For example, a protocol might like to support editable (replaceable) postings as well as "permanent" postings using the same kind.
• Users can't specify custom lifetimes for events of existing protocols. For example, you could imagine sending an ephemeral kind 1 message, as a "disappearing" temporary message that would pretty much just work on any existing client.
• The 10k blocks of ranges are arbitrarily sized, and they could be either too small (new ranges would have to be reserved in the future) or too big (prime kind real-estate will forever go unused).
• Users who aren't aware of the special handling of a particular kind mind start using it and get unexpected event lifetimes.

Importance of parameterised replaceable events

I believe NIP-33 is a very important specification and will be heavily used by future protocols.

The NIP itself doesn't contain any use-cases, but as a basic example, I could imagine using these types of event lifetimes for sharding and segregating your kind-3 contact lists. You could imagine having multiple different contact lists, like family, friends, co-workers, randos1, and randos2. This would future-proof contact lists growing too large, improve the efficiency of updates, and provide a natural organisation schema for users.

Furthermore, if it were possible to use parameterised replaceable events on kind 3, this would all work with minimal client changes and a reasonable backwards compatibility story. I'm not saying this is a good idea and we should do it, just pointing out the possibility.

Note that replaceable events are just a special case of parameterised replaceable events with a static replacement key.

Proposal

Here are two minor changes that I believe would generalise the protocol:

• Extend NIP-40 to special-case an expiration tag with value 0. This would indicate that the event is intended to be ephemeral, and should be treated as described in NIP-16, regardless of kind.
• Support the d tag on events of any kind. This indicates the event is to be treated as a parameterised replaceable event as described in NIP-33, using the value of this tag as a replacement key. This works for any kind, but the kinds of the replaced and replacing events must still match. The absence of a d tag does not result in an implicit empty d tag as described in NIP-33.

Disadvantages

• Events that want replaceable behaviour become slightly larger, since they need to include a d tag.
• A user could forget to include a d tag and unintentionally have multiple events of the same kind on a relay. However, when querying, clients that expect replaceable behaviour could take care to select the latest one (by created_at). Additionally, the user could correct this situation using a NIP-09 deletion.

Back-compat

The two above changes would not break any of the existing NIPs. However, I think it could make sense to create a NIP that supersedes NIPs 16 and 33, and mark the coupling of event lifetime with the 10k kind-ranges deprecated.

Additionally, here is how a relay could internally implement backwards compatibility using the d and expiration primitives:

• Kinds 20000-29999 have an implicit expiration tag with value 0 added to the front of the tag list. NIP-40 is clarified to specify that in the case of multiple expiration tags, the first tag's value is to be used.
• Kinds 0, 3, 41, 10000-19999, and 30000-39999 have implicit "d" tags with "" values added to the end of their tags list.

History

• The original version of this proposal suggested using a replace tag instead of NIP-33's d tag, but this would not have been queryable as pointed out by @barkyq below.

[barkyq]

I think the "d" tag is good to be queryable.

That way a client can ask for a given note by referring to the "d" tag rather than the changing event id.

I have no idea what "d" stands for, maybe "document", as I think one of the original use-cases for P.R,E. was editable documents.

[barkyq]

As an aside/old-man-complaint... I am not convinced of the utility of general purpose "expiration" tag. Just send a deletion event when you want your event to expire. Puts unnecessary burden on the relays IMO. Feel like most users will opt to not use the expiration "feature" and the relay will just always be checking for events to delete without ever being able to delete very much.

On the other hand, ephemeral events is super relay friendly 👍

Agreed that it would be nice to have an ephemeral event related to kind 1. Naively could just define 20001 as ephemeral kind 1. I guess it is not a very principled solution though. and many unanswered questions viz UI

[barkyq]

Also love strfry super impressive work, wish I understood C++ better. Very very good performance on wss://nos.lol

[hoytech]

Thank you!

You're completely right about the d tag being useful for querying. Especially if the user has a large number of PREs of the same kind. I'll edit the above proposal to get rid of my proposed replace tag in favour of using the NIP-33 d tag -- my mistake.

I partially agree about not using expiration and instead just sending a delete message when you want your events to go away since deletion is more general purpose. However there are some arguments in favour of expiration:

• You might not know all the relays that your event has propagated to, and there is no way to ensure that your delete message will reach all these relays too.
• You might not be available to sign the delete message right at the expiration time (network down, computer crashed, whatever). Even if you do sign it right on time, it won't necessarily propagate to all the relays on time. Clients that understand the expiration tag can immediately stop showing it when the time comes (and could even display a countdown or whatever too).

It does add a bit of extra load on the relays, but in strfry I've added an index on this field so the periodic checking for expired events is very inexpensive.

Good point that an event of 20001 could be standardised as an "ephemeral kind 1". However, clients would have to be re-coded to query for this kind, and display it. My proposal allows "ephemeralness" to compose with any kind so it would be more likely to work with existing clients.

[fiatjaf]

I like this. It is sad that we made bad choices in the past.

[fiatjaf]

@Semisol @cameri

[barkyq]

Seems easy enough to implement. Will need to reserve the d tag as a special tag for all events; do not suppose that should be added to "generic tag queries" NIP (something like the tags p, e, d are reserved). Would be a shame if some other NIP started using d tags on non 30000-39999 kinds.

Also expiration being less than created_at should automatically imply the event is treated as ephemeral? Seems like an easy enough trigger to implement in SQL or any relay implementation.

Seems like expiration < created_at would be totally useless unless expiration was queryable. If it was queryable, then setting low expiration values would give a defacto integer "kind" one could associate to ephemeral events. Most relays implementing NIP-40 will probably already have indices for the expiration. Although the way the queryable tags work, it would be very awkward to query for, say, all notes which will expire in 2 days.

Perhaps could add custom query rules to NIP-40, something like, in the filter JSON keys like:

"expiration": ["at", "1676088797"]
"expiration": ["before", "1676088797"]
"expiration": ["after", "1676088797"]

Obvious interpretation as filters on the "expiration" tag. Events which do not have the expiration tag should not be returned etc etc.

Would be kind of cool.

[barkyq]

Also, to reduce long-lived spam, popular people like jack could say stuff like, "I am filtering by expiration in less than 24 hours, let's have a discussion about github on nostr etc etc". Kind of a stoner thought and certainly many things potentially problematic but curious to hear what the author and others think.

[hoytech]

Also expiration being less than created_at should automatically imply the event is treated as ephemeral?

EDIT (mis-read this). This could work too. What would the different possible values for expiration represent, how ephemeral it is? :)

Maybe instead of overloading expiration, we should instead use a new tag like ephemeral. If present, the event should be considered ephemeral. This would probably have a bit better backwards compatibility actually, for relays that have implemented NIP-40 but not the 0 overload.

TBH I can't think of many use-cases for users querying by expiration and/or ephemeralness. I agree your method for querying numeric ranges is cool, but I don't think we should burden relay implementors with that unless we have some really solid use-cases.

[barkyq]

Yes I just figured that since it would be straightforward to check if expiration < created_at, it would add a new spot to put some extra information, without really increasing the size of the event very much.

Re: querying for expiring events. Yes I agree that it would be annoying to coordinate all the different relay codebases to add a custom filter parsing rule just to query expiration tags... The part which motivated me was that probably the DBs already have an index for this expiration integer. But yes a bit convoluted and not very intuitive.

Re: changing to ephemeral tag. This tag ephemeral would not take any arguments? Seems pretty elegant IMO.

---

If a d tag is present, the event is considered replaceable, and the second argument in the tag is considered as the "identifier" for the event (which can persist, i.e., the title of a manuscript / news article or something). Any kind can have a d tag.

One potential source of conflict is if people put d tags (with different values) on events which were historically considered replaceable like 0, 3, 41, 10000-19999. Like trying to implement a collection of kind 3 events for family, friends, etc. What do you think about that?

Also a bit strange if kind 5 (event deletions) got d tags, as it would be like deleting a deletion event, which I think is supposed to not be allowed.

---

If the ephemeral tag is present, the event is considered ephemeral. Are there any event kinds which should definitely not have an ephemeral tag? Cannot think of any right now.

Perhaps there could be a "fake" relay hint added to e tags when replying to ephemeral events. This could signal to clients to not bother looking for a given event (or even display some little box where the event should be saying "ephemeral event"). Probably beyond the scope of this NIP though.

One more question. Do you specify if there is an event with d tag and ephemeral tag? Two options, probably depending on the order people write SQL triggers. Guess one canonical order should be specified. Probably the old event with same d tag and kind gets deleted then the new one gets broadcasted without being stored (or gets stored very briefly for strfry)? At the end of the process there is no event with the same d tag, kind, and pubkey?

[hoytech]

One potential source of conflict is if people put d tags (with different values) on events which were historically considered replaceable like 0, 3, 41, 10000-19999

I think it makes the most sense for events with default-replaceable kinds to be treated as NIP-33 events. So if there is no d tag it effectively uses an empty string as the replacement key. That way the interpretation of events without d tags does not change. If people start adding events with non-empty d-tags, then these new events will exist in parallel with the events without d-tags. I agree this might cause some confusion for clients that only expect one kind 3 event, but clients really ought to handle this case anyway, since a relay does not have to implement NIP-02 (and even then, NIP-02 says replacement is "SHOULD").

Also a bit strange if kind 5 (event deletions) got d tags, as it would be like deleting a deletion event, which I think is supposed to not be allowed.

Agreed, deletions is an interesting case. NIP-09 says "Publishing a deletion event against a deletion has no effect. Clients and relays are not obliged to support "undelete" functionality."

Which doesn't seem definitive either way to me. Personally I see no reason not to support "undeletes" (either with subsequent NIP-09 deletions or with d-tags). The relay-side code is even easier that way.

Deletes are actually another interesting use-case for composing ephemeral events. The natural processing of an ephemeral deletion would be to delete the referenced events but not store the deletion event itself -- this might be useful for cleaning up old junk from relays without leaving a bunch of delete events lying around. Of course, somebody could re-submit the old events if they kept copies of them, but that's not always a concern.

I can't really think of any other events that shouldn't compose with ephemeral events. As mentioned above, I believe you can always emulate ephemeralness with a NIP-09 deletion anyway.

Somebody making a reply to an ephemeral event -- that's actually a tricky one and I could see it causing problems, although these problems are much the same as exist now when replying to an event that subsequently gets deleted. I think your suggestion about flagging ephemeral events in the UI is a good one.

If d and ephemeral are on the same event I think the natural order of processing is as you describe: Replacement happens, the event is broadcast, and then the event is deleted (or never stored). You're right this should probably be made explicit.

Thank you for thinking through all these edge cases!

[barkyq]

Alright. I am on board! Excited to hear what others think. 👍

[ryzizub]

I love this 🔥 Thanks @hoytech for pointing this out

[cameri]

I think the "d" tag is good to be queryable.

That way a client can ask for a given note by referring to the "d" tag rather than the changing event id.

I have no idea what "d" stands for, maybe "document", as I think one of the original use-cases for P.R,E. was editable documents.

d is for deduplication

[cameri]

@hoytech @fiatjaf I am pretty much onboard with these changes.

[scsibug]

I'm in agreement as well.

[fiatjaf]

OK, I think that gives us enough quorum to proceed with this.

[fiatjaf]

On the other hand I think this is a bad idea now. We don't want to create unnecessary burden on clients by allowing every event to become instantly replaceable. If kind:1 notes are replaceable that would mean a disproportionate amount of complexity added to otherwise simple social clients.

[scsibug]

Is it better for kinds to opt-in (as part of their NIP) to non-replaceability? And then have relays ignore the d tag (if one exists) in those cases? Agreed that kind:1 replaceability is not desirable.

[peterzion45]

Who understands these things except for the developers/programmers/and coders?
500ef1fdb07f3ac3a9d6c660c27df6af03acf661ba8fb9d651914a5c87e5bdf9