Add empty tag-only message support and an increased size limit for tags #287

Merged
merged 25 commits into from Feb 16, 2017

Projects

None yet

9 participants

@jwheare
Member
jwheare commented Dec 20, 2016 edited
  • Specs a TAGMSG tag-only event. Enables things like content-less reactions/faving, message deletion, message seen notifications (ircv3/ircv3-ideas#1)
  • Increases the size limit for tags to 4096
  • Cap name version bump to draft/message-tags-0.2
@jwheare jwheare added this to the Roadmap milestone Dec 20, 2016
core/message-tags-3.3.md
+### Size limit
+
+The size limit for message tags is increased to 512 to 4096 bytes, including the leading
+`@` and trailing space characters, leaving 3094 bytes for tags themselves. The size limit
@dequis
dequis Dec 20, 2016 Contributor

s/3094/4094/

core/message-tags-3.3.md
@@ -96,6 +96,9 @@ no message body is provided. These events MUST NOT be delivered to clients who h
negotiated the message tags capability. If neither client-only tags nor a message body
are provided, servers MAY respond with a standard `ERR_NOTEXTTOSEND` error.
+Clients that receive empty events MUST not display empty messages as-is in the message history
@DarthGandalf
DarthGandalf Dec 20, 2016 Member

s/not/NOT/

core/message-tags-3.3.md
+An alternate form of an empty message sent by a client with the `+example-client-tag` client-only tag, where the last parameter is omitted
+
+```
+@+example-client-tag=example-value PRIVMSG #channel
@DarthGandalf
DarthGandalf Dec 20, 2016 Member

This allows omitting the last parameter of PRIVMSG.
What other commands can omit it now?

Note that this change is pretty far away from tags themselves.

I don't think clients should be allowed to send empty messages at all, even if the actual message is not in last parameter anymore, but in the tag. That way old clients have some way to show the conversation. Clients which use such tag should duplicate the message in the text... which makes the new tag useless.

@dequis
dequis Dec 21, 2016 Contributor

As mentioned above in the diff, the only other command is NOTICE.

even if the actual message is not in last parameter anymore, but in the tag

This isn't about moving the actual message to a tag. If it's a message that has to be displayed as such, it still belongs in the last parameter. This is about opening another channel of communication that is only visible to clients that support this spec and won't share the same behavior of public messages. See for example #289, about reactions. Another example is typing notifications.

@jwheare
jwheare Dec 21, 2016 Member

@DarthGandalf other than the mentioned use cases for content-less messages, is your objection that this is allowing PRIVMSG #channel rather than PRIVMSG #channel :?

I could change it to require the last param but allow it to be empty. But since it's allowing that param to effectively be ignored, I'm not sure it's needed.


I see the reason for allowing this change is so that servers can keep the standard PRIVMSG/NOTICE routing for message delivery to channels/users, and use them for pure tag delivery. Another alternative is to define a new message type that just carries tags, but since empty messages aren't allowed/useful normally, it makes sense to use them.

@SaberUK
SaberUK Dec 21, 2016 Contributor

Another alternative is to define a new message type that just carries tags,

I'll start the bikeshedding for the name: METAMSG

@jwheare
Member
jwheare commented Jan 1, 2017

Any more input on this?

core/message-tags-3.3.md
+
+---
+
+An alternate form of an empty message sent by a client with the `+example-client-tag` client-only tag, where the last parameter is omitted
@SaberUK
SaberUK Jan 2, 2017 Contributor

IMO this should be the only form allowed. Theres no point adding alternative methods just for the sake of adding alternatives.

@attilamolnar
attilamolnar Jan 2, 2017 Member

+1 for only allowing one form

@attilamolnar
Member

The version of the draft cap should be bumped so early implementations can discover if a server is capable of empty PRIVMSG support

@DarthGandalf

Or even :#channel

Contributor

That's already covered by the core protocol; also, no need to make people accidentally think you can have spaces in that parameter.

@jwheare jwheare requested a review from attilamolnar Jan 6, 2017
core/message-tags-3.3.md
@@ -80,6 +89,24 @@ Individual tag keys MUST only be used a maximum of once per message. Clients
receiving messages with more than one occurrence of a tag key SHOULD discard all
but the final occurrence.
+### Empty tagged messages
+
+Servers MUST accept `PRIVMSG` and `NOTICE` events sent with client-only tags even if
@DarthGandalf
DarthGandalf Jan 7, 2017 Member

The difference between PRIVMSG and NOTICE is whether bots should reply to the message.
Should reactions be as PRIVMSG or NOTICE?

I prefer @SaberUK 's new METAMSG verb and not to change syntax of widely used PRIVMSG/NOTICE

@jwheare
jwheare Jan 7, 2017 Member

Since there's a CAP to enable this behaviour, I think it's OK. Bots that enable it can make sure they don't reply (and there'll be no text to reply to anyway).

A key benefit of using PRIVMSG/NOTICE is that servers already route them appropriately. Adding another verb feels heavyweight to me, but happy to discuss further.

@DarthGandalf
DarthGandalf Jan 7, 2017 Member

servers already route them appropriately

That's the exact problem. I think the majority of server implementations assume that there is a text for PRIVMSG. For example, https://github.com/znc/znc/blob/master/include/znc/Message.h#L297 has such assumption.
By adding a new verb, old code which handles PRIVMSG can continue assuming that there is always a text.

@jwheare
jwheare Jan 7, 2017 Member

To be clear, I'm not suggesting servers won't have to update their code to allow this. Whether it's by making PRIVMSG/NOTICE accept empty messages or by adding a new verb to be routed in exactly the same way as those two messages is the question. I feel like the former would be simpler, but maybe not in your case.

@attilamolnar
attilamolnar Jan 8, 2017 Member

I'm indifferent about this. Strictly speaking on the protocol level there is no reason to introduce a new verb. Servers have to apply nearly the same restriction checks for tag-only messages as for normal messages e.g. callerid, reg only speak, flood checks, mute on channels, etc. so code that interacts with tag-only messages will need to be reviewed to ensure it handles them correctly regardless of the actual protocol used.

Another thing to consider is echo-message being already defined for PRIVMSG but adding a new verb requires either changing the definition of echo-message to support it or define it with "built-in" echo. The latter will create a new corner case with self METAMSG in labeled responses that will need addressing. None of these are huge changes, though.

@jwheare
jwheare Jan 8, 2017 Member

Yes, this is my main concern with using a new verb. Having to enumerate in the spec all the ways in which it needs to be handled to match the way PRIVMSG and NOTICE are handled seems unnecessarily risky.

@DarthGandalf
DarthGandalf Jan 9, 2017 Member

Alternative idea to consider: use an empty PRIVMSG instead (@whatever PRIVMSG #chan :) which also doesn't require syntax changes, but doesn't require new verb.

@SaberUK
SaberUK Jan 9, 2017 Contributor

That still requires syntax changes because an empty last parameter for PRIVMSG and NOTICE is explicitly banned, see ERR_NOTEXTTOSEND.

@DarthGandalf
DarthGandalf Jan 9, 2017 Member

Right, but scope of this change is smaller.
I mean, all implementations which use some analog of std::string, don't need to change the whole codebase to std::optional<std::string> anymore. The same for "" vs None in Python

@jwheare
jwheare Jan 9, 2017 Member

Out of interest, how many places in the znc codebase would this affect? I ended up just following the robustness principle and accept both forms in my client/server implementations. In fact, I was already treating them as equivalent in the client.

@DarthGandalf
DarthGandalf Jan 9, 2017 Member

how many places in the znc codebase would this affect?

All modules, including external ones. Because the module callback has been virtual EModRet OnChanMsg(CNick& Nick, CChan& Channel, CString& sMessage);. This can't be called for PRIVMSG without text as it would be indistinguishable from PRIVMSG with empty text.

It's not only about ZNC. KVIrc would suffer from similar problems.

@jwheare
jwheare Jan 9, 2017 Member

This can't be called for PRIVMSG without text as it would be indistinguishable from PRIVMSG with empty text.

What's the consequence of losing this distinction? Is there a practical issue with "casting" a missing param to an empty param before calling the callback?

@DarthGandalf
DarthGandalf Jan 9, 2017 Member
SRV→ZNC: @tags :n!u@h PRIVMSG #chan
ZNC→CLI: @tags :n!u@h PRIVMSG #chan :

The requirement to not render the message for the client won't apply anymore.

@jwheare
jwheare Jan 9, 2017 Member

OK. This was my original motivation for allowing both forms. So either I can restore that or we can make the empty but present last param the only allowed form (which will probably attract comments that it seems unnecessary)

@DarthGandalf
DarthGandalf Jan 10, 2017 Member

+1 for "the empty but present last param the only allowed form"

@jwheare
jwheare Jan 10, 2017 Member

How's this? 304a03c

@attilamolnar
attilamolnar Jan 10, 2017 Member

+1 for "the empty but present last param the only allowed form"

How's this? 304a03c

I'm fine with this.

core/message-tags-3.3.md
+error.
+
+Clients that receive events without a message text MUST NOT display empty messages as-is
+in the message history unless they are using the attached tags according to their own
@DarthGandalf
DarthGandalf Jan 7, 2017 Member

I can't parse this, sorry. Who are "their own"? tags? clients? messages?

@jwheare
jwheare Jan 7, 2017 Member

The tags. I can try rewriting it.

@DarthGandalf
DarthGandalf Jan 7, 2017 Member

But... how else can they use those tags? According to specs of some other unrelated tags?

@jwheare
jwheare Jan 7, 2017 Member

Does this make more sense? 8667e48

@DarthGandalf
DarthGandalf Jan 7, 2017 Member

Yes, it does.

core/message-tags-3.3.md
+
+### Size limit
+
+The size limit for message tags is increased from 512 to 4096 bytes, including the leading
@DarthGandalf
DarthGandalf Jan 7, 2017 Member

I'm afraid that this creates an incentive for third parties to create custom extensions which move text of message to @example.com/text, to workaround the limit of 512

@jwheare
jwheare Jan 7, 2017 Member

That would be in direct contravention of the recommendation in the spec.

There is no guarantee that other clients will support or display the client-only tags they receive, so these tags should only be used to enhance messages with non-critical information. Messages should still make sense to clients with no support for tags.

If clients do that with their own prefixed tags, then they should expect it not to be widely adopted. I don't see it as a very effective incentive.

@attilamolnar

LGTM

@ProgVal
Contributor
ProgVal commented Jan 10, 2017

Enables things like content-less reactions/faving, message deletion, message seen notifications

Why do you want to use tags instead of a verb for these?
They clearly are actions, as much as AWAY, MODE, TOPIC, etc.

@attilamolnar
Member

Why do you want to use tags instead of a verb for these?
They clearly are actions, as much as AWAY, MODE, TOPIC, etc.

So features like what you quoted won't depend on server updates once client to client tags are supported.

@ProgVal
Contributor
ProgVal commented Jan 10, 2017

Message deletion and seen notifications will have to be supported by the server, for #292, and maybe faving too.
It may make sense for reactions to always be attached to a PRIVMSG, since they are a message; but it would have to be discussed in a separate spec.

I'm worried about the two possibilities discussed here, because not having a command would break one the most basic conventions of IRC, and using a content-less PRIVMSG would strip PRIVMSG of its meaning (why PRIVMSG and not NOTICE or MODE?).

From a less philosophical point of view, I am worried that using an empty PRIVMSG would either break third-party plugins of my bot which do not handle that case (if the bot negotiates the capability) or prevent other plugins from using the empty PRIVMSG from usin the feature (if the bot does not negotiate it).
And not using a command would break any third-party plugin that assumes the parser outputs a command.

What would you think of adding new verb(s?) for the sole purpose of carrying tags, on a channel, user-to-user, or from a network admin to all the users?

@jwheare
Member
jwheare commented Jan 10, 2017
  1. Can you explain why deletion/seen/faving would require server support? The chathistory can just store the empty messages and clients can replay them as normal.

  2. What does this mean?

not having a command would break one the most basic conventions of IRC

  1. If you're concerned, your bot plugin framework could require plugins to enable support for the CAP via some internal config, and then you can forward/filter them to individual plugins as necessary.

  2. Reasons for not adding a new verb have been discussed a few times already in this thread (search in page for "METAMSG")

@ProgVal
Contributor
ProgVal commented Jan 10, 2017 edited
@attilamolnar
Member
attilamolnar commented Jan 10, 2017 edited

@ProgVal I don't understand the issue. The bot can have two events, one for regular PRIVMSG the other one for tag only PRIVMSG. Pass regular PRIVMSGs to the first, tag only PRIVMSGs to the second. Exactly as you would do with a third verb. Did I miss something?

@ProgVal
Contributor
ProgVal commented Jan 10, 2017

Yes, I could do that. I am not sure how I would name them, because all callbacks have the name of the command, so I can't have two callbacks for the same command with this naming scheme. That's why I would prefer a new verb.

@jwheare
Member
jwheare commented Jan 10, 2017

There isn't a deletion spec at the moment, so this is slightly off-topic speculation. If one is proposed that uses client tags it would have to be an honour system. Servers would probably be allowed to delete them from chanhistory, etc, but it wouldn't be enforced. Given it's not possible to delete people's logs remotely, I don't see how any deletion system that doesn't rely on honour would work. The spec would have to caveat that strongly, that it wouldn't be a way to completely redact sensitive data, just allow honourable clients to mitigate the damage, or if nothing else, just a way to clean up cosmetic double posting etc, same with editing.


Re: naming schemes, do you have a separate callback for /me ACTION messages or do all plugins have to parse the CTCP from a PRIVMSG themselves?

In the IRCCloud API, we have PRIVMSG -> buffer_msg, NOTICE -> notice, ACTION -> buffer_me_msg, and empty tag messages -> empty_message. Our mobile apps don't yet support empty_message and so safely ignore them.

The motivation and scope for using empty existing verbs is to make integration at the IRC protocol level simple for a wide range of implementors. Since you have your own API interface, you're well placed to create your own abstractions that are more suited to your specific domain.


Don't servers already factor code for PRIVMSG and NOTICE? If yes, adding a third verb routed exactly like them shouldn't be a problem

It would potentially be a lot of extra clauses in a lot of different places. Not just routing but as mentioned previously "callerid, reg only speak, flood checks, mute" just for it to work. Those cases might need to be reviewed to be tag aware if that becomes an issue in light of new specs, but for base compatibility with this spec, pretty much only the ERR_NOTEXTTOSEND check is absolutely necessary for a server.

@ProgVal
Contributor
ProgVal commented Jan 10, 2017

Re: naming schemes, do you have a separate callback for /me ACTION messages or do all plugins have to parse the CTCP from a PRIVMSG themselves?

There is no separate callback, but they can call a function to know whether it's a CTCP.

Since you have your own API interface, you're well placed to create your own abstractions that are more suited to your specific domain.

For now, someone who knows the IRC commands can guess the name of the callback without looking at the doc or the code.
Adding the kind of abstraction you suggest would require a new name for this callback that cannot be guessed without looking at the doc.

It would potentially be a lot of extra clauses in a lot of different places. Not just routing but as mentioned previously "callerid, reg only speak, flood checks, mute" just for it to work. Those cases might need to be reviewed to be tag aware if that becomes an issue in light of new specs, but for base compatibility with this spec, pretty much only the ERR_NOTEXTTOSEND check is absolutely necessary for a server.

Can't you just grep for PRIVMSG and NOTICE and replace all references to them by calls common functions that can handle the three verbs?

@jwheare
Member
jwheare commented Jan 10, 2017

I don't think ircv3 can commit to only speccing features that are guessable without docs.

@ProgVal
Contributor
ProgVal commented Jan 10, 2017

My point is that content-less PRIVMSG complicates client APIs.

So from my point of view, METAMSG vs content-less PRIVMSG is: simplicity of client APIs (+ consistency of the semantic of PRIVMSG) vs less code to adapt for existing servers

@jwheare
Member
jwheare commented Jan 10, 2017

I suppose my point is, some complexity will always exist either way. And it won't necessarily be evenly distributed. I also think it's easy to make assumptions about what will and won't be complex for different software to implement.

Luckily, as this is a draft that relies on implementation before being ratified, we don't need to assume. So for now we should go with majority consensus, and after we've had more implementations we can discuss it more.

@DarthGandalf
Member

I would much prefer METAMSG, for the same reasons as @ProgVal but I can accept empty PRIVMSG instead if really needed.

@jwheare jwheare referenced this pull request Jan 11, 2017
Open

Reaction client tag #289

@jwheare
Member
jwheare commented Jan 13, 2017

After finding a few bugs in my server implementation where I was sending empty PRIVMSGs to other clients that didn't have the cap enabled, I'm going to try out an implementation using a different verb: TAGMSG.

@jwheare
Member
jwheare commented Jan 13, 2017

I've implemented this and I now agree it's much better and less error prone.

@jwheare jwheare requested a review from attilamolnar Jan 13, 2017
@MuffinMedic
MuffinMedic commented Jan 13, 2017 edited

For naming purposes, is this new verb intended to be used only for sending tags without any user-text (TAGMSG) or any instance where we might want to send metadata (METAMSG)?

(If no use cases exist where metadata is sent outside of tags, then this becomes a moot point.)

@DanielOaks
Member

Metadata's handled with its' own command(s), so that shouldn't be an issue for this verb.

@jwheare
Member
jwheare commented Jan 16, 2017

Barring no other objections, I'll merge this tomorrow.

core/message-tags-3.3.md
+
+A new event `TAGMSG` is defined for sending messages with tags but no text content.
+This event MUST be delivered to targets in the same way as `PRIVMSG` and `NOTICE`
+events, taking into account channel membership and modes.
@attilamolnar
attilamolnar Jan 17, 2017 Member

How does echo-message interact with this? This should be documented explicitly with examples.

@attilamolnar
attilamolnar Jan 17, 2017 Member

Same goes for status TAGMSG, e.g. TAGMSG @#chan.

@jwheare
jwheare Jan 17, 2017 Member

Is there a statusmsg spec we can refer to? Is this covered by "this behaves exactly the same as PRIVMSG/NOTICE"? If not, what else needs an explicit mention?

@dequis
dequis Jan 17, 2017 Contributor

Is there a statusmsg spec we can refer to?

It's mentioned very briefly in the ISUPPORT drafts.

I'd say it counts as behaving exactly the same as PRIVMSG/NOTICE, but explicit mentions don't hurt.

@jwheare
jwheare Jan 17, 2017 Member

Yeah can definitely add some more examples, just wondering where to draw the line. Should all possible target types be included as in the PRIVMSG examples from 1459?

core/message-tags-3.3.md
+If this event is sent without any tags, servers SHOULD reject it with a standard
+`ERR_NEEDMOREPARAMS` error numeric.
+
+See `PRIVMSG` for more details on replies and examples.
@attilamolnar
attilamolnar Jan 17, 2017 Member

What is referenced here?

@jwheare
jwheare Jan 17, 2017 Member

I mean, that sentence is taken from the end of NOTICE. Should I link the word PRIVMSG to https://tools.ietf.org/html/rfc1459#section-4.4.1 ?

@jwheare
Member
jwheare commented Jan 19, 2017

Added some examples and links. If there are no further comments, I'll merge this on Monday.

@jwheare
Member
jwheare commented Jan 24, 2017

@attilamolnar happy to approve these changes?

@attilamolnar
Member
attilamolnar commented Jan 25, 2017 edited

@jwheare Please add 2 more examples with TAGMSG interacting with labeled-response and echo-message in the normal msg case and in the self message case for complete clarity

Edit: there is already an example for echo-message, never mind that part of my comment.

core/message-tags-3.3.md
+These events MUST NOT be delivered to clients that haven't negotiated the message tags
+capability and MAY be rejected if no tags are included.
+
+If this event is sent without any tags, servers SHOULD reject it with a standard
@attilamolnar
attilamolnar Jan 25, 2017 edited Member

Not that important, but any reason this is not a MUST? i.e. if servers choose to disallow tagless TAGMSG then they must use the given numeric for that. As currently stands servers are allowed to do whatever they want with a tagless TAGMSG making this a useless requirement.

@jwheare
jwheare Jan 25, 2017 Member

It's a SHOULD in case servers don't want to bother to check the tags on TAGMSG for simplicity, since clients receiving them would be a no-op anyway.

How about this:

If this event is sent without any tags, servers SHOULD reject it. If they do, the standard ERR_NEEDMOREPARAMS error numeric MUST be used.

@jwheare
jwheare Jan 25, 2017 Member

Actually no, I hadn't re-read the previous paragraph, I'll find a better wording.

core/message-tags-3.3.md
+capability and MAY be rejected if no tags are included.
+
+If this event is sent without any tags, servers SHOULD reject it with a standard
+`ERR_NEEDMOREPARAMS` error numeric.
@attilamolnar
attilamolnar Jan 25, 2017 Member

Please expand the examples to demonstrate the correct server behavior or behaviors in this corner case.

core/message-tags-3.3.md
+These events MUST NOT be delivered to clients that haven't negotiated the message tags
+capability and MAY be rejected if no tags are included.
+
+If this event is sent without any tags, servers SHOULD reject it with a standard
@attilamolnar
attilamolnar Jan 25, 2017 Member

Yeah this is what I had in mind.

@jwheare
Member
jwheare commented Jan 25, 2017

Is a repetition of the label/echo/self messages examples really needed here? The labeled-response spec is not specific to PRIVMSG/NOTICE, but should apply to all messages. I don't think we need to document it on every new event we define.

@jwheare
Member
jwheare commented Jan 25, 2017 edited

I added a @label tag to the echo-message example, but I don't think the obscure and hard to describe self-message corner case needs repeating here.

@jwheare
Member
jwheare commented Jan 28, 2017 edited

TODO

  • Add error numeric for tag length limit exceeded.
  • Swap "event" for "command"
@jwheare
Member
jwheare commented Jan 28, 2017 edited
  1. The word "command" and "message" are used somewhat interchangeably in the RFCs ("event" is not). The distinction seems to be that a "command" refers to the numeric/verb itself or the act of issuing or processing it, while a "message" refers to the complete message with prefix and parameters and the act of delivering it. Somewhat subtle but I tried to use them appropriately.

  2. I changed the link to 2812 for PRIVMSG examples because it has a better definition of what a <msgtarget> is.

  3. Added an explicit error numeric for tags that exceed the limit. ERR_INPUTTOOLONG 417 is used in ircu and derivatives for lines that exceed the traditional untagged 512 limit. I think it's much more robust to explicitly error on this case instead of having rules or undefined server behaviour around truncation, leading to arbitrary data loss.

@dequis
Contributor
dequis commented Jan 28, 2017

Added an explicit error numeric for tags that exceed the limit. ERR_INPUTTOOLONG 417 is used in ircu and derivatives for lines that exceed the traditional untagged 512 limit. I think it's much more robust to explicitly error on this case instead of having rules or undefined server behaviour around truncation, leading to arbitrary data loss.

For a sec I thought this was #281, where requiring that numeric may or may not happen. But it seems uncontroversial in the context of the new message-tags, with all its new required behaviors, and particularly about exceeding the tag length limit. So yeah, sure.

I think @DanielOaks was interested in reviewing the usage of that numeric first.

@jwheare
Member
jwheare commented Jan 31, 2017

I've discovered a slightly edge casey issue while implementing the ERR_INPUTTOOLONG error. A client might send a message that comes in under the limit, but the server then adds tags that push it over the limit.

The server could respond with the numeric anyway, but there are a few problems:

  1. The server needs to precalculate the length of any tags it might add. If limit checking happens at the raw input parsing stage, this adds complexity further down the process.
  2. The server can't wait until it has generated the full response to detect this and convert to an error, because side effects like forwarding the message to other clients will have already happened.
  3. But, there may theoretically be tags introduced in future that mean it's not always possible to know the length of all the tags until those messages have been delivered, or at least checked, creating more complexity. e.g. (off the top of my head) if we introduce a tag that indicates how many targets received the message, or if different recipients get different tag sets.

Because of this, we have a situation where tags might again need to be truncated unexpectedly.

One solution is to allow for truncation in the spec, perhaps adding a @truncated=<length> tag at output processing time to let clients know what happened. This would of course result in even more truncation. It also weakens the point of having an error numeric in the first place.

Another idea is that we separate the limits. So, for example the total limit could be 4608, with 512 reserved for server tags and 4096 for client tags. It allows input parsing to remain simpler, safe and at the edges for both client and server with no unexpected data loss due to truncation.

Any issues with that? Also, any thoughts on the numbers? Perhaps we still need more than 512 for server tags, but client tags are the main motivation for increasing the limit.

@jwheare
Member
jwheare commented Feb 11, 2017

Added the split limit details in dba7cd5

@jwheare jwheare merged commit c7ab1c4 into ircv3:master Feb 16, 2017
@jwheare jwheare deleted the jwheare:message-tags-empty+size branch Feb 16, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment