BUGFIX: longer tokens violate maximum SMTP command line length

[AlesKrajnik]

This BUGFIX for issue #69 fixes the case when the XOAuth2 is longer than 497 bytes. Such tokens must not be sent in the initial response (the AUTH XOAUTH2 <token> command), but rather send an empty initial response (AUTH XOAUTH2), waiting for the server to reply with 334 and then sending the token. (https://datatracker.ietf.org/doc/html/rfc4954)

[AlesKrajnik]

The unit tests for the original implementation of XOAUTH2 do not exist and I'm afraid it would require a major refactor. I tested it locally on my computer against Dovecot, but I'd feel more comfortable if it could be tested more. How to proceed?

[AlesKrajnik]

This should return 334 and empty message. I wonder if we should somehow test that the message from server is empty?

[schengawegga]

@aleshaczech
Thanks you for your pull request.
At the moment, i try to reproduce the issue, but have no success with productive mailservers.
GMail, for example, did not response an error with a OAuth base64 larger than 512 bytes.
So i will try to install my own dovecot an a local development machine and see, if it works and if we need to check that the multiline response is empty or not.
I´ll keep you on track. ;-)

[AlesKrajnik]

Hi @schengawegga,

let me add a few more details, hopefully it helps to better review and test my PR.

TLDR: Dovecot, when acting as a SMTP client, follows the specification strictly and their implementation can be used to verify my PR. However, when Dovecot acts as a SMTP server, it's probably more tolerant, hence the testing needs to be done with fairly long XOAuth2 tokens in order to observe the buggy behaviour.

---

In Dovecot, this obviously appears only with submission-login service. I am not sure if it affects any setup or only some particular configuration, but some hints are in the Dovecot's changelog for the latest version 2.3.19:

• submission-login: Long credentials, such as OAUTH2 tokens, were refused during SASL interactive due to submission server applying line length limits.
• submission-login: When proxying to remote host, authentication was not using interactive SASL when logging in using long credentials such as OAUTH2 tokens. This caused authentication to fail due to line length constraints in SMTP protocol.

…and also for the previous version 2.3.18:

• submission-login: Authentication does not accept OAUTH2 token (or other very long credentials) because it considers the line to be too long.

As mentioned in a Dovecot's forum's thread, the affected versions were 2.3.18 and 2.3.19. In my case, the problem was observed with the version 2.3.19. The aforementioned thread also points to a commit that shows how Dovecot handles the SMTP XOAUTH2 when acting as a SMTP client (either when proxying submission to the submission server, or when acting as a generic SMTP client) - the same case my PR's is addressing.

From https://github.com/dovecot/core/blob/release-2.3.19/src/submission-login/submission-proxy.c#L267 (and similarly in https://github.com/dovecot/core/blob/release-2.3.19/src/lib-smtp/smtp-client-connection.c#L963):

if (str_len(sasl_output_base64) == 0)
	str_append(str, " =");
else if ((5 + strlen(mech_name) + 1 + str_len(sasl_output_base64)) >
	 SMTP_BASE_LINE_LENGTH_LIMIT)
	client->proxy_sasl_ir = i_strdup(str_c(sasl_output_base64));
else {
	str_append_c(str, ' ');
	str_append_str(str, sasl_output_base64);
}

From https://github.com/dovecot/core/blob/release-2.3.19/src/lib-smtp/smtp-common.h#L10:

#define SMTP_BASE_LINE_LENGTH_LIMIT (512 - 2)

---

It's worth mentioning that I observed Dovecot being able to receive tokens longer than 512 bytes. I believe Dovecot is more tolerant than the specification demands when acting as a server, but it's following the specification strictly when acting as a client.

I think the relevant values are somehow related to https://github.com/dovecot/core/blob/release-2.3.19/src/lib-smtp/smtp-command.h#L4, but I didn't test it thoroughly enough to pinpoint the exact threshold when Dovecot's submission server refuses the token.

If I am not wrong, that means that testing needs to be done with tokens way longer than 512 bytes to observe the behaviour, I vaguely remember tokens over 1.000 bytes still being okay for Dovecot. I observed the issue when the JWT's payload was long and signed with either 2.048-bit or 4.096-bit RSA keys.

Dovecot's test suite actually uses tokens of length 1.792 bytes as can be seen here: https://github.com/dovecot/core/blob/release-2.3.19/src/lib-smtp/test-smtp-server-errors.c#L831.

Hope this helps.

[AlesKrajnik]

Hi @schengawegga,

I continued a bit with the research and found out that Dovecot accepts SMTP AUTH command parameters of up to 1.026 bytes length, not including the CRLF (that is whatever follows after AUTH before CRLF). I guess there's some bug in Dovecot regarding the length, it should probably be 1.024 bytes, anyway...

Given that XOAUTH2 authentication mechanism command starts with AUTH XOAUTH2 , that would make the total maximum length of the XOAUTH2 token up to 1.018 bytes.

Anything longer should result in:

500 5.5.2 Line too long

I created a testing Dovecot Docker container, feel free to use it.

[thomascube]

@aleshaczech The propose change looks good to me.

[schengawegga]

@aleshaczech Sorry for the long waiting time. I will try to test your request with your docker container in the next days.

[schengawegga]

@aleshaczech thanks for the dovecot container. now i can reproduce the behavior of dovecot with auth parameters over 1.016 length (on my test runs), and your working bugfix. But i have one question. you wrote the RFC 4954 in the comment for the maximum token length of 512 bytes. but i can´t find any description in this RFC about 512 bytes, or any specific byte length. Can you explain, how you choose the length of 512 bytes to separate the auth command? I only want to understand, if it is possible that any other SMTP server has the limit of 512 bytes, and not at 1.016 or 1.026 as dovecot.

At least, thanks for your great research and pull request. I will think about, if it is really necessary to do an if-statement in the code. All other SASL send out the authentification method and the credentials in base64 separately. Only XOAuth2 tries to send it as a combination of auth method and credentials. The combination will safe request time, but the question is, is the limit of 512 bytes (497 bytes) a safe limit for all common SMTP servers?

Or, should we first put the combinated auth, and if that resolves in an "limit to long" message, do another try with the separated alternative. This will require more time, but it will be more close on the actual behavior of the code.

@aleshaczech what is your opinion?
@jparise what do you think about this two alternatives?

[AlesKrajnik]

Hi @schengawegga ,

thank you for testing it and confirming. Your findings of the maximum length of 1.016 bytes make sense, I guess I made a mistake measuring, maybe something to do with the <CRLF>.

Regarding the maximum length of SMTP commands, it's mentioned there in the (RFC 4954, section 4) indirectly as following:

Note that the AUTH command is still subject to the line length limitations defined in [SMTP]. If use of the initial response argument would cause the AUTH command to exceed this length, the client MUST NOT use the initial response parameter (and instead proceed as defined in Section 5.1 of [SASL]).

and later in the same section regarding the size of the token:

Note that these [BASE64] strings can be much longer than normal SMTP commands. Clients and servers MUST be able to handle the maximum encoded size of challenges and responses generated by their supported authentication mechanisms. This requirement is independent of any line length limitations the client or server may have in other parts of its protocol implementation. (At the time of writing of this document, 12288 octets is considered to be a sufficient line length limit for handling of deployed authentication mechanisms.) If, during an authentication exchange, the server receives a line that is longer than the server's authentication buffer, the server fails the AUTH command with the 500 reply. Servers using the enhanced status codes extension [ESMTP-CODES] SHOULD return an enhanced status code of 5.5.6 in this case.

(It's not very clearly said, but I believe that refers to the token length when sent outside of the initial response, that is when not part of the AUTH command but rather when sent on its own line, or when received by client).

The referenced "SMTP" is RFC 2821, section 4.5.3.1 and states:

The maximum total length of a command line including the command word and the <CRLF> is 512 characters. SMTP extensions may be used to increase this limit.

---

There's another definition of length in RFC 4954, section 3, however, that's unrelated to the issue I opened. It's the following:

An optional parameter using the keyword "AUTH" is added to the MAIL FROM command, and extends the maximum line length of the MAIL FROM command by 500 characters.

That only extends the size of the authentication when part of the MAIL FROM command like this:

MAIL FROM:<e=mc2@example.com> AUTH=e+3Dmc2@example.com

I am mentioning it since it might be the reason why some implementations decided to support longer AUTH command, such as Dovecot and their 1.024 bytes or so, perhaps to address frequently broken clients due to misinterpretation of the RFCs?

---

It's quite difficult to check all the RFCs and how they link to each other, so I'm not pretending to be sure. And, apparently, there are mismatches between RFC and reality, such as the case of Dovecot that seems to be more tolerant than the RFC requires.

But exactly for that reason I'd suggest to follow the strict interpretation of RFC when sending data and go with 512 bytes for the AUTH command with the initial reponse.

Regarding the idea not to use the initial response at all, that's related to this part of the RFC 4954, section 4:

The optional initial response argument to the AUTH command is used to save a round-trip when using authentication mechanisms that support an initial client response. If the initial response argument is omitted and the chosen mechanism requires an initial client response, the server MUST proceed as defined in Section 5.1 of [SASL]. In SMTP, a server challenge that contains no data is defined as a 334 reply with no text part. Note that there is still a space following the reply code, so the complete response line is "334 ".

So, indeed it's an optimisation. I wouldn't mind keeping it, it can save a bit, but it makes the logic a bit more complex and error prone, so I don't have a strong opinion here. At least in my use case (JWT tokens signed with 2.048 bytes or 4.096 RSA keys) the token is almost always longer, I can imagine more people having similar use case, so perhaps the saving in practice are not that big.

On the other hand, for short tokens, the behaviour would stay exactly as it is now, if I get it right; I mean that keeping this optimisation would keep the current behaviour for short-enough tokens, which could reduce the risk of running into another set of problems (e.g. what if some servers do not support receiving the token on a separate line due to their broken implementation of the RFC?).

Regarding trying to send the token in the initial response and then handling the "too long" error, I would suggest to avoid it for a couple of reasons:

1. I am not sure how different servers would deal with it. Would they keep the connection and let the client retry?
2. Firewalls could have issues with that too. They would see a failed authentication and could act upon it.
3. It could make life of the administrators more difficult when inspecting logs.
4. Finally, and perhaps most importantly, I believe that would make the Net_SMTP library deliberately violating the RFC and expecting the servers to handle it. I would rather stick to the standards.

Hope this helps.

[schengawegga]

Hi @aleshaczech ,
thanks for your detailed explanations and support.
I´ve merged your pull request and released a new version of PEAR/Net_SMTP.