TextParser: Tolerate trailing whitespace

[beorn7]

Apparently, trailing whitespace is not tolerated, see prometheus/prometheus#1469 .

[beorn7]

Depending on the decision on prometheus/docs#543, it might be alright to not tolerate trailing whitespace.

[beorn7]

Un-assigning from myself as I intend to focus on other Prometheus project. Right now, others are way more involved in text format affairs than I am.

[RalfJung]

I just ran into this. The error message with a trailing space is really confusing. It says text format parsing error in line 1: expected integer as timestamp, got \"\". But when you add a timestamp it says that's wrong, too.

[brian-brazil]

I don't think we'll be making any changes here, if you use a client library it'll handle it for you plus OpenMetrics will supersede all this.

[beorn7]

I guess a typical scenario where to run into this is the textfile collector of the node exporter. That's where all kind of whitespace usage will be encountered because the textfile snippets are often created in a makeshift way. From that perspective, I see value in allowing trailing whitespace.

A different story is if and when the textfile collector will switch to OpenMetrics. If it does, we'd impose the strict whitespace requirement of OpenMetrics to all the creators of textfile snippets. (In principle, I'd love to let the textfile collector converge on OpenMetrics, too, but I'd also prefer OpenMetrics to be liberal with whitespace, which isn't going to happen.)

[brian-brazil]

My thoughts are that changing the Prometheus text format this late in the game is unwise, anything that it currently does is now defacto the standard. Keep in mind that there are parsers other than the ones we maintain, and this would be a breaking change.

[beorn7]

We wouldn't change the creation code, we would only make the parser more tolerant. It doesn't have to be part of the standard, it's just following the https://en.wikipedia.org/wiki/Robustness_principle .

Obviously, it would be bad if we saw the parser as a test to certify standard compliance of Prometheus text format generators. I wouldn't see it that way, simply because the Prometheus text format is not a publicly advertised standard. (That would be OpenMetrics, for which those tests exist or will exist.)

[brian-brazil]

Obviously, it would be bad if we saw the parser as a test to certify standard compliance of Prometheus text format generators

This code is the official parser. If it parses it, it's in compliance.

We wouldn't change the creation code, we would only make the parser more tolerant.

Thus that would be changing the standard.

it's just following the https://en.wikipedia.org/wiki/Robustness_principle

This turns out to be a bad idea, due to things like this.

I wouldn't see it that way, simply because the Prometheus text format is not a publicly advertised standard.

I don't see how that's relevant. We've always defined this code to be the reference implementation.

[beorn7]

I don't follow that line of argument. It all hinges on the premise that the parser is a compliance test and that the parser code defines the standard.

For better or worse, that was never standardized or advertized.

We are in a gray area here. I would like to navigate this ambiguity to the benefit of our users. In this case, allowing trailing whitespace is what most people will expect (as proved by multiple people that have run into this). We'll help all of them.

The counter case to bring up here is that somebody might have implemented a parser that doesn't tolerate trailing whitespace, too. If now somebody creates text format with trailing whitespace that then somehow makes it to that 3rd party parser, there will be a problem. That's a lot of If's:

• A 3rd party parser has been implemented (reasonably likely), and it also rejects trailing whitespace (which is less likely – even our own intention was to accept trailing whitespace, it came as a surprise that we didn't, note that this issue is filed as a bug almost four years ago).
• Then somebody creates Prometheus text format with trailing whitespaces (likely, that's the whole point of fixing this issue), which is then fed to a parser that accepts trailing whitespace, and then the same text snippet is fed to a parser that doesn't accept trailing whitespace (which should be extremely rare, it's not that there are many opportunities to pass around makeshift constructed Prometheus text format – the most prominent example is the textfile collector, but those snippets are always ever read by the textfile collector).

[brian-brazil]

For better or worse, that was never standardized or advertized.

It's been stated multiple times over the years. For example if promtool (which uses this) parses your /metrics then it is valid.

It has always been the reference implementation.

The counter case to bring up here is that somebody might have implemented a parser that doesn't tolerate trailing whitespace, too.

This describes both the current version of this parser and the parser inside Prometheus.

Differing versions of Prometheus, the node exporter or pgw doing different things for what is nominally all the v0.0.4 format is not desirable behaviour. We might have gotten away with this a few years back, but today changing this would require defining a v0.0.5 - which I don't think we're going to do.

as proved by multiple people that have run into this

Also as far as I'm aware we've only ever had one report related to this, for a user who has (presumably) long resolved their issue. It seems a bit disproportionate to introduce such risk/complexity for a use case that effectively doesn't exist. It's not practical to change the format every time a user comes up with a new form of invalid input.

[beorn7]

Most people who run into this issue will probably realize after a while that they made a mistake. Very few will come here and complain. It is still surprising that no trailing whitespaces are accepted, which is made worse by the confusing error message. I would expect many brain cycles of humanity wasted on this.

If the Prometheus server rejected trailing whitespace, then fixing this issue would indeed also require changing Prometheus because I do think we should be consistent within the project, including the degree of tolerance our parsers apply. I'd see it as an argument to change neither the Prometheus server's behavior nor the text parser at this point.

But what if the Prometheus server accepted trailing whitespace? :philosoraptor:

[brian-brazil]

Our tolerance does vary, the Prometheus one is for speed rather than correctness, but this is not the sort of we should vary on. What does Prometheus do currently?

…

On Mon 20 Jan 2020, 15:32 Björn Rabenstein, ***@***.***> wrote: Most people who run into this issue will probably realize after a while that they made a mistake. Very few will come here and complain. It is still surprising that no trailing whitespaces are accepted, which is made worse by the confusing error message. I would expect many brain cycles of humanity wasted on this. If the Prometheus server rejected trailing whitespace, then fixing this issue would indeed also require changing Prometheus because I do think we should be consistent within the project, including the degree of tolerance our parsers apply. I'd see it as an argument to not change the Prometheus server's behavior at this point. But what if the Prometheus server accepted trailing whitespace? :philosoraptor: — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#33?email_source=notifications&email_token=ABWJG5SQIVLFKSNURAFFHA3Q6W7ZHA5CNFSM4B5OOPP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJNAIPI#issuecomment-576324669>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABWJG5V7YHJC235ZSMDDNBLQ6W7ZHANCNFSM4B5OOPPQ> .

[beorn7]

The Prometheus server currently tolerates trailing whitespace.

[brian-brazil]

That is a problem then.

[RalfJung]

Not accepting trailing whitespaces would not have been a big problem if the error message would not have said the exact opposite of what is actually the problem. That was the real problem.

[brian-brazil]

The official parser and the Prometheus parser doing different things here is something that needs resolving, one way or the other. The Prometheus parser is wrong by the letter of the law, but this could be considered by some to be a breaking change.

Improving error messages is also good.

[roidelapluie]

2.15 and master might differ on this.

[brian-brazil]

Oh? Is this a recent regression?

[roidelapluie]

mmh no. I was thinking that maybe the new parser was more strict about this. it is not.

[beorn7]

The Prometheus parser is wrong by the letter of the law

This is again based on the premise that the expfmt parser sets the standard. I already challenged that, so you could at least acknowledge that your understanding is not the common one.

IMHO, the letter of the law is https://prometheus.io/docs/instrumenting/exposition_formats/ . It says:

Within a line, tokens can be separated by any number of blanks and/or tabs (and must be separated by at least one if they would otherwise merge with the previous token). Leading and trailing whitespace is ignored.

That's crystal clear. The expfmt parser has a bug, and the prometheus/prometheus parser is right.

[aaaidan]

Hello, just dropping in here to weigh in. I'm a newly minted prometheus user, though I'm have a fair bit of development experience.

Please, allow extra whitespace! 🙏 I'm glad the docs seem to come down on the right side of this.

The current behavior is confusing and discouraging. A format in this simple human-readable style sends a strong message that extra whitespace is ignored. At the moment, it's hard to figure it out that it's not.

I almost gave up on an exporter script because of this. I'm glad I pushed through. Not everyone will not have the time and energy to do so. This comment is the only reason I considered whitespace was my issue (thanks @beorn7!).

A friendly/helpful error message may be good low hanging fruit, but I'd love to see the underlying whitespace problem sorted. Hope this is useful, and let me know if I can help any other way.

[roidelapluie]

Yes it's a bug because the docs say otherwise as implemented.