-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Define quote escaping semantics. #1645
Comments
For reference see related issue on jira and linked use cases (csv quote, exec command, mutate string) |
@wiibaa Thanks for linking in that jira ticket ❤️ |
Seems to be missing from the proposal. Without this, there's no way to encode the sequence backslash-n without getting a newline instead.
Will be problematic, since that looks exactly like the start of a legitimate multi-line string (single quote, followed by whitespace, then arbitrary text). If multi-line strings aren't legal, or have additional escaping requirements, then that should be documented. |
minimal strawman for elastic#1645
Ugh markdown ruined my escaping, I guess. I did not intend to propose that On Tuesday, August 19, 2014, Neil Gentleman notifications@github.com
|
Just giving my 2 cents, I have the feeling that the "historical reason" is that a regex-string would requires escaping of all backslashes to be interpreted correctly by the filter using them a lot (multiline and GROK) and it would be silly as stated by Jordan in the past, so the escaping was built-in in the config parser and a TODO was added in logstash 1.0.x and 1.1.x to manage proper regex config element. Maybe allowing real regex object where it is due and then allowing string to be string (as mostly expected in ruby/java/bash) would be an alternative. |
@nigelzor I fixed the rendering problem markdown had. It was turning quote-backslash-backslash-quote into a single backslash. |
@wiibaa in logstash 1.2 and beyond you can do I always intended to interpret backslash-as-escapes but never actually got around to it. It burdens users infrequently, but as we have more users, probability makes that affect a larger population of users, so I wanted to file about fixing it now. |
@jordansissel what I meant is that you cannot do it inside filter config to disambigue between a pure string and a regex. So internally, sometimes a string is interpreted as a regex, sometimes it is not.
or
|
is this issue related to this issue? |
@splashx yep :) |
@jordansissel oh no 😆 |
+1 as I just hit this today too |
Hi @jordansissel, are there any progress about this issue? Is there a timetable when it's fixed? |
+1 hit today (same as #3239 6 months ago) |
This change is likely to be resolved eventually, but not right now. There's a lot of work going on for the new centralized configuration feature that's coming soon and will impact how we store and represent the configuration of a logstash agent. I'd rather fix this problem at that time than try to fix it now and break everyone who is using the current not-so-good behavior. |
Thank you for this information 👍 |
I'm not sure what to make of this. Could someone confirm that this is the current behavior? Upon encountering a quote character, the Logstash config parser interprets this as a string literal, which is terminated by the next quote character of the same type as the opening quote. This string literal then represents the string of characters between the start and end quote characters in the surface syntax. This means that a character sequence
If |
I've updated the issue's description to include some possible workarounds for specific characters. See the section under "Workarounds today:" |
@jordansissel +1 let's try to target 5.2 for this. Can you please own this feature? |
Glad this issue is somewhat on your radar. The existing workarounds are helpful, but one thing is apparently still impossible to do with the current configuration parser: Using mutate+gsub to replace something matched with a double-quote character . i.e.
just like mentioned in logstash-plugins/logstash-filter-mutate#40. |
@jordansissel @acchen97 any news on this? |
No news. There are many workarounds that can still be used today. I want to
add this feature, but we are overloaded with tasks as it is and it may take
some time before this gets worked on.
…On Fri, Mar 31, 2017 at 12:14 AM Jakob Reiter ***@***.***> wrote:
@jordansissel <https://github.com/jordansissel> @acchen97
<https://github.com/acchen97> any new on this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1645 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAIC6socvU57pCb87SLRb3l8u1TOrfPPks5rrKfKgaJpZM4CYjY9>
.
|
This makes it impossible to insert e.g. control characters. |
I have a branch where I am working on this as time permits. No ETA. https://github.com/elastic/logstash/compare/feature/config-string-escapes |
I was able to work around this by manually inserting the character literal. 😄 |
PR is open for this: #7442 |
New boolean setting `config.support_escapes` which defaults to false (the historical behavior). When set to true, the following escapes are handled: * backslash doublequote -> doublequote * backslash quote -> quote * backslash n -> newline (ascii 10) * backslash r -> carriage return (ascii 13) * backslash backslash -> backslash * backslash t -> tab (ascii 9) This will solve #1645.
New boolean setting `config.support_escapes` which defaults to false (the historical behavior). When set to true, the following escapes are handled: * backslash doublequote -> doublequote * backslash quote -> quote * backslash n -> newline (ascii 10) * backslash r -> carriage return (ascii 13) * backslash backslash -> backslash * backslash t -> tab (ascii 9) This will solve #1645.
New boolean setting `config.support_escapes` which defaults to false (the historical behavior). When set to true, the following escapes are handled: * backslash doublequote -> doublequote * backslash quote -> quote * backslash n -> newline (ascii 10) * backslash r -> carriage return (ascii 13) * backslash backslash -> backslash * backslash t -> tab (ascii 9) This will solve #1645.
New boolean setting `config.support_escapes` which defaults to false (the historical behavior). When set to true, the following escapes are handled: * backslash doublequote -> doublequote * backslash quote -> quote * backslash n -> newline (ascii 10) * backslash r -> carriage return (ascii 13) * backslash backslash -> backslash * backslash t -> tab (ascii 9) This will solve #1645.
New boolean setting `config.support_escapes` which defaults to false (the historical behavior). When set to true, the following escapes are handled: * backslash doublequote -> doublequote * backslash quote -> quote * backslash n -> newline (ascii 10) * backslash r -> carriage return (ascii 13) * backslash backslash -> backslash * backslash t -> tab (ascii 9) This will solve #1645.
New boolean setting `config.support_escapes` which defaults to false (the historical behavior). When set to true, the following escapes are handled: * backslash doublequote -> doublequote * backslash quote -> quote * backslash n -> newline (ascii 10) * backslash r -> carriage return (ascii 13) * backslash backslash -> backslash * backslash t -> tab (ascii 9) This will solve #1645.
New boolean setting `config.support_escapes` which defaults to false (the historical behavior). When set to true, the following escapes are handled: * backslash doublequote -> doublequote * backslash quote -> quote * backslash n -> newline (ascii 10) * backslash r -> carriage return (ascii 13) * backslash backslash -> backslash * backslash t -> tab (ascii 9) This will solve #1645.
New boolean setting `config.support_escapes` which defaults to false (the historical behavior). When set to true, the following escapes are handled: * backslash doublequote -> doublequote * backslash quote -> quote * backslash n -> newline (ascii 10) * backslash r -> carriage return (ascii 13) * backslash backslash -> backslash * backslash t -> tab (ascii 9) This will solve #1645.
New boolean setting `config.support_escapes` which defaults to false (the historical behavior). When set to true, the following escapes are handled: * backslash doublequote -> doublequote * backslash quote -> quote * backslash n -> newline (ascii 10) * backslash r -> carriage return (ascii 13) * backslash backslash -> backslash * backslash t -> tab (ascii 9) This will solve #1645.
New boolean setting `config.support_escapes` which defaults to false (the historical behavior). When set to true, the following escapes are handled: * backslash doublequote -> doublequote * backslash quote -> quote * backslash n -> newline (ascii 10) * backslash r -> carriage return (ascii 13) * backslash backslash -> backslash * backslash t -> tab (ascii 9) This will solve #1645. Fixes #7516
Fixed in #7442 |
Hmm. I configure fresh ELK system and it took me few days to find why my text contain \n (lteral \ then literal n). Well, I wrote
|
This is requested numerous places, but I was reminded from #895
Escaping quotes and control characters are impossible to do today in Logstash config. "\n" is literally backslash and lowercase n. """ is literally backslash and doublequote.
Logstash supports single and double-quoted text values, and we should support escapes in both the same exact way:
Proposal
Add a new setting,
config.support_escapes
that, when set, will enable the following behaviors on quoted strings in the Logstash config:Workarounds today:
setting => 'I said, "Hello!"'
setting => "I said, 'Hello!'"
setting => "C:\some\path"
The text was updated successfully, but these errors were encountered: