NIFI-5826 Fix back-slash escaping at Lexers

[ijokarumawak]

Summary

Current Lexers convert a back-slash plus another character sequence (e.g. \[) to double back-slash plus the next character (e.g. \\[).
But from detailed analysis (see below), it seems the conversion is wrong and it should leave such characters as it is.

Details

I debugged how Lexer works, and found that:

• The ESC fragment handles an escaped special character in String representation. I.e. String \t will be converted to actual tab character.
• The string values user input from NiFi UI are passed to RecordPath.compile method as it is. E.g. the input string replaceRegex(/name, '\[', '') is passed to as is, then the single back-slash is converted to double back-slash by the ESC fragment line 155.
• I believe the line 153-156 is aimed to preserve escaped characters as it is, because such escape doesn't mean anything for the RecordPath/AttrExpLang spec. And those should be unescaped later by underlying syntaxes such as RegEx.
  • And current line 155 does it wrongly. It should append a single back-slash..
  • Other Lexers (AttributeExpressionLexer.g and HL7QueryLexer.g) have the same issue.
• So, I think we should fix all Lexers instead of adding another conversion.

Here is the Lexer code for reference:

143 fragment
144 ESC
145   :  '\\'
146     (
147         '"'    { setText("\""); }
148       |  '\''  { setText("\'"); }
149       |  'r'   { setText("\r"); }
150       |  'n'   { setText("\n"); }
151       |  't'   { setText("\t"); }
152       |  '\\'  { setText("\\\\"); }
153       |  nextChar = ~('"' | '\'' | 'r' | 'n' | 't' | '\\')
154        {
155          StringBuilder lBuf = new StringBuilder(); lBuf.append("\\\\").appendCodePoint(nextChar); setText(lBuf.toString());
156        }
157    )
158  ;

NiFi template for test

Here is a NiFi flow template to test how before/after this change.
https://gist.github.com/ijokarumawak/b6bdca8074a4457bc4a425b90a6b17f0

In order to try the template, you need to build this PR as NiFi 1.9.0-SNAPSHOT, then download following 1.8.0 nars in your SNAPSHOT's lib dir, so that both versions can be used in the flow.

• https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-standard-nar/1.8.0/nifi-standard-nar-1.8.0.nar
• https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-update-attribute-nar/1.8.0/nifi-update-attribute-nar-1.8.0.nar

Test result

UpdateAttribute test for backward compatibility

GenerateFlowFile generates FlowFiles with attribute a whose value is:

this is new line
and this is just a backslash \n

Result
1.8.0

1.9.0-SNAPSHOT

UpdateRecord test illustrating the NIFI-5826 issue is addressed

GenerateFlowFile generates content:

key,value
on[e,1
[two,2

Result
1.8.0
Regex compilation error as reported

1.9.0-SNAPSHOT
The square brackets are converted successfully

---

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

• Is there a JIRA ticket associated with this PR? Is it referenced
  in the commit message?

• Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

• Has your PR been rebased against the latest commit within the target branch (typically master)?

• Is your initial contribution a single, squashed commit?

For code changes:

• Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
• Have you written or updated unit tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
• If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
• If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

[ottobackwards]

There should be new tests that go along with this

[bdesert]

@ijokarumawak
I agree with @ottobackwards regarding adding test cases.
Other than that it looks fine and works as expected.
Initially I also had changed Lexer. While reviewing your solution and trying to understand why the same didn't work for me, I finally found another bug that prevented me to get proper results.
I opened JIRA for that bug, if you wish to take a look.
For this fix i've tested:
regression in EL for various backslash sequence combination in regex and in NON-regex EL functions.
I've also tested RecordPath based functions and found that this PR fixes one more defect - how backslash sequences being handled in non-regex based functions (all string literals that are processed by ANTLR).

[ijokarumawak]

@bdesert @ottobackwards Thanks for reviewing. I've added unit tests.

As a reference, I run the added tests with Lexer before this PR. Following tests failed with current Lexer, but passes with the updated Lexer:

This test failed with original Lexer because [\s] is converted to [\\s] by Lexer and didn't match.

[ERROR] testReplaceRegexEscapedCharacters(org.apache.nifi.record.path.TestRecordPath)  Time elapsed: 0.007 s  <<< FAILURE!
org.junit.ComparisonFailure:
Replacing whitespace to new line expected:<John[
]Doe> but was:<John[ ]Doe>
        at org.apache.nifi.record.path.TestRecordPath.testReplaceRegexEscapedCharacters(TestRecordPath.java:1046)

This test failed with original Lexer because \[ is converted to \\[ by Lexer and produced RegEx syntax error.

[ERROR] testReplaceRegexEscapedBrackets(org.apache.nifi.record.path.TestRecordPath)  Time elapsed: 0.001 s  <<< ERROR!
org.apache.nifi.record.path.exception.RecordPathException:
java.util.regex.PatternSyntaxException: Unclosed character class near index 2
\\[
  ^
        at org.apache.nifi.record.path.TestRecordPath.testReplaceRegexEscapedBrackets(TestRecordPath.java:1149)

Both tests failed with the same cause, and fixed by this PR.

[bdesert]

+1 LGTM, ran unit tests, tested fix and ran regression tests on live instance with this fix. All works as expected. Thanks for the fix! Merged to master