Add patterns and grok command

[joshuali925]

Description

see #814 , #815

• add java-grok source code into sql.common
  • java-grok is an independent library that's used for the PPL's grok command, but in the latest release Grok is not serializable (and SQL grok expression cannot be pushed down). After discussing with peng I'm tentatively moving the source code into sql.common module.
• add grok PPL command, similar to how parse currently works with regex
• add patterns command, which removes unwanted characters from a log line to create a new field as the pattern of that log line
• patterns currently supports default pattern (remove all alphanumerical characters) or user can define what characters should be removed

Issues Resolved

closes #814
closes #815

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #813 (9ec7825) into 2.x (8245943) will increase coverage by 0.09%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                2.x     opensearch-project/observability#813      +/-   ##
============================================
+ Coverage     94.87%   94.97%   +0.09%     
- Complexity     2956     3004      +48     
============================================
  Files           291      294       +3     
  Lines          7869     8017     +148     
  Branches        572      586      +14     
============================================
+ Hits           7466     7614     +148     
  Misses          349      349              
  Partials         54       54              

Flag	Coverage Δ
query-workbench	62.76% <ø> (ø)
sql-engine	97.85% <100.00%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...rg/opensearch/sql/analysis/ExpressionAnalyzer.java	100.00% <ø> (ø)
...ensearch/sql/expression/ExpressionNodeVisitor.java	100.00% <ø> (ø)
...ain/java/org/opensearch/sql/analysis/Analyzer.java	100.00% <100.00%> (ø)
...c/main/java/org/opensearch/sql/expression/DSL.java	100.00% <100.00%> (ø)
...pensearch/sql/expression/parse/GrokExpression.java	100.00% <100.00%> (ø)
...ensearch/sql/expression/parse/ParseExpression.java	100.00% <100.00%> (ø)
...earch/sql/expression/parse/PatternsExpression.java	100.00% <100.00%> (ø)
...ensearch/sql/expression/parse/RegexExpression.java	100.00% <100.00%> (ø)
...ensearch/sql/planner/physical/ProjectOperator.java	100.00% <100.00%> (ø)
...main/java/org/opensearch/sql/utils/ParseUtils.java	100.00% <100.00%> (ø)
... and 9 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[penghuo]

@joshuali925 as comments in #814. I think we could add grok as a method in parse command for now.

[penghuo]

we should sanitize rawString when logging

[penghuo]

is it warn? or debug?

[joshuali925]

changed to debug. rawString is unstructured text not sure how to sanitize, let me know if i should remove it from log

[penghuo]

we should replace rawString as ***.

[joshuali925]

hard coded it to ***

[penghuo]

files in common/test/resources/nasa is required? seems like split file

[joshuali925]

files in common/test/resources/nasa is required? seems like split file

yes it's split file, used here

sql/common/src/test/java/org/opensearch/sql/common/grok/ApacheTest.java

Lines 52 to 67 in f331e09

	public void test002_nasa_httpd_access() throws GrokException, IOException {
	Grok grok = compiler.compile("%{COMMONAPACHELOG}");
	System.out.println("Starting test with nasa log -- may take a while");
	BufferedReader br;
	String line;
	File dir = new File(LOG_DIR_NASA);
	for (File child : dir.listFiles()) {
	br = new BufferedReader(new FileReader(LOG_DIR_NASA + child.getName()));
	while ((line = br.readLine()) != null) {
	Match gm = grok.match(line);
	final Map<String, Object> capture = gm.capture();
	Assertions.assertThat(capture).doesNotContainKey("Error");
	}
	br.close();
	}
	}

[dai-chen]

Thanks for the changes!

[vamsi-amazon]

What if the pattern itself contains single quotes and double quotes.

[joshuali925]

user should escape quotes in quotes