NIFI-10067 enable use of script, dynamic_templates and _bulk headers for PutElasticsearchJson/Record processors #6628

ChrisSamo632 · 2022-11-07T21:01:48Z

Summary

NIFI-10067 enable use of script when updating documents in Elasticsearch via PutElasticsearchJson or PutElasticsearchRecord

NIFI-3262 enable use of dynamic_templates in Elasticsearch _bulk operations

NIFI-3262 enable use of bulk header fields in Elasticsearch _bulk operations via BULK: dynamic headers in PutElasticsearchJson and PutElasticsearchRecord

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Apache NiFi Jira issue created

Pull Request Tracking

Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

Pull Request based on current revision of the main branch
Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

Build completed using mvn clean install -P contrib-check
- JDK 8
- JDK 11
- JDK 17

Licensing

~~[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy~~
~~[ ] New dependencies are documented in applicable LICENSE and NOTICE files~~

Documentation

Documentation formatting appears as expected in rendered files

MikeThomsen · 2022-11-08T23:35:35Z

Are you planning to update the UpdateByQueryElasticsearch processor to support scripts? I was planning to do that and didn't see it getting updated in your PR.

ChrisSamo632 · 2022-11-09T06:28:12Z

Are you planning to update the UpdateByQueryElasticsearch processor to support scripts? I was planning to do that and didn't see it getting updated in your PR.

I don't believe there's anything that needs doing - the processor accepts arbitrary json as the "query" and sends that to elasticsearch, so you can already provide a script object within that

Or have you tried it and found something not working/you think could be improved?

Hopefully all that's left for this PR now is a rebase and finishing the additional docs for the PUT processors to describe the new user of dynamic properties to populate the bulk api request headers

MikeThomsen · 2022-11-09T15:55:06Z

Or have you tried it and found something not working/you think could be improved?

My vision for that processor would be a query builder that allows you to break down the process. Ex specifying the script or script id and then the update query. That combined with the ability to just use the flowfile body as a raw document to punt to update_by_query.

ChrisSamo632 · 2022-11-09T15:59:27Z

Or have you tried it and found something not working/you think could be improved?

My vision for that processor would be a query builder that allows you to break down the process. Ex specifying the script or script id and then the update query. That combined with the ability to just use the flowfile body as a raw document to punt to update_by_query.

That (to me) sounds like a (potentially breaking) change to the processor's API, best left to another ticket/discussion

I'm happy to update the existing processor's documentation to clarify that the query property for UpdateByQuery is currently the request body that's sent to Elasticsearch for the _update_by_query endpoint, i.e. if you want to run a Script, write your JSON like {"script": {"source": "..."}, "query": {"match_all":{}}}

ChrisSamo632 · 2022-12-29T11:07:18Z

My vision for that processor...

@MikeThomsen FYI I raised NIFI-11016 to cover your future update to the UpdateByQueryElasticsearch processor if you wanted to do that in future

gresockj

Thanks for this improvement, @ChrisSamo632. A couple minor naming suggestions, and then I'd like to test this out.

gresockj · 2023-02-12T19:22:39Z

...-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchJson.java

+
+    static final PropertyDescriptor DYNAMIC_TEMPLATES = new PropertyDescriptor.Builder()
+            .name("put-es-json-dynamic_templates")
+            .displayName("Script")


I think this is a copy/paste error

gresockj · 2023-02-12T19:27:17Z

...rocessors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecord.java

@@ -190,6 +201,26 @@ public class PutElasticsearchRecord extends AbstractPutElasticsearch {
        .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
        .build();

+    static final PropertyDescriptor SCRIPT_RECORD_PATH = new PropertyDescriptor.Builder()
+            .name("put-es-record-script-path")
+            .displayName("script Record Path")


Perhaps capital S

ChrisSamo632 · 2023-03-31T07:53:46Z

A community member has asked on Slack whether this change can also incorporate the option for scripted_upsert.

Options to set this flag for the PutElasticsearchRecord and PutElasticsearchJson processors should be added as part of this PR (it would make sense to do so, should be simple and hopefully completes the current set of Elasticsearch Update API options available to clients), to set the scripted_upsert: true flag in the Bulk API operation when wanted.

ChrisSamo632 · 2023-03-31T15:00:52Z

scripted_upsert option added to the PutElasticsearchJson and PutElasticsearchRecord processors (with associated Integration Tests in the ElasticsearchClientServiceImpl

…rch via PutElasticsearchJson or PutElasticsearchRecord NIFI-3262 enable use of dynamic_templates in Elasticsearch _bulk operations NIFI-3262 enable use of bulk header fields in Elasticsearch _bulk operations via BULK: dynamic headers in PutElasticsearchJson and PutElasticsearchRecord

…ElasticsearchRecord processors

gresockj

Ok, I tested out the latest changes -- thanks for the updates! I'll go ahead and merge.

…arch via PutElasticsearchJson or PutElasticsearchRecord NIFI-3262: enable use of dynamic_templates in Elasticsearch _bulk operations Signed-off-by: Joe Gresock <jgresock@gmail.com> This closes apache#6628.

…arch via PutElasticsearchJson or PutElasticsearchRecord NIFI-3262: enable use of dynamic_templates in Elasticsearch _bulk operations Signed-off-by: Joe Gresock <jgresock@gmail.com> This closes #6628.

ChrisSamo632 marked this pull request as draft November 7, 2022 21:02

ChrisSamo632 force-pushed the NIFI-10067 branch from 6718170 to 903f3f7 Compare November 7, 2022 21:03

ChrisSamo632 force-pushed the NIFI-10067 branch 2 times, most recently from 8eb0329 to 4afda58 Compare November 10, 2022 20:36

ChrisSamo632 marked this pull request as ready for review November 10, 2022 20:43

ChrisSamo632 requested review from MikeThomsen and gresockj November 10, 2022 20:44

ChrisSamo632 force-pushed the NIFI-10067 branch from 4afda58 to bab380d Compare November 22, 2022 08:30

ChrisSamo632 force-pushed the NIFI-10067 branch 2 times, most recently from 9e6ff72 to 2de7ff5 Compare January 6, 2023 09:43

ChrisSamo632 removed request for MikeThomsen and gresockj February 10, 2023 23:44

ChrisSamo632 changed the title ~~NIFI-10067 enable use of script for Elasticsearch updates~~ NIFI-10067 enable use of script, dynamic_templates and _bulk headers for PutElasticsearchJson/Record processors Feb 13, 2023

ChrisSamo632 force-pushed the NIFI-10067 branch from 2de7ff5 to a757ca2 Compare February 13, 2023 20:35

ChrisSamo632 force-pushed the NIFI-10067 branch from a757ca2 to ffede7e Compare March 20, 2023 20:43

gresockj reviewed Mar 29, 2023

View reviewed changes

ChrisSamo632 force-pushed the NIFI-10067 branch from ffede7e to f4f0bb0 Compare March 31, 2023 14:59

ChrisSamo632 force-pushed the NIFI-10067 branch from f4f0bb0 to 35760bf Compare April 2, 2023 16:35

ChrisSamo632 and others added 2 commits April 25, 2023 20:05

NIFI-10067 add scripted_upsert option to PutElasticsearchJson and Put…

0699c68

…ElasticsearchRecord processors

ChrisSamo632 force-pushed the NIFI-10067 branch from 35760bf to 0699c68 Compare April 25, 2023 20:14

gresockj approved these changes May 2, 2023

View reviewed changes

asfgit closed this in 3a3b9ed May 2, 2023

ChrisSamo632 deleted the NIFI-10067 branch May 2, 2023 16:01

ChrisSamo632 mentioned this pull request May 2, 2023

NIFI-10067: enable use of script when updating documents in Elasticsearch via PutElasticsearchJson or PutElasticsearchRecord #7224

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NIFI-10067 enable use of script, dynamic_templates and _bulk headers for PutElasticsearchJson/Record processors #6628

NIFI-10067 enable use of script, dynamic_templates and _bulk headers for PutElasticsearchJson/Record processors #6628

ChrisSamo632 commented Nov 7, 2022 •

edited

MikeThomsen commented Nov 8, 2022

ChrisSamo632 commented Nov 9, 2022

MikeThomsen commented Nov 9, 2022

ChrisSamo632 commented Nov 9, 2022

ChrisSamo632 commented Dec 29, 2022

gresockj left a comment

gresockj Feb 12, 2023

gresockj Feb 12, 2023

ChrisSamo632 commented Mar 31, 2023 •

edited

ChrisSamo632 commented Mar 31, 2023

gresockj left a comment

NIFI-10067 enable use of script, dynamic_templates and _bulk headers for PutElasticsearchJson/Record processors #6628

NIFI-10067 enable use of script, dynamic_templates and _bulk headers for PutElasticsearchJson/Record processors #6628

Conversation

ChrisSamo632 commented Nov 7, 2022 • edited

Summary

Tracking

Issue Tracking

Pull Request Tracking

Pull Request Formatting

Verification

Build

Licensing

Documentation

MikeThomsen commented Nov 8, 2022

ChrisSamo632 commented Nov 9, 2022

MikeThomsen commented Nov 9, 2022

ChrisSamo632 commented Nov 9, 2022

ChrisSamo632 commented Dec 29, 2022

gresockj left a comment

Choose a reason for hiding this comment

gresockj Feb 12, 2023

Choose a reason for hiding this comment

gresockj Feb 12, 2023

Choose a reason for hiding this comment

ChrisSamo632 commented Mar 31, 2023 • edited

ChrisSamo632 commented Mar 31, 2023

gresockj left a comment

Choose a reason for hiding this comment

ChrisSamo632 commented Nov 7, 2022 •

edited

ChrisSamo632 commented Mar 31, 2023 •

edited