NIFI-11480 add option to group PutElasticsearchRecord errors by Elasticsearch _bulk error type

[ChrisSamo632]

Summary

NIFI-11480 - add option to group PutElasticsearchRecord errors by Elasticsearch _bulk error type

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using mvn clean install -P contrib-check
  • JDK 17

Licensing

• [ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
• [ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• [ ] Documentation formatting appears as expected in rendered files

[mattyb149]

How do you test different errors in a NiFi instance (not a unit test)? Given the Operation I can only generate one kind of error at a time. This looks good though, just want to verify I can get groups of errors then I'll merge

[ChrisSamo632]

How do you test different errors in a NiFi instance (not a unit test)? Given the Operation I can only generate one kind of error at a time. This looks good though, just want to verify I can get groups of errors then I'll merge

@mattyb149 my suggestion (although I'll not be in a position to try it for a while), would be to create an index with a numeric field mapping, index a document with a known id, then run the processor in update mode with 2 records - 1 with a mismatched id (should result in a not_found error), the 2nd with the correct id but a non-numeric value for the numeric field (should give a "format exception")

From memory, this is the kind of thing I've tried to show in the unit/integration tests

This could possibly all be done with a single FlowFile and run off the processor - the index operation can be set by the value of a field within a record. So something like:

1. Op=index, id=1, fields: foo="bar", baz=1
2. Op=update, id=unknown, fields: foo="zap"
3. Op=update, id=1, fields: baz="not a number"

Index=bulk_test, Index Op record path=/Op, id record path=/id

That's a chance Elasticsearch will set the "baz" fields mapping to string by default, but hopefully it'll choose Integer. If so, we'll need to separately define the index mappings and pre-create the index with an InvokeHTTP call (for example). If you don't get to it first, I'll try to put a Flow Definition together and attach it to the jira ticket as an example

[ChrisSamo632]

@mattyb149 I've added an example Flow Definition to the Jira ticket that exhibits this behaviour, for reference

[mattyb149]

+1 LGTM, thanks for the improvement and the reproduction flow! Merging to support/nifi-1.x and main