NIFI-12411: Update PublishAMQP to read AMQP headers value from FlowFile attributes and amq$headers string

[umarhussain15]

Summary

NIFI-12411

Processor can now also create message headers using attributes matching the given Regex property. The name of the attribute is used as header key. In addition to that it can read values from two sources i.e. attributes matching Regex and the headers present in amq$headers string. In case of key duplication in both sources, the precedence property decide which value will be used.

AMQP attributes keys are now set as constants strings and reused where needed.

Added onScheduled to PublishAMQP and ConsumeAMQP to read all properties which don't change during running state.

Test cases added for PublishAMQP headers source changes.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using mvn clean install -P contrib-check
  • JDK 21

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[exceptionfactory]

Thanks for the proposed improvement @umarhussain15.

The general focus on the change makes sense, but there are some other stylistic recommendations I noted. It is helpful to improve the overall Processor implementation through the definition and reuse of static values for things like FlowFile attributes, so thanks for including those changes.

[joewitt]

@umarhussain15 Can you please rebase and consider the feedback shared from @exceptionfactory ?

Thanks

[umarhussain15]

Hi @joewitt @exceptionfactory,
I have reverted the changes of making class level variables based on processor properties.
Also, the branch is rebased on latest changes from main

[umarhussain15]

Changes applied in new commits

[umarhussain15]

@exceptionfactory @joewitt

I have added one more change, which will allow users to set amqp processor's password via env variables.
I think this will be helpful in cases where we use AMQP processors in multiple NiFI instances. Instead of copying and pasting different passwords in processor properties, we can reference the env variable via its name.
Let me know your thoughts on it.
Thanks.

[exceptionfactory]

@exceptionfactory @joewitt

I have added one more change, which will allow users to set amqp processor's password via env variables. I think this will be helpful in cases where we use AMQP processors in multiple NiFI instances. Instead of copying and pasting different passwords in processor properties, we can reference the env variable via its name. Let me know your thoughts on it. Thanks.

Thanks for calling out this particular change @umarhussain15. Given the scope of the current pull request, I recommend considering that change separately.

In general, sensitive properties such as passwords do not support reading from environment variables. The reason is that environment variables are also accessible through Expression Language, so they are not necessarily secure. The use case is still supported through Parameter Contexts and Parameter Providers, where the Environment Variable Parameter Provider can be used. This is an additional layer of indirection, but it follows the general convention for other sensitive properties.

[umarhussain15]

Thanks for the explanation, @exceptionfactory. I will revert the change here then.

[umarhussain15]

Reverted the AbstractAMQPProcessor change to support expression language in password field.

[exceptionfactory]

Thanks for following up @umarhussain15. Please note, the Checkstyle rules were recently updated to include consistency checking for whitespace, so the latest static analysis build noted these issues:

Warning:  src/main/java/org/apache/nifi/amqp/processors/PublishAMQP.java:[115,38] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
Warning:  src/main/java/org/apache/nifi/amqp/processors/PublishAMQP.java:[307,76] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
Warning:  src/main/java/org/apache/nifi/amqp/processors/PublishAMQP.java:[314,76] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
Warning:  src/main/java/org/apache/nifi/amqp/processors/PublishAMQP.java:[316,76] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
Warning:  src/main/java/org/apache/nifi/amqp/processors/PublishAMQP.java:[380,68] (whitespace) WhitespaceAround: '{' is not preceded with whitespace.
Warning:  src/main/java/org/apache/nifi/amqp/processors/ConsumeAMQP.java:[349,69] (whitespace) WhitespaceAround: '{' is not preceded with whitespace.

[umarhussain15]

Checkstyle reported issues fixed

[exceptionfactory]

@umarhussain15 This looks close to completion. There is one minor merge conflict right now related to logging that should be easy to address in a rebase and force push. The only other suggested change is to use Map.of() instead of anonymous HashMaps in the test methods.

[exceptionfactory]

Thanks for the updates @umarhussain15. On further review, the implementation and property names raise a few more questions, particular in terms of the Precedence property. If you can address the syntax issues, that would be helpful, and consider whether the Precedence property is necessary.

[umarhussain15]

Hi @exceptionfactory,
I have applied your suggestions and also removed the precedence option from the processor.
I have also updated the additional documentation to describe the headers source setting of the processor.

[exceptionfactory]

Thanks for working through the feedback and making adjustments @umarhussain15, the latest version looks ready to go. +1 merging