NIFI-12350 Implement Azure Active Directory authentication option whe…

[vuong2407]

…n using Apache Kafka in Azure Event Hubs

Summary

NIFI-12350

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using mvn clean install -P contrib-check
  • JDK 21

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[exceptionfactory]

Thanks for the contribution @vuong2407. This feature for Azure AD authentication looks useful in general, and aligns with optional supports for AWS MSK. However, the initial implementation raises several concerns about shared configuration properties. Work is in progress to redesign the Kafka integration strategy using an extensible Controller Service to handle authentication, and that may provide a better foundation for this kind of feature. For this pull request, some further consideration is necessary to determine whether it can fit in the existing structure given the dependencies required.

[exceptionfactory]

The nifi-kafka-shared library is intended to avoid direct dependencies on Kafka libraries, so the implementation approach needs to be refactored to avoid this dependency on kafka-clients in this shared library.

[exceptionfactory]

This class and others are missing the Apache license header.

[exceptionfactory]

Setting static variables is not suitable for multiple instances of this Processor, although the way the Kafka library instantiates the class makes it difficult to configure. Some further design consideration will be necessary.

[exceptionfactory]

Thanks for your work on this improvement @vuong2407.

On further review of these changes, it would be best to evaluate an alternative that is not specific to the Azure EventHubs Kafka environment. The OAUTHBEARER authentication strategy is generic, as described in KIP-255, although some implementations may have certain particular approaches. The proposed changes would add a direct dependency on msal4j, which would not apply to many Kafka client deployments.

One option is to revisit this capability after completion of new Kafka components, described in NIFI-11259. There is no precise timeline for that implementation, so the other option is to evaluate an alternative approach that avoids vendor-specific dependencies. That may not be achievable without introducing some new Controller Service extension interface, which again points back to the redesign. As it stands, however, introducing these substantive changes to the current components does not appear to be the best way forward in terms of future maintainability.

[exceptionfactory]

Thanks again for the work on this change @vuong2407.

In light of the current conflicts, and the work in progress on refactored Kafka Processors for NIFI-11259 now in review, it would be better to revisit this implementation after the refactored Processors have been reviewed and merged. That should provide a much better baseline for integrating alternative authentication strategies.