NIFI-11231 Stateless NiFi sensitive parameter context support

[slambrose]

Summary

NIFI-11231

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using mvn clean install -P contrib-check
  • JDK 11
  • JDK 17

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy - N/A
• New dependencies are documented in applicable LICENSE and NOTICE files - N/A

Documentation

• Documentation formatting appears as expected in rendered files

[exceptionfactory]

Thanks for the contribution @slambrose!

I noted a handful a small naming issues, but the basic approach follows the example of other components.

Can you provide some additional background on the intended use case? The Stateless execution environment has a different approach for loading flow parameters using Parameter Value Providers. This is different than Parameter Providers in traditional NiFi, which is part of the reason that Parameter Providers are not supported right now.

@gresockj May also have some additional background from work on both Parameter Providers and Parameter Value Providers.

[slambrose]

Thanks for the feedback/review! I implemented all of the requested changes. For ParameterProviderLogObserver, I found there was already an existing class by this name doing exactly what my class was doing, but in a strange location in the core package. I moved it to the framework->logging package where all of the other LogObserver classes are, and implemented that instead. I also removed my duplicate class.

[slambrose]

Use case:
We are attempting to use stateless NiFi in k8s to process data while providing a horizontally scaled approach. Some of our users' flows are using sensitive parameters, so we have to be able to support this in our implementation. @Dye357 is the project lead for the effort, and can give a more robust explanation if needed. I tested this with an SFTP processor using the secure environment variable parameter context, which worked successfully.

[exceptionfactory]

Use case: We are attempting to use stateless NiFi in k8s to process data while providing a horizontally scaled approach. Some of our users' flows are using sensitive parameters, so we have to be able to support this in our implementation. @Dye357 is the project lead for the effort, and can give a more robust explanation if needed. I tested this with an SFTP processor using the secure environment variable parameter context, which worked successfully.

Thanks for addressing the code comments and providing some background on the use case @slambrose, that is helpful.

Have you or @Dye357 reviewed the Parameter Value Provider implementations for NiFi Stateless? The Parameter Value Providers support the use case of supplying sensitive parameter values, which should make the proposed changes unnecessary. If there is some feature gap in the Parameter Value Providers, we should evaluate that for improvement, as opposed to introducing Parameter Providers, intended for traditional NiFi deployments.

[slambrose]

Use case: We are attempting to use stateless NiFi in k8s to process data while providing a horizontally scaled approach. Some of our users' flows are using sensitive parameters, so we have to be able to support this in our implementation. @Dye357 is the project lead for the effort, and can give a more robust explanation if needed. I tested this with an SFTP processor using the secure environment variable parameter context, which worked successfully.

Thanks for addressing the code comments and providing some background on the use case @slambrose, that is helpful.

Have you or @Dye357 reviewed the Parameter Value Provider implementations for NiFi Stateless? The Parameter Value Providers support the use case of supplying sensitive parameter values, which should make the proposed changes unnecessary. If there is some feature gap in the Parameter Value Providers, we should evaluate that for improvement, as opposed to introducing Parameter Providers, intended for traditional NiFi deployments.

Hmm.. I'm not sure Parameter Value Provider would work for us since these have to be passed in on runtime or in the properties file. We're going to be supporting a variety of flows stored in Registry. The goal is to run stateless NiFi with any given Registry url, bucket id, flow id, and flow version. So those flows that utilize sensitive parameter contexts need to be able to work under any running statless nifi pod/container. We wouldn't be changing the run command or properties files for each flow that's processed with stateless NiFi. Instead, they would have to be pulled from some secrets manager or k8s secrets. Let me chat some more with @Dye357 to see if there's anything I'm missing.

[gresockj]

Hmm.. I'm not sure Parameter Value Provider would work for us since these have to be passed in on runtime and known beforehand. We're going to be supporting a variety of flows stored in Registry. The goal is to run stateless NiFi with any given Registry url, bucket id, flow id, and flow version without any knowledge of what properties/params users have configured in their flows. So those flows that utilize sensitive parameter contexts need to be able to work under any running statless nifi pod/container. We wouldn't be changing the run command or properties files for each flow that's processed with stateless NiFi. Let me chat some more with @Dye357 to see if there's anything I'm missing.

Hi @slambrose, what you describe should be possible with the existing EnvironmentVariableParameterValueProvider. Let me know if you have any other questions about the setup.

[exceptionfactory]

Closing based on discussion of recommended alternatives. If there are any additional questions on the current implementation, the discussion can be continued under the linked Jira issue: NIFI-11231