NIFI-12616: Added some Use Case docs for Python processors and update…

[markap14]

…d Runtime Manifests to include Python based processors as well as Use Case/MultiProcessorUseCase documentation elements. Refactored/cleaned up some of the Python code and added unit tests.

Summary

NIFI-00000

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using mvn clean install -P contrib-check
  • JDK 21

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[bbende]

Overall looks good, verified the python processors are in the runtime manifest retrieved from the REST API, just one minor thing...

After building with tests I noticed git was showing these as untracked:

nifi-nar-bundles/nifi-py4j-bundle/nifi-python-extension-api/src/main/python/src/nifiapi/__pycache__/
	nifi-nar-bundles/nifi-py4j-bundle/nifi-python-framework/src/main/python/framework/__pycache__/
	nifi-nar-bundles/nifi-py4j-bundle/nifi-python-framework/src/test/python/framework/__pycache__/

I'm assuming they got created as part of running some tests, can we update .gitignore and add __pycache__ ?

[bbende]

@markap14 it seems like there is a consistent failure in the Windows build for this python related test...

======================================================================
ERROR: TestPythonProcessorAdapter (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: TestPythonProcessorAdapter
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\unittest\loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\unittest\loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "D:\a\nifi\nifi\nifi-nar-bundles\nifi-py4j-bundle\nifi-python-framework\src\test\python\framework\TestPythonProcessorAdapter.py", line 17, in <module>
    import ProcessorInspection
ModuleNotFoundError: No module named 'ProcessorInspection'


----------------------------------------------------------------------
Ran 1 test in 0.009s

FAILED (errors=1)
Error:  Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:881)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:841)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:447)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Error:  Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.1.1:exec (python-test) on project nifi-python-framework: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

[exceptionfactory]

Thanks the improvements @markap14, and thanks for the review @bbende. The latest version with the optional build profile for Python unit tests looks good as a build solution at this point. +1 merging

[exceptionfactory]

Reviewing these changes in light of the recent build failures, it looks like this plugin execution should be optional so that it does not result in failures on systems that do not have the python3 executable installed.