NIFI-5895 - Add a processor to start an Oozie workflow by pvillard31 · Pull Request #3218 · apache/nifi

pvillard31 · 2018-12-14T09:48:16Z

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically master)?
Is your initial contribution a single, squashed commit?

For code changes:

Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
Have you written or updated unit tests to verify your changes?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

pvillard31 · 2018-12-14T10:14:06Z

This PR has been tested against an Oozie 4.2.0 server instance in the following conditions:

Plaintext
Plaintext + SSL
Kerberos
Kerberos + SSL

A very basic example has been used:
https://github.com/apache/oozie/tree/branch-4.3/examples/src/main/apps/shell

NiFi workflow:
https://gist.github.com/pvillard31/4a0a7b33dc27c11538cec0b5e80c0324

The workflow.xml from the Oozie repo must be pushed in HDFS in a directory and that directory is used in the configuration of the processor. All the properties given in the job.properties file from the Oozie repo must be passed through the configuration of the processor or the attributes of the flow files.

zenfenan · 2018-12-16T12:45:01Z

...ifi-oozie-processors/src/main/java/org/apache/nifi/processors/oozie/SubmitOozieWorkflow.java

+
+    public static final PropertyDescriptor APP_PATH = new PropertyDescriptor.Builder()
+            .name("oozie-app-path")
+            .displayName("HDFS Application Path")


I would recommend it to be changed to something like Workflow Path.

zenfenan · 2018-12-16T12:47:29Z

nifi-nar-bundles/nifi-oozie-bundle/nifi-oozie-processors/pom.xml

+    <packaging>jar</packaging>
+
+    <dependencies>
+		<dependency>


Pure nitpick: Would like to see proper indentation. :)

pvillard31 · 2018-12-17T10:14:55Z

Thanks for the review @zenfenan - just pushed a commit to address your comments.

mattyb149 · 2019-01-30T15:23:21Z

...ifi-oozie-processors/src/main/java/org/apache/nifi/processors/oozie/SubmitOozieWorkflow.java

+                String principal = credentialsService.getPrincipal();
+                String keyTab = credentialsService.getKeytab();
+                Configuration conf = new Configuration();
+                conf.set("hadoop.security.authentication", "kerberos");


Are there any other Oozie-or HDFS-related properties that should come from an oozie-site.xml or other site config file? Just wondering if there should be a property like the other Hadoop processors to specify configuration file(s).

Not to my knowledge - IIRC this Kerberos property is to enable SPNEGO authN against the Oozie server REST API but there should not be any other requirement. The files hdfs-site, core-site, etc, are loaded by the Oozie server anyway.

mattyb149 · 2019-01-30T15:27:52Z

nifi-nar-bundles/nifi-oozie-bundle/pom.xml

+
+    <properties>
+    	<oozie.version>4.3.1</oozie.version>
+    	<hadoop.version>3.0.0</hadoop.version>


For consistency, we should probably document that hadoop.version should match the one in nifi-hadoop-libraries, unless there's some way to inherit it (see my comment in the NAR POM)

Yep true. Didn't look too long (switching planes in airport) but didn't find a way to inherit the version from the nifi-hadoop-libraries so I just added a comment about the version.

mattyb149 · 2019-01-30T15:28:56Z

nifi-nar-bundles/nifi-oozie-bundle/nifi-oozie-nar/pom.xml

+    <dependencies>
+        <dependency>
+            <groupId>org.apache.nifi</groupId>
+            <artifactId>nifi-standard-services-api-nar</artifactId>


For consistency, shouldn't we have nifi-hadoop-libraries-nar as a parent here? Then the hadoop-common and other dependencies in the processors POM could be marked as provided and we'd be sure to be using the same versions across the Hadoop-related processors.

mattyb149 · 2019-01-30T15:29:45Z

nifi-nar-bundles/nifi-oozie-bundle/nifi-oozie-nar/src/main/resources/META-INF/NOTICE

@@ -0,0 +1,136 @@
+nifi-kafka-nar


Copy/pasta typo here, should be nifi-oozie-nar. Did you check the L&N to make sure it's correct for Oozie (vs wherever it came from)?

Good catch - I tried to be exhaustive on the Oozie deps, hopefully didn't miss one.

joewitt · 2021-03-25T17:55:44Z

closing due to inactivity

NIFI-5895 - Add a processor to start an Oozie workflow

ef6b789

pvillard31 force-pushed the oozie branch from bbcdf5d to ef6b789 Compare December 14, 2018 10:03

zenfenan suggested changes Dec 16, 2018

View reviewed changes

review

e5cf6b8

pvillard31 force-pushed the oozie branch from a65afe1 to e5cf6b8 Compare December 17, 2018 10:13

mattyb149 reviewed Jan 30, 2019

View reviewed changes

pvillard31 added 2 commits January 30, 2019 22:02

Merge branch 'master' of github.com:apache/nifi into oozie

58cfae7

review comments

fc14bc2

joewitt closed this Mar 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NIFI-5895 - Add a processor to start an Oozie workflow#3218

NIFI-5895 - Add a processor to start an Oozie workflow#3218
pvillard31 wants to merge 4 commits intoapache:masterfrom
pvillard31:oozie

pvillard31 commented Dec 14, 2018 •

edited

Loading

Uh oh!

pvillard31 commented Dec 14, 2018 •

edited

Loading

Uh oh!

zenfenan Dec 16, 2018

Uh oh!

zenfenan Dec 16, 2018

Uh oh!

pvillard31 commented Dec 17, 2018

Uh oh!

mattyb149 Jan 30, 2019

Uh oh!

pvillard31 Feb 1, 2019

Uh oh!

mattyb149 Jan 30, 2019

Uh oh!

pvillard31 Feb 1, 2019

Uh oh!

mattyb149 Jan 30, 2019

Uh oh!

mattyb149 Jan 30, 2019

Uh oh!

pvillard31 Feb 1, 2019

Uh oh!

joewitt commented Mar 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pvillard31 commented Dec 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

For all changes:

For code changes:

For documentation related changes:

Note:

Uh oh!

pvillard31 commented Dec 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pvillard31 commented Dec 17, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joewitt commented Mar 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pvillard31 commented Dec 14, 2018 •

edited

Loading

pvillard31 commented Dec 14, 2018 •

edited

Loading