NIFI-5922: Stateless NiFi, an alternative runtime for NiFi flows #3241

SamHjelmfelt · 2019-01-03T01:26:28Z

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically master)?
Is your initial contribution a single, squashed commit?

For code changes:

Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
Have you written or updated unit tests to verify your changes?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

SamHjelmfelt · 2019-01-03T01:31:39Z

A pre-built docker image can be found on docker hub with this tag: samhjelmfelt/nifi-fn:latest

alopresto · 2019-01-03T01:33:26Z

nifi-fn/.gitignore

@@ -0,0 +1 @@
+target/


I think traditionally we just inherit the root .gitignore file.

Makes sense. removed.

alopresto · 2019-01-03T01:35:37Z

nifi-fn/pom.xml

+
+    <groupId>org.apache.nifi.fn</groupId>
+    <artifactId>nifi-fn</artifactId>
+    <version>1.0-SNAPSHOT</version>


I think you would want to change this to three-component versioning as we follow semantic versioning.

Makes sense. Updated.

SamHjelmfelt · 2019-01-03T02:03:48Z

Here is a Apache OpenWhisk quick-start that comes preconfigured for NiFi-Fn. I will work to improve the documentation of this runtime option.
https://github.com/SamHjelmfelt/OpenWhisk-YarnDeployment

Note that this project also has support for OpenWhisk on YARN, making event-driven, ephemeral NiFi-Fn flows on Hadoop possible.

alopresto · 2019-01-03T18:35:48Z

nifi-fn/pom.xml

+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <groupId>org.apache.nifi.fn</groupId>


I think the groupId should just be org.apache.nifi unless you expect to have a number of artifacts here.

Agreed. Fixed.

joewitt · 2019-01-04T03:47:18Z

nifi-fn/pom.xml

+
+    <properties>
+        <java.version>1.8</java.version>
+        <nifi.version>1.9.0-SNAPSHOT</nifi.version>


should just copy/paste 1.9.0-SNAPSHOT. this approach to property based version management fights with the maven release versions handling so it should be avoided.

makes sense. fixed

ottobackwards · 2019-01-06T03:41:52Z

Is there any design documentation on this? There is no javadoc

markap14 · 2019-01-11T13:37:49Z

nifi-fn/src/main/java/org/apache/nifi/fn/core/FnConfigurationContext.java

+
+    @Override
+    public Map<PropertyDescriptor, String> getProperties() {
+        return new HashMap<>(this.properties);


Given that properties is injected in and never modified, we should probably just use this.properties = Collections.unmodifiableMap(properties); in the constructor and avoid copying the map here. As-is, this can result in quite a lot of garbage.

SamHjelmfelt · 2019-04-03T21:20:29Z

Bringing in changes from @markap14. Still more to be done such as docker.

ottobackwards · 2019-04-06T12:09:49Z

Still no design doc. The documentation in the readme or somewhere else should do more than describe the parameters of this module, they should document what it addresses and why you would use it.

"There are times when you want to do ${THING} and an environment like %{ENV}, but using NIFI ${PROBLEMS}. Nifi-FN addresses this by adding the ability to ${SIZZLE} and .........."

alopresto · 2019-04-08T22:56:19Z

Still no design doc. The documentation in the readme or somewhere else should do more than describe the parameters of this module, they should document what it addresses and why you would use it.

"There are times when you want to do ${THING} and an environment like %{ENV}, but using NIFI ${PROBLEMS}. Nifi-FN addresses this by adding the ability to ${SIZZLE} and .........."

Sam has shared this with a few of us offline and it's a very compelling story. Hopefully he can condense it into an appropriate form to share here.

joewitt · 2019-04-08T23:07:04Z

how is this different than was in his proposal email thread?

we have that thread, the jira, a pr with lots of code and collab going...

alopresto · 2019-04-08T23:10:40Z

Sorry, I was referring to the mailing list email. I think with the provided code, the email has more context and makes sense. I remember reading the email and not understanding the environment this would fit into.

SamHjelmfelt · 2019-04-10T02:04:24Z

A copy of the latest docker image can be found here: samhjelmfelt/nifi-stateless:1.10.0-SNAPSHOT. It is 1.71GB, and the NARs are already un-packed. The API is the same as before and documented in the README file.

Not sure what is needed to push this into apache/nifi-stateless on docker hub.

- Separated into nifi-fn-core, nifi-fn-bootstrap, nifi-fn-nar, nifi-fn-assembly; fully unpacks nars and runs flows - Rebased against master and updated to version 1.10.0-SNAPSHOT - Removed dependency on nifi-framework-core - Added LICENSE/NOTICE files

…es; code cleanup

markap14 · 2019-05-21T15:50:18Z

@SamHjelmfelt I know this has taken a while to get everything resolved, but it's a great contribution! Many thanks for sticking with it until we got it all sorted out. I've now merged to master! There's still plenty to do here, of course, but I've run several flows with it, and I think it's in good enough shape to go ahead and merge. Thanks again!

SamHjelmfelt · 2019-05-21T16:07:02Z

Merged!

alopresto reviewed Jan 3, 2019

View reviewed changes

joewitt reviewed Jan 4, 2019

View reviewed changes

markap14 reviewed Jan 11, 2019

View reviewed changes

SamHjelmfelt force-pushed the nifi-fn branch from 65fa4d4 to 872802c Compare April 3, 2019 21:17

SamHjelmfelt changed the title ~~NIFI-5922: NiFi-Fn, an alternative runtime for NiFi flows~~ NIFI-5922: Stateless NiFi, an alternative runtime for NiFi flows Apr 8, 2019

SamHjelmfelt and others added 10 commits May 20, 2019 12:17

First Commit

a1e0e42

first round of updates

8397b9d

pom fixes

17d37c4

NIFI-5922: Addressed checkstyle violations; added to README

76cdaf4

Renaming from NiFi-Fn to NiFi-Stateless

26a289a

Fixed docker image and moved it into the nifi-docker project

b2e5f7d

Fixed Docker container, YARN runtime, and OpenWhisk runtime

7ca7fb2

NIFI-5922: Bug fixes; initialize, setup, and enable controller servic…

b4b910b

…es; code cleanup

Small fix after rebase

36b8587

SamHjelmfelt force-pushed the nifi-fn branch from a72d559 to 36b8587 Compare May 20, 2019 23:07

SamHjelmfelt closed this May 21, 2019

SamHjelmfelt deleted the nifi-fn branch May 24, 2019 00:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NIFI-5922: Stateless NiFi, an alternative runtime for NiFi flows #3241

NIFI-5922: Stateless NiFi, an alternative runtime for NiFi flows #3241

SamHjelmfelt commented Jan 3, 2019

SamHjelmfelt commented Jan 3, 2019 •

edited

Loading

alopresto Jan 3, 2019

SamHjelmfelt Jan 3, 2019

alopresto Jan 3, 2019

SamHjelmfelt Jan 3, 2019

SamHjelmfelt commented Jan 3, 2019

alopresto Jan 3, 2019

SamHjelmfelt Jan 5, 2019

joewitt Jan 4, 2019

SamHjelmfelt Jan 4, 2019

ottobackwards commented Jan 6, 2019

markap14 Jan 11, 2019

SamHjelmfelt commented Apr 3, 2019

ottobackwards commented Apr 6, 2019

alopresto commented Apr 8, 2019

joewitt commented Apr 8, 2019

alopresto commented Apr 8, 2019

SamHjelmfelt commented Apr 10, 2019

markap14 commented May 21, 2019

SamHjelmfelt commented May 21, 2019

		@@ -0,0 +1 @@
		target/

NIFI-5922: Stateless NiFi, an alternative runtime for NiFi flows #3241

NIFI-5922: Stateless NiFi, an alternative runtime for NiFi flows #3241

Conversation

SamHjelmfelt commented Jan 3, 2019

For all changes:

For code changes:

For documentation related changes:

Note:

SamHjelmfelt commented Jan 3, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SamHjelmfelt commented Jan 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ottobackwards commented Jan 6, 2019

Choose a reason for hiding this comment

SamHjelmfelt commented Apr 3, 2019

ottobackwards commented Apr 6, 2019

alopresto commented Apr 8, 2019

joewitt commented Apr 8, 2019

alopresto commented Apr 8, 2019

SamHjelmfelt commented Apr 10, 2019

markap14 commented May 21, 2019

SamHjelmfelt commented May 21, 2019

SamHjelmfelt commented Jan 3, 2019 •

edited

Loading