NIFI-3446 FlowConfigurationDAO Changes by patricker · Pull Request #1478 · apache/nifi

patricker · 2017-02-07T04:53:25Z

Allows for Flow Configuration (flow.xml.gz) to be pluggable. There should be no "functional" changes as a result of this code change.

For all changes:

Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically master)?
Is your initial contribution a single, squashed commit?

For code changes:

Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
Have you written or updated unit tests to verify your changes?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

pvillard31

Hey @patricker, a minor comment but it LGTM.

pvillard31 · 2017-02-11T14:56:26Z

nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java


    // core properties
    public static final String PROPERTIES_FILE_PATH = "nifi.properties.file.path";
+    public static final String FLOW_CONFIGURATION_IMPLEMENTATION = "nifi.flow.configuration.implementation";


This needs to be added in nifi.properties file in nifi-resources.

joewitt

Left several comments but they're all around the same theme. The locking logic is now needing to be cleaned up. If that is restored to work as it did before where the caller handled ensuring thread safety since the dao itself is said to be non-thread safe then I think this is fairly clean and easy to reason over and improves putting logic back in one place over our previous implementation. The rest of it seems good including using default implementation if not specified and so on.

joewitt · 2017-02-14T04:26:43Z

...mework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java

-        } finally {
-            writeLock.unlock();
-        }
+        dao.save(controller);


this no longer appears to have write lock protection

joewitt · 2017-02-14T04:26:57Z

...mework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java

-        } finally {
-            writeLock.unlock();
-        }
+        dao.save(controller, outStream);


this no longer has write lock protection

joewitt · 2017-02-14T04:27:06Z

...mework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java

-        } finally {
-            writeLock.unlock();
-        }
+        dao.overwriteFlow(is);


this no longer has write lock protection

joewitt · 2017-02-14T04:29:13Z

...mework/nifi-framework-core/src/main/java/org/apache/nifi/controller/StandardFlowService.java

    @Override
    public void start() throws LifeCycleStartException {
-        writeLock.lock();
+       dao.writeProtectFlow();


this is leaking responsibilities. The 'dao' should either be said to be thread safe or not. The previous approach was not thread safe and thus we had locking around it as was in the previous implementation. Here we have some cases guarded and some not but we're actually invoking a lock within the non-thread safe class now. This approach is harder to reason over in my opinion. We should either make the class thread safe and the caller does not need to do anything special or it should be not thread safe and the calling class should protect it (as it was). Though this method shows we need to have some of that logic external so it makes sense to keep the locking as it was (controlled by the caller with the callee/dao being not thread safe by design)

Thanks @joewitt, I'll move responsibility back over to the caller, along with your other feedback, and push an update.

joewitt · 2017-02-14T04:36:32Z

...work/nifi-framework-core/src/main/java/org/apache/nifi/persistence/FlowConfigurationDAO.java

+    /**
+     * Locks the Flow to prevent cross-thread Writeing
+     */
+    void writeProtectFlow();


i mentioned it in a previous comment but the need to make locking calls available by callers into this class indicates something a bit awkward happening. See previous comments.

joewitt · 2017-02-14T04:36:50Z

...work/nifi-framework-core/src/main/java/org/apache/nifi/persistence/FlowConfigurationDAO.java

+    /**
+     * Locks the Flow to prevent cross-thread Writeing
+     */
+    void readProtectFlow();


copy/paste error on the read/unread protection javadocs.

patricker · 2017-02-17T18:46:32Z

@joewitt @pvillard31 I made the changes and squashed the commits since about 50% of my changes were basically rolled back to the original code.

One issue... it doesn't work, NiFI can't find and load the class on startup.

I dug around in the code some more and found that all Class Loading is done using ExtensionManager. All of the providers are registered with ExtensionManager and it handles locating classes for these known registered types.

I tried registering FlowConfigurationDAO but it's in the wrong project. Moving it from nifi-framework-core to nifi-framework-api is no small task... there are a lot of dependencies and I don't think it really makes sense to move them all. Thoughts?

patricker · 2017-02-21T19:41:05Z

I'm going to just close it as unworkable right now. If someone wants to salvage this later feel free to use my code or start over.

joewitt · 2017-02-21T19:48:56Z

thanks peter. i think this will be much easier and better aligned with the intent as the flow registry work progresses.

pvillard31 requested changes Feb 11, 2017

View reviewed changes

joewitt requested changes Feb 14, 2017

View reviewed changes

patricker force-pushed the NIFI-3446 branch from c0843c2 to 2c8b4e2 Compare February 17, 2017 17:31

NIFI-3446

ce94a60

patricker force-pushed the NIFI-3446 branch from 2c8b4e2 to ce94a60 Compare February 17, 2017 18:43

patricker closed this Feb 21, 2017

patricker deleted the NIFI-3446 branch March 30, 2017 07:17

Conversation

patricker commented Feb 7, 2017

For all changes:

For code changes:

For documentation related changes:

Note:

Uh oh!

pvillard31 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patricker commented Feb 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patricker commented Feb 21, 2017

Uh oh!

joewitt commented Feb 21, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

patricker commented Feb 17, 2017 •

edited

Loading