NIFI-2933 Remote input/output ports at any PG

[ijokarumawak]

NOTE: NiFi Registry needs to be updated in order to support Flow versioning with the added 'Allow Remote Access' port configuration.

---

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

• Is there a JIRA ticket associated with this PR? Is it referenced
  in the commit message?

• Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

• Has your PR been rebased against the latest commit within the target branch (typically master)?

• Is your initial contribution a single, squashed commit?

For code changes:

• Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
• Have you written or updated unit tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
• If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
• If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

[markap14]

Thanks @ijokarumawak! Will review...

[markap14]

@ijokarumawak I've been testing and reviewing this - great work! I tried to do some weird things to see if I could cause any errors, but you seem to have caught all of the corner cases that I could think of :)
I left a handful of comments in the PR, mostly just minor things that need to be addressed, or naming conventions that I felt could be made more clear. Nothing big.

I did notice a couple of things during testing that I wanted to call out though:

• If I configure a Root Group Port, I have the checkbox for "Allow Remote Access." But if I uncheck it and hit Apply, I get an error. Since we don't allow the user to disable Remote Access, we should not allow them to change that in the UI.

• If we're going to have the ability to drag a Port on and then configure to set "Allow Remote Access," I feel like the dialog that asks for a name should also include that checkbox so that it can be set when creating the Port.

• I was a little surprised by the implementation, that a single Local Port can be configured to allow for Remote Access or not, but in either case it still functions as a Local Port. This feels a little odd to me, because a single component on the graph is serving two very different purposes. I feel like it should be either a Local Port or a Remote Port, not both. Thoughts on this?

• While sending data to a port on a secure instance, I revoked my node's permissions to send data to the port. This worked as expected but produced a confusing error message:

2019-03-07 13:44:51,046 ERROR [Timer-Driven Process Thread-1] o.a.n.r.p.socket.SocketClientProtocol SocketClientProtocol[CommsID=43359ff0-85be-42d3-9487-ed2c675c75b2] received unexpected response User Not Authorized: org.apache.nifi.remote.StandardPublicPort@2c831d3f authorization failed for user CN=localhost, OU=NIFI because Unable to modify data transfers to Input Port with ID 5970d42e-0169-1000-ffff-ffffbe062e3b. from Peer[url=nifi://localhost:9001] when negotiating Codec

A quick search through the PR for the phrase "modify data transfers" did not yield any results, so it's possible that this error message has always been like this. If that's the case, then I don't want to hold up this PR for it, but we can probably create a JIRA to make that more clear.

[ijokarumawak]

Thanks @markap14 for reviewing this!

The last commit resolves most review comments except the big one, separate private LocalPort and public LocalPort.

I've been feeling the same oddness with current implementation as Mark called out. Now I think the 'Allow Remote Access' property should be immutable once the LocalPort is created.

BTW, port type names are confusing. It may be useful if we document it somewhere.. Currently, terminologies are like these with this branch:

• RemotePort: Resides in a RemoteProcessGroup, representing a port at the remote Site-to-Site target NiFi instance.
• RootGroupPort: Resides in the root ProcessGroup of a local flow. Always accessible remotely.
• LocalPort:
  • public LocalPort: Enabled 'Allow Remote Access'. Only accepts incoming connections via S2S.
  • private LocalPort: Only accepts incoming connections via local connections.

I will refactor this PR as follows, to make 'Allow Remote Access' immutable:

• Show 'Allow Remote Access' as checkbox at the initial port creation dialog
• After creation, show it as a label in the Port configuration dialog instead of checkbox
• Hide 'Concurrent Tasks' for Local private port.
• Revert some UI Javascripts those were needed to support dynamic port type change
• Modify server side validations those were needed to support dynamic port type change

For authorization error message improvement, I created https://issues.apache.org/jira/browse/NIFI-6114

[markap14]

@ijokarumawak sounds good. Though I think rather than a checkbox in the UI indicating whether or not we allow Remote Access... I would lean more toward a drop-down that allows user to choose the type of port: 'Local Port' vs 'Site to Site Port'.

I also agree that the port terminology is getting a big confusing. I wonder if we should rename RemotePort to RemoteGroupPort. And then we can probably get rid of RootGroupPort all together and instead just have LocalPort and SiteToSitePort (or whatever we want to call it). The RootGroupPort really was meant to differentiate between a local port and one that is capable of performance site-to-site functions. So I think it makes sense to instead just use that sort of terminology, no?

[ijokarumawak]

Refactored UI.

@markap14 Yes, I agree with you. If we can remove RootGroupPort, then those component names will be much cleaner. I'm going to do further refactoring with that approach.

[mcgilman]

Thanks for the PR @ijokarumawak! I've left a few comments here. Another observation I have that wasn't part of this changeset but noticed when testing is the number of ports we list on the [R]PG on the canvas reflects the ports eligible for connections. This may no longer be the case if all ports are remote and the user is attempting a local connection. I believe this approach is the correct one but we likely need to clarify the UX around it.

@moranr Any thoughts?

[mcgilman]

Can we make this dialog larger so that there is no scrolling? The contents appear a little tight.

Also when opening on the root group, can we hide this setting since we only allow for one option?

[ijokarumawak]

Made the dialog larger. So there's no scrolling border. Also hide the combo on the root group.

I also wanted to change the dialog size on the root group or on child groups, however, dialog elements height and min-highet are updated automatically (I assume JQuery does that). I gave up changing the dialog size. There are unnecessary space if the dialog is shown on the root group. I hope it's still acceptable.

[ijokarumawak]

@mcgilman Thanks for reviewing. I hope the last commit addresses all your comments, except the one about the number of port on the canvas.

I'm not sure if I understand it correctly, but I found that a child PG shows total number of ports regardless of public/local, while it only shows total num of FlowFiles and received bytes for local connections.
I think the child PG on the parent PG should show the total num of FlowFiles and received bytes including both public and local connections as its In value.

Is this the same concern you mentioned above?

[mcgilman]

@ijokarumawak My comment regarding the port count was focused on the number of available ports. Without drilling into the group, that figure currently represents the number of ports the user could connect to if they dragged a connection to/from it. Now that is not the case. With this PR, if there were 2 remote ports and 0 local ports that figure would report 2 even though there would be none available if the user dragged a connection to/from it. I'd like to get @moranr and @andrewmlim thoughts on this.

[ijokarumawak]

Thanks for the clarification @mcgilman. Then I will wait for remarks from @moranr and @andrewmlim.

[andrewmlim]

@moranr and I started our review on the UX impact of this. Still working through and will likely ask Matthew Clarke (reporter for https://issues.apache.org/jira/browse/NIFI-2933) for his input, especially around the FlowFile stats shown on the child process group.

In the meantime, wanted to note what looks to be a bug. When a child group has no local output ports, connecting the group to a component on the canvas shows the following pop-up

:

But my environment was not secure and has no authorizations.

[ijokarumawak]

Thanks @andrewmlim for finding that wrong message! I fixed that.

[ekovacs]

Hi @ijokarumawak

i checked out the PR and played around with it. There are a few remarks / questions i have:

1. any port dragged onto the root canvas is public by default. I only have the ability to select the type of the port if i am already in a PG.

2. if i create a simple, 2 part flow like this:
   a.) "generate flowfile" => "Remote Process Group" and
   b.) "input port" => "logAttribute"

and then i want to group 'b.)' part of the flow into a PG, i'll get a:
"Cannot move Ports from the Root Group to a Non-Root Group".
i see that this limitation is not brought in by this PR, but existed previously in the master, along with some other checks:

nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java

Line 2770 in d3d4326

throw new IllegalStateException("Cannot move Ports from the Root Group to a Non-Root Group");

do we want to keep these limitations?

3. I create a PG, and in that PG i created a public port. Modified a.) part of the flow.: Disabled transmission. Deleted Connection. Created a new connection: "to input", selected the: "public port in child". Enabled transmission. FLowfiles started to flow to the PG's public port, this is good.
   While transitting, i created another connection: "to input" selected: "port-on-root-canvas". Flow files are being queued up for this connection. unless i stop transmission and start transmission for the Remote Process Group. see attached picture.
   

4. After 3., i deleted the connection with the queued flowfiles. RPG still shows "XYZ -> 2". yet it transmits only to one public port (one that is in the child PG)
   

[moranr]

@andrewmlim and I reviewed and would like recommend the following changes.

For the Process Group canvas component
Include a count of remote process groups to the In/Out stat lines. Being able to distinguish local from remote at this level can be useful. It also helps provide a some clarity when trying to connect to a group and seeing the no local ports message. Here is an example showing the additional remote port count:

For the Add (input) Port dialog

Change dialog title to:

Add Input Port

Change "Receive Data From" label to:

Receive From

Change "Receive From" label tooltip to:

Specify where FlowFiles are received from

Change "Local connections" menu option tooltip to:

Receive FlowFiles from components in parent process groups

Change "Site-to-Site connections" menu option to:

Remote connections (site-to-site)

Change "Remote connections (site-to-site)" menu option tooltip to:

Receive FlowFiles from remote process group (site-to-site)

For the Add (output) Port dialog

Change dialog title to:

Add Output Port

Change "Transfer Data To" label to:

Send To

Change "Send To" label tooltip to:

Specify where FlowFiles are sent

Change "Local connections" menu option tooltip to:

Send FlowFiles to components in parent process groups

Change "Site-to-Site connections" menu option to:

Remote connections (site-to-site)

Change "Remote connections (site-to-site)" menu option tooltip to:

Send FlowFiles to remote process group (site-to-site)

For the no name error dialog

Change dialog title to:

Configuration Error

Change dialog message to:

The port name must be specified.

The error dialog should open on top of the Add Port dialog. Clicking OK should take the user back to the Add Port dialog. The current implementation of the Add Remote Process Group dialog work this way (screenshot below).

For the duplicate name error dialog

Change dialog title to:

Configuration Error

Change dialog message to:

The port name must be unique throughout the flow.

Dialog behavior should be the same as described in the previous example (no name error dialog).

[andrewmlim]

@moranr and I did get some feedback from Matt Clarke. One issue he brought up is the following.

When creating a connection to a remote port within a child process group, within the Create Connection dialog, the port is labeled as being associated with the root process group not the child process. To illustrate:

On my root PG, I have:

-a child process group ("child_pg") that contains a remote input port ("remote_input_port")
-a RPG that points to the same NiFi instance
-a funnel

If I connect the funnel to the RPG, I see the following:

The input port is not within the group "NiFi Flow" (the root pg), so it should say "child_pg".

If there was a local input port in the child process group, if the funnel was connected to the child process group, the proper distinction is made that the local input port is within "child_pg".

[ijokarumawak]

Hi @ekovacs Here are the answers for your questions:

1. any port dragged onto the root canvas is public by default. I only have the ability to select the type of the port if i am already in a PG.

This is by design. Having local port in the root PG doesn't make sense because no connection can be made from/to such local port since there won't be a parent PG of the root PG.

2. if i create a simple, 2 part flow like this:
   "Cannot move Ports from the Root Group to a Non-Root Group".
   do we want to keep these limitations?

Good catch, thanks. I will consider removing this limitation.

3. I create a PG, and in that PG i created a public port. Modified a.) part of the flow.: Disabled

This is expected. Adding new connection to RemoteProcessGroup while it is transmitting, doesn't start transmission for the newly connected remote port. Each port connection has its own transmission status. You can see it by right clicking the RPG and select 'Manage remote ports' and start transmitting individually from there.

4. After 3., i deleted the connection with the queued flowfiles. RPG still shows "XYZ -> 2". yet it transmits only to one public port (one that is in the child PG)

This is expected. The '2' representing the number of available ports. Not necessary mean actively transmitting. In your case there are two possible ports (in the root and in the child PG).

[ijokarumawak]

Thanks @moranr and @andrewmlim for the UX related feedback. I've started incorporating them. Will let you guys know when I finished updating all.

One thing, port name/id duplication have been implemented as 409 (conflict) error at the server side, while empty port name error returned with 400 (bad request) status code.
Generally, NiFi UI dialogs close if it receives error other than 400.

There are few options, which one do you think the best??

1. Change server's return code from 409 to 400 for port duplication. I can do this, but not confident enough, because 409 also does make sense in case if another port has been added by a different user..
2. Keeps the port new/config dialog open with empty port name, but close it with duplicate error or any other errors than 400. This is the common behavior for UI.
3. Keeps the port new/config dialog open regardless of the status code. This can be a weird UX sometimes. For example, when admin revokes the user's privilege before user completed the dialog, but the dialog should be closed because retrying won't success..

[andrewmlim]

Thanks @ijokarumawak for providing more context around the 409/400 behavior in the application. Can you provide some other UI examples where a 409 error occurs and the dialog closes?

[ijokarumawak]

@andrewmlim I checked other dialogs and found that:

• Connection configuration dialog closes regardless of the result code. Even closes the dialog 400. Probably we should update this to provide consistent UX.
• Processor / RPG dialog closes with only successful 200. All dialogs should follow this.

I was confused with the current implementation of the Port configuration dialog.
Currently (before this PR), Port config dialog source code has following comments and closes dialog when it receives non-successful response codes other than 400.

// handle bad request locally to keep the dialog open, allowing the user
// to make changes. if the request fails for another reason, the dialog
// should be closed so the issue can be addressed (stale flow for instance)

On the other hand, RemoteProcessGroup (or Processor for example) configuration dialog does NOT close with status code such as 400 (bad request), 403 (forbidden) or 409 (conflict). It closes the dialog with only 200 (success) response.

I think we should follow RemoteProcessGroup dialog style. Close the dialog only it receives 200. This PR already does so.

[ijokarumawak]

@andrewmlim About the feedback from Matthew Clarke posted earlier.

All public ports are exposed to RPGs as a flat structure. ProcessGroup hierarchy is not shared to remote instances. The root process group name is the only known PG name at the S2S client side.

Maybe the component name RemoteProcessGroup is getting a bit confusing once we allow any PG to have public port.. I wish it could have been RemoteNiFi.
In order to share process group hierarchy, we need to alter S2S protocol. I don't think that's a good idea.

Instead, we can change label text in case connection source/target is a RemoteGroupPort.

I'd like to ask @mcgilman or @markap14 their opinion if possible.

[andrewmlim]

@ijokarumawak Thanks for your latest comments. I filed two Jiras to make the UX consistent during Processor Configuration (https://issues.apache.org/jira/browse/NIFI-6160) and for Connection Configuration when required fields are missing (https://issues.apache.org/jira/browse/NIFI-6170).

I agree that we should hold off/discuss further the improvement Matt Clarke raised if a change to the S2S protocol is required to implement it. Thanks for providing that context. Changing the label text is an interesting workaround for now. Will wait for thoughts from @mcgilman and @markap14.

[markap14]

Personally I am fine with the idea of changing "Within Group" to indicate "Remote NiFi instance." Would even consider adding the remote instance's URL ("Remote NiFi instance http://other-nifi.org.com:8080").

I am also in favor of keeping the dialog open on any failure condition.

[ijokarumawak]

Thanks @markap14 for sharing your thoughts.

I've implemented experiment code to change the label text and show remote url.
@andrewmlim @moranr How do you think?

Connection to a Remote Input Port

Connection from a Remote Output Port

[moranr]

I like the addition of the URL.

However, for a connection to a remote input port, I would suggest keeping as it was before and simply add an indicator that it's "remote." So instead of Remote NiFi Instance in the screenshot above, something like:

Within Group (Remote)

This is consistent with the the other scenario and the labels still read like a narrative as they were designed (e.g., From/To..., Within..., etc.).

[ijokarumawak]

@moranr Thanks, that makes sense, too. I've updated the code to make the labels narrative. How about this?

Connection to a Remote Input Port

Connection from a Remote Output Port

[andrewmlim]

I vote for Within Remote Group first. Within Group (Remote) second.

[ijokarumawak]

I vote for Within Remote Group. Updated the code.

[markap14]

Hey @ijokarumawak it looks like this PR now has some conflicts. Can you rebase against master so that we can finish up the review?

[ijokarumawak]

Thanks @markap14 for checking. Rebased with the latest master.

[markap14]

Thanks for rebasing, @ijokarumawak. Things seemed to work OK on a standalone, insecure nifi. Then I tried starting up a 2-node secure cluster. When I attempted to create a Process Group, I got an NPE:

2019-04-29 15:30:27,425 ERROR [NiFi Web Server-25] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.lang.RuntimeException: Failed to serialize entity for cluster replication. Returning Internal Server Error response.
java.lang.RuntimeException: Failed to serialize entity for cluster replication
	at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.serializeEntity(OkHttpReplicationClient.java:259)
	at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.createRequestBody(OkHttpReplicationClient.java:233)
	at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.prepareRequest(OkHttpReplicationClient.java:98)
	at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.performVerification(ThreadPoolRequestReplicator.java:619)
	at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicate(ThreadPoolRequestReplicator.java:445)
	at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicate(ThreadPoolRequestReplicator.java:301)
	at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicate(ThreadPoolRequestReplicator.java:227)
	at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicate(ThreadPoolRequestReplicator.java:164)
	at org.apache.nifi.web.api.ApplicationResource.replicateNodeResponse(ApplicationResource.java:1081)
	at org.apache.nifi.web.api.ApplicationResource.replicateNodeResponse(ApplicationResource.java:1053)
	at org.apache.nifi.web.api.ApplicationResource.replicate(ApplicationResource.java:1044)
	at org.apache.nifi.web.api.ApplicationResource.replicate(ApplicationResource.java:1028)
	at org.apache.nifi.web.api.ProcessGroupResource.createProcessGroup(ProcessGroupResource.java:1716)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
	at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:60)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:60)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:60)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:99)
	at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:58)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
	at org.apache.nifi.web.security.headers.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:48)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.apache.nifi.web.security.headers.XSSProtectionFilter.doFilter(XSSProtectionFilter.java:48)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.apache.nifi.web.security.headers.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.apache.nifi.web.security.headers.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:48)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:531)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
	at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.apache.nifi.web.api.entity.ProcessGroupEntity["component"]->org.apache.nifi.web.api.dto.ProcessGroupDTO["inputPortCount"])
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
	at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:727)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
	at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2655)
	at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381)
	at org.apache.nifi.cluster.coordination.http.replication.okhttp.JsonEntitySerializer.serialize(JsonEntitySerializer.java:39)
	at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.serializeEntity(OkHttpReplicationClient.java:253)
	... 115 common frames omitted
Caused by: java.lang.NullPointerException: null
	at org.apache.nifi.web.api.dto.ProcessGroupDTO.getInputPortCount(ProcessGroupDTO.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
	... 125 common frames omitted

[ijokarumawak]

Thanks for reviewing @markap14 and sorry for the delay. I was taking vacation. Fixed the NPE and DTO deserialization issue which happened after fixing NPE. Please try it again.

[markap14]

@ijokarumawak thanks for the update. Tried testing again today. Now, the secure cluster is working as expected. Unfortunately, though, I'm now running into problems with a standalone, insecure node. I created a simple flow that will use GenerateFlowFile -> RPG (back to http://localhost:8080, using RAW site-to-site protocol). Then created a Process Group that has a 'Public Input Port' in it just sending to an UpdateAttribute. When I Enable Transmission on the RPG, the data doesn't go anywhere. Just stays there. If I attempt to disable transmission on the RPG, it just blocks and never stops.

[ijokarumawak]

@markap14 Thanks for testing this again. I tried reproducing the stuck situation you encountered with a non-secure standalone NiFi, but to no avail. Public Input Ports in a child Process Group work fine. Tested:

• Restart NiFi while GenerateFlowFile -> RPG keep sending data
• Stop the port and start it again
• Back-pressure, and recover from it
• Delete and recreate the same port name

If that is reproducible with your NiFi, would you be able to capture stack-trace for further analysis? Thanks!

[markap14]

@ijokarumawak I cannot replicate the issue anymore. Which leads me to think that it was probably a user error (perhaps I had the specific port disabled in the RPG or something...??) But all is looking good at this point. Tried to do some testing of corner cases, stop port while transmitting, stop rpg while transmitting, force backpressure and timeouts, etc. Everything recovered as it should. Great work! Many thanks for this contribution - I know many people will be very happy about it! Have merged to master.