Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NIFI-7407 Refactored SSL context generation throughout framework and extensions. #4263

Closed
wants to merge 12 commits into from

Conversation

alopresto
Copy link
Contributor

Thank you for submitting a contribution to Apache NiFi.

Please provide a short description of the PR here:

Description of PR

In order to streamline the creation of SSLContext and SSLSocket objects throughout the application, I refactored the various near-duplicate but slightly-different factory objects. This also involved refactoring duplicate enums and removing legacy modules which were no longer necessary. Additional comments, unit tests, and regression tests were introduced.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

  • Is there a JIRA ticket associated with this PR? Is it referenced
    in the commit message?

  • Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

  • Has your PR been rebased against the latest commit within the target branch (typically master)?

  • Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

  • Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
  • Have you written or updated unit tests to verify your changes?
  • Have you verified that the full build is successful on both JDK 8 and JDK 11?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
  • If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
  • If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

alopresto added 2 commits May 10, 2020 18:26
…2" (in shared constant).

Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
@alopresto
Copy link
Contributor Author

alopresto commented May 11, 2020

High level description of changes:

  • Previously there were SslContextFactory implementations in multiple modules. I enhanced the one in nifi-security-utils which is now used throughout the project, and removed the implementations in nifi-framework and nifi-socket-utils.
    • Part of this refactoring was removing public static methods which created an SSLContext object from various combinations of explicit keystore and truststore properties. These were being used in an inconsistent manner. I introduced a container object called TlsConfiguration which wraps the state of the configuration and provides internal validation checks. This encapsulates the need to check for different combinations of configuration presence/validity in each use case (components, framework, etc.) and relieves the calling developer of re-implementing this logic every time.
    • I also provided static convenience methods like getX509TrustManager() and createSSLSocketFactory() because in most cases that is what the calling code needs, rather than an intermediate SSLContext object they need to further configure. This reduced the need for Tuple<> return values throughout the code.
    • Duplicate code to transform the various return values and configure the OkHttpClient and its Builder were refactored to utility methods.
  • Duplicate enums were refactored.
  • Unnecessary code dealing with client authentication settings when creating a client connection/socket were removed (these settings would be ignored, as only an SSL/TLS server can decide to enforce/request client authentication).
  • Some tests were refactored to make mocking easier.
  • Removed extraneous file loading during NiFiProperties construction in many tests.
  • Enforced modern TLS protocol versions in various internal socket creations.

The easiest way to test these changes is to configure and deploy a secured cluster (see Apache NiFi Walkthroughs: Creating and Securing a NiFi Cluster with the TLS Toolkit) and run a flow which handles incoming secured connections such as ListenHTTP, HandleHttpRequest, etc.

Copy link
Contributor

@pvillard31 pvillard31 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some unit tests failing. It looks like order matters when asserting that two arrays are equal. I suggested to use Set instead. (Note that's the only thing I looked at for now - just to get green builds)

@alopresto
Copy link
Contributor Author

Still some failures because the Java 8 on the GitHub Actions containers does not support TLSv1.3. Will resolve.

@thenatog
Copy link
Contributor

Reviewing..

Added external timing check to timing test assertion.
@alopresto
Copy link
Contributor Author

alopresto commented May 14, 2020

I made the dropdown for RestrictedSSLContextService more explicit where it now provides TLS, TLSv1.2 on Java 8 and TLS, TLSv1.2, TLSv1.3 on Java 11. Selecting TLS will allow connections over TLSv1.2 and TLSv1.3 (on Java 11 only. Java 8 does not support TLSv1.3).

With TLSv1.2 selected:


# TLSv1.2 is successful

 ..oolkit-1.11.4   master ●  echo Q | openssl s_client -connect node1.nifi:9999 -key nifi-key.key -cert nifi-cert.pem -CAfile nifi-cert.pem -tls1_2
CONNECTED(00000003)
depth=1 OU = NIFI, CN = ca.nifi
verify return:1
depth=0 OU = NIFI, CN = node1.nifi
verify return:1
---
Certificate chain
 0 s:OU = NIFI, CN = node1.nifi
   i:OU = NIFI, CN = ca.nifi
 1 s:OU = NIFI, CN = ca.nifi
   i:OU = NIFI, CN = ca.nifi
---
...
---
SSL handshake has read 2289 bytes and written 1464 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: BA2FC4...0D2790
    Session-ID-ctx:
    Master-Key: C773AC...A85A19
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1589478477
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
DONE

# TLSv1.3 fails

 ..oolkit-1.11.4   master ●  echo Q | openssl s_client -connect node1.nifi:9999 -key nifi-key.key -cert nifi-cert.pem -CAfile nifi-cert.pem -tls1_3
CONNECTED(00000003)
4570201536:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1544:SSL alert number 70
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 234 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
 ✘  ..oolkit-1.11.4   master ● 

With TLS selected:


### TLSv1.3 is successful

 ..oolkit-1.11.4   master ●  echo Q | openssl s_client -connect node1.nifi:9999 -key nifi-key.key -cert nifi-cert.pem -CAfile nifi-cert.pem -tls1_3
CONNECTED(00000003)
depth=1 OU = NIFI, CN = ca.nifi
verify return:1
depth=0 OU = NIFI, CN = node1.nifi
verify return:1
---
Certificate chain
 0 s:OU = NIFI, CN = node1.nifi
   i:OU = NIFI, CN = ca.nifi
 1 s:OU = NIFI, CN = ca.nifi
   i:OU = NIFI, CN = ca.nifi
---
...
---
SSL handshake has read 2510 bytes and written 1800 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE

# TLSv1.2 is successful

 ..oolkit-1.11.4   master ●  echo Q | openssl s_client -connect node1.nifi:9999 -key nifi-key.key -cert nifi-cert.pem -CAfile nifi-cert.pem -tls1_2
CONNECTED(00000003)
depth=1 OU = NIFI, CN = ca.nifi
verify return:1
depth=0 OU = NIFI, CN = node1.nifi
verify return:1
---
Certificate chain
 0 s:OU = NIFI, CN = node1.nifi
   i:OU = NIFI, CN = ca.nifi
 1 s:OU = NIFI, CN = ca.nifi
   i:OU = NIFI, CN = ca.nifi
---
...
---
SSL handshake has read 2293 bytes and written 1464 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 7E5D46...1F4E63
    Session-ID-ctx:
    Master-Key: AB80DE...4FCC9A
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1589478427
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
DONE

Copy link
Contributor

@markap14 markap14 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @alopresto! I know this was a huge and tedious undertaking. But it dramatically simplifies things when dealing with TLS and will help to ensure that we are both correct and consistent in dealing with these configurations in the future.

I reviewed the code and all looks good from that perspective. I created a 2-node cluster that was secured with self-signed certificates. Tested clustering and UI access. Tested load-balanced connections and secure site-to-site (both http-based and raw socket based). Everything appears to work exactly as expected.

I'm a +1 but @thenatog indicated that he is reviewing also so will not merge it to master yet. Thanks again!

@thenatog
Copy link
Contributor

thenatog commented May 14, 2020

Looks like there's currently a test error for JDK11.

My testing:

Java 8

Java 11

Saw errors with site to site when using the HTTP protocol. I'm not certain if it's related to these changes or not:
"2020-05-14 15:16:06,799 WARN [Timer-Driven Process Thread-9] o.apache.nifi.remote.client.PeerSelector Could not communicate with node0.com:9551 to determine which nodes exist in the remote NiFi cluster, due to javax.net.ssl.SSLPeerUnverifiedException: Certificate for <node0.com> doesn't match any of the subject alternative names: [node1.com]"
It's possible these errors only happen for a cluster hosted on the same machine/localhost.

@alopresto
Copy link
Contributor Author

Thanks @markap14 and @thenatog for the extensive testing. I pushed another commit which enables TLSv1.3 for the Java 11 UI/API port and should resolve the test error.

I can reproduce the S2S issue mentioned above on a secure 3 node cluster pointing back to itself when all nodes are hosted on the same machine. I don't think the PR changed how this worked, so I suspect this existed previously, but I'll try to address it here as well. I also encountered trouble retrieving S2S peers so I will add some unit tests there to see what I can isolate and fix.

@alopresto alopresto force-pushed the NIFI-7407_rebased branch from 5f7d2b1 to b85436e Compare May 15, 2020 02:12
@alopresto
Copy link
Contributor Author

Force pushed as I had to fix the referenced Jira number in the recent commit messages.

@alopresto
Copy link
Contributor Author

I decided to do the S2S refactor in a separate Jira as it grew larger than anticipated. @thenatog if you're satisfied with what's here, please give a +1 and I'll merge. Thanks.

@thenatog
Copy link
Contributor

Looks like there might be a small issue with JDK11 tests, if we can fix that I'll +1. Thanks for the huge contribution, Andy! Definitely valuable changes here - the SSL Contexts have needed this improvement for a long time.

@alopresto
Copy link
Contributor Author

There was another Java 11 unit test failure. Resolved that.

@alopresto
Copy link
Contributor Author

Running a full build this time because some of the tests were failing on ordering.

… classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.
@alopresto
Copy link
Contributor Author

Thanks for finding all the edge cases @thenatog & @markap14. I think this is ready for your +1. I'll then merge.

@thenatog
Copy link
Contributor

+1, looks good. Thanks Andy!

@asfgit asfgit closed this in 441781c May 19, 2020
phuthientran pushed a commit to FerrelBurn/nifi that referenced this pull request Jan 8, 2021
…2" (in shared constant).

Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.

This closes apache#4263.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
driesva pushed a commit to driesva/nifi that referenced this pull request Mar 19, 2021
…2" (in shared constant).

Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.

This closes apache#4263.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
hoavho added a commit to hoavho/nifi that referenced this pull request Jul 19, 2023
* NIFI-7135 - Fix Java 11 build with com.puppycrawl.tools:checkstyle:jar:8.29 dependency

This closes #4050.

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7165 Fixed TLS Toolkit Guide flags with default validity days.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4064.

* NIFI-7175 removed appveyor and travis builds which have become unreliable.  Added Github Actions based CI build for pull requests and pushes. Removed unit tests which make unreadable or unaccesible dirs. Includes windows build as well as ubuntu. we can do localization based builds later as this is all proven stable.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4058.

* NIFI-7175 README.md

* NIFI-7175 Fixed core attributes formatting in Developer Guide. (#4066)

* NIFI-7175 Fixed core attributes formatting in Developer Guide.

* NIFI-7175 Made core attribute names more consistent.

* NIFI-7026 Add kerberos password property to NiFi HortonworksSchemaRegistry

* NIFI-7114: Fix file leaks in StandardCommsSession and S2S Reporting components

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7114: This closes #4069. Update time-sensitive tests in TestLuceneEventIndex

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7181 convert integration test to proper name so it isnt run as unit test

* NIFI-7178 - Handle the case when schema is not available

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4067.

* NIFI-7185 relaxed timing constrained for builds on lower resources environments like our Github CI builds

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4075.

* NIFI-7183 - This closes #4073. Improve ReplaceText when removing FF's content

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7184 - This closes #4074. Added mime type property to GenerateFlowFile

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7133 - This closes #4049.  Clarification of EnforceOrder description

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7007: Add update functionality to the PutCassandraRecord processor.

NIFI-7007: Add additional unit tests that hit non-happy path

NIFI-7007: Use AllowableValue instead of string

NIFI-7007: Add the use of attributes for the update method, statement type and batch statement type

NIFI-7007: Add additional tests, mainly for the use of attributes

NIFI-7007: add some ReadsAttribute properties to the PutCassandraRecord processor

NIFI-7007: additional update keys validation logic

NIFI-7007: fix imports

NIFI-7007: Convert fieldValue to long in separate method

NIFI-7007: Add new style of tests checking actual CQL output

NIFI-7007: add license to new test file

NIFI-7007: add customValidate to check for certain incompatible property combinations

NIFI-7007: remove check on updateMethod and replace Set.of with java 8 compatible replacmenet

NIFI-7007: Add test for failure with empty update method via attributes

NIFI-7007: remove unused variable

NIFI-7007: Fix customValidate that incorrectly invalidated a valid config

Fix Checkstyle

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3977

* NIFI-5924 Labels should be searchable

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4070

* NIFI-6363 Refactors sensitive properties, adds additional providers.

NIFI-6363 Additional fixes.

NIFI-6363 Fix Hadoop compile problem.  Add GCP IT instructions.

NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase.

NIFI-6363 - Added some docs and experimental tag to the relevant classes.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4080.

* Revert "NIFI-6363 Refactors sensitive properties, adds additional providers."

This reverts commit 479fcfdc0bbd97dd0635bbec0273cc25fcf9cc03.

It does not build properly as shown in Github Actions.

* NIFI-7163 - added RulesEngine and RulesEngineProvider interfaces, enhanced easy rules to support provider interface and refactored to extract rules engine implementation

NIFI-7163 - updated documentation and comments

NIFI-7163 - fix checkstyle issues

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4081

* NIFI-7173 - This closes #4084. add nifi-metrics jar to scripting bundle

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7152 Added custom ExceptionMappers to handle invalid Remote Process Group port value - (#4085)

JsonContentConversionExceptionMapper, JsonMappingExceptionMapper, JsonParseExceptionMapper.
Registered the custom ExceptionMappers.
Added unit tests to throw Exception for string port value and sanitize script input. Handled null or empty JsonMappingException reference path.
Added the Apache license to Groovy Test.

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7164 Upgrade shyiko/mysql-binlog-connector-java dependency

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4076.

* NIFI-7201 - Update build to latest apache-maven-parent and split Github Actions builds to include OSX and distribute localization

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4091.

* NIFI-7139 Add record.error.message on failure of a record reader or writer

Handle scenario where message might be null.

Update to test case that was failing because adding attributes modified a flow file even if you don't change the contents.

Fixed Style Issues and Updated WritesAttributes.

Added Test Case for Error Message

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4052

* NIFI-7205 NIFI-7206

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4093.

* NIFI-6856 - Make client ID a non-required field for the MQTTConsume and MQTTProduce processors. Generates a
random ID if not set.

Also add group ID field to ConsumeMQTT processor. Allows consumer to join consumer group at $share/<group_id>/<topic_filter>

add expression language support for the MQTT client ID

Setting client id in publish test fails because it is not a flowfile attribute.
Remove client id and autogenerate it when testing.

Since the evaluation is done in onScheduled, there is no flow file available and we're not using the attributes to make the expression language evaluation. You can change the scope to use the Variable Registry.

Co-Authored-By: Pierre Villard <pierre.villard.fr@gmail.com>
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3879.

* NIFI-6791 Add UUID3 and UUID5 functions to Expression Language

This closes #4031

* Bump commons-lang3 and commons-codec versions

* NIFI-7018: Initial commit of processors extending AbstractHadoopProcessor supporting kerberos passwords
AbstractHadoopProcessor will always authenticate the principal with a KerberosUser implementation and a UGI will be acquired from the Subject associated with the KerberosUser implementation
AbstractHadoopProcessor's getUserGroupInformation method will now attempt to check the TGT and relogin if a KerberosUser impelmentation is available, otherwise it will return the UGI referenced in the HdfsResource instance
Updated AbstractHadoopProcessor's customValidate method to consider the provided password and updated validation failure explanations when a KerberosCredentialsService is specified together with a principal, password, or keytab
Added toString method override to AbstractKerberosUser
Updated Hive/HBase components to be compatible with the KerberosProperties.validatePrincipalWithKeytabOrPassword method
Fixed null ComponentLog in GetHDFSSequenceFileTest

Added package-protected accessor method (getAllowExplicitKeytabEnvironmentVariable) to AbstractHadoopProcessor for determining if the environment variable "NIFI_ALLOW_EXPLICIT_KEYTAB" has been set
AbstractHadoopProcessor will now only fail validation when the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable is set to false if a keytab is provided to allow the user to specify a principal and password
Added AbstractHadoopProcessorSpec to verify validation of principal/keytab/password/kerberos credential service combinations

This closes #4095.

* NIFI-7025: Initial commit adding Kerberos Password feature for Hive components
Kerberos Password property should not support EL, this includes a change to KerberosProperties which is also used by the HDFS processors (AbstractHadoopProcessor)
Added wiring in a KerberosContext to a TestRunner's MockProcessorInitializationContext
Removed synchronization blocks around KerberosUser.checkTGTAndRelogin, since that method is already synchronized
Updated AbstractHadoopProcessor to have a boolean accessor method to determine if explicit keytab configuration is allowed
Removed synchronization block from HiveConnectionPool's getConnection method (in Hive, Hive_1_1, Hive3 modules), since new TGT ticket acquisition is handled by the KerberosUser implementation.  If UGI is used to relogin, synchronization is handled internally by UGI.
Added Kerberos Principal and Kerberos Password properties to Hive, Hive_1_1, and Hive3 components
Hive, Hive_1_1, and Hive3 components now use KerberosUser implementations to authenticate with a KDC

Updated handling of the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable in Hive and Hive3 components.  An accessor method has been added that uses Boolean.parseBoolean, which returns true if the environment variable is set to true, and false otherwise (including when the environment variable is unset).

Addressing PR feedback

Addressing PR feedback

This closes #4102.

* NIFI-7024: Added Kerberos Password support to HBase_1_1_2_ClientService and HBase_2_ClientService

This closes #4103.

* NIFI-7029 Add kerberos password property to NiFi Kudu components

This closes #4097.

* NIFI-7019 Add kerberos principal and password properties to NiFi DBPCConnectionPool

This closes #4087.

* NIFI-7030 Add Kerberos principal and password properties to Solr processors
Updating validation logic to be consistent with other password-based kerberos processors, removing getPassword from KerberosPasswordUser as it was only used from testing

This closes #4062.

* NIFI-5644 Fixed typo in getWrappedQuery method of AbstractDatabaseFetchProcessor class

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4106.

* NIFI-7218 Fixed typo in Overview docs. (#4107)

* NIFI-7121 Updated comment to state a 'static' salt is used in the constructor. (#4098)

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-4970 - EOF Exception in InvokeHttp when body's response is empty with gzip

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4109.

* NIFI-7025: Wrap Hive 3 calls with UGI.doAs
Updated PutHive3Streaming to wrap calls to Hive in UGI.doAs methods
Fixed misleading logging message after the principal has been authenticated with the KDC
When connecting to unsecured Hive 3, a UGI with "simple" auth will be used

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4108

* NIFI-7208: Fixed PutSQL/JdbcCommon handling of timestamps (nanoseconds, e.g.)

* NIFI-7227 Fixed typo in Global Access Policy table (#4112)

Co-authored-by: spius <57421336+spius@users.noreply.github.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7055 handle empty split evaluations, which contain only ,

add explict test for " , "

updated with counting validator

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4012

* NIFI-7055: Removed unit test that is now covered by ListValidator

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4114.

* NIFI-7222 Cleaned up API for FTP/SFTP remote file retrieval and ensure we close remote file resources for SFTP pulls in particular

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4115.

* NIFI-7226: Add Connection Factory configuration properties to PublishJMS and ConsumeJMS processors

Some JMS client libraries may not work with the existing controller services due to incompatible
classloader handling between the 3rd party library and NiFi.
Via configuring the Connection Factory on the processor itself, only the processor's and its
children's classloaders will be used which eliminates the mentioned incompatibility.

This closes #4110.

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7232 if users do not supply a remote path PutSFTP with conflict resolution will fail with an NPE

* Fixed unit test failed with NIFI-7232

This closes #4117.

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7191 Conditionally disable docker integration tests

Honor Maven properties skipTests and maven.test.skip for the
shell script integration tests that verifiy docker images.

* Fiz writeBatch to avoid cycle in provenance graph

* NIFI-7231: move controller service validation out of synchronized block for enabling

This closes #4118.

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7197 - In-place replacement in LookupRecord processor

This closes #4088

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7224 Protecting against possible NPE in ImportFlowVersion command in CLI

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4121.

* NIFI-7195 - Catch MongoException to route flow files to failure

This closes #4089

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-7210 - added PG path in bulletins for S2S Bulletin RT

Added group path to BULLETIN table for QueryNiFiReportingTask

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4100

* NIFI-7179 Documented Download flow option in Process group context menu (#4124)

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7200: Revert "NIFI-6530 - HTTP SiteToSite server returns 201 in case no data is available"

This reverts commit f01668e66ad2e45197915769e966a4be27e1592e.

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-4970 - Add a property to deal with empty GZIP HTTP response

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4127.

* NIFI-7239: Upgrade the Hive 3 bundle to use Apache Hive 3.1.2

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4129.

* NIFI-7050 ConsumeJMS is not yielded in case of exception

This closes #4004.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7245: JMS processors yield when connection factory initialisation failed

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4133.

* unit test reproducing the issue

* Fixed bug in JsonRowRecordReader when reading a 'raw' record with a schema that indicates that a field should be a Map. Also updated unit test to explicitly define schema, since schema inference will never return a Map but rather a Record

* NIFI-7242: When a Parameter is changed, any property referencing that parameter should have its #onPropertyModified method called. Also renamed Accumulo tests to integration tests because they start embedded servers and connect to them, which caused failures in my environment. Also fixed a bug in TestLengthDelimitedJournal because it was resulting in failures when building locally as well.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4134.

* NIFI-7119 Implement boundary checking for Argon2 cost parameters (#4111)

* NIFI-7119 Implemented parameter boundary enforcement for Argon2SecureHasher constructor.
Added unit tests for validating each parameter check.

* NIFI-7119 Refactored parameter validations. Added more test sizes to boundary checkers. Changed logger severity to error and added bounds to messages.

* NIFI-7119 Refactored Argon2 parameter data types to handle unsigned integer boundary values.
Updated unit tests.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7241: When updating Process Group to match VersionedProcessGroup, remove any connections before recursing into child groups. This ensures that if a Port exists in child group A and is connected to a port in child group B, if the VersionedProcessGroup indicates to remove the port, that connection will be removed before attempting to remove the port. Updating and adding connections must still be done last, after all components have been added. But missing connections can be removed earlier.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4136.

* NIFI-7248: Atlas reporting task handles PutHive3Streaming

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4138.

* NIFI-7244 Updated all tests which dont run well on windows to either work or be ignored on windows
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4132.

* NIFI-6742 Use JUnit TemporaryFolder when creating test databases

- Add @Rule for TemporaryFolder
- Replace use of previous target/db with TemporaryFolder/db
- Remove use of ~/test db (in home directory)
- Remove System.out lines

Signed-off-by: Marc Parisi <phrocker@apache.org>

This closes #4137.

* NIFI-7251: Upgrade hadoop-client version to 3.2.1 to avoid the regression bug

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4141.

* NIFI-7229 - Upgrade jackson-databind direct dependencies

This closes #4113

* NIFI-7249: Force String keys in maps in DataTypeUtils.inferDataType()

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4139.

* NIFI-7250 activate user.timezone appropriate to each region
NIFI-7250 fix a test which appears brittle at least on windows builds on slow environments
NIFI-7250 activated a timezone run for AU Australia/Melbourne which exposed a poor magic number and needless assertion but interesting results worth keeping

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4140.

* NIFI-7256: This closes #4142. Fixed thresholds in unit test. Instead of assuming that multiple runs of the processor will occur within 100 milliseconds, allowed the multiple runs to occur within 3 mins of one another.

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7208: Restore default timezone in JdbcCommon

* NIFI-7223 - Fixed a minor issue where the OkHttpReplicationClient class loaded blank properties as empty string instead of an expected null value. Added a isNotBlank check. Added unit tests for replication client and HTTPNotificationService.
NIFI-7223 - Renamed some variables and methods.
NIFI-7223 - Removed unused dependency. Corrected security properties in administration-guide.

* NIFI-7223 [WIP] Resolved compilation issues in unit test on OpenJDK 11 by removing Sun security class references.
Added OkHttpReplicationClient#isTLSConfigured() method.
Added unit test.
NIFI-7223 Fixed remaining unit tests for TLS regression.
Renamed tests for clarity.

* NIFI-7223 - Added another test for when keyPasswd is not present.

* NIFI-7223 Resolved merge conflicts from additional test case for null key password.

This closes #4145.

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7258 - fix overflow in PutAzureEventHub when not configured correctly

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4146.

* NIFI-7267 - Upgrade spring-data-redis in Redis bundle (#4150)

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… (#4151)

* NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.favre.lib.bcrypt library.
Updated LICENSE and NOTICE files to reflect changes.
Updated unit tests.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7268 Fixed typo in Javadoc.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

* NIFI-6293

Add support to Mongo Extended JSON v2
Add org.json lib
Replace evil json
Replace evil json for alternative
Include testExtendedJsonSupport
Style adjustment
Remove unecessary new JSON parser
Fix query in testExtendedJsonSupport
Parse with Jackson and BSON
Back to default MONGO_URI

This closes #4068

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-7264 Make jsonPath Expression Logging More Reasonable
add special handling of PathNotFoundExceptions to log to debug
fix spelling error
wrap debug log in guard per review

This closes #4148

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-7187 adding missing version strings from accumulo bundle pom
- Removed Cat X JSON.org dep inclusion which seems to not be necessary
- Updated a ton of easier/safer looking deps
- Updated tika due to CVE

This closes #4086

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7221 Initial work

* NIFI-7221 Support v2 and v3 protocol version for Hortonworks Schema Registry
- Update nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/serialization/SchemaRegistryRecordSetWriter.java
- Addressing review feedback

This closes #4120.

* NIFI-7274 add time for test conditions to be met

* NIFI-7271 Make command timeout configurable for ShellUserGroupProvider
- Changing ShellRunner to use a separate thread for reading the output of the process
- Removing unused member variable
- Addressing review feedback

This closes #4154.

* NIFI-7257 Added HadoopDBCPConnectionPool
- Updated InstanceClassLoader to resolve files that are in the instance urls or additional urls
- Updated nifi-mock to support KerberosContext and removeProperty for ControllerServices
- Added unit test for HadoopDBCPConnectionPool
- Addressing review feedback

This closes #4149.

* NIFI-7269 - Upgrade solrj version to 7 in nifi-solr-processors

Remove unused imports

Use the latest solrj version(8.4.1)

Setup default schemaFactory for tests
The default schemaFactory ManagedIndexSchemaFactory creates
additional files in test's resources directory. Change it to
ClassicIndexSchemaFactory for classic behavior.

This closes #4152.

Signed-off-by: Bryan Bende <bbende@apache.org>

* NIFI-7238 Improve Caching for Github CI and relax core usage to not max out cores all moving toward more stable builds.

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7278 Adding support for SCRAM-SHA-512 to Kafka 2.0 processors

* NIFI-5925: Added controller services to set of components that are searched

NIFI-5925: cleanup, add negative test

NIFI-5925: fixed checkstyle

This closes #4105

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-7238 Continue to improve Github Actions CI stability

* NIFI-7238 remove no longer needed delete command

* NIFI-7238 printing maven version info in same build command

* NIFI-7281 This closes #4159. Use BufferedInputStream in StandardSocketChannelRecordReader in order to support mark/reset

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7279 This closes #4160. Protect against NPE in RedisDistributedMapCacheClientService when value is null

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7153 Adds ContentLengthFilter to enforce configurable maximum length on incoming HTTP requests.
Adds DoSFilter to enforce configurable maximum on incoming HTTP requests per second.
Redirected log messages for ContentLengthFilter to nifi-app.log in logback.xml.

This closes #4125.

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7286 ListenTCPRecord cleanup changed from @OnStopped to @OnUnscheduled

* NIFI-7287: Move services-api dependency from Prometheus reporting task to its NAR

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4162

* NIFI-7290: omit transitive that is causing the build to fail and not needed for the test scope

* NIFI-7291 updated enforcement of dependency rules and build command

Reviewed by markap14
This closes #4166.

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7294 Address deprecation issues in solrj and httpclient

Some calls to deprecated methods in httpclient were resulting in
UnsupportedOperationException. Use the new API calls in both httpclient
and solrj. Add an integration test to include test coverage for
org.apache.nifi.processors.solr.SolrUtils.createClient

This closes #4171.

* NIFI-7297 add available()

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4172.

* NIFI-7103 Adding PutDataLakeStorage Processor to provide native support for Azure Data Lake Storage Gen 2 Storage.

added data-lake dependency
NIFI-7103 fixed indentation
Update to add IllegalArgumentException
Fixed indentation and logging
nifi-7103 review changes
nifi-7103 root directory and exception change

This closes #4126.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7293 Add in-memory janusgraph implementation of GraphClientService to help with live testing.

Added new in memory janus graph client for testing.
Added integration test to ExecuteGraphQuery.

NIFI-7293 Added missing getter.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4168

* add validator for lists that ensure the element validator is called for empty entries

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4116

* NIFI-7311 adding additional cleanup and output to github actions yml and not running nifi-system-test module

Self merging wo review as part of github ci actions stability.

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7188 Extending UI search with filters and refactoring existing solution

This closes #4123.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping (#4156)

* NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping

* NIFI-7273: Changed method name, fix handling when analytics not enabled

* NIFI-7273: Removed attachment header from Prometheus metrics endpoint

* NIFI-7273: Removed unused variable

* NIFI-7317 - make .java files non-executable

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4181.

* NIFI-7126 Increased test iterations to 10,000 in Argon2SecureHasherTe… (#4187)

* NIFI-7126 Increased test iterations to 10,000 in Argon2SecureHasherTest#testDefaultCostParamsShouldBeSufficient to avoid JVM warmup issues.

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7326 updated URL to find splunk artifacts (#4188)

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7314 HandleHttpRequest stops Jetty in OnUnscheduled instead of OnStopped. Also reject pending request and clean their queue when shutting down.

NIFI-7314 In HandleHttpRequest returning 503 when rejecting pending requests before shutdown.
NIFI-7314 In HandleHttpRequest add logs and better response message during cleanup.

This closes #4191.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7259 Adding DeleteDataLakeStorage Processor to provide native support for Azure Data lake Gen 2 Storage.

Updated to remove unused variables
NIFI-7259 import and property description changes

This closes #4189.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7345: Fixed Hive database and table names case insensitivity in Atlas reporting task

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4198.

* NIFI-6849: Reworked how nodes inherit cluster information when joining a cluster. Now, if there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's flow is inherited.
 - Refactored Flow Synchronization to make code cleaner
 - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty.
 - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start
- Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state.
- Added additional log statements to aid in debugging

NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior

NIFI-6849: Addressed review feedback

NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup

This closes #3891

* NIFI-7087: Use FlowManager.findAllConnections() when available

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4026

* NIFI-7341 Updated certificate commands and source code formatting in Toolkit Guide. (#4196)

* NIFI-7339: Fixed bug that caused Write Ahead Provenance Repository not to rollover event files after specified time. Code cleanup. Updated some default properties.

* NIFI-7346: Ensure that the Provenance Repository doesn't delete the Active Event File

* NIFI-7319 Add walkthrough document (#4193)

* NIFI-7319 Added first draft of walkthroughs doc.

* NIFI-7319 Added instructions and screenshots for securing standalone NiFi instance.

* NIFI-7319 Added instructions and screenshots for instructing OS & browser to trust self-signed certificate.

* NIFI-7319 Added instructions and screenshots for securing NiFi with externally-provided certificates.

* NIFI-7319 Added instructions and screenshots for building NiFi from source.

* NIFI-7319 [WIP] Converting secure cluster instructions to match format.
Fixed instructions regarding embedded ZooKeeper configuration.

* NIFI-7319 Completed secure cluster walkthrough.

* NIFI-7319 Added walkthroughs to documentation navigation list.

* NIFI-7319 Incorporated PR feedback on broken links.

* NIFI-7319 Removed line number helpers from update sections.

* NIFI-7319 Incorporated final PR review items.

Co-authored-by: Sandra Pius <spiusapache@gmail.com>

* NIFI-7354: Allow analytics properties to be set via environment variables in docker scripts

This closes #4203.

Signed-off-by: Aldrin Piri <aldrin@apache.org>

* NIFI-7347: Fixed NullPointerException that can happen if a bin is merged due to timeout and has no records

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4210.

* NIFI-6977 - Change the reporting behavior of Azure Reporting task to report report the time when metrics are generated

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4211.

* [NIFI-7358] - Fix: Sorting on 'Estimated Time to Back Pressure' in the Connection summary table does not work properly
- fix style issues
- review feedback

This closes #4208

* NIFI-7292 Preventing file listing from fail because of insufficient privileges

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4195.

* Added Jira and security reporting links to README.md

* NIFI-7359 Fix parent id on process metrics for Prometheus

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4209

* NIFI-7348 Wait - Removes WAIT_START_TIMESTAMP after expiration

This closes #4201.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>

* NIFI-7334 Adding FetchDataLakeStorage Processor to provide native support for Azure Data lake Gen 2 Storage.

NIFI-7334 Update to FetchDataLakeStorage Processor

This closes #4212.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7366 - ConsumeEWS Processor parse EML

https://issues.apache.org/jira/browse/NIFI-7366

This commit allows to retrieve ItemAttachement (such as EML) file when pulling mail.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4215.

* NIFI-7318 - Allow 'docker stop' to gracefully shutdown

As it is issuing 'docker stop' will immediatly exit the container.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4182.

* NIFI-7300 Allowing narrow numeric types to fit againt schema check with wider type; Allowing doubles with value within float precision to be considered as valid floats (NIFI-7302)

* NIFI-7375: This closes #4218. Fixed a bug that caused Provenance Events not to show up in specific situations when clicking View Provenance for a Processor.
- Added System-level tests for Provenance repository to reproduce behavior.
- Added a Provenance Client to the CLI, which is necessary for System-level tests.
- Added small additional configuration for Provenance repository to simplify development of system tests
- Minor improvements to system tests (such as ability to destroy environment between tests) needed for Provenance repository based system tests

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7280 ReportLineageToAtlas recognizes 'atlas.metadata.namespace' from Atlas config file.

Still recognizes 'atlas.cluster.name' as well, but takes lower precedence than the new property.
Also Atlas URL can be provided via the 'atlas.rest.address' property in the atlas-application.properties.

NIFI-7280 In ReportLineageToAtlas improved documentation and adjusted property ordering for better user experience. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended documentation. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended more documentation. More minor refactor.
NIFI-7280 - In Atlas reporting: complete clusterName -> namespace overhaul where appropriate.

This closes #4213.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7378: Ensure label values are not null in Prometheus metrics (#4219)

* Do not update status for stopping a deleted node

* NIFI-7389 Makes Missable heartbeat counts configurable

This closes #4236.

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7377 Cleaned up nifi-stateless logs.
Refactored masking logic to CipherUtility and indicated masking with label and Base64 output.
Added JSON masking logic to nifi-stateless module.
Added argument masking functionality to Program.
Moved groovy unit tests to proper Maven directory structure.
Modified plain argument output to use filtering/masking methods in provided utility.
Refactored utility methods.
Updated unit tests.

This closes #4222.

Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7298: PutAzureDataLakeStorage tests.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4227.

* NIFI-7379: Support multiple instances of Prometheus registries/metrics (#4229)

* NIFI-7379: Support multiple instances of Prometheus registries/metrics

* NIFI-7379: Refactored Prometheus objects to support multiple instances

* NIFI-7394: Add support for sending Multipart/FORM data to InvokeHTTP.
By using dynamic properties with a prefix naming scheme, allow
definition of the parts, including the name to give the Flowfile content
part, and optionally it's file name.
After review:
- change so that we can send just the form content or just form data
  without the flowfile
- change the content name and content file name from dynamic properties
  to properties
- change the dynamic name to be an invalid http header "post:form:xxxx"
- add validation and more tests

This closes #4234.

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7408 - added percent used metrics for connections

NIFI-7408 - return double value for utilization

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4240

* NIFI-7170:
- Adding a flag to nifi.properties to disable anonymous authentication.

NIFI-7170:
- Fixing checkstyle issues.

NIFI-7170:
- Adding missing license header.

NIFI-7170:
- Initial PR feedback.

NIFI-7170:
- Fixing broken integration tests.
- Creating new integration tests for verifying allowing and preventing anonymous access.

NIFI-7170:
- Ensuring the new anonymous authentication property is considered for proxied requests.

NIFI-7170 - Fixed comment.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4099.

* Added note about unique initial user identity names to walkthrough doc.

* NIFI-6149: Azure EventHub Managed identities support patch

review changes
additional review changes
NIFI-6149: typo fixes

This closes #4226.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7416: Update travis-ci to GitHub Actions in the PR template

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4247.

* NIFI-7415: Add .asf.yaml to configure GitHub integrations

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4246.

* NIFI-7404: Fixed invalid script processors upon thread termination

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4238.

* NIFI-7412: Fixed provenance event types in Azure Fetch/Delete processors

This closes #4245.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7414: Escape user-defined values that contain invalid XML characters before writing flow.xml.gz

NIFI-7414: Updated StandardFlowSerializerTest to include testing for variable names and values being filtered

This closes #4244

* NIFI-7425 Log Message for ReplaceText Over Buffer Size

Adds a log message when ReplaceText sends a flowfile to the failure relationship because
it is larger than the max buffer size.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4255.

* NIFI-7420 remove the http.param attribute. It contained both the query parameters already captured in http.query.param _and_ the multipart form data names and values, which are captured in the part data, and could be very very large

This closes #4251.

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7428: Switch hive.version property to set Hive 3 version

This closes #4259

* NIFI-7398 Upgraded jackson-databind dependency version to 2.9.10.4 at root pom.xml.
Upgraded tika-parsers dep in nifi-media-processors.
Upgraded jackson-databind dep in nifi-graph-processors.
Upgraded jackson-databind dep in nifi-elasticsearch-client-service-api.
Upgraded jackson-databind dep in in nifi-easyrules-service.
Upgraded calcite-core dep in nifi-sql-reporting-tasks.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4252.

* NIFI-7390 Covering Avro type conversion in case of map withing Record

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4256

* NIFI-7423 Upgraded jquery dependency version.

NIFI-7423 Upgraded jquery dependency version to latest 3.5.1.

This closes #4258

* NIFI-6913: PutAzureBlobStorage processor will create container if not exists

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4237.

* NIFI-7413: Documented REMOTE_INVOCATION provenance event type in user/dev guides

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4267

* NIFI-7321 - Allow NiFi admins to configure whether Jetty will send the Jetty server version in responses.
Fixed a checkstyle error.
Added property to nifi.properties.
Changed property to a variable that is set with the pom.xml.
Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context.
Fixed typo error.

This closes #4192.

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7367: Add tests for FetchAzureDataLakeStorage

NIFI-7367: Negative test cases for expression language in FetchAzureDataLakeStorage
FetchAzureDataLakeStorage throws exception when filesystem or filename is blank.
Fixed logged error messages in all 3 of the Delete, Fetch and Put ADLS processors.
testFetchDirectory test case marked as ignored.

This closes #4257.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7448: Fix quoting of DDL table name in PutORC

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4269.

* NIFI-7460: Avoid NPE when a VersionedProcessor has a null value for autoTerminatedRelationships. Added additional logging and improved error handling around syncing with invalid flows

* NIFI-7437 - created separate thread for preloading predictions, refactors for performance

NIFI-7437 - reduced scheduler to 15 seconds, change cache to expire after no access vs expire after write

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4274

* NIFI-6497: Allow FreeFormTextRecordSetWriter to access FlowFile Attributes

This closes #4275.

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7380 - fix for controller service validation in NiFi Stateless

This closes #4264.

Signed-off-by: Matthieu Cauffiez <matthieu.cauffiez@bell.ca>
Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7446: FetchAzureDataLakeStorage processor now throws exception when the specified path points to a directory

A newer version (12.1.1) of azure-storage-file-datalake is imported.

This closes #4273.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7331 Fixed grammatical errors in log output.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4283.

* NIFI-7409: Azure managed identity support to Azure Datalake processors

NIFI-7409: review changes
NIFI-7409: ordering import statements
NIFI-7409: changed validateCredentialProperties logic

This closes #4249.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7336: Add tests for DeleteAzureDataLakeStorage

DeleteAzureDataLakeStorage now throws exception if fileSystem or fileName is empty string

NIFI-7336: Add tests for DeleteAzureDataLakeStorage - typos fixed
NIFI-7336: Add tests for DeleteAzureDataLakeStorage - fixed a test case

This closes #4272.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-6911 Removed default Blob value for PutAzureBlobStorage

This closes #3906

Signed-off-by: Joey Frazee <jfrazee@apache.org>

* NIFI-7407 Replaced SSLContextFactory references to "TLS" with "TLSv1.2" (in shared constant).
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.

This closes #4263.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7471 fix bug with property validation

* NIFI-6571 Check token length on TLS toolkit server startup

This closes #3659.

Signed-off-by: Joey Frazee <jfrazee@apache.org>

* Fixed a couple of typos in the RecordPath guide

* NIFI-7463
Create empty relationship for RunMongoAggregation

Fix default autoterminate and condition to redirect to REL_EMPTY

Change from new relationship to write an empty FlowFile to RESULT

Fix MONGO_URI

This closes #4281

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-7462: This adds a way to convert or cast a choice object into a valid type for use with calcite query functions

NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types

NIFI-7462: Fixed checkstyle violation, removed documentation around the CAST functions that were no longer needed

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4282

* NIFI-7482 Changed InvokeHTTP to be extensible.
Added unit test.

This closes #4291.

Signed-off-by: Arpad Boda <aboda@apache.org>

* Updated pull request template to separate JDK 8 and 11 questions

* NIFI-6255 NIFI-6287: Hash function for expression language and record path.
NIFI-6255 NIFI-6287: Rebased to match the new expression language interface
NIFI-6255 NIFI-6287: Fix wildcard imports and unused imports
NIFI-6255 NIFI-6287: Move to the common codec DigetUtils
Update commons-codec

This closes #3624

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-6672 PlusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc
NIFI-6672 Removed test code.

This closes #3738

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-6673 MultiplyEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY or NEGATIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc

This closes #3739

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-6674 MinusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY

MinusEvaluator throws an Arithmetic Exception in case of Long overflow.
TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY

The behaviour change is reverted until further investigations.
The overflow behaviour is still enforced by unit tests and documented in the expression language doc

fixed mispositioned # in doc

This closes #3740

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-7211 Added @Ignore with warning message to a test that randomly fails due to timing issues.

This closes #4296

* NIFI-6785 Support Deflate Compression
NIFI-6785 Remove unused imports

This closes #3822

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-7453 In PutKudu creating a new Kudu client when refreshing TGT

NIFI-7453 Creating a new Kudu client when refreshing TGT in KerberosPasswordUser as well. (Applied to KerberosKeytabUser only before.)
NIFI-7453 Safely closing old Kudu client before creating a new one.
NIFI-7453 Visibility adjustment.

This closes #4276.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7485 Updated commons-configuration2.
NIFI-7485 Found more instances that needed updating.

This closes #4295

* NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor

NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor
Made warning and error messages more informative.
Refactored flowFile assertion in the tests.

This closes #4287.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7487 - Added batch support and displayName to ModifyBytes processor

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4302.

* NIFI-7483 - Remove description about 'Rolling strategy' in TailFile's docs

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4293.

* NIFI-7484:fix ListFTP and FetchFTP docs. Change 'SFTP' to 'FTP' in description

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4294.

* NIFI-7422: Support aws_s3_pseudo_dir in Atlas reporting task

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4292.

* NIFI-6701 - Fix for PublishGCPPubSub

* NIFI-7430 - LookupRecord change coordinate key for in-place replacement

* NIFI-6666 Add Useragent Header to InvokeHTTP requests

This closes #3734

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* Updated KEYS with new key after previous one expired

* NIFI-7403:Add a function that adjust if the result is failed before we call the onFailed or onCompleted function. If the result is failed, return true and do sth

NIFI-7403:Add an extension point to adjust the result, if the result is failed then process onFailed function

NIFI-7403:Implement the AdjustFailed Function, if PutSQL set the SUPPORT_TRANSACTIONS true, then check whether the result contains REL_RETRY or REL_FAILURE.If it contains that, reroute the result and return true.

NIFI-7403: fix reroute logic in AdjustFailed function

NIFI-7403:Add and modify some unit test for PutSQL's SUPPORT_TRANSACTIONS property

NIFI-7403:Update for PR recheck

NIFI-7403:Add documentation on the Support Fragmented Transactions property to indicate the transactions rollback behavior

NIFI-7403: Fix Checkstyle issue

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4266

* NIFI-7477 Optionally adding validation details as a new attribute of the flowfile

NIFI-7477 Improving description and unit test now verifies attribute content

NIFI-7477: Fixed checkstyle errors

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4301

* NIFI-7447: When returning an object from a Controller Service, if that object is defined as an interface, proxy that interface. This way, any method call into the object will also change the classloader to the appropriate classloader.

* NIFI-7312: Enable search in variable registry of root process group

This closes #4303.

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7369 Adding decimal support for record handling in order to avoid missing precision when reading in records

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7369: Consider DECIMAL type as a numeric type when using a CHOICE type in QueryRecord

This closes #4223.

* NIFI-7299 Add basic OAuth2 token provider service that can fetch access tokens when supplied with appropriate credentials.

Added skeleton of oauth2 provider.
Added copy of our code.
Refactored a few things.
Updated apis to better match flow descriptions.
Updated poms and other artifacts.
Updated copyright notice.
Updated LICENSE.

This closes #4173

Signed-off-by: Jeremy Dyer <jeremydyer@apache.org>

* NIFI-7476: Implemented FlowFileGating / FlowFileConcurrency at the ProcessGroup level
Added FlowFileOutboundPolicy to ProcessGroups and updated LocalPort to make use of it
Persisted FlowFile Concurrency and FlowFile Output Policy to flow.xml.gz and included in flow fingerprint
Added configuration for FlowFile concurrency and outbound policy to UI for configuration of Process Groups
Added system tests. Fixed a couple of bugs that were found
Fixed a couple of typos in the RecordPath guide

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4306.

* NIFI-7508: Reset classloader after running TestStandardControllerServiceInvocationHandler
and fix checkstyle violation on NiFiSystemIT

* NIFI-7467 Refactored S2S peer selection logic.
Removed list structure for peer selection as it was unnecessary and often wasteful (most clusters are 3 - 7 nodes, the list was always 128 elements).
Changed integer percentages to double to allow for better normalization.
Removed 80% cap on remote peers as it was due to legacy requirements.
Added unit tests for non-deterministic distribution calculations.
Added unit tests for edge cases due to rounding errors, single valid remotes, unbalanced clusters, and peer queue consecutive selection tracking.
Migrated all legacy PeerSelector unit tests to new API.
Removed unused System time manipulation as tests no longer need it.
Added class-level Javadoc to PeerSelector.
Removed S2S details request replication, as the responses were not being merged, which led to incorrect ports being returned and breaking S2S peer retrieval.
Fixed copy/paste error where input ports were being listed as output ports during remote flow refresh.
Fixed comments and added unbalanced cluster test scenarios.
Removed unnecessary marker interface.
Removed commented code.
Changed weighting & penalization behavior.
Changed dependency scope to test.

This closes #4289.

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7393: Add max idle time and idle connections to InvokeHTTP

This closes #4233.

Signed-off-by: Joey Frazee <jfrazee@apache.org>

* NIFI-7507: Added section to User Guide on configuring a Process Group

NIFI-7507: Fixed Flowfile Expiration header in doc

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4318

* NIFI-7511 In ControllerServiceProxyWrapper extended documentation. Minor refactor in StandardControllerServiceInvocationHandler. Also removed an unused import from NiFiSystemIT.

This closes #4317.

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7385 Provided reverse-indexed TokenCache implementation.
Cleaned up code style.
Unit test was failing on Windows 1.8 GitHub Actions build but no other environment. Increased artificial delay to avoid timing issues.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

This closes #4271.

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7514:
- Ensuring the group id is always set in the properties table when loading properties.
- Using a common approach to getting parameters in nfControllerService.
- Code clean up.
- Addressing review feedback.
- Ensuring the service dialog is closed when navigating to the parameter context dialog.

This closes #4322

* NIFI-7490 - Add optional raw field to Syslog readers

review

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4299

* NIFI-7313:fix bug on 'Quote Table Identifiers'
NIFI-7313:add test by wanghongqi
NIFI-7313:edit test

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4185

* NIFI-7434: Add endpoint suffix to Azure storage processors

This closes #4265.

Signed-off-by: Joey Frazee <jfrazee@apache.org>

* NIFI-7442 Add CLI commands to the registry in order to support automatic registry setup

* NIFI-7442 Added missing use cases (list users and user groups), made update-access-policy use case more in line with the NiFi side. Added some tests. Additional refactor, documentation revision.

This closes #4329.

* NIFI-7527 AbstractKuduProcessorrefresh TGT deadlock fix: Redesigned locking.

NIFI-7527 Fixed StackOverFlowError due to pacing issue (recursive login before loggedIn flag is set).
NIFI-7527 Refactor: removed redundant kudu client creation.

This closes #4330.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7035 The first curator connection issue is logged as ERROR until reconnect

* NIFI-7539: When capturing diagnostics information, capture a thread dump once and then provide this information to ProcessorNode when capturing active threads. Previously, each processor captured a thread dump itself. When this is done thousands of times it can result in a very long delay.

* NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment (#4341)

* NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment

This also fixes NIFI-4760.

* NIFI-7540: Remove duplicate mail.smtp.starttls.enable from TestListenSMTP

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7537 - Small fix to make the LDAP connection timeout property a String rather than a Long. Fixes cast error in LDAP libs.

* NIFI-6094 - Added the X-Content-Type-Options header to all web responses. (#4307)

NIFI-6094 - Added the mime/content type for ttf files.

* NIFI-7551 Add support for VARCHAR to Kudu NAR bundle
 - update Kudu dependencies to Kudu 1.12.0
 - add VARCHAR to Kudu Lookup Service and Processor
 - add tests for VARCHAR columns

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4347.

* NIFI-7536: Fix to improve performance when determining the run status of processors when needing to wait for all processors to stop for updating parameter context, etc.

* NIFI-7509: Added optional Record Writer property to all List* Processors

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4315.

* NIFI-7566: Avoid using Thread.sleep() to wait for Site-to-Site connection to be handled. Instead, use TimeUnit.timedWait and use Object.notifyAll when setting the beingServiced flag. This significantly reduces latency and improves throughput for small-batch site-to-site communications

This closes #4353.

Signed-off-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7529 Removed OS and Java information from InvokeHttp's UserAgent field so that it's removed regardless of whether or not this field is kept.

This closes #4332

* NIFI-7516: Catch and log SingularMatrixExceptions in OrdinaryLeastSquares model (#4323)

* NIFI-7501
Update nf-context-menu.js for an intuitive road to parameters
When rightclicking a process group the variables are shown, but parameters are not. This makes sense as they have a prerequisite, in the form of a parameter context. This change gives a more consistent experience for finding the functionality regarding parameters by ensuring the contextmenu shows the possibility to configure a parameter context. Once the paramater context has been created for a process group, the parameters text shows, so this is no longer visible. People would then need to click configure to change the context, just as they would be required to do now.

Added generateflowfile load tag and description
Added GenerateFlowFile load tag to be consistent with DuplicateFlowFile and updated description to refer to DuplicateFlowFile.

Revert "Update nf-context-menu.js for an intuitive road to parameters"

This reverts commit 3c44b1661f09fb6ae11d2f088550f81fb7a4b393.

This closes #4333

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-7558 Fixed CatchAllFilter init logic by calling super.init().
Renamed legacy terms.
Updated documentation.

This closes #4351.

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-7542 Override jackson-databind version.
NIFI-7542 Override additional jackson-databind versions.
NIFI-7542 Upgrade jackson-databind dependency to 2.9.10.5 in the root pom.xml.

This closes #4343

Signed-off-by: Mike Thomsen <mthomsen@apache.org>

* NIFI-7523: Use SSL Context Service for Atlas HTTPS connection in Atla… (#4348)

* NIFI-7523: Use SSL Context Service for Atlas HTTPS connection in Atlas reporting task

Also fixing ControllerServiceDisabledException-s when validating the Kerberos config

* NIFI-7523: Fixed test failure on Windows

* NIFI-7523: Added license headers.

* NIFI-7523: Fixed another test failure on Windows

* NIFI-7523: Review changes

* NIFI-7576 ListenHTTP: Honor primary node only
ListenHTTP processor now binds port and creates a HTTP connection only
if one of the following conditions apply:
- Primary node execution is 'false'
- Primary node execution is 'true' and node is elected as primary node.

Changes:
- Connection is established in 'onTrigger' annotated method instead of
  'onSchedule'. (This is similar to how handleHTTPRequest processor
  handles connections.)
- 'onPrimaryNodeStateChange' annotated method is introduced to tear down
  server on reelection of primary node

This closes #4356.

Signed-off-by: Mark Payne <markap14@hotmail.com>

* NIFI-6163 Reporting task cannot be set to running when in INVALID state

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4334.

* NIFI-7577 Upgrade angular version.

* NIFI-7577 Update jquery usages.

This closes #4357

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

* NIFI-7586 In CassandraSesionProvider added properties to set socket-level read timeout and connect timeout.

In QueryCassandra when writing flowfile to the sesion it's done on the raw OutputStream.
Wrapped it in a BufferedOutputStream for better performance.

This closes #4368.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7578 nifi-toolkit CLI Process Group Create command
- Remove unused imports
- Fix checkstyle errors

This closes #4358.

* NIFI-7587 This closes #4372. Increased tolerance for non-deterministic unit test.

Signed-off-by: Joe Witt <joewitt@apache.org>

* NIFI-7590 In 'CassandraSessionProvider.onDisabled' setting Cassandra-related references properly to null after closing them so that they can be renewed in 'onEnabled' (which creates them only if set to 'null', leaving them closed otherwise).

NIFI-7590 Removed 'CassandraSessionProvider.onStopped'.

This closes #4373.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7513 Added custom DNS resolution steps to walkthrough (#4359)

* NIFI-7563: Optimize the usage of JMS sessions and message producers

The introduced changes prevent creating unnecesary sessions and producers
in some scenarios.

This closes #4378.

Signed-off-by: Joey Frazee <jfrazee@apache.org>

* NIFI-7594 In HandleHttpRequest deleting multipart file resources after processing.

This closes #4379.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

* NIFI-7332 Added method to log available claim names from the ID provider response when the OIDC Identifying User claim is not found. Revised log message to print available claims.
Added new StandardOidcIdentityProviderGroovyTest file.
Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test.
Added comments in getAvailableClaims() method.
Fixed typos in NiFi Docs Admin Guide.
Added license to Groovy test.
Fixed a checkstyle error.
Refactor exchangeAuthorizationCode method.
Added unit tests.
Verified all unit tests added so far are passing.
Refactored code. Added unit tests.
Refactored OIDC provider to decouple constructor & network-dependent initialization.
Added unit tests.
Added unit tests.
Refactored OIDC provider to separately authorize the client. Added unit tests.
Added unit tests.

NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve the NiFi JWT.

Signed-off-by: Natha…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants