NIFI-7018: Initial commit of processors extending AbstractHadoopProce… #4095
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ssor supporting kerberos passwords AbstractHadoopProcessor will always authenticate the principal with a KerberosUser implementation and a UGI will be acquired from the Subject associated with the KerberosUser implementation AbstractHadoopProcessor's getUserGroupInformation method will now attempt to check the TGT and relogin if a KerberosUser impelmentation is available, otherwise it will return the UGI referenced in the HdfsResource instance Updated AbstractHadoopProcessor's customValidate method to consider the provided password and updated validation failure explanations when a KerberosCredentialsService is specified together with a principal, password, or keytab Added toString method override to AbstractKerberosUser Updated Hive/HBase components to be compatible with the KerberosProperties.validatePrincipalWithKeytabOrPassword method Fixed null ComponentLog in GetHDFSSequenceFileTest
bbende
reviewed
Feb 27, 2020
| return UserGroupInformation.getUGIFromSubject(subject); | ||
| }); | ||
| } catch (PrivilegedActionException | LoginException e) { | ||
| throw new IOException("Unable to acquire UGI for KerberosUser: " + e.getLocalizedMessage(), |
There was a problem hiding this comment.
In the case of PrivilegedActionException, the message is always null because they expect you to get the message from first calling getException() or getCause() and then getting the message from that, so you may want to do that first here depending which type of exception.
There was a problem hiding this comment.
Updated the code to use the message from the wrapped exception, and split the unified catch into two distinct catch clauses.
| .build()); | ||
| } | ||
|
|
||
| final String allowExplicitKeytabVariable = System.getenv(ALLOW_EXPLICIT_KEYTAB); | ||
| if ("false".equalsIgnoreCase(allowExplicitKeytabVariable) && (explicitPrincipal != null || explicitKeytab != null)) { | ||
| if ("false".equalsIgnoreCase(allowExplicitKeytabVariable) && (explicitPrincipal != null || explicitKeytab != null || explicitPassword != null)) { |
There was a problem hiding this comment.
I'm not sure if we want to include password into this logic. If ALLOW_EXPLICIT_KEYTAB=false, it seems valid to still use principal+password.
There was a problem hiding this comment.
Updated the code to allow that scenario, and added a unit test to verify the validation logic in AbstractHadoopProcessor. This works in conjunction with the added test cases in TestKerberosProperties to validate principal/keytab/password combinations.
| @@ -69,6 +79,37 @@ public static synchronized UserGroupInformation loginKerberos(final Configuratio | |||
| return UserGroupInformation.getCurrentUser(); | |||
| } | |||
|
|
|||
| public static synchronized UserGroupInformation loginKerberosWithPassword(final Configuration config, final String principal, final String password) throws IOException { | |||
There was a problem hiding this comment.
I think this unused at this point since the KerberoUser's are created outside and everything goes through getUgiFromKerberosUser, but I'm not against keeping the method if you want.
There was a problem hiding this comment.
Agreed on removing it. We don't want, at this point anyway, to get a UGI without being able to associate the KerberosUser from which it was created. I removed the method.
…nmentVariable) to AbstractHadoopProcessor for determining if the environment variable "NIFI_ALLOW_EXPLICIT_KEYTAB" has been set AbstractHadoopProcessor will now only fail validation when the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable is set to false if a keytab is provided to allow the user to specify a principal and password Added AbstractHadoopProcessorSpec to verify validation of principal/keytab/password/kerberos credential service combinations
bbende
reviewed
Feb 28, 2020
| final String allowExplicitKeytabVariable = System.getenv(ALLOW_EXPLICIT_KEYTAB); | ||
| if ("false".equalsIgnoreCase(allowExplicitKeytabVariable) && (explicitPrincipal != null || explicitKeytab != null)) { | ||
| final String allowExplicitKeytabVariable = getAllowExplicitKeytabEnvironmentVariable(); | ||
| if ("false".equalsIgnoreCase(allowExplicitKeytabVariable) && explicitKeytab != null) { |
There was a problem hiding this comment.
Do we want this line to be unchanged from the original logic?
Before it was && (explicitPrincipal != null || explicitKeytab != null)
There was a problem hiding this comment.
Never mind this comment, I see now that we need to change the logic for the case where ALLOW_EXPLICIT_KEYTAB is false and they use a principal + password.
|
Everything looks good here, going to merge |
driesva
pushed a commit
to driesva/nifi
that referenced
this pull request
Mar 19, 2021
…ssor supporting kerberos passwords AbstractHadoopProcessor will always authenticate the principal with a KerberosUser implementation and a UGI will be acquired from the Subject associated with the KerberosUser implementation AbstractHadoopProcessor's getUserGroupInformation method will now attempt to check the TGT and relogin if a KerberosUser impelmentation is available, otherwise it will return the UGI referenced in the HdfsResource instance Updated AbstractHadoopProcessor's customValidate method to consider the provided password and updated validation failure explanations when a KerberosCredentialsService is specified together with a principal, password, or keytab Added toString method override to AbstractKerberosUser Updated Hive/HBase components to be compatible with the KerberosProperties.validatePrincipalWithKeytabOrPassword method Fixed null ComponentLog in GetHDFSSequenceFileTest Added package-protected accessor method (getAllowExplicitKeytabEnvironmentVariable) to AbstractHadoopProcessor for determining if the environment variable "NIFI_ALLOW_EXPLICIT_KEYTAB" has been set AbstractHadoopProcessor will now only fail validation when the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable is set to false if a keytab is provided to allow the user to specify a principal and password Added AbstractHadoopProcessorSpec to verify validation of principal/keytab/password/kerberos credential service combinations This closes apache#4095.
hoavho
added a commit
to hoavho/nifi
that referenced
this pull request
Jul 19, 2023
* NIFI-7135 - Fix Java 11 build with com.puppycrawl.tools:checkstyle:jar:8.29 dependency This closes #4050. Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7165 Fixed TLS Toolkit Guide flags with default validity days. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4064. * NIFI-7175 removed appveyor and travis builds which have become unreliable. Added Github Actions based CI build for pull requests and pushes. Removed unit tests which make unreadable or unaccesible dirs. Includes windows build as well as ubuntu. we can do localization based builds later as this is all proven stable. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4058. * NIFI-7175 README.md * NIFI-7175 Fixed core attributes formatting in Developer Guide. (#4066) * NIFI-7175 Fixed core attributes formatting in Developer Guide. * NIFI-7175 Made core attribute names more consistent. * NIFI-7026 Add kerberos password property to NiFi HortonworksSchemaRegistry * NIFI-7114: Fix file leaks in StandardCommsSession and S2S Reporting components Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7114: This closes #4069. Update time-sensitive tests in TestLuceneEventIndex Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7181 convert integration test to proper name so it isnt run as unit test * NIFI-7178 - Handle the case when schema is not available Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4067. * NIFI-7185 relaxed timing constrained for builds on lower resources environments like our Github CI builds Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4075. * NIFI-7183 - This closes #4073. Improve ReplaceText when removing FF's content Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7184 - This closes #4074. Added mime type property to GenerateFlowFile Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7133 - This closes #4049. Clarification of EnforceOrder description Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7007: Add update functionality to the PutCassandraRecord processor. NIFI-7007: Add additional unit tests that hit non-happy path NIFI-7007: Use AllowableValue instead of string NIFI-7007: Add the use of attributes for the update method, statement type and batch statement type NIFI-7007: Add additional tests, mainly for the use of attributes NIFI-7007: add some ReadsAttribute properties to the PutCassandraRecord processor NIFI-7007: additional update keys validation logic NIFI-7007: fix imports NIFI-7007: Convert fieldValue to long in separate method NIFI-7007: Add new style of tests checking actual CQL output NIFI-7007: add license to new test file NIFI-7007: add customValidate to check for certain incompatible property combinations NIFI-7007: remove check on updateMethod and replace Set.of with java 8 compatible replacmenet NIFI-7007: Add test for failure with empty update method via attributes NIFI-7007: remove unused variable NIFI-7007: Fix customValidate that incorrectly invalidated a valid config Fix Checkstyle Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #3977 * NIFI-5924 Labels should be searchable Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4070 * NIFI-6363 Refactors sensitive properties, adds additional providers. NIFI-6363 Additional fixes. NIFI-6363 Fix Hadoop compile problem. Add GCP IT instructions. NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase. NIFI-6363 - Added some docs and experimental tag to the relevant classes. Signed-off-by: Nathan Gough <thenatog@gmail.com> This closes #4080. * Revert "NIFI-6363 Refactors sensitive properties, adds additional providers." This reverts commit 479fcfdc0bbd97dd0635bbec0273cc25fcf9cc03. It does not build properly as shown in Github Actions. * NIFI-7163 - added RulesEngine and RulesEngineProvider interfaces, enhanced easy rules to support provider interface and refactored to extract rules engine implementation NIFI-7163 - updated documentation and comments NIFI-7163 - fix checkstyle issues Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4081 * NIFI-7173 - This closes #4084. add nifi-metrics jar to scripting bundle Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7152 Added custom ExceptionMappers to handle invalid Remote Process Group port value - (#4085) JsonContentConversionExceptionMapper, JsonMappingExceptionMapper, JsonParseExceptionMapper. Registered the custom ExceptionMappers. Added unit tests to throw Exception for string port value and sanitize script input. Handled null or empty JsonMappingException reference path. Added the Apache license to Groovy Test. Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7164 Upgrade shyiko/mysql-binlog-connector-java dependency Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4076. * NIFI-7201 - Update build to latest apache-maven-parent and split Github Actions builds to include OSX and distribute localization Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4091. * NIFI-7139 Add record.error.message on failure of a record reader or writer Handle scenario where message might be null. Update to test case that was failing because adding attributes modified a flow file even if you don't change the contents. Fixed Style Issues and Updated WritesAttributes. Added Test Case for Error Message Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4052 * NIFI-7205 NIFI-7206 Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4093. * NIFI-6856 - Make client ID a non-required field for the MQTTConsume and MQTTProduce processors. Generates a random ID if not set. Also add group ID field to ConsumeMQTT processor. Allows consumer to join consumer group at $share/<group_id>/<topic_filter> add expression language support for the MQTT client ID Setting client id in publish test fails because it is not a flowfile attribute. Remove client id and autogenerate it when testing. Since the evaluation is done in onScheduled, there is no flow file available and we're not using the attributes to make the expression language evaluation. You can change the scope to use the Variable Registry. Co-Authored-By: Pierre Villard <pierre.villard.fr@gmail.com> Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #3879. * NIFI-6791 Add UUID3 and UUID5 functions to Expression Language This closes #4031 * Bump commons-lang3 and commons-codec versions * NIFI-7018: Initial commit of processors extending AbstractHadoopProcessor supporting kerberos passwords AbstractHadoopProcessor will always authenticate the principal with a KerberosUser implementation and a UGI will be acquired from the Subject associated with the KerberosUser implementation AbstractHadoopProcessor's getUserGroupInformation method will now attempt to check the TGT and relogin if a KerberosUser impelmentation is available, otherwise it will return the UGI referenced in the HdfsResource instance Updated AbstractHadoopProcessor's customValidate method to consider the provided password and updated validation failure explanations when a KerberosCredentialsService is specified together with a principal, password, or keytab Added toString method override to AbstractKerberosUser Updated Hive/HBase components to be compatible with the KerberosProperties.validatePrincipalWithKeytabOrPassword method Fixed null ComponentLog in GetHDFSSequenceFileTest Added package-protected accessor method (getAllowExplicitKeytabEnvironmentVariable) to AbstractHadoopProcessor for determining if the environment variable "NIFI_ALLOW_EXPLICIT_KEYTAB" has been set AbstractHadoopProcessor will now only fail validation when the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable is set to false if a keytab is provided to allow the user to specify a principal and password Added AbstractHadoopProcessorSpec to verify validation of principal/keytab/password/kerberos credential service combinations This closes #4095. * NIFI-7025: Initial commit adding Kerberos Password feature for Hive components Kerberos Password property should not support EL, this includes a change to KerberosProperties which is also used by the HDFS processors (AbstractHadoopProcessor) Added wiring in a KerberosContext to a TestRunner's MockProcessorInitializationContext Removed synchronization blocks around KerberosUser.checkTGTAndRelogin, since that method is already synchronized Updated AbstractHadoopProcessor to have a boolean accessor method to determine if explicit keytab configuration is allowed Removed synchronization block from HiveConnectionPool's getConnection method (in Hive, Hive_1_1, Hive3 modules), since new TGT ticket acquisition is handled by the KerberosUser implementation. If UGI is used to relogin, synchronization is handled internally by UGI. Added Kerberos Principal and Kerberos Password properties to Hive, Hive_1_1, and Hive3 components Hive, Hive_1_1, and Hive3 components now use KerberosUser implementations to authenticate with a KDC Updated handling of the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable in Hive and Hive3 components. An accessor method has been added that uses Boolean.parseBoolean, which returns true if the environment variable is set to true, and false otherwise (including when the environment variable is unset). Addressing PR feedback Addressing PR feedback This closes #4102. * NIFI-7024: Added Kerberos Password support to HBase_1_1_2_ClientService and HBase_2_ClientService This closes #4103. * NIFI-7029 Add kerberos password property to NiFi Kudu components This closes #4097. * NIFI-7019 Add kerberos principal and password properties to NiFi DBPCConnectionPool This closes #4087. * NIFI-7030 Add Kerberos principal and password properties to Solr processors Updating validation logic to be consistent with other password-based kerberos processors, removing getPassword from KerberosPasswordUser as it was only used from testing This closes #4062. * NIFI-5644 Fixed typo in getWrappedQuery method of AbstractDatabaseFetchProcessor class Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4106. * NIFI-7218 Fixed typo in Overview docs. (#4107) * NIFI-7121 Updated comment to state a 'static' salt is used in the constructor. (#4098) Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-4970 - EOF Exception in InvokeHttp when body's response is empty with gzip Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4109. * NIFI-7025: Wrap Hive 3 calls with UGI.doAs Updated PutHive3Streaming to wrap calls to Hive in UGI.doAs methods Fixed misleading logging message after the principal has been authenticated with the KDC When connecting to unsecured Hive 3, a UGI with "simple" auth will be used Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4108 * NIFI-7208: Fixed PutSQL/JdbcCommon handling of timestamps (nanoseconds, e.g.) * NIFI-7227 Fixed typo in Global Access Policy table (#4112) Co-authored-by: spius <57421336+spius@users.noreply.github.com> Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7055 handle empty split evaluations, which contain only , add explict test for " , " updated with counting validator Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4012 * NIFI-7055: Removed unit test that is now covered by ListValidator Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4114. * NIFI-7222 Cleaned up API for FTP/SFTP remote file retrieval and ensure we close remote file resources for SFTP pulls in particular Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4115. * NIFI-7226: Add Connection Factory configuration properties to PublishJMS and ConsumeJMS processors Some JMS client libraries may not work with the existing controller services due to incompatible classloader handling between the 3rd party library and NiFi. Via configuring the Connection Factory on the processor itself, only the processor's and its children's classloaders will be used which eliminates the mentioned incompatibility. This closes #4110. Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7232 if users do not supply a remote path PutSFTP with conflict resolution will fail with an NPE * Fixed unit test failed with NIFI-7232 This closes #4117. Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7191 Conditionally disable docker integration tests Honor Maven properties skipTests and maven.test.skip for the shell script integration tests that verifiy docker images. * Fiz writeBatch to avoid cycle in provenance graph * NIFI-7231: move controller service validation out of synchronized block for enabling This closes #4118. Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7197 - In-place replacement in LookupRecord processor This closes #4088 Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7224 Protecting against possible NPE in ImportFlowVersion command in CLI Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4121. * NIFI-7195 - Catch MongoException to route flow files to failure This closes #4089 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-7210 - added PG path in bulletins for S2S Bulletin RT Added group path to BULLETIN table for QueryNiFiReportingTask Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4100 * NIFI-7179 Documented Download flow option in Process group context menu (#4124) Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7200: Revert "NIFI-6530 - HTTP SiteToSite server returns 201 in case no data is available" This reverts commit f01668e66ad2e45197915769e966a4be27e1592e. Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-4970 - Add a property to deal with empty GZIP HTTP response Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4127. * NIFI-7239: Upgrade the Hive 3 bundle to use Apache Hive 3.1.2 Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4129. * NIFI-7050 ConsumeJMS is not yielded in case of exception This closes #4004. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7245: JMS processors yield when connection factory initialisation failed Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4133. * unit test reproducing the issue * Fixed bug in JsonRowRecordReader when reading a 'raw' record with a schema that indicates that a field should be a Map. Also updated unit test to explicitly define schema, since schema inference will never return a Map but rather a Record * NIFI-7242: When a Parameter is changed, any property referencing that parameter should have its #onPropertyModified method called. Also renamed Accumulo tests to integration tests because they start embedded servers and connect to them, which caused failures in my environment. Also fixed a bug in TestLengthDelimitedJournal because it was resulting in failures when building locally as well. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4134. * NIFI-7119 Implement boundary checking for Argon2 cost parameters (#4111) * NIFI-7119 Implemented parameter boundary enforcement for Argon2SecureHasher constructor. Added unit tests for validating each parameter check. * NIFI-7119 Refactored parameter validations. Added more test sizes to boundary checkers. Changed logger severity to error and added bounds to messages. * NIFI-7119 Refactored Argon2 parameter data types to handle unsigned integer boundary values. Updated unit tests. Co-authored-by: Andy LoPresto <alopresto@apache.org> Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7241: When updating Process Group to match VersionedProcessGroup, remove any connections before recursing into child groups. This ensures that if a Port exists in child group A and is connected to a port in child group B, if the VersionedProcessGroup indicates to remove the port, that connection will be removed before attempting to remove the port. Updating and adding connections must still be done last, after all components have been added. But missing connections can be removed earlier. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4136. * NIFI-7248: Atlas reporting task handles PutHive3Streaming Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4138. * NIFI-7244 Updated all tests which dont run well on windows to either work or be ignored on windows Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4132. * NIFI-6742 Use JUnit TemporaryFolder when creating test databases - Add @Rule for TemporaryFolder - Replace use of previous target/db with TemporaryFolder/db - Remove use of ~/test db (in home directory) - Remove System.out lines Signed-off-by: Marc Parisi <phrocker@apache.org> This closes #4137. * NIFI-7251: Upgrade hadoop-client version to 3.2.1 to avoid the regression bug Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4141. * NIFI-7229 - Upgrade jackson-databind direct dependencies This closes #4113 * NIFI-7249: Force String keys in maps in DataTypeUtils.inferDataType() Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4139. * NIFI-7250 activate user.timezone appropriate to each region NIFI-7250 fix a test which appears brittle at least on windows builds on slow environments NIFI-7250 activated a timezone run for AU Australia/Melbourne which exposed a poor magic number and needless assertion but interesting results worth keeping Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4140. * NIFI-7256: This closes #4142. Fixed thresholds in unit test. Instead of assuming that multiple runs of the processor will occur within 100 milliseconds, allowed the multiple runs to occur within 3 mins of one another. Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7208: Restore default timezone in JdbcCommon * NIFI-7223 - Fixed a minor issue where the OkHttpReplicationClient class loaded blank properties as empty string instead of an expected null value. Added a isNotBlank check. Added unit tests for replication client and HTTPNotificationService. NIFI-7223 - Renamed some variables and methods. NIFI-7223 - Removed unused dependency. Corrected security properties in administration-guide. * NIFI-7223 [WIP] Resolved compilation issues in unit test on OpenJDK 11 by removing Sun security class references. Added OkHttpReplicationClient#isTLSConfigured() method. Added unit test. NIFI-7223 Fixed remaining unit tests for TLS regression. Renamed tests for clarity. * NIFI-7223 - Added another test for when keyPasswd is not present. * NIFI-7223 Resolved merge conflicts from additional test case for null key password. This closes #4145. Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7258 - fix overflow in PutAzureEventHub when not configured correctly Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4146. * NIFI-7267 - Upgrade spring-data-redis in Redis bundle (#4150) Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… (#4151) * NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.favre.lib.bcrypt library. Updated LICENSE and NOTICE files to reflect changes. Updated unit tests. Co-authored-by: Andy LoPresto <alopresto@apache.org> * NIFI-7268 Fixed typo in Javadoc. Co-authored-by: Andy LoPresto <alopresto@apache.org> * NIFI-6293 Add support to Mongo Extended JSON v2 Add org.json lib Replace evil json Replace evil json for alternative Include testExtendedJsonSupport Style adjustment Remove unecessary new JSON parser Fix query in testExtendedJsonSupport Parse with Jackson and BSON Back to default MONGO_URI This closes #4068 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-7264 Make jsonPath Expression Logging More Reasonable add special handling of PathNotFoundExceptions to log to debug fix spelling error wrap debug log in guard per review This closes #4148 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-7187 adding missing version strings from accumulo bundle pom - Removed Cat X JSON.org dep inclusion which seems to not be necessary - Updated a ton of easier/safer looking deps - Updated tika due to CVE This closes #4086 Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7221 Initial work * NIFI-7221 Support v2 and v3 protocol version for Hortonworks Schema Registry - Update nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/serialization/SchemaRegistryRecordSetWriter.java - Addressing review feedback This closes #4120. * NIFI-7274 add time for test conditions to be met * NIFI-7271 Make command timeout configurable for ShellUserGroupProvider - Changing ShellRunner to use a separate thread for reading the output of the process - Removing unused member variable - Addressing review feedback This closes #4154. * NIFI-7257 Added HadoopDBCPConnectionPool - Updated InstanceClassLoader to resolve files that are in the instance urls or additional urls - Updated nifi-mock to support KerberosContext and removeProperty for ControllerServices - Added unit test for HadoopDBCPConnectionPool - Addressing review feedback This closes #4149. * NIFI-7269 - Upgrade solrj version to 7 in nifi-solr-processors Remove unused imports Use the latest solrj version(8.4.1) Setup default schemaFactory for tests The default schemaFactory ManagedIndexSchemaFactory creates additional files in test's resources directory. Change it to ClassicIndexSchemaFactory for classic behavior. This closes #4152. Signed-off-by: Bryan Bende <bbende@apache.org> * NIFI-7238 Improve Caching for Github CI and relax core usage to not max out cores all moving toward more stable builds. Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7278 Adding support for SCRAM-SHA-512 to Kafka 2.0 processors * NIFI-5925: Added controller services to set of components that are searched NIFI-5925: cleanup, add negative test NIFI-5925: fixed checkstyle This closes #4105 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-7238 Continue to improve Github Actions CI stability * NIFI-7238 remove no longer needed delete command * NIFI-7238 printing maven version info in same build command * NIFI-7281 This closes #4159. Use BufferedInputStream in StandardSocketChannelRecordReader in order to support mark/reset Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7279 This closes #4160. Protect against NPE in RedisDistributedMapCacheClientService when value is null Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7153 Adds ContentLengthFilter to enforce configurable maximum length on incoming HTTP requests. Adds DoSFilter to enforce configurable maximum on incoming HTTP requests per second. Redirected log messages for ContentLengthFilter to nifi-app.log in logback.xml. This closes #4125. Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7286 ListenTCPRecord cleanup changed from @OnStopped to @OnUnscheduled * NIFI-7287: Move services-api dependency from Prometheus reporting task to its NAR Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4162 * NIFI-7290: omit transitive that is causing the build to fail and not needed for the test scope * NIFI-7291 updated enforcement of dependency rules and build command Reviewed by markap14 This closes #4166. Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7294 Address deprecation issues in solrj and httpclient Some calls to deprecated methods in httpclient were resulting in UnsupportedOperationException. Use the new API calls in both httpclient and solrj. Add an integration test to include test coverage for org.apache.nifi.processors.solr.SolrUtils.createClient This closes #4171. * NIFI-7297 add available() Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4172. * NIFI-7103 Adding PutDataLakeStorage Processor to provide native support for Azure Data Lake Storage Gen 2 Storage. added data-lake dependency NIFI-7103 fixed indentation Update to add IllegalArgumentException Fixed indentation and logging nifi-7103 review changes nifi-7103 root directory and exception change This closes #4126. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7293 Add in-memory janusgraph implementation of GraphClientService to help with live testing. Added new in memory janus graph client for testing. Added integration test to ExecuteGraphQuery. NIFI-7293 Added missing getter. Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4168 * add validator for lists that ensure the element validator is called for empty entries Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4116 * NIFI-7311 adding additional cleanup and output to github actions yml and not running nifi-system-test module Self merging wo review as part of github ci actions stability. Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7188 Extending UI search with filters and refactoring existing solution This closes #4123. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping (#4156) * NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping * NIFI-7273: Changed method name, fix handling when analytics not enabled * NIFI-7273: Removed attachment header from Prometheus metrics endpoint * NIFI-7273: Removed unused variable * NIFI-7317 - make .java files non-executable Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4181. * NIFI-7126 Increased test iterations to 10,000 in Argon2SecureHasherTe… (#4187) * NIFI-7126 Increased test iterations to 10,000 in Argon2SecureHasherTest#testDefaultCostParamsShouldBeSufficient to avoid JVM warmup issues. Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7326 updated URL to find splunk artifacts (#4188) Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7314 HandleHttpRequest stops Jetty in OnUnscheduled instead of OnStopped. Also reject pending request and clean their queue when shutting down. NIFI-7314 In HandleHttpRequest returning 503 when rejecting pending requests before shutdown. NIFI-7314 In HandleHttpRequest add logs and better response message during cleanup. This closes #4191. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7259 Adding DeleteDataLakeStorage Processor to provide native support for Azure Data lake Gen 2 Storage. Updated to remove unused variables NIFI-7259 import and property description changes This closes #4189. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7345: Fixed Hive database and table names case insensitivity in Atlas reporting task Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4198. * NIFI-6849: Reworked how nodes inherit cluster information when joining a cluster. Now, if there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's flow is inherited. - Refactored Flow Synchronization to make code cleaner - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty. - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start - Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state. - Added additional log statements to aid in debugging NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior NIFI-6849: Addressed review feedback NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup This closes #3891 * NIFI-7087: Use FlowManager.findAllConnections() when available Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4026 * NIFI-7341 Updated certificate commands and source code formatting in Toolkit Guide. (#4196) * NIFI-7339: Fixed bug that caused Write Ahead Provenance Repository not to rollover event files after specified time. Code cleanup. Updated some default properties. * NIFI-7346: Ensure that the Provenance Repository doesn't delete the Active Event File * NIFI-7319 Add walkthrough document (#4193) * NIFI-7319 Added first draft of walkthroughs doc. * NIFI-7319 Added instructions and screenshots for securing standalone NiFi instance. * NIFI-7319 Added instructions and screenshots for instructing OS & browser to trust self-signed certificate. * NIFI-7319 Added instructions and screenshots for securing NiFi with externally-provided certificates. * NIFI-7319 Added instructions and screenshots for building NiFi from source. * NIFI-7319 [WIP] Converting secure cluster instructions to match format. Fixed instructions regarding embedded ZooKeeper configuration. * NIFI-7319 Completed secure cluster walkthrough. * NIFI-7319 Added walkthroughs to documentation navigation list. * NIFI-7319 Incorporated PR feedback on broken links. * NIFI-7319 Removed line number helpers from update sections. * NIFI-7319 Incorporated final PR review items. Co-authored-by: Sandra Pius <spiusapache@gmail.com> * NIFI-7354: Allow analytics properties to be set via environment variables in docker scripts This closes #4203. Signed-off-by: Aldrin Piri <aldrin@apache.org> * NIFI-7347: Fixed NullPointerException that can happen if a bin is merged due to timeout and has no records Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4210. * NIFI-6977 - Change the reporting behavior of Azure Reporting task to report report the time when metrics are generated Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4211. * [NIFI-7358] - Fix: Sorting on 'Estimated Time to Back Pressure' in the Connection summary table does not work properly - fix style issues - review feedback This closes #4208 * NIFI-7292 Preventing file listing from fail because of insufficient privileges Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4195. * Added Jira and security reporting links to README.md * NIFI-7359 Fix parent id on process metrics for Prometheus Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4209 * NIFI-7348 Wait - Removes WAIT_START_TIMESTAMP after expiration This closes #4201. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org> * NIFI-7334 Adding FetchDataLakeStorage Processor to provide native support for Azure Data lake Gen 2 Storage. NIFI-7334 Update to FetchDataLakeStorage Processor This closes #4212. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7366 - ConsumeEWS Processor parse EML https://issues.apache.org/jira/browse/NIFI-7366 This commit allows to retrieve ItemAttachement (such as EML) file when pulling mail. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4215. * NIFI-7318 - Allow 'docker stop' to gracefully shutdown As it is issuing 'docker stop' will immediatly exit the container. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4182. * NIFI-7300 Allowing narrow numeric types to fit againt schema check with wider type; Allowing doubles with value within float precision to be considered as valid floats (NIFI-7302) * NIFI-7375: This closes #4218. Fixed a bug that caused Provenance Events not to show up in specific situations when clicking View Provenance for a Processor. - Added System-level tests for Provenance repository to reproduce behavior. - Added a Provenance Client to the CLI, which is necessary for System-level tests. - Added small additional configuration for Provenance repository to simplify development of system tests - Minor improvements to system tests (such as ability to destroy environment between tests) needed for Provenance repository based system tests Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7280 ReportLineageToAtlas recognizes 'atlas.metadata.namespace' from Atlas config file. Still recognizes 'atlas.cluster.name' as well, but takes lower precedence than the new property. Also Atlas URL can be provided via the 'atlas.rest.address' property in the atlas-application.properties. NIFI-7280 In ReportLineageToAtlas improved documentation and adjusted property ordering for better user experience. Minor refactor. NIFI-7280 In ReportLineageToAtlas amended documentation. Minor refactor. NIFI-7280 In ReportLineageToAtlas amended more documentation. More minor refactor. NIFI-7280 - In Atlas reporting: complete clusterName -> namespace overhaul where appropriate. This closes #4213. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7378: Ensure label values are not null in Prometheus metrics (#4219) * Do not update status for stopping a deleted node * NIFI-7389 Makes Missable heartbeat counts configurable This closes #4236. Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7377 Cleaned up nifi-stateless logs. Refactored masking logic to CipherUtility and indicated masking with label and Base64 output. Added JSON masking logic to nifi-stateless module. Added argument masking functionality to Program. Moved groovy unit tests to proper Maven directory structure. Modified plain argument output to use filtering/masking methods in provided utility. Refactored utility methods. Updated unit tests. This closes #4222. Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com> Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7298: PutAzureDataLakeStorage tests. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4227. * NIFI-7379: Support multiple instances of Prometheus registries/metrics (#4229) * NIFI-7379: Support multiple instances of Prometheus registries/metrics * NIFI-7379: Refactored Prometheus objects to support multiple instances * NIFI-7394: Add support for sending Multipart/FORM data to InvokeHTTP. By using dynamic properties with a prefix naming scheme, allow definition of the parts, including the name to give the Flowfile content part, and optionally it's file name. After review: - change so that we can send just the form content or just form data without the flowfile - change the content name and content file name from dynamic properties to properties - change the dynamic name to be an invalid http header "post:form:xxxx" - add validation and more tests This closes #4234. Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7408 - added percent used metrics for connections NIFI-7408 - return double value for utilization Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4240 * NIFI-7170: - Adding a flag to nifi.properties to disable anonymous authentication. NIFI-7170: - Fixing checkstyle issues. NIFI-7170: - Adding missing license header. NIFI-7170: - Initial PR feedback. NIFI-7170: - Fixing broken integration tests. - Creating new integration tests for verifying allowing and preventing anonymous access. NIFI-7170: - Ensuring the new anonymous authentication property is considered for proxied requests. NIFI-7170 - Fixed comment. Signed-off-by: Nathan Gough <thenatog@gmail.com> This closes #4099. * Added note about unique initial user identity names to walkthrough doc. * NIFI-6149: Azure EventHub Managed identities support patch review changes additional review changes NIFI-6149: typo fixes This closes #4226. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7416: Update travis-ci to GitHub Actions in the PR template Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4247. * NIFI-7415: Add .asf.yaml to configure GitHub integrations Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4246. * NIFI-7404: Fixed invalid script processors upon thread termination Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4238. * NIFI-7412: Fixed provenance event types in Azure Fetch/Delete processors This closes #4245. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7414: Escape user-defined values that contain invalid XML characters before writing flow.xml.gz NIFI-7414: Updated StandardFlowSerializerTest to include testing for variable names and values being filtered This closes #4244 * NIFI-7425 Log Message for ReplaceText Over Buffer Size Adds a log message when ReplaceText sends a flowfile to the failure relationship because it is larger than the max buffer size. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4255. * NIFI-7420 remove the http.param attribute. It contained both the query parameters already captured in http.query.param _and_ the multipart form data names and values, which are captured in the part data, and could be very very large This closes #4251. Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7428: Switch hive.version property to set Hive 3 version This closes #4259 * NIFI-7398 Upgraded jackson-databind dependency version to 2.9.10.4 at root pom.xml. Upgraded tika-parsers dep in nifi-media-processors. Upgraded jackson-databind dep in nifi-graph-processors. Upgraded jackson-databind dep in nifi-elasticsearch-client-service-api. Upgraded jackson-databind dep in in nifi-easyrules-service. Upgraded calcite-core dep in nifi-sql-reporting-tasks. Signed-off-by: Nathan Gough <thenatog@gmail.com> This closes #4252. * NIFI-7390 Covering Avro type conversion in case of map withing Record Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4256 * NIFI-7423 Upgraded jquery dependency version. NIFI-7423 Upgraded jquery dependency version to latest 3.5.1. This closes #4258 * NIFI-6913: PutAzureBlobStorage processor will create container if not exists Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4237. * NIFI-7413: Documented REMOTE_INVOCATION provenance event type in user/dev guides Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4267 * NIFI-7321 - Allow NiFi admins to configure whether Jetty will send the Jetty server version in responses. Fixed a checkstyle error. Added property to nifi.properties. Changed property to a variable that is set with the pom.xml. Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context. Fixed typo error. This closes #4192. Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7367: Add tests for FetchAzureDataLakeStorage NIFI-7367: Negative test cases for expression language in FetchAzureDataLakeStorage FetchAzureDataLakeStorage throws exception when filesystem or filename is blank. Fixed logged error messages in all 3 of the Delete, Fetch and Put ADLS processors. testFetchDirectory test case marked as ignored. This closes #4257. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7448: Fix quoting of DDL table name in PutORC Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4269. * NIFI-7460: Avoid NPE when a VersionedProcessor has a null value for autoTerminatedRelationships. Added additional logging and improved error handling around syncing with invalid flows * NIFI-7437 - created separate thread for preloading predictions, refactors for performance NIFI-7437 - reduced scheduler to 15 seconds, change cache to expire after no access vs expire after write Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4274 * NIFI-6497: Allow FreeFormTextRecordSetWriter to access FlowFile Attributes This closes #4275. Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7380 - fix for controller service validation in NiFi Stateless This closes #4264. Signed-off-by: Matthieu Cauffiez <matthieu.cauffiez@bell.ca> Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7446: FetchAzureDataLakeStorage processor now throws exception when the specified path points to a directory A newer version (12.1.1) of azure-storage-file-datalake is imported. This closes #4273. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7331 Fixed grammatical errors in log output. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4283. * NIFI-7409: Azure managed identity support to Azure Datalake processors NIFI-7409: review changes NIFI-7409: ordering import statements NIFI-7409: changed validateCredentialProperties logic This closes #4249. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7336: Add tests for DeleteAzureDataLakeStorage DeleteAzureDataLakeStorage now throws exception if fileSystem or fileName is empty string NIFI-7336: Add tests for DeleteAzureDataLakeStorage - typos fixed NIFI-7336: Add tests for DeleteAzureDataLakeStorage - fixed a test case This closes #4272. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-6911 Removed default Blob value for PutAzureBlobStorage This closes #3906 Signed-off-by: Joey Frazee <jfrazee@apache.org> * NIFI-7407 Replaced SSLContextFactory references to "TLS" with "TLSv1.2" (in shared constant). Changed JettyServer default SSL initialization and updated unit test. Removed SecurityStoreTypes (unused). Added StringUtils inverted blank and empty checks. Added TlsConfiguration container object. Enhanced KeystoreType enum. Added clean #createSSLContext() method to serve as base method for special cases/other method signatures. Added utility methods in KeyStoreUtils. Added generic TlsException for callers that cannot resolve TLS-specific exceptions. Added utility methods for component object debugging. Enforced TLS protocol version on cluster comms socket creation. Added utility method for SSL server socket creation. Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version). Removed duplicate SSLContextFactory. Switched duplicate SslContextFactory to wrap shared SSLContextFactory. Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test). Added further validation & boundary checking in uses of TlsConfiguration. Provided SSLSocketFactory accessor in SslContextFactory. Refactored OkHttpReplicationClient tuple method. Refactored OcspCertificateValidator TLS logic. Added utility method to apply TLS configs to OkHttpClientBuilder. Removed references to duplicate SslContextFactory. Removed unnecessary SslContextFactory. Moved OkHttpClientUtils to nifi-web-util module. Updated module dependencies. Removed now empty nifi-security module. Enforced TLS protocol selection on LB server socket. Enforced TLS protocol selection on S2S server socket. Applied specified TLS protocol versions to S2S socket creation. Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory. Replaced references to creation methods throughout codebase. Replaced references to unnecessary NiFiProperties file reads throughout tests. Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth. Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners. Cleaned up legacy code. Added external timing check to timing test assertion. Made RestrictedSSLContextService TLS protocol versions allowable values explicit. Enabled TLSv1.3 on Java 11. Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService. Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task. This closes #4263. Signed-off-by: Nathan Gough <thenatog@gmail.com> Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7471 fix bug with property validation * NIFI-6571 Check token length on TLS toolkit server startup This closes #3659. Signed-off-by: Joey Frazee <jfrazee@apache.org> * Fixed a couple of typos in the RecordPath guide * NIFI-7463 Create empty relationship for RunMongoAggregation Fix default autoterminate and condition to redirect to REL_EMPTY Change from new relationship to write an empty FlowFile to RESULT Fix MONGO_URI This closes #4281 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-7462: This adds a way to convert or cast a choice object into a valid type for use with calcite query functions NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types NIFI-7462: Fixed checkstyle violation, removed documentation around the CAST functions that were no longer needed Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4282 * NIFI-7482 Changed InvokeHTTP to be extensible. Added unit test. This closes #4291. Signed-off-by: Arpad Boda <aboda@apache.org> * Updated pull request template to separate JDK 8 and 11 questions * NIFI-6255 NIFI-6287: Hash function for expression language and record path. NIFI-6255 NIFI-6287: Rebased to match the new expression language interface NIFI-6255 NIFI-6287: Fix wildcard imports and unused imports NIFI-6255 NIFI-6287: Move to the common codec DigetUtils Update commons-codec This closes #3624 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-6672 PlusEvaluator throws an Arithmetic Exception in case of Long overflow. TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY The behaviour change is reverted until further investigations. The overflow behaviour is still enforced by unit tests and documented in the expression language doc NIFI-6672 Removed test code. This closes #3738 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-6673 MultiplyEvaluator throws an Arithmetic Exception in case of Long overflow. TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY or NEGATIVE_INFINITY The behaviour change is reverted until further investigations. The overflow behaviour is still enforced by unit tests and documented in the expression language doc This closes #3739 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-6674 MinusEvaluator throws an Arithmetic Exception in case of Long overflow. TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY MinusEvaluator throws an Arithmetic Exception in case of Long overflow. TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY The behaviour change is reverted until further investigations. The overflow behaviour is still enforced by unit tests and documented in the expression language doc fixed mispositioned # in doc This closes #3740 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-7211 Added @Ignore with warning message to a test that randomly fails due to timing issues. This closes #4296 * NIFI-6785 Support Deflate Compression NIFI-6785 Remove unused imports This closes #3822 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-7453 In PutKudu creating a new Kudu client when refreshing TGT NIFI-7453 Creating a new Kudu client when refreshing TGT in KerberosPasswordUser as well. (Applied to KerberosKeytabUser only before.) NIFI-7453 Safely closing old Kudu client before creating a new one. NIFI-7453 Visibility adjustment. This closes #4276. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7485 Updated commons-configuration2. NIFI-7485 Found more instances that needed updating. This closes #4295 * NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor Made warning and error messages more informative. Refactored flowFile assertion in the tests. This closes #4287. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7487 - Added batch support and displayName to ModifyBytes processor Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4302. * NIFI-7483 - Remove description about 'Rolling strategy' in TailFile's docs Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4293. * NIFI-7484:fix ListFTP and FetchFTP docs. Change 'SFTP' to 'FTP' in description Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4294. * NIFI-7422: Support aws_s3_pseudo_dir in Atlas reporting task Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4292. * NIFI-6701 - Fix for PublishGCPPubSub * NIFI-7430 - LookupRecord change coordinate key for in-place replacement * NIFI-6666 Add Useragent Header to InvokeHTTP requests This closes #3734 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * Updated KEYS with new key after previous one expired * NIFI-7403:Add a function that adjust if the result is failed before we call the onFailed or onCompleted function. If the result is failed, return true and do sth NIFI-7403:Add an extension point to adjust the result, if the result is failed then process onFailed function NIFI-7403:Implement the AdjustFailed Function, if PutSQL set the SUPPORT_TRANSACTIONS true, then check whether the result contains REL_RETRY or REL_FAILURE.If it contains that, reroute the result and return true. NIFI-7403: fix reroute logic in AdjustFailed function NIFI-7403:Add and modify some unit test for PutSQL's SUPPORT_TRANSACTIONS property NIFI-7403:Update for PR recheck NIFI-7403:Add documentation on the Support Fragmented Transactions property to indicate the transactions rollback behavior NIFI-7403: Fix Checkstyle issue Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4266 * NIFI-7477 Optionally adding validation details as a new attribute of the flowfile NIFI-7477 Improving description and unit test now verifies attribute content NIFI-7477: Fixed checkstyle errors Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4301 * NIFI-7447: When returning an object from a Controller Service, if that object is defined as an interface, proxy that interface. This way, any method call into the object will also change the classloader to the appropriate classloader. * NIFI-7312: Enable search in variable registry of root process group This closes #4303. Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7369 Adding decimal support for record handling in order to avoid missing precision when reading in records Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7369: Consider DECIMAL type as a numeric type when using a CHOICE type in QueryRecord This closes #4223. * NIFI-7299 Add basic OAuth2 token provider service that can fetch access tokens when supplied with appropriate credentials. Added skeleton of oauth2 provider. Added copy of our code. Refactored a few things. Updated apis to better match flow descriptions. Updated poms and other artifacts. Updated copyright notice. Updated LICENSE. This closes #4173 Signed-off-by: Jeremy Dyer <jeremydyer@apache.org> * NIFI-7476: Implemented FlowFileGating / FlowFileConcurrency at the ProcessGroup level Added FlowFileOutboundPolicy to ProcessGroups and updated LocalPort to make use of it Persisted FlowFile Concurrency and FlowFile Output Policy to flow.xml.gz and included in flow fingerprint Added configuration for FlowFile concurrency and outbound policy to UI for configuration of Process Groups Added system tests. Fixed a couple of bugs that were found Fixed a couple of typos in the RecordPath guide Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4306. * NIFI-7508: Reset classloader after running TestStandardControllerServiceInvocationHandler and fix checkstyle violation on NiFiSystemIT * NIFI-7467 Refactored S2S peer selection logic. Removed list structure for peer selection as it was unnecessary and often wasteful (most clusters are 3 - 7 nodes, the list was always 128 elements). Changed integer percentages to double to allow for better normalization. Removed 80% cap on remote peers as it was due to legacy requirements. Added unit tests for non-deterministic distribution calculations. Added unit tests for edge cases due to rounding errors, single valid remotes, unbalanced clusters, and peer queue consecutive selection tracking. Migrated all legacy PeerSelector unit tests to new API. Removed unused System time manipulation as tests no longer need it. Added class-level Javadoc to PeerSelector. Removed S2S details request replication, as the responses were not being merged, which led to incorrect ports being returned and breaking S2S peer retrieval. Fixed copy/paste error where input ports were being listed as output ports during remote flow refresh. Fixed comments and added unbalanced cluster test scenarios. Removed unnecessary marker interface. Removed commented code. Changed weighting & penalization behavior. Changed dependency scope to test. This closes #4289. Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7393: Add max idle time and idle connections to InvokeHTTP This closes #4233. Signed-off-by: Joey Frazee <jfrazee@apache.org> * NIFI-7507: Added section to User Guide on configuring a Process Group NIFI-7507: Fixed Flowfile Expiration header in doc Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4318 * NIFI-7511 In ControllerServiceProxyWrapper extended documentation. Minor refactor in StandardControllerServiceInvocationHandler. Also removed an unused import from NiFiSystemIT. This closes #4317. Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7385 Provided reverse-indexed TokenCache implementation. Cleaned up code style. Unit test was failing on Windows 1.8 GitHub Actions build but no other environment. Increased artificial delay to avoid timing issues. Co-authored-by: Andy LoPresto <alopresto@apache.org> This closes #4271. Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7514: - Ensuring the group id is always set in the properties table when loading properties. - Using a common approach to getting parameters in nfControllerService. - Code clean up. - Addressing review feedback. - Ensuring the service dialog is closed when navigating to the parameter context dialog. This closes #4322 * NIFI-7490 - Add optional raw field to Syslog readers review Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4299 * NIFI-7313:fix bug on 'Quote Table Identifiers' NIFI-7313:add test by wanghongqi NIFI-7313:edit test Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4185 * NIFI-7434: Add endpoint suffix to Azure storage processors This closes #4265. Signed-off-by: Joey Frazee <jfrazee@apache.org> * NIFI-7442 Add CLI commands to the registry in order to support automatic registry setup * NIFI-7442 Added missing use cases (list users and user groups), made update-access-policy use case more in line with the NiFi side. Added some tests. Additional refactor, documentation revision. This closes #4329. * NIFI-7527 AbstractKuduProcessorrefresh TGT deadlock fix: Redesigned locking. NIFI-7527 Fixed StackOverFlowError due to pacing issue (recursive login before loggedIn flag is set). NIFI-7527 Refactor: removed redundant kudu client creation. This closes #4330. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7035 The first curator connection issue is logged as ERROR until reconnect * NIFI-7539: When capturing diagnostics information, capture a thread dump once and then provide this information to ProcessorNode when capturing active threads. Previously, each processor captured a thread dump itself. When this is done thousands of times it can result in a very long delay. * NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment (#4341) * NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment This also fixes NIFI-4760. * NIFI-7540: Remove duplicate mail.smtp.starttls.enable from TestListenSMTP Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7537 - Small fix to make the LDAP connection timeout property a String rather than a Long. Fixes cast error in LDAP libs. * NIFI-6094 - Added the X-Content-Type-Options header to all web responses. (#4307) NIFI-6094 - Added the mime/content type for ttf files. * NIFI-7551 Add support for VARCHAR to Kudu NAR bundle - update Kudu dependencies to Kudu 1.12.0 - add VARCHAR to Kudu Lookup Service and Processor - add tests for VARCHAR columns Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4347. * NIFI-7536: Fix to improve performance when determining the run status of processors when needing to wait for all processors to stop for updating parameter context, etc. * NIFI-7509: Added optional Record Writer property to all List* Processors Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4315. * NIFI-7566: Avoid using Thread.sleep() to wait for Site-to-Site connection to be handled. Instead, use TimeUnit.timedWait and use Object.notifyAll when setting the beingServiced flag. This significantly reduces latency and improves throughput for small-batch site-to-site communications This closes #4353. Signed-off-by: Andy LoPresto <alopresto@apache.org> * NIFI-7529 Removed OS and Java information from InvokeHttp's UserAgent field so that it's removed regardless of whether or not this field is kept. This closes #4332 * NIFI-7516: Catch and log SingularMatrixExceptions in OrdinaryLeastSquares model (#4323) * NIFI-7501 Update nf-context-menu.js for an intuitive road to parameters When rightclicking a process group the variables are shown, but parameters are not. This makes sense as they have a prerequisite, in the form of a parameter context. This change gives a more consistent experience for finding the functionality regarding parameters by ensuring the contextmenu shows the possibility to configure a parameter context. Once the paramater context has been created for a process group, the parameters text shows, so this is no longer visible. People would then need to click configure to change the context, just as they would be required to do now. Added generateflowfile load tag and description Added GenerateFlowFile load tag to be consistent with DuplicateFlowFile and updated description to refer to DuplicateFlowFile. Revert "Update nf-context-menu.js for an intuitive road to parameters" This reverts commit 3c44b1661f09fb6ae11d2f088550f81fb7a4b393. This closes #4333 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-7558 Fixed CatchAllFilter init logic by calling super.init(). Renamed legacy terms. Updated documentation. This closes #4351. Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-7542 Override jackson-databind version. NIFI-7542 Override additional jackson-databind versions. NIFI-7542 Upgrade jackson-databind dependency to 2.9.10.5 in the root pom.xml. This closes #4343 Signed-off-by: Mike Thomsen <mthomsen@apache.org> * NIFI-7523: Use SSL Context Service for Atlas HTTPS connection in Atla… (#4348) * NIFI-7523: Use SSL Context Service for Atlas HTTPS connection in Atlas reporting task Also fixing ControllerServiceDisabledException-s when validating the Kerberos config * NIFI-7523: Fixed test failure on Windows * NIFI-7523: Added license headers. * NIFI-7523: Fixed another test failure on Windows * NIFI-7523: Review changes * NIFI-7576 ListenHTTP: Honor primary node only ListenHTTP processor now binds port and creates a HTTP connection only if one of the following conditions apply: - Primary node execution is 'false' - Primary node execution is 'true' and node is elected as primary node. Changes: - Connection is established in 'onTrigger' annotated method instead of 'onSchedule'. (This is similar to how handleHTTPRequest processor handles connections.) - 'onPrimaryNodeStateChange' annotated method is introduced to tear down server on reelection of primary node This closes #4356. Signed-off-by: Mark Payne <markap14@hotmail.com> * NIFI-6163 Reporting task cannot be set to running when in INVALID state Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4334. * NIFI-7577 Upgrade angular version. * NIFI-7577 Update jquery usages. This closes #4357 Signed-off-by: Scott Aslan <scottyaslan@gmail.com> * NIFI-7586 In CassandraSesionProvider added properties to set socket-level read timeout and connect timeout. In QueryCassandra when writing flowfile to the sesion it's done on the raw OutputStream. Wrapped it in a BufferedOutputStream for better performance. This closes #4368. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7578 nifi-toolkit CLI Process Group Create command - Remove unused imports - Fix checkstyle errors This closes #4358. * NIFI-7587 This closes #4372. Increased tolerance for non-deterministic unit test. Signed-off-by: Joe Witt <joewitt@apache.org> * NIFI-7590 In 'CassandraSessionProvider.onDisabled' setting Cassandra-related references properly to null after closing them so that they can be renewed in 'onEnabled' (which creates them only if set to 'null', leaving them closed otherwise). NIFI-7590 Removed 'CassandraSessionProvider.onStopped'. This closes #4373. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7513 Added custom DNS resolution steps to walkthrough (#4359) * NIFI-7563: Optimize the usage of JMS sessions and message producers The introduced changes prevent creating unnecesary sessions and producers in some scenarios. This closes #4378. Signed-off-by: Joey Frazee <jfrazee@apache.org> * NIFI-7594 In HandleHttpRequest deleting multipart file resources after processing. This closes #4379. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org> * NIFI-7332 Added method to log available claim names from the ID provider response when the OIDC Identifying User claim is not found. Revised log message to print available claims. Added new StandardOidcIdentityProviderGroovyTest file. Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test. Added comments in getAvailableClaims() method. Fixed typos in NiFi Docs Admin Guide. Added license to Groovy test. Fixed a checkstyle error. Refactor exchangeAuthorizationCode method. Added unit tests. Verified all unit tests added so far are passing. Refactored code. Added unit tests. Refactored OIDC provider to decouple constructor & network-dependent initialization. Added unit tests. Added unit tests. Refactored OIDC provider to separately authorize the client. Added unit tests. Added unit tests. NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve the NiFi JWT. Signed-off-by: Natha…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…ssor supporting kerberos passwords
AbstractHadoopProcessor will always authenticate the principal with a KerberosUser implementation and a UGI will be acquired from the Subject associated with the KerberosUser implementation
AbstractHadoopProcessor's getUserGroupInformation method will now attempt to check the TGT and relogin if a KerberosUser impelmentation is available, otherwise it will return the UGI referenced in the HdfsResource instance
Updated AbstractHadoopProcessor's customValidate method to consider the provided password and updated validation failure explanations when a KerberosCredentialsService is specified together with a principal, password, or keytab
Added toString method override to AbstractKerberosUser
Updated Hive/HBase components to be compatible with the KerberosProperties.validatePrincipalWithKeytabOrPassword method
Fixed null ComponentLog in GetHDFSSequenceFileTest
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables X functionality; fixes bug NIFI-YYYY.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically
master)?Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not
squashor use--forcewhen pushing to allow for clean monitoring of changes.For code changes:
mvn -Pcontrib-check clean installat the rootnififolder?LICENSEfile, including the mainLICENSEfile undernifi-assembly?NOTICEfile, including the mainNOTICEfile found undernifi-assembly?.displayNamein addition to .name (programmatic access) for each of the new properties?For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.