Master #1

…r:8.29 dependency This closes #4050. Signed-off-by: Andy LoPresto <alopresto@apache.org>

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4064.

…able. Added Github Actions based CI build for pull requests and pushes. Removed unit tests which make unreadable or unaccesible dirs. Includes windows build as well as ubuntu. we can do localization based builds later as this is all proven stable. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4058.

* NIFI-7175 Fixed core attributes formatting in Developer Guide. * NIFI-7175 Made core attribute names more consistent.

…istry

…omponents Signed-off-by: Joe Witt <joewitt@apache.org>

…neEventIndex Signed-off-by: Joe Witt <joewitt@apache.org>

…nit test

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4067.

…vironments like our Github CI builds Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4075.

… content Signed-off-by: Joe Witt <joewitt@apache.org>

…owFile Signed-off-by: Joe Witt <joewitt@apache.org>

Signed-off-by: Joe Witt <joewitt@apache.org>

NIFI-7007: Add additional unit tests that hit non-happy path NIFI-7007: Use AllowableValue instead of string NIFI-7007: Add the use of attributes for the update method, statement type and batch statement type NIFI-7007: Add additional tests, mainly for the use of attributes NIFI-7007: add some ReadsAttribute properties to the PutCassandraRecord processor NIFI-7007: additional update keys validation logic NIFI-7007: fix imports NIFI-7007: Convert fieldValue to long in separate method NIFI-7007: Add new style of tests checking actual CQL output NIFI-7007: add license to new test file NIFI-7007: add customValidate to check for certain incompatible property combinations NIFI-7007: remove check on updateMethod and replace Set.of with java 8 compatible replacmenet NIFI-7007: Add test for failure with empty update method via attributes NIFI-7007: remove unused variable NIFI-7007: Fix customValidate that incorrectly invalidated a valid config Fix Checkstyle Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #3977

Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4070

NIFI-6363 Additional fixes. NIFI-6363 Fix Hadoop compile problem. Add GCP IT instructions. NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase. NIFI-6363 - Added some docs and experimental tag to the relevant classes. Signed-off-by: Nathan Gough <thenatog@gmail.com> This closes #4080.

…viders." This reverts commit 479fcfd. It does not build properly as shown in Github Actions.

…anced easy rules to support provider interface and refactored to extract rules engine implementation NIFI-7163 - updated documentation and comments NIFI-7163 - fix checkstyle issues Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4081

Signed-off-by: Joe Witt <joewitt@apache.org>

…ess Group port value - (#4085) JsonContentConversionExceptionMapper, JsonMappingExceptionMapper, JsonParseExceptionMapper. Registered the custom ExceptionMappers. Added unit tests to throw Exception for string port value and sanitize script input. Handled null or empty JsonMappingException reference path. Added the Apache license to Groovy Test. Signed-off-by: Andy LoPresto <alopresto@apache.org>

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4076.

…ub Actions builds to include OSX and distribute localization Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4091.

…riter Handle scenario where message might be null. Update to test case that was failing because adding attributes modified a flow file even if you don't change the contents. Fixed Style Issues and Updated WritesAttributes. Added Test Case for Error Message Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4052

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4093.

…nd MQTTProduce processors. Generates a random ID if not set. Also add group ID field to ConsumeMQTT processor. Allows consumer to join consumer group at $share/<group_id>/<topic_filter> add expression language support for the MQTT client ID Setting client id in publish test fails because it is not a flowfile attribute. Remove client id and autogenerate it when testing. Since the evaluation is done in onScheduled, there is no flow file available and we're not using the attributes to make the expression language evaluation. You can change the scope to use the Variable Registry. Co-Authored-By: Pierre Villard <pierre.villard.fr@gmail.com> Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #3879.

This closes #4031

…ssor supporting kerberos passwords AbstractHadoopProcessor will always authenticate the principal with a KerberosUser implementation and a UGI will be acquired from the Subject associated with the KerberosUser implementation AbstractHadoopProcessor's getUserGroupInformation method will now attempt to check the TGT and relogin if a KerberosUser impelmentation is available, otherwise it will return the UGI referenced in the HdfsResource instance Updated AbstractHadoopProcessor's customValidate method to consider the provided password and updated validation failure explanations when a KerberosCredentialsService is specified together with a principal, password, or keytab Added toString method override to AbstractKerberosUser Updated Hive/HBase components to be compatible with the KerberosProperties.validatePrincipalWithKeytabOrPassword method Fixed null ComponentLog in GetHDFSSequenceFileTest Added package-protected accessor method (getAllowExplicitKeytabEnvironmentVariable) to AbstractHadoopProcessor for determining if the environment variable "NIFI_ALLOW_EXPLICIT_KEYTAB" has been set AbstractHadoopProcessor will now only fail validation when the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable is set to false if a keytab is provided to allow the user to specify a principal and password Added AbstractHadoopProcessorSpec to verify validation of principal/keytab/password/kerberos credential service combinations This closes #4095.

…omponents Kerberos Password property should not support EL, this includes a change to KerberosProperties which is also used by the HDFS processors (AbstractHadoopProcessor) Added wiring in a KerberosContext to a TestRunner's MockProcessorInitializationContext Removed synchronization blocks around KerberosUser.checkTGTAndRelogin, since that method is already synchronized Updated AbstractHadoopProcessor to have a boolean accessor method to determine if explicit keytab configuration is allowed Removed synchronization block from HiveConnectionPool's getConnection method (in Hive, Hive_1_1, Hive3 modules), since new TGT ticket acquisition is handled by the KerberosUser implementation. If UGI is used to relogin, synchronization is handled internally by UGI. Added Kerberos Principal and Kerberos Password properties to Hive, Hive_1_1, and Hive3 components Hive, Hive_1_1, and Hive3 components now use KerberosUser implementations to authenticate with a KDC Updated handling of the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable in Hive and Hive3 components. An accessor method has been added that uses Boolean.parseBoolean, which returns true if the environment variable is set to true, and false otherwise (including when the environment variable is unset). Addressing PR feedback Addressing PR feedback This closes #4102.

…ce and HBase_2_ClientService This closes #4103.

NIFI-7026 Add kerberos password property to NiFi HortonworksSchemaReg…

This closes #4097.

…ConnectionPool This closes #4087.

…essors Updating validation logic to be consistent with other password-based kerberos processors, removing getPassword from KerberosPasswordUser as it was only used from testing This closes #4062.

…chProcessor class Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4106.

…structor. (#4098) Signed-off-by: Andy LoPresto <alopresto@apache.org>

… with gzip Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4109.

Updated PutHive3Streaming to wrap calls to Hive in UGI.doAs methods Fixed misleading logging message after the principal has been authenticated with the KDC When connecting to unsecured Hive 3, a UGI with "simple" auth will be used Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4108

…s, e.g.)

Co-authored-by: spius <57421336+spius@users.noreply.github.com> Signed-off-by: Andy LoPresto <alopresto@apache.org>

add explict test for " , " updated with counting validator Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4012

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4114.

…e we close remote file resources for SFTP pulls in particular Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4115.

…JMS and ConsumeJMS processors Some JMS client libraries may not work with the existing controller services due to incompatible classloader handling between the 3rd party library and NiFi. Via configuring the Connection Factory on the processor itself, only the processor's and its children's classloaders will be used which eliminates the mentioned incompatibility. This closes #4110. Signed-off-by: Mark Payne <markap14@hotmail.com>

…resolution will fail with an NPE

This closes #4117. Signed-off-by: Joe Witt <joewitt@apache.org>

Honor Maven properties skipTests and maven.test.skip for the shell script integration tests that verifiy docker images.

…ck for enabling This closes #4118. Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #4088 Signed-off-by: Mark Payne <markap14@hotmail.com>

…d in CLI Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4121.

This closes #4089 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

NIFI-7233 - Fix Mongo Processors to avoid cycle in provenance graph

Added group path to BULLETIN table for QueryNiFiReportingTask Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4100

…nu (#4124) Signed-off-by: Andy LoPresto <alopresto@apache.org>

…case no data is available" This reverts commit f01668e. Signed-off-by: Joe Witt <joewitt@apache.org>

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4127.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4129.

This closes #4004. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

…n failed Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4133.

…chema that indicates that a field should be a Map. Also updated unit test to explicitly define schema, since schema inference will never return a Map but rather a Record

… parameter should have its #onPropertyModified method called. Also renamed Accumulo tests to integration tests because they start embedded servers and connect to them, which caused failures in my environment. Also fixed a bug in TestLengthDelimitedJournal because it was resulting in failures when building locally as well. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4134.

* NIFI-7119 Implemented parameter boundary enforcement for Argon2SecureHasher constructor. Added unit tests for validating each parameter check. * NIFI-7119 Refactored parameter validations. Added more test sizes to boundary checkers. Changed logger severity to error and added bounds to messages. * NIFI-7119 Refactored Argon2 parameter data types to handle unsigned integer boundary values. Updated unit tests. Co-authored-by: Andy LoPresto <alopresto@apache.org> Signed-off-by: Andy LoPresto <alopresto@apache.org>

…, remove any connections before recursing into child groups. This ensures that if a Port exists in child group A and is connected to a port in child group B, if the VersionedProcessGroup indicates to remove the port, that connection will be removed before attempting to remove the port. Updating and adding connections must still be done last, after all components have been added. But missing connections can be removed earlier. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4136.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4138.

…work or be ignored on windows Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4132.

- Add @rule for TemporaryFolder - Replace use of previous target/db with TemporaryFolder/db - Remove use of ~/test db (in home directory) - Remove System.out lines Signed-off-by: Marc Parisi <phrocker@apache.org> This closes #4137.

…sion bug Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4141.

This closes #4113

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4139.

NIFI-7250 fix a test which appears brittle at least on windows builds on slow environments NIFI-7250 activated a timezone run for AU Australia/Melbourne which exposed a poor magic number and needless assertion but interesting results worth keeping Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4140.

…of assuming that multiple runs of the processor will occur within 100 milliseconds, allowed the multiple runs to occur within 3 mins of one another. Signed-off-by: Joe Witt <joewitt@apache.org>

NIFI-6791 Add UUID3 and UUID5 functions to Expression Language

…ss loaded blank properties as empty string instead of an expected null value. Added a isNotBlank check. Added unit tests for replication client and HTTPNotificationService. NIFI-7223 - Renamed some variables and methods. NIFI-7223 - Removed unused dependency. Corrected security properties in administration-guide.

…1 by removing Sun security class references. Added OkHttpReplicationClient#isTLSConfigured() method. Added unit test. NIFI-7223 Fixed remaining unit tests for TLS regression. Renamed tests for clarity.

… key password. This closes #4145. Signed-off-by: Joe Witt <joewitt@apache.org>

…ectly Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4146.

Signed-off-by: Andy LoPresto <alopresto@apache.org>

#4151) * NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.favre.lib.bcrypt library. Updated LICENSE and NOTICE files to reflect changes. Updated unit tests. Co-authored-by: Andy LoPresto <alopresto@apache.org> * NIFI-7268 Fixed typo in Javadoc. Co-authored-by: Andy LoPresto <alopresto@apache.org>

Add support to Mongo Extended JSON v2 Add org.json lib Replace evil json Replace evil json for alternative Include testExtendedJsonSupport Style adjustment Remove unecessary new JSON parser Fix query in testExtendedJsonSupport Parse with Jackson and BSON Back to default MONGO_URI This closes #4068 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

add special handling of PathNotFoundExceptions to log to debug fix spelling error wrap debug log in guard per review This closes #4148 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

- Removed Cat X JSON.org dep inclusion which seems to not be necessary - Updated a ton of easier/safer looking deps - Updated tika due to CVE This closes #4086 Signed-off-by: Mark Payne <markap14@hotmail.com>

…egistry - Update nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/serialization/SchemaRegistryRecordSetWriter.java - Addressing review feedback This closes #4120.

- Changing ShellRunner to use a separate thread for reading the output of the process - Removing unused member variable - Addressing review feedback This closes #4154.

- Updated InstanceClassLoader to resolve files that are in the instance urls or additional urls - Updated nifi-mock to support KerberosContext and removeProperty for ControllerServices - Added unit test for HadoopDBCPConnectionPool - Addressing review feedback This closes #4149.

Remove unused imports Use the latest solrj version(8.4.1) Setup default schemaFactory for tests The default schemaFactory ManagedIndexSchemaFactory creates additional files in test's resources directory. Change it to ClassicIndexSchemaFactory for classic behavior. This closes #4152. Signed-off-by: Bryan Bende <bbende@apache.org>

…ax out cores all moving toward more stable builds. Signed-off-by: Joe Witt <joewitt@apache.org>

…arched NIFI-5925: cleanup, add negative test NIFI-5925: fixed checkstyle This closes #4105 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

…tChannelRecordReader in order to support mark/reset Signed-off-by: Joe Witt <joewitt@apache.org>

…apCacheClientService when value is null Signed-off-by: Joe Witt <joewitt@apache.org>

…ngth on incoming HTTP requests. Adds DoSFilter to enforce configurable maximum on incoming HTTP requests per second. Redirected log messages for ContentLengthFilter to nifi-app.log in logback.xml. This closes #4125. Signed-off-by: Andy LoPresto <alopresto@apache.org>

…eduled

…k to its NAR Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4162

…needed for the test scope

Reviewed by markap14 This closes #4166. Signed-off-by: Joe Witt <joewitt@apache.org>

Some calls to deprecated methods in httpclient were resulting in UnsupportedOperationException. Use the new API calls in both httpclient and solrj. Add an integration test to include test coverage for org.apache.nifi.processors.solr.SolrUtils.createClient This closes #4171.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4172.

…rt for Azure Data Lake Storage Gen 2 Storage. added data-lake dependency NIFI-7103 fixed indentation Update to add IllegalArgumentException Fixed indentation and logging nifi-7103 review changes nifi-7103 root directory and exception change This closes #4126. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

…ce to help with live testing. Added new in memory janus graph client for testing. Added integration test to ExecuteGraphQuery. NIFI-7293 Added missing getter. Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4168

…or empty entries Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4116

…and not running nifi-system-test module Self merging wo review as part of github ci actions stability. Signed-off-by: Joe Witt <joewitt@apache.org>

…olution This closes #4123. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

#4156) * NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping * NIFI-7273: Changed method name, fix handling when analytics not enabled * NIFI-7273: Removed attachment header from Prometheus metrics endpoint * NIFI-7273: Removed unused variable

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4181.

#4187) * NIFI-7126 Increased test iterations to 10,000 in Argon2SecureHasherTest#testDefaultCostParamsShouldBeSufficient to avoid JVM warmup issues. Signed-off-by: Andy LoPresto <alopresto@apache.org>

Signed-off-by: Andy LoPresto <alopresto@apache.org>

…nStopped. Also reject pending request and clean their queue when shutting down. NIFI-7314 In HandleHttpRequest returning 503 when rejecting pending requests before shutdown. NIFI-7314 In HandleHttpRequest add logs and better response message during cleanup. This closes #4191. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

…pport for Azure Data lake Gen 2 Storage. Updated to remove unused variables NIFI-7259 import and property description changes This closes #4189. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

…Atlas reporting task Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4198.

…g a cluster. Now, if there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's flow is inherited. - Refactored Flow Synchronization to make code cleaner - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty. - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start - Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state. - Added additional log statements to aid in debugging NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior NIFI-6849: Addressed review feedback NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup This closes #3891

Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4026

…Toolkit Guide. (#4196)

…t to rollover event files after specified time. Code cleanup. Updated some default properties.

…ctive Event File

* NIFI-7319 Added first draft of walkthroughs doc. * NIFI-7319 Added instructions and screenshots for securing standalone NiFi instance. * NIFI-7319 Added instructions and screenshots for instructing OS & browser to trust self-signed certificate. * NIFI-7319 Added instructions and screenshots for securing NiFi with externally-provided certificates. * NIFI-7319 Added instructions and screenshots for building NiFi from source. * NIFI-7319 [WIP] Converting secure cluster instructions to match format. Fixed instructions regarding embedded ZooKeeper configuration. * NIFI-7319 Completed secure cluster walkthrough. * NIFI-7319 Added walkthroughs to documentation navigation list. * NIFI-7319 Incorporated PR feedback on broken links. * NIFI-7319 Removed line number helpers from update sections. * NIFI-7319 Incorporated final PR review items. Co-authored-by: Sandra Pius <spiusapache@gmail.com>

…bles in docker scripts This closes #4203. Signed-off-by: Aldrin Piri <aldrin@apache.org>

…ged due to timeout and has no records Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4210.

…report report the time when metrics are generated Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4211.

…e Connection summary table does not work properly - fix style issues - review feedback This closes #4208

…rivileges Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4195.

Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4209

This closes #4201. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>

…port for Azure Data lake Gen 2 Storage. NIFI-7334 Update to FetchDataLakeStorage Processor This closes #4212. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

https://issues.apache.org/jira/browse/NIFI-7366 This commit allows to retrieve ItemAttachement (such as EML) file when pulling mail. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4215.

As it is issuing 'docker stop' will immediatly exit the container. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4182.

…th wider type; Allowing doubles with value within float precision to be considered as valid floats (NIFI-7302)

…ts not to show up in specific situations when clicking View Provenance for a Processor. - Added System-level tests for Provenance repository to reproduce behavior. - Added a Provenance Client to the CLI, which is necessary for System-level tests. - Added small additional configuration for Provenance repository to simplify development of system tests - Minor improvements to system tests (such as ability to destroy environment between tests) needed for Provenance repository based system tests Signed-off-by: Joe Witt <joewitt@apache.org>

…from Atlas config file. Still recognizes 'atlas.cluster.name' as well, but takes lower precedence than the new property. Also Atlas URL can be provided via the 'atlas.rest.address' property in the atlas-application.properties. NIFI-7280 In ReportLineageToAtlas improved documentation and adjusted property ordering for better user experience. Minor refactor. NIFI-7280 In ReportLineageToAtlas amended documentation. Minor refactor. NIFI-7280 In ReportLineageToAtlas amended more documentation. More minor refactor. NIFI-7280 - In Atlas reporting: complete clusterName -> namespace overhaul where appropriate. This closes #4213. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

)

This closes #4236. Signed-off-by: Andy LoPresto <alopresto@apache.org>

Refactored masking logic to CipherUtility and indicated masking with label and Base64 output. Added JSON masking logic to nifi-stateless module. Added argument masking functionality to Program. Moved groovy unit tests to proper Maven directory structure. Modified plain argument output to use filtering/masking methods in provided utility. Refactored utility methods. Updated unit tests. This closes #4222. Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com> Signed-off-by: Andy LoPresto <alopresto@apache.org>

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4227.

#4229) * NIFI-7379: Support multiple instances of Prometheus registries/metrics * NIFI-7379: Refactored Prometheus objects to support multiple instances

By using dynamic properties with a prefix naming scheme, allow definition of the parts, including the name to give the Flowfile content part, and optionally it's file name. After review: - change so that we can send just the form content or just form data without the flowfile - change the content name and content file name from dynamic properties to properties - change the dynamic name to be an invalid http header "post:form:xxxx" - add validation and more tests This closes #4234. Signed-off-by: Mark Payne <markap14@hotmail.com>

NIFI-7408 - return double value for utilization Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4240

- Adding a flag to nifi.properties to disable anonymous authentication. NIFI-7170: - Fixing checkstyle issues. NIFI-7170: - Adding missing license header. NIFI-7170: - Initial PR feedback. NIFI-7170: - Fixing broken integration tests. - Creating new integration tests for verifying allowing and preventing anonymous access. NIFI-7170: - Ensuring the new anonymous authentication property is considered for proxied requests. NIFI-7170 - Fixed comment. Signed-off-by: Nathan Gough <thenatog@gmail.com> This closes #4099.

review changes additional review changes NIFI-6149: typo fixes This closes #4226. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4247.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4246.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4238.

This closes #4245. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

…ters before writing flow.xml.gz NIFI-7414: Updated StandardFlowSerializerTest to include testing for variable names and values being filtered This closes #4244

Adds a log message when ReplaceText sends a flowfile to the failure relationship because it is larger than the max buffer size. Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4255.

…y parameters already captured in http.query.param _and_ the multipart form data names and values, which are captured in the part data, and could be very very large This closes #4251. Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #4259

… root pom.xml. Upgraded tika-parsers dep in nifi-media-processors. Upgraded jackson-databind dep in nifi-graph-processors. Upgraded jackson-databind dep in nifi-elasticsearch-client-service-api. Upgraded jackson-databind dep in in nifi-easyrules-service. Upgraded calcite-core dep in nifi-sql-reporting-tasks. Signed-off-by: Nathan Gough <thenatog@gmail.com> This closes #4252.

Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4256

NIFI-7423 Upgraded jquery dependency version to latest 3.5.1. This closes #4258

… exists Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4237.

…/dev guides Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4267

…e Jetty server version in responses. Fixed a checkstyle error. Added property to nifi.properties. Changed property to a variable that is set with the pom.xml. Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context. Fixed typo error. This closes #4192. Signed-off-by: Andy LoPresto <alopresto@apache.org>

NIFI-7367: Negative test cases for expression language in FetchAzureDataLakeStorage FetchAzureDataLakeStorage throws exception when filesystem or filename is blank. Fixed logged error messages in all 3 of the Delete, Fetch and Put ADLS processors. testFetchDirectory test case marked as ignored. This closes #4257. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4269.

…utoTerminatedRelationships. Added additional logging and improved error handling around syncing with invalid flows

…tors for performance NIFI-7437 - reduced scheduler to 15 seconds, change cache to expire after no access vs expire after write Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4274

…butes This closes #4275. Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #4264. Signed-off-by: Matthieu Cauffiez <matthieu.cauffiez@bell.ca> Signed-off-by: Mark Payne <markap14@hotmail.com>

…hen the specified path points to a directory A newer version (12.1.1) of azure-storage-file-datalake is imported. This closes #4273. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4283.

NIFI-7409: review changes NIFI-7409: ordering import statements NIFI-7409: changed validateCredentialProperties logic This closes #4249. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

DeleteAzureDataLakeStorage now throws exception if fileSystem or fileName is empty string NIFI-7336: Add tests for DeleteAzureDataLakeStorage - typos fixed NIFI-7336: Add tests for DeleteAzureDataLakeStorage - fixed a test case This closes #4272. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

This closes #3906 Signed-off-by: Joey Frazee <jfrazee@apache.org>

…2" (in shared constant). Changed JettyServer default SSL initialization and updated unit test. Removed SecurityStoreTypes (unused). Added StringUtils inverted blank and empty checks. Added TlsConfiguration container object. Enhanced KeystoreType enum. Added clean #createSSLContext() method to serve as base method for special cases/other method signatures. Added utility methods in KeyStoreUtils. Added generic TlsException for callers that cannot resolve TLS-specific exceptions. Added utility methods for component object debugging. Enforced TLS protocol version on cluster comms socket creation. Added utility method for SSL server socket creation. Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version). Removed duplicate SSLContextFactory. Switched duplicate SslContextFactory to wrap shared SSLContextFactory. Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test). Added further validation & boundary checking in uses of TlsConfiguration. Provided SSLSocketFactory accessor in SslContextFactory. Refactored OkHttpReplicationClient tuple method. Refactored OcspCertificateValidator TLS logic. Added utility method to apply TLS configs to OkHttpClientBuilder. Removed references to duplicate SslContextFactory. Removed unnecessary SslContextFactory. Moved OkHttpClientUtils to nifi-web-util module. Updated module dependencies. Removed now empty nifi-security module. Enforced TLS protocol selection on LB server socket. Enforced TLS protocol selection on S2S server socket. Applied specified TLS protocol versions to S2S socket creation. Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory. Replaced references to creation methods throughout codebase. Replaced references to unnecessary NiFiProperties file reads throughout tests. Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth. Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners. Cleaned up legacy code. Added external timing check to timing test assertion. Made RestrictedSSLContextService TLS protocol versions allowable values explicit. Enabled TLSv1.3 on Java 11. Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService. Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task. This closes #4263. Signed-off-by: Nathan Gough <thenatog@gmail.com> Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #3659. Signed-off-by: Joey Frazee <jfrazee@apache.org>

Create empty relationship for RunMongoAggregation Fix default autoterminate and condition to redirect to REL_EMPTY Change from new relationship to write an empty FlowFile to RESULT Fix MONGO_URI This closes #4281 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

…valid type for use with calcite query functions NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types NIFI-7462: Fixed checkstyle violation, removed documentation around the CAST functions that were no longer needed Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4282

Added unit test. This closes #4291. Signed-off-by: Arpad Boda <aboda@apache.org>

… path. NIFI-6255 NIFI-6287: Rebased to match the new expression language interface NIFI-6255 NIFI-6287: Fix wildcard imports and unused imports NIFI-6255 NIFI-6287: Move to the common codec DigetUtils Update commons-codec This closes #3624 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

…g overflow. TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY The behaviour change is reverted until further investigations. The overflow behaviour is still enforced by unit tests and documented in the expression language doc NIFI-6672 Removed test code. This closes #3738 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

… Long overflow. TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to POSITIVE_INFINITY or NEGATIVE_INFINITY The behaviour change is reverted until further investigations. The overflow behaviour is still enforced by unit tests and documented in the expression language doc This closes #3739 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

…ng overflow. TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY MinusEvaluator throws an Arithmetic Exception in case of Long overflow. TestQuery checks that Long overflow is detected and Double overflow is correctly promoted to NEGATIVE_INFINITY The behaviour change is reverted until further investigations. The overflow behaviour is still enforced by unit tests and documented in the expression language doc fixed mispositioned # in doc This closes #3740 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

…fails due to timing issues. This closes #4296

NIFI-6785 Remove unused imports This closes #3822 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

NIFI-7453 Creating a new Kudu client when refreshing TGT in KerberosPasswordUser as well. (Applied to KerberosKeytabUser only before.) NIFI-7453 Safely closing old Kudu client before creating a new one. NIFI-7453 Visibility adjustment. This closes #4276. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

NIFI-7485 Found more instances that needed updating. This closes #4295

…e processor NIFI-7445: Add Conflict Resolution property to PutAzureDataLakeStorage processor Made warning and error messages more informative. Refactored flowFile assertion in the tests. This closes #4287. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4302.

… docs Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4293.

…scription Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4294.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4292.

This closes #3734 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

…e call the onFailed or onCompleted function. If the result is failed, return true and do sth NIFI-7403:Add an extension point to adjust the result, if the result is failed then process onFailed function NIFI-7403:Implement the AdjustFailed Function, if PutSQL set the SUPPORT_TRANSACTIONS true, then check whether the result contains REL_RETRY or REL_FAILURE.If it contains that, reroute the result and return true. NIFI-7403: fix reroute logic in AdjustFailed function NIFI-7403:Add and modify some unit test for PutSQL's SUPPORT_TRANSACTIONS property NIFI-7403:Update for PR recheck NIFI-7403:Add documentation on the Support Fragmented Transactions property to indicate the transactions rollback behavior NIFI-7403: Fix Checkstyle issue Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4266

…the flowfile NIFI-7477 Improving description and unit test now verifies attribute content NIFI-7477: Fixed checkstyle errors Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4301

…t object is defined as an interface, proxy that interface. This way, any method call into the object will also change the classloader to the appropriate classloader.

This closes #4303. Signed-off-by: Mark Payne <markap14@hotmail.com>

…d missing precision when reading in records Signed-off-by: Mark Payne <markap14@hotmail.com>

…E type in QueryRecord This closes #4223.

…ss tokens when supplied with appropriate credentials. Added skeleton of oauth2 provider. Added copy of our code. Refactored a few things. Updated apis to better match flow descriptions. Updated poms and other artifacts. Updated copyright notice. Updated LICENSE. This closes #4173 Signed-off-by: Jeremy Dyer <jeremydyer@apache.org>

…ocessGroup level Added FlowFileOutboundPolicy to ProcessGroups and updated LocalPort to make use of it Persisted FlowFile Concurrency and FlowFile Output Policy to flow.xml.gz and included in flow fingerprint Added configuration for FlowFile concurrency and outbound policy to UI for configuration of Process Groups Added system tests. Fixed a couple of bugs that were found Fixed a couple of typos in the RecordPath guide Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4306.

…iceInvocationHandler and fix checkstyle violation on NiFiSystemIT

Removed list structure for peer selection as it was unnecessary and often wasteful (most clusters are 3 - 7 nodes, the list was always 128 elements). Changed integer percentages to double to allow for better normalization. Removed 80% cap on remote peers as it was due to legacy requirements. Added unit tests for non-deterministic distribution calculations. Added unit tests for edge cases due to rounding errors, single valid remotes, unbalanced clusters, and peer queue consecutive selection tracking. Migrated all legacy PeerSelector unit tests to new API. Removed unused System time manipulation as tests no longer need it. Added class-level Javadoc to PeerSelector. Removed S2S details request replication, as the responses were not being merged, which led to incorrect ports being returned and breaking S2S peer retrieval. Fixed copy/paste error where input ports were being listed as output ports during remote flow refresh. Fixed comments and added unbalanced cluster test scenarios. Removed unnecessary marker interface. Removed commented code. Changed weighting & penalization behavior. Changed dependency scope to test. This closes #4289. Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #4233. Signed-off-by: Joey Frazee <jfrazee@apache.org>

NIFI-7507: Fixed Flowfile Expiration header in doc Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4318

…nor refactor in StandardControllerServiceInvocationHandler. Also removed an unused import from NiFiSystemIT. This closes #4317. Signed-off-by: Mark Payne <markap14@hotmail.com>

Cleaned up code style. Unit test was failing on Windows 1.8 GitHub Actions build but no other environment. Increased artificial delay to avoid timing issues. Co-authored-by: Andy LoPresto <alopresto@apache.org> This closes #4271. Signed-off-by: Andy LoPresto <alopresto@apache.org>

- Ensuring the group id is always set in the properties table when loading properties. - Using a common approach to getting parameters in nfControllerService. - Code clean up. - Addressing review feedback. - Ensuring the service dialog is closed when navigating to the parameter context dialog. This closes #4322

review Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4299

NIFI-7313:add test by wanghongqi NIFI-7313:edit test Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes #4185

This closes #4265. Signed-off-by: Joey Frazee <jfrazee@apache.org>

…tic registry setup

…update-access-policy use case more in line with the NiFi side. Added some tests. Additional refactor, documentation revision. This closes #4329.

…ocking. NIFI-7527 Fixed StackOverFlowError due to pacing issue (recursive login before loggedIn flag is set). NIFI-7527 Refactor: removed redundant kudu client creation. This closes #4330. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

… reconnect

…ump once and then provide this information to ProcessorNode when capturing active threads. Previously, each processor captured a thread dump itself. When this is done thousands of times it can result in a very long delay.

…ment (#4341) * NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment This also fixes NIFI-4760. * NIFI-7540: Remove duplicate mail.smtp.starttls.enable from TestListenSMTP Signed-off-by: Andy LoPresto <alopresto@apache.org>

…String rather than a Long. Fixes cast error in LDAP libs.

…ses. (#4307) NIFI-6094 - Added the mime/content type for ttf files.

- update Kudu dependencies to Kudu 1.12.0 - add VARCHAR to Kudu Lookup Service and Processor - add tests for VARCHAR columns Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4347.

… of processors when needing to wait for all processors to stop for updating parameter context, etc.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4315.

…tion to be handled. Instead, use TimeUnit.timedWait and use Object.notifyAll when setting the beingServiced flag. This significantly reduces latency and improves throughput for small-batch site-to-site communications This closes #4353. Signed-off-by: Andy LoPresto <alopresto@apache.org>

… field so that it's removed regardless of whether or not this field is kept. This closes #4332

…ares model (#4323)

Update nf-context-menu.js for an intuitive road to parameters When rightclicking a process group the variables are shown, but parameters are not. This makes sense as they have a prerequisite, in the form of a parameter context. This change gives a more consistent experience for finding the functionality regarding parameters by ensuring the contextmenu shows the possibility to configure a parameter context. Once the paramater context has been created for a process group, the parameters text shows, so this is no longer visible. People would then need to click configure to change the context, just as they would be required to do now. Added generateflowfile load tag and description Added GenerateFlowFile load tag to be consistent with DuplicateFlowFile and updated description to refer to DuplicateFlowFile. Revert "Update nf-context-menu.js for an intuitive road to parameters" This reverts commit 3c44b16. This closes #4333 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

Renamed legacy terms. Updated documentation. This closes #4351. Signed-off-by: Mark Payne <markap14@hotmail.com>

NIFI-7542 Override additional jackson-databind versions. NIFI-7542 Upgrade jackson-databind dependency to 2.9.10.5 in the root pom.xml. This closes #4343 Signed-off-by: Mike Thomsen <mthomsen@apache.org>

#4348) * NIFI-7523: Use SSL Context Service for Atlas HTTPS connection in Atlas reporting task Also fixing ControllerServiceDisabledException-s when validating the Kerberos config * NIFI-7523: Fixed test failure on Windows * NIFI-7523: Added license headers. * NIFI-7523: Fixed another test failure on Windows * NIFI-7523: Review changes

ListenHTTP processor now binds port and creates a HTTP connection only if one of the following conditions apply: - Primary node execution is 'false' - Primary node execution is 'true' and node is elected as primary node. Changes: - Connection is established in 'onTrigger' annotated method instead of 'onSchedule'. (This is similar to how handleHTTPRequest processor handles connections.) - 'onPrimaryNodeStateChange' annotated method is introduced to tear down server on reelection of primary node This closes #4356. Signed-off-by: Mark Payne <markap14@hotmail.com>

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes #4334.

This closes #4357 Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

…evel read timeout and connect timeout. In QueryCassandra when writing flowfile to the sesion it's done on the raw OutputStream. Wrapped it in a BufferedOutputStream for better performance. This closes #4368. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

- Remove unused imports - Fix checkstyle errors This closes #4358.

…c unit test. Signed-off-by: Joe Witt <joewitt@apache.org>

…related references properly to null after closing them so that they can be renewed in 'onEnabled' (which creates them only if set to 'null', leaving them closed otherwise). NIFI-7590 Removed 'CassandraSessionProvider.onStopped'. This closes #4373. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

The introduced changes prevent creating unnecesary sessions and producers in some scenarios. This closes #4378. Signed-off-by: Joey Frazee <jfrazee@apache.org>

…r processing. This closes #4379. Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>

…der response when the OIDC Identifying User claim is not found. Revised log message to print available claims. Added new StandardOidcIdentityProviderGroovyTest file. Updated deprecated methods in StandardOidcIdentityProvider. Changed log output to print all available claim names from JWTClaimsSet. Added unit test. Added comments in getAvailableClaims() method. Fixed typos in NiFi Docs Admin Guide. Added license to Groovy test. Fixed a checkstyle error. Refactor exchangeAuthorizationCode method. Added unit tests. Verified all unit tests added so far are passing. Refactored code. Added unit tests. Refactored OIDC provider to decouple constructor & network-dependent initialization. Added unit tests. Added unit tests. Refactored OIDC provider to separately authorize the client. Added unit tests. Added unit tests. NIFI-7332 Refactored exchangeAuthorizationCode method to separately retrieve the NiFi JWT. Signed-off-by: Nathan Gough <thenatog@gmail.com> This closes #4344.

Master #1

Master #1

Commits on Feb 20, 2020

Commits on Feb 21, 2020

Commits on Feb 22, 2020

Commits on Feb 23, 2020

Commits on Feb 24, 2020

Commits on Feb 25, 2020

Commits on Feb 26, 2020

Commits on Feb 27, 2020

Commits on Feb 28, 2020

Commits on Mar 2, 2020

Commits on Mar 3, 2020

Commits on Mar 4, 2020

Commits on Mar 5, 2020

Commits on Mar 6, 2020

Commits on Mar 7, 2020

Commits on Mar 8, 2020

Commits on Mar 9, 2020

Commits on Mar 10, 2020

Commits on Mar 11, 2020

Commits on Mar 12, 2020

Commits on Mar 13, 2020

Commits on Mar 15, 2020

Commits on Mar 16, 2020

Commits on Mar 17, 2020

Commits on Mar 18, 2020

Commits on Mar 19, 2020

Commits on Mar 20, 2020

Commits on Mar 22, 2020

Commits on Mar 24, 2020

Commits on Mar 25, 2020

Commits on Mar 26, 2020

Commits on Mar 27, 2020

Commits on Mar 30, 2020

Commits on Mar 31, 2020

Commits on Apr 1, 2020

Commits on Apr 2, 2020

Commits on Apr 3, 2020

Commits on Apr 6, 2020

Commits on Apr 7, 2020

Commits on Apr 8, 2020

Commits on Apr 9, 2020

Commits on Apr 11, 2020

Commits on Apr 13, 2020

Commits on Apr 15, 2020

Commits on Apr 16, 2020

Commits on Apr 17, 2020

Commits on Apr 18, 2020

Commits on Apr 20, 2020

Commits on Apr 22, 2020

Commits on Apr 25, 2020

Commits on Apr 27, 2020

Commits on Apr 28, 2020

Commits on Apr 29, 2020

Commits on Apr 30, 2020

Commits on May 1, 2020

Commits on May 4, 2020

Commits on May 6, 2020

Commits on May 7, 2020

Commits on May 8, 2020

Commits on May 9, 2020

Commits on May 12, 2020

Commits on May 13, 2020

Commits on May 14, 2020

Commits on May 15, 2020

Commits on May 18, 2020

Commits on May 19, 2020

Commits on May 20, 2020

Commits on May 21, 2020

Commits on May 22, 2020

Commits on May 25, 2020

Commits on May 26, 2020

Commits on May 27, 2020

Commits on May 29, 2020

Commits on May 30, 2020

Commits on Jun 1, 2020

Commits on Jun 2, 2020

Commits on Jun 4, 2020

Commits on Jun 5, 2020