nifi oracle cdc changes using xstream #3306
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mattyb149
reviewed
Feb 14, 2019
| <version>1.9.0-SNAPSHOT</version> | ||
| </parent> | ||
|
|
||
| <artifactId>nifi-oraclecdc-nar</artifactId> |
There was a problem hiding this comment.
Don't forget to add this to the nifi-assembly.pom so it's included in the distro. If it is a particularly large NAR, consider adding a profile (deactivated by default), see the include-atlas or include-hive3 profiles as an example.
There was a problem hiding this comment.
they have been added to the assembly pom
…m being handled correctly. NIFI-6228 Added unit test for SSL configuration. Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes apache#3535
NIFI-5537 Renamed bundle based on review comments NIFI-5537 Updated neo4j setup and version documentation and default values based on review comments NIFI-5537 - Changed exception, logging, and json parser based on review comments NIFI-5537 - Check debug enabled before logging at debug level NIFI-5537 - Updated property displayname/description and corrected test file names NIFI-5537 - Added note about dummy password required in case authentication is disabled on server NIFI-5537 - Use SSLContext
NIFI-5537 Added skeletons of services, apis, etc. NIFI-5537 Renamed classes to reflect that they are for cypher and not neo4j specific NIFI-5537 Added start of client API for doing streaming execution. NIFI-5537 Completed refactor of unit and integration tests. NIFI-5537 Added OpenCypherClientService. NIFI-5537 Updated to gremlin-core 3.3.5 to get better SSL configuration options. NIFI-5537 Added SSL support to OpenCypher client service. NIFI-5537 Updated L&N. NIFI-5537 Updated to 1.10.0-SNAPSHOT. NIFI-5537 Removed Mac-specific documentation in int test. NIFI-5537 Updated a few properties to add EL. NIFI-5537 Added GremlinClientService to support gremlin as well. NIFI-5537 refactored everything to be a graph bundle. NIFI-5537 Updated documentation. NIFI-5537 Fixed assembly. NIFI-5537 Updated Jackson. NIFI-5537 Moved MIT section per code review guidance. NIFI-5537 Changed provenance event and made it more explicit. NIFI-5537 Updated processor config file. NIFI-5537 Made changes requested in a code review. NIFI-5537 Added documentation about driver configuration. NIFI-5537 Remove output flowfile if an exception is thrown. NIFI-5537 Updated configuration properties. NIFI-5537 Made port and path configurable in the tinkerpop family of components. NIFI-5537 Upgraded to OpenCypher cypher-gremlin-neo4j-driver 0.9.13 because it's the most recent version that JanusGraph supports. NIFI-5537 Added warning about Janus 0.3.X to int test. NIFI-5537 Added TinkerPopClientService interface. NIFI-5537 Changed to mockito-core from mockito-all. NIFI-5537 Added documentation about Gremlin for first time users w/ NiFi.
NIFI-5973 More comments and better defaults for the shell provider. NIFI-5973 Fixed bug where user was being retrieved by identifier when identity was provided. NIFI-5973 Fixed a formatting string in the OS X shell commands. Updated testing conditions to run IT in OS X environment. Changed unit test to provide identity rather than identifier. This closes apache#3537. Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-6004 Improve testing of PutFile file and directory permissions NIFI-6004 Typo in regex NIFI-6004 Updates based on review This closes apache#3294. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
modifying test cases, fixing file processing. Testing done: - Unit tests - Manual run against a hadoop instance This closes apache#3558
This closes apache#3539. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
…h a Thread could throw an Exception, then another Thread could update the Journal before the first thread closes it. Added unit test to replicate. This closes apache#3561
…heus-nar Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes apache#3565
NIFI-6387 RetryFlowFile NIFI-6387 Maximum Retries support FLOWFILE_ATTRIBUTES scope NIFI-6387 Fixed reuses descriptions for clarity This closes apache#3541. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
This closes apache#3513. Signed-off-by: Mark Payne <markap14@hotmail.com>
- Add additional doc about best practices. This closes apache#3540. Signed-off-by: Mark Payne <markap14@hotmail.com>
… code requires it. NIFI-5562 - Removed unnecessary usage of Guava NIFI-5562 - Updated Guava version to 28.0-jre NIFI-5562 - Upgraded Guava to 28.0-jre and Curator to 4.2.0 This closes apache#3577 Signed-off-by: Mike Thomsen <mthomsen@apache.org>
…o output flowfiles when Output Batch Size is set NIFI-6271, fix incoming flowfile attributes don't copy into output flowfiles when Output Batch Size is set NIFI-6271, fix incoming flowfile attributes don't copy into output flowfiles when Output Batch Size is set replace getAttribute(uuid) with getAttribute(CoreAttributes.UUID.key() fix checkstyle violation This closes apache#3575. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
…r is using the "Use Segment Names" option in the ExtractHL7Attributes processor and the HL7 message being processed contains custom user segments (e.g., Z Segments), the code will downgrade to returning the Segment Label as it does when this option is not used. Without this a null pointer is thrown at line 288. Signed-off-by: Joe Witt <joewitt@apache.org>
…tingTask and PrometheusReportingTask (apache#3554) * NIFI-5417: Add missing component status and metrics to S2SReportingTask and PrometheusReportingTask * NIFI-5417: Added executionNode to schema and doc This closes apache#3554 Signed-off-by: Yolanda M. Davis <yolanda.m.davis@gmail.com>
Updated pom files regarding groovy dependencies. groovy-all was changed to a pom type dependency Added Groovy Bintray as a plugin repository in root pom Upgraded spock-core to version 1.3-groovy-2.5 and added exclusion of groovy dependencies to force spock to use the version from dependencyManagement Updated groovy-eclipse-batch to use the groovy.version property to determine the version Updated groovy-eclipse-compiler to 3.4.0-01 Updated maven-compiler-plugin to use maven.compiler.source and maven.compiler.target properties for source and target configuration properties Removed configuration and dependencies sections from nifi-toolkit-admin and nifi-toolkit-encrypt-config maven-compiler-plugin configurations so that the configuration from the root pom's maven-compiler-plugin is inherited Removed dependencyManagement from nifi-jetty module, version of groovy modules will be inherited from root pom Removed maven-compiler-plugin configuration from nifi-toolkit-api so that the configuration from the root pom's maven-compiler-plugin is inherited Updated spock-core dependencyManagement to 1.3-groovy-2.5 Fixed AESSensitivePropertyProviderTest issue with Groovy creating KEY_256_HEX slice of the wrong size due to BigDecimal being used as the result of the division; using intdiv to force an integer result creates the correctly sized array Added groovy-json test dependency to nifi-web-security Removed maven-compiler-plugin configuration from nifi-lookup-services, nifi-mock-record-utils, and nifi-web-utils so that the configuration from the root pom's maven-compiler-plugin is inherited Updated root pom pluginManagement to specify version 3.8.0 of maven-compiler-plugin Added maven-compiler-plugin config to nifi-toolkit-admin and nifi-toolkit-encrypt-config to use groovy-eclipse-compiler during the compile phase so that the groovy-based tools are compiled Addressed deprecated CliBuilder and OptionAccessor usage in nifi-toolkit-encrypt-config, those classes were moved from groovy.util to groovy.cli.commons Removed getInner() usage from nifi-toolkit-encrypt-config, method no longer exists causing the tests to crash Updated CryptographicHashAttributeTest to use java.time classes instead of java.util.Date Updated nifi root POM's groovy-test dependency to be test-scoped Added properties for specifying groovy versions for several modules: nifi, nifi-groovyx-bundle, nifi-scripting-bundle, nifi-toolkit Established dependency management for groovy-all:pom:2.5.4, and added that dependency to several modules: nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-admin, nifi-toolkit-encrypt-config Added groovy version property usage to several modules that established a dependency on a groovy submodule that was not listed in its own or inherited dependency management Removed unused build-helper-maven-plugin from nifi-toolkit-api's POM Removed unnecessary groovy-eclipse-compiler build plugin config from nifi-web-utils' POM to use the inherited config for that plugin Updated several modules' NOTICEs to include appropriate Groovy NOTICE content Updated to list groovy-all:pom:2.5.4 and its transitive submodule dependencies to nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, and nifi-toolkit-assembly NOTICEs Added missing groovy-all:jar:2.1.6 NOTICE to nifi-hive-nar and nifi-hive_1_1-nar NOTICEs Added missing groovy-all:jar:2.4.11 NOTICE to nifi-hive3-nar NOTICE Updated to list groovy-all:jar:2.4.16 nifi-other-graph-services-nar NOTICE Removed Groovy NOTICE content from nifi-record-serialization-services-nar NOTICE, no Groovy modules are included in the NAR NIFI-5254 Updated several modules' NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with Groovy modules which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-hive-nar, nifi-hive3-nar, nifi-hive_1_1-nar, nifi-other-graph-services-nar, nifi-scripting-nar, nifi-toolkit-assembly NIFI-5254 Updated NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with modules used by Groovy 2.5.4 which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-assembly. The following modules' LICENSE and NOTICE content were added: Apache Ant, Apache Commons CLI, JLine, JUnit Platform/Jupiter This closes apache#3547 Signed-off-by: Mike Thomsen <mthomsen@apache.org>
…s in data loss This closes apache#3573. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
This closes apache#3568. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
…MapCacheService and HBase_2_ClientMapCacheService so that they use the column family and qualifier specified in the service This closes apache#3581. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes apache#3588.
…port setObject() without type Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes apache#3585.
This closes apache#3589. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
…rrectly when set 'Use Avro Logical Types' to true This closes apache#3584. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Signed-off-by: Bryan Bende <bbende@apache.org>
…e is used, bump registry client version This closes apache#3126.
…e - ensure non null content type This closes apache#3596. Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
…lection when a Diagnostics Dump is performed This closes apache#3809
Update the links to the quartz documentation in the User Guide section for cron trigger scheduling. This closes apache#3802. Signed-off-by: Kevin Doran <kdoran@apache.org>
…ssary Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes apache#3803.
…or Kafka 2.0 processors Signed-off-by: Joe Witt <joewitt@apache.org>
…chFile Signed-off-by: Joe Witt <joewitt@apache.org>
…ession/decompression NIFI-6773 Adding two unit tests for lz4-framed compression/decompression NIFI-6773 Adding compressed lz4 file for decompression test Signed-off-by: Joe Witt <joewitt@apache.org>
…fferent value for hostname/port for web api or cluster protocol, assume that node is correct about itself instead of assuming that Cluster Coordinator knows best about the other node NIFI-6779: Remove any conflicting Node Identifiers when a new Node ID is encountered This closes apache#3819
NIFI-6703: Fixed extracted nar directory and marked api as experimental NIFI-6703: Moving nifi-stateless into nifi-framework NIFI-6703: Refactored to fix jetty/spring issues NIFI-6703: checkstyle fix NIFI-6703: updated to mirror traditional NiFi's bootstrap process and java11 dependency management NIFI-6703: minor changes NIFI-6703: Documentation fixes This closes apache#3795. Signed-off-by: Bryan Bende <bbende@apache.org>
|
@rkarthik29 please review the conflict. Also as Burgess suggested please don't have the nar included by default. Have it assembly activated. We cannot really afford more nars included by default |
tpalfy
reviewed
Oct 18, 2019
nifi-assembly/pom.xml
Outdated
| <dependency> | ||
| <groupId>org.apache.nifi</groupId> | ||
| <artifactId>nifi-oraclecdc-nar</artifactId> | ||
| <version>1.9.0-SNAPSHOT</version> |
There was a problem hiding this comment.
Shouldn't versions be 1.10.0?
There was a problem hiding this comment.
yes, changed to 1.10.0-SNAPSHOT
| <parent> | ||
| <groupId>org.apache.nifi</groupId> | ||
| <artifactId>nifi-oraclecdc-bundle</artifactId> | ||
| <version>1.9.0-SNAPSHOT</version> |
There was a problem hiding this comment.
Shouldn't versions be 1.10.0?
There was a problem hiding this comment.
yes, changed to 1.10.0-SNAPSHOT
| <dependency> | ||
| <groupId>org.apache.nifi</groupId> | ||
| <artifactId>nifi-api</artifactId> | ||
| <version>1.9.0-SNAPSHOT</version> |
There was a problem hiding this comment.
Shouldn't versions be 1.10.0?
There was a problem hiding this comment.
yes, changed to 1.10.0-SNAPSHOT
...a/org/apache/nifi/processors/oraclecdc/controller/impl/StandardOracleClassLoaderService.java
Outdated
Show resolved
Hide resolved
| @Tags({ "dbcp", "jdbc", "database", "connection", "pooling", "store" }) | ||
| @CapabilityDescription("Provides a shareable classloader for the Oracle CDC connection pool") | ||
|
|
||
| public class StandardOracleClassLoaderService extends AbstractControllerService implements OracleClassLoaderService { |
There was a problem hiding this comment.
If I understand correctly the purpose of this service is to save loading the jdbc driver over and over.
I think we have a great opportunity to make this completely generic with very little extra effort.
The only Oracle-specific setting here is the driver name. That could come from a property as well.
There was a problem hiding this comment.
hi @tpalfy , the purpose is to make sure that we can attach the cdc listener to multiple xstream servers from a single nifi instance. The xstream stuff needs some oracle native libraries. Since native libraries can only be loaded once inside JVM, i had to take this alternative. I do agree it could be utilized elsewhere, especially with the HDFS processors.
...main/java/org/apache/nifi/processors/oraclecdc/controller/impl/StandardOracleCDCService.java
Show resolved
Hide resolved
...main/java/org/apache/nifi/processors/oraclecdc/controller/impl/StandardOracleCDCService.java
Outdated
Show resolved
Hide resolved
...main/java/org/apache/nifi/processors/oraclecdc/controller/impl/StandardOracleCDCService.java
Outdated
Show resolved
Hide resolved
...main/java/org/apache/nifi/processors/oraclecdc/controller/impl/StandardOracleCDCService.java
Outdated
Show resolved
Hide resolved
...lecdc-processors/src/main/java/org/apache/nifi/processors/oraclecdc/OracleChangeCapture.java
Show resolved
Hide resolved
Signed-off-by: Andy LoPresto <alopresto@apache.org>
…s scan exclusions Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com> This closes apache#3793.
- Upgrading nimbus-jose-jwt. This closes apache#3830. Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-6778 - added comments and component description annotations NIFI-6778 - added license and notice details NIFI-6778 - added additional details section and updated description of service NIFI-6778 - fixed checkstyle error NIFI-6778 - addressed review comments for documentation and description corrections NIFI-6778 - added include-rules profile for assembly build NIFI-6778 - add MVEL and ASM to license and notice NIFI-6778 - switch to use Jackson for NiFi Rules Json deserialization Signed-off-by: Matthew Burgess <mattyb149@apache.org> This closes apache#3824
… implementation. Added skeleton implementation of EncryptedFileSystemRepository. Added new impl to META-INF registry. Added investigation comments to FileSystemRepository. Implemented RepositoryObject block and stream encryptors. Added passing unit test for encryption and decryption of multiple content writes (large buffered file) for AES-CTR encryptor. Refactored shared logic from AES CTR and G/CM encryptors to abstract parent. Added working unit test for writing/reading via encrypted file system repository. Added stream wrappers. Added encryptor. Added working unit test for writing/reading multiple pieces of content via encrypted file system repository. Added unit test skeleton for writing/reading multiple pieces of content with different keys via encrypted file system repository. Implemented key management skeleton for encrypted content repository. Multiple content claims can now be encrypted with different keys on the same resource claim and retrieved. Implemented validation on setting active key id. Added content repository encryption properties to NiFiProperties. Implemented configuration of encryption services from NiFiProperties. Refactored NiFiPropertiesLoader functionality to CryptoUtils for availability in other modules. Added RepositoryEncryptionConfiguration and repo-specific subclasses for data containers. Continued refactoring of CryptoUtils and RepositoryEncryptorUtils library methods. Exposed some internal state of FileSystemRepository via protected getters so encrypted implementation could access. Refactored EncryptedFileSystemRepository to extend rather than duplicate FSR. Refactored EFSR to use ECROS which now extends extracted ContentRepositoryOutputStream protected inner class in FSR. Added unit test to encrypt & decrypt image resource. Added smaller image resource for easier unit test debugging. Added importFrom method to resolve issue where GetFile would not encrypt content persisted to repository. Added text test resource for tests around exporting claim subsets. Added exportTo methods to handle decrypting encrypted content. Performed large unit test refactoring, moving shared logic to helper methods. Added unit test for merged content claim with header/footer/demarcator. Added unit test for merging content claims each encrypted with a different key. Ignored non-deterministically failing firewall DNS test. Added documentation to User and Admin Guide for Encrypted Content Repository. Added image. Added refactored utility method for shared ROEM extraction and validation logic in AbstractAESEncryptor. Replaced ad-hoc generation of ciphertext stream and byte[] for testing with static initialization from pre-generated serialized form for performance. Cleaned up unused test code. Cleaned up Javadoc and code comments. Refactored shared logic. Fixed checkstyle issue. Fixed test failure due to error message change. Added experimental warning to repository implementation classes and User Guide documentation. Signed-off-by: Joe Witt <joewitt@apache.org>
…impls (apache#3826) * NIFI-6780: Introduce RecordSinkService to separate format and destination, refactor common S2S utils Added QueryNiFiReportingTask to SQL query NiFi status and metrics Add PROCESSOR_STATUS and PROCESS_GROUP_STATUS Add CONNECTION_STATUS_PREDICTIONS check for null predictions Add ConnectionStatusRecursiveIterator Fix issue w/ duplicate iterator outputs Refactored query interfaces, fixed assembly POM Rebased v master, fixed isBackPressureEnabled and Checkstyle/RAT errors Add DatabaseRecordSink service (apache#13) * Add DatabaseRecordSink service * Incorporated review comments * NIFI-6780: Add/fix docs, cleanup warnings, fixed some table definitions * Added bundle profile, remove predictions table if not enabled * Added doc for which tables are available when This closes apache#3826.
… nifi-nar-bundle pom
|
@joewitt all conflicts and build issues resolved. |
|
@rkarthik29 Can you please consolidate this PR? |
|
#3858, open a new PR, closing this one |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically master)?
Is your initial contribution a single, squashed commit?
For code changes:
For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.