NIFI-6032 - CDC for Oracle using xstream

[rkarthik29]

Thank you for submitting a contribution to Apache NiFi.

Please provide a short description of the PR here:

Description of PR

Enables X functionality; fixes bug NIFI-YYYY.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

• Is there a JIRA ticket associated with this PR? Is it referenced
  in the commit message?

• Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

• Has your PR been rebased against the latest commit within the target branch (typically master)?

• Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

• Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
• Have you written or updated unit tests to verify your changes?
• Have you verified that the full build is successful on both JDK 8 and JDK 11?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
• If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
• If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

[shivamkh90]

Eagerly waiting for this feature 👍

[pvillard31]

@shivamkh90 - that would greatly help if you can checkout this pull request, give it a try and report your results here. An extensive set of tests with your flow definitions and screenshots of the results and expectations would help others to also chime in. We have tons of PR and we need help from the community to get the PRs merged into NiFi.

[davyam]

@rkarthik29 Thanks for creating this PR!
ASAP I will test this contribution and report the results.

[davyam]

@rkarthik29
I built this PR, but the OracleChangeCapture did not appear on the processor list. Commands executed:

git clone https://github.com/rkarthik29/nifi-1.git 
cd nifi-1
git checkout NIFI-6032
export MAVEN_OPTS="-Xms1024m -Xmx3076m -XX:MaxPermSize=256m"
mvn -T C2.0 clean install -Pinclude-grpc
cd nifi-assembly/target/
tar -xvzf nifi-1.11.0-SNAPSHOT-bin.tar.gz
sh nifi-1.11.0-SNAPSHOT/bin/nifi.sh run

So I had to copy the nars from nifi-1/nifi-nar-bundles/nifi-oraclecdc-bundle to the nifi-1.11.0-SNAPSHOT/lib directory manually.

Please, check the parent pom.xml.

[rkarthik29]

@rkarthik29
I built this PR, but the OracleChangeCapture did not appear on the processor list. Commands executed:

git clone https://github.com/rkarthik29/nifi-1.git 
cd nifi-1
git checkout NIFI-6032
export MAVEN_OPTS="-Xms1024m -Xmx3076m -XX:MaxPermSize=256m"
mvn -T C2.0 clean install -Pinclude-grpc
cd nifi-assembly/target/
tar -xvzf nifi-1.11.0-SNAPSHOT-bin.tar.gz
sh nifi-1.11.0-SNAPSHOT/bin/nifi.sh run

So I had to copy the nars from nifi-1/nifi-nar-bundles/nifi-oraclecdc-bundle to the nifi-1.11.0-SNAPSHOT/lib directory manually.

Please, check the parent pom.xml.

@davyam

There is a profile config in the nifi parent pom.xml. Please add -p include-oraclecdc to maven command to package this piece as part of nifi tar.

include-oraclecdc false org.apache.nifi nifi-oraclecdc-nar 1.11.0-SNAPSHOT nar org.apache.nifi nifi-oraclecdcservice-api-nar 1.11.0-SNAPSHOT nar

[davyam]

Hi @rkarthik29

I am getting the following error:
org.apache.nifi.processor.exception.ProcessException: cannot attach to outbound server: nfxout

Steps that I did:

BUILD NIFI

git clone https://github.com/rkarthik29/nifi-1.git -b NIFI-6032

cd nifi-1

export MAVEN_OPTS="-Xms1024m -Xmx3076m -XX:MaxPermSize=256m"

mvn -T C2.0 clean install -Pinclude-grpc -Pinclude-oraclecdc

# Error:
[ERROR] Failed to execute goal on project nifi-splunk-processors: Could not resolve dependencies for project org.apache.nifi:nifi-splunk-processors...Could not find artifact com.splunk:splunk:jar:1.6.5.0 in splunk...

# Solution:
https://feitam.es/nifi-1-11-4-error-maven-could-not-resolve-dependencies-update-interval

mvn -T C2.0 clean install -Pinclude-grpc -Pinclude-oraclecdc -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -Dmaven.wagon.http.ssl.ignore.validity.dates=true

cd nifi-assembly/target/

tar -xvzf nifi-1.11.0-SNAPSHOT-bin.tar.gz

sh nifi-1.11.0-SNAPSHOT/bin/nifi.sh run

INSTALL ORACLE DRIVER

# Download driver:
https://www.oracle.com/br/database/technologies/instant-client/linux-x86-64-downloads.html

yum install oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm

CONFIG LIBS

vim ~./bash_profile
    ...
    export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib:$LD_LIBRARY_PATH

source ~./bash_profile

vim nifi-1.11.0-SNAPSHOT/conf/bootstrap.conf
    ...
    java.arg.17=-Djava.library.path=/usr/lib/oracle/12.2/client64/lib
    ...

sh nifi-1.11.0-SNAPSHOT/bin/nifi.sh run

CREATE ORACLE XSTREAM OUTBOUND SERVER

# Create Recovery Area dir:
mkdir -p /opt/oracle/oradata/recovery_area

chown -R oracle: /opt/oracle

# Enable GoldenGate replication and archive log mode:
su - oracle

sqlplus /nolog

CONNECT sys AS SYSDBA

alter system set db_recovery_file_dest_size = 5G;

alter system set db_recovery_file_dest = '/opt/oracle/oradata/recovery_area' scope=spfile;

alter system set enable_goldengate_replication=true;

shutdown immediate

startup mount

alter database archivelog;

alter database open;

# Should show "Database log mode: Archive Mode":
archive log list

exit;

# Create Tablespace dir:
mkdir -p /opt/oracle/oradata/tbs

# Create an XStream admin user:
sqlplus / as sysdba

CREATE TABLESPACE xstream_adm_tbs DATAFILE '/opt/oracle/oradata/tbs/xstream_adm_tbs.dbf'
  SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;

CREATE USER xstrmadmin IDENTIFIED BY xsa
  DEFAULT TABLESPACE xstream_adm_tbs
  QUOTA UNLIMITED ON xstream_adm_tbs;

GRANT CREATE SESSION TO xstrmadmin;

BEGIN
   DBMS_XSTREAM_AUTH.GRANT_ADMIN_PRIVILEGE(
      grantee                 => 'xstrmadmin',
      privilege_type          => 'CAPTURE',
      grant_select_privileges => TRUE
   );
END;
/

exit;

# Create XStream user:
sqlplus / as sysdba

CREATE TABLESPACE xstream_tbs DATAFILE '/opt/oracle/oradata/tbs/xstream_tbs.dbf'
  SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;

CREATE USER xstrm IDENTIFIED BY xs
  DEFAULT TABLESPACE xstream_tbs
  QUOTA UNLIMITED ON xstream_tbs;

GRANT CREATE SESSION TO xstrm;
GRANT SELECT ON V_$DATABASE to xstrm;
GRANT FLASHBACK ANY TABLE TO xstrm;

exit; 

# Create NiFi user:
sqlplus / as sysdba

CREATE TABLESPACE nifi_tbs DATAFILE '/opt/oracle/oradata/tbs/nifi_tbs.dbf'
  SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;

CREATE USER nifi IDENTIFIED BY nifi
  DEFAULT TABLESPACE nifi_tbs
  QUOTA UNLIMITED ON nifi_tbs;

CREATE TABLE nifi.tb_teste (id NUMBER PRIMARY KEY, name VARCHAR2(150));

ALTER TABLE nifi.tb_teste ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;

exit; 

# Create an XStream Outbound server:
sqlplus xstrmadmin/xsa

DECLARE
  tables  DBMS_UTILITY.UNCL_ARRAY;
  schemas DBMS_UTILITY.UNCL_ARRAY;
BEGIN
    tables(1)  := NULL;
    schemas(1) := 'nifi';
  DBMS_XSTREAM_ADM.CREATE_OUTBOUND(
    server_name     =>  'nfxout',
    table_names     =>  tables,
    schema_names    =>  schemas);
END;
/

exit;

# Alter the XStream Outbound server to allow the xstrm user to connect to it:
sqlplus / as sysdba

BEGIN
  DBMS_XSTREAM_ADM.ALTER_OUTBOUND(
    server_name  => 'nfxout',
    connect_user => 'xstrm');
END;
/

exit;

# Insert data:
INSERT INTO nifi.tb_teste VALUES (1, 'Davy');

NiFi Processor:

NiFi Services:

Oracle views:

NiFi log:

2020-10-16 23:39:24,282 INFO [NiFi Web Server-110] o.a.n.c.s.StandardProcessScheduler Starting OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51]
2020-10-16 23:39:24,284 INFO [NiFi Web Server-110] o.a.n.controller.StandardProcessorNode Starting OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51]
2020-10-16 23:39:24,381 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@25224e98 // Another save pending = false
2020-10-16 23:40:03,611 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2020-10-16 23:40:03,612 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2020-10-16 23:40:03,774 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@3d5b2c1e checkpointed with 0 Records and 0 Swap Files in 23 milliseconds (Stop-the-world time = 5 milliseconds, Clear Edit Logs time = 4 millis), max Transaction ID -1
2020-10-16 23:40:24,290 WARN [Monitor Processor Lifecycle Thread-1] o.a.n.controller.StandardProcessorNode Timed out while waiting for OnScheduled of OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51] to finish. An attempt is made to cancel the task via Thread.interrupt(). However it does not guarantee that the task will be canceled since the code inside current OnScheduled operation may have been written to ignore interrupts which may result in a runaway thread. This could lead to more issues, eventually requiring NiFi to be restarted. This is usually a bug in the target Processor 'OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51]' that needs to be documented, reported and eventually fixed.
2020-10-16 23:40:24,291 ERROR [Timer-Driven Process Thread-1] o.a.n.p.o.c.i.StandardOracleCDCService StandardOracleCDCService[id=32c9c2cd-0175-1000-c663-73fb836e27c7] cannot attach to outbound server: nfxout java.sql.SQLException: Cannot get a connection, general error
2020-10-16 23:40:24,293 ERROR [Timer-Driven Process Thread-1] o.a.n.p.oraclecdc.OracleChangeCapture OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to org.apache.nifi.processor.exception.ProcessException: cannot attach to outbound server: nfxout: org.apache.nifi.processor.exception.ProcessException: cannot attach to outbound server: nfxout
org.apache.nifi.processor.exception.ProcessException: cannot attach to outbound server: nfxout
	at org.apache.nifi.processors.oraclecdc.controller.impl.StandardOracleCDCService.attach(StandardOracleCDCService.java:198)
	at sun.reflect.GeneratedMethodAccessor400.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
	at com.sun.proxy.$Proxy89.attach(Unknown Source)
	at org.apache.nifi.processors.oraclecdc.OracleChangeCapture.onScheduled(OracleChangeCapture.java:137)
	at sun.reflect.GeneratedMethodAccessor399.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
	at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1532)
	at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2020-10-16 23:40:24,296 WARN [Timer-Driven Process Thread-1] o.a.n.p.o.c.i.StandardOracleCDCService StandardOracleCDCService[id=32c9c2cd-0175-1000-c663-73fb836e27c7] cannot detach from the outbound server: oracle.streams.XStreamOut
2020-10-16 23:40:24,297 ERROR [Timer-Driven Process Thread-1] org.apache.nifi.util.ReflectionUtils Failed while invoking annotated method 'public void org.apache.nifi.processors.oraclecdc.OracleChangeCapture.shutdown(org.apache.nifi.processor.ProcessContext)' with arguments '[org.apache.nifi.processor.StandardProcessContext@1d5c0a7c]'.
java.lang.reflect.InvocationTargetException: null
	at sun.reflect.GeneratedMethodAccessor402.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
	at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotations(ReflectionUtils.java:268)
	at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotation(ReflectionUtils.java:90)
	at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1576)
	at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.nifi.processor.exception.ProcessException: cannot detach from the outbound server: oracle.streams.XStreamOut
	at org.apache.nifi.processors.oraclecdc.controller.impl.StandardOracleCDCService.detach(StandardOracleCDCService.java:210)
	at sun.reflect.GeneratedMethodAccessor403.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
	at com.sun.proxy.$Proxy89.detach(Unknown Source)
	at org.apache.nifi.processors.oraclecdc.OracleChangeCapture.shutdown(OracleChangeCapture.java:143)
	... 15 common frames omitted
2020-10-16 23:41:45,272 INFO [NiFi Web Server-103] o.a.n.c.s.StandardProcessScheduler Stopping OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51]
2020-10-16 23:41:45,273 INFO [NiFi Web Server-103] o.a.n.controller.StandardProcessorNode Stopping processor: OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51]
2020-10-16 23:41:45,455 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@25224e98 // Another save pending = false
2020-10-16 23:41:54,308 WARN [Monitor Processor Lifecycle Thread-2] o.a.n.controller.StandardProcessorNode Timed out while waiting for OnScheduled of OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51] to finish. An attempt is made to cancel the task via Thread.interrupt(). However it does not guarantee that the task will be canceled since the code inside current OnScheduled operation may have been written to ignore interrupts which may result in a runaway thread. This could lead to more issues, eventually requiring NiFi to be restarted. This is usually a bug in the target Processor 'OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51]' that needs to be documented, reported and eventually fixed.
2020-10-16 23:41:54,308 ERROR [Timer-Driven Process Thread-2] o.a.n.p.o.c.i.StandardOracleCDCService StandardOracleCDCService[id=32c9c2cd-0175-1000-c663-73fb836e27c7] cannot attach to outbound server: nfxout java.sql.SQLException: Cannot get a connection, general error
2020-10-16 23:41:54,309 ERROR [Timer-Driven Process Thread-2] o.a.n.p.oraclecdc.OracleChangeCapture OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to org.apache.nifi.processor.exception.ProcessException: cannot attach to outbound server: nfxout: org.apache.nifi.processor.exception.ProcessException: cannot attach to outbound server: nfxout
org.apache.nifi.processor.exception.ProcessException: cannot attach to outbound server: nfxout
	at org.apache.nifi.processors.oraclecdc.controller.impl.StandardOracleCDCService.attach(StandardOracleCDCService.java:198)
	at sun.reflect.GeneratedMethodAccessor400.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
	at com.sun.proxy.$Proxy89.attach(Unknown Source)
	at org.apache.nifi.processors.oraclecdc.OracleChangeCapture.onScheduled(OracleChangeCapture.java:137)
	at sun.reflect.GeneratedMethodAccessor399.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
	at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1532)
	at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2020-10-16 23:41:54,312 WARN [Timer-Driven Process Thread-2] o.a.n.p.o.c.i.StandardOracleCDCService StandardOracleCDCService[id=32c9c2cd-0175-1000-c663-73fb836e27c7] cannot detach from the outbound server: oracle.streams.XStreamOut
2020-10-16 23:41:54,312 ERROR [Timer-Driven Process Thread-2] org.apache.nifi.util.ReflectionUtils Failed while invoking annotated method 'public void org.apache.nifi.processors.oraclecdc.OracleChangeCapture.shutdown(org.apache.nifi.processor.ProcessContext)' with arguments '[org.apache.nifi.processor.StandardProcessContext@3803bfd3]'.
java.lang.reflect.InvocationTargetException: null
	at sun.reflect.GeneratedMethodAccessor402.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
	at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotations(ReflectionUtils.java:268)
	at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotation(ReflectionUtils.java:90)
	at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1576)
	at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.nifi.processor.exception.ProcessException: cannot detach from the outbound server: oracle.streams.XStreamOut
	at org.apache.nifi.processors.oraclecdc.controller.impl.StandardOracleCDCService.detach(StandardOracleCDCService.java:210)
	at sun.reflect.GeneratedMethodAccessor403.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
	at com.sun.proxy.$Proxy89.detach(Unknown Source)
	at org.apache.nifi.processors.oraclecdc.OracleChangeCapture.shutdown(OracleChangeCapture.java:143)
	... 15 common frames omitted
2020-10-16 23:42:03,613 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2020-10-16 23:42:03,613 INFO [pool-12-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2020-10-16 23:42:03,796 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@3d5b2c1e checkpointed with 0 Records and 0 Swap Files in 21 milliseconds (Stop-the-world time = 5 milliseconds, Clear Edit Logs time = 4 millis), max Transaction ID -1

Is something wrong?

[rkarthik29]

@davyam

looks like there is an issue with getConnection(). is it probably timing out???? this is the error log

o.a.n.controller.StandardProcessorNode Timed out while waiting for OnScheduled of OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51] to finish. An attempt is made to cancel the task via Thread.interrupt(). However it does not guarantee that the task will be canceled since the code inside current OnScheduled operation may have been written to ignore interrupts which may result in a runaway thread. This could lead to more issues, eventually requiring NiFi to be restarted. This is usually a bug in the target Processor 'OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51]' that needs to be documented, reported and eventually fixed.
2020-10-16 23:40:24,291 ERROR [Timer-Driven Process Thread-1] o.a.n.p.o.c.i.StandardOracleCDCService StandardOracleCDCService[id=32c9c2cd-0175-1000-c663-73fb836e27c7] cannot attach to outbound server: nfxout java.sql.SQLException: Cannot get a connection, general error
2020-10-16 23:40:24,293 ERROR [Timer-Driven Process Thread-1] o.a.n.p.oraclecdc.OracleChangeCapture OracleChangeCapture[id=32c99420-0175-1000-0e97-840ff38cca51] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to org.apache.nifi.processor.exception.ProcessException: cannot attach to outbound

that error seems to come from DBCPConectionPool. Best way to figure out is to run the ITOracleChangeCaptureTest via eclipse,you can then put a breakpoint in standardoraclecdcservice.attach() to get more details on why this is happening. This is probably needs to be handled.. https://issues.apache.org/jira/browse/DBCP-459. You have to investigate why that timeout is happening though.

Make sure you are using a thick client for connection to oracle. I don't think that is the cause of this error, but ensure that you are doing that.

Karthik

[rkarthik29]

@davyam
there is some instructions on how to set the stream server here https://github.com/rkarthik29/oracle_cdc
You also need the oracle native libraries included in the java_library_path. You can set that in nifi bootstrap.conf. Dont the error is related to this as you will get the error at a later step further down, right around the attach invocation on xstreamout.

Karthik

[davyam]

@davyam
there is some instructions on how to set the stream server here https://github.com/rkarthik29/oracle_cdc
You also need the oracle native libraries included in the java_library_path. You can set that in nifi bootstrap.conf. Dont the error is related to this as you will get the error at a later step further down, right around the attach invocation on xstreamout.

Karthik

Hi @rkarthik29

The instructions that I followed were almost the same, but I will try again following yours.

About the libraries, they are defined in bootstrap.conf, I just forget to show you in my previous comment.

Thanks for the quick answers.

Davy Machado

[davyam]

Hi @rkarthik29

I got it! It is working!

I think that the problem was the StandardOracleClassLoaderService config. I changed the value of Database Driver Location(s) to /usr/lib/oracle/12.2/client64/lib, instead of the jar path.

Now I will do more tests.

Thanks!

Davy Machado

[github-actions]

We're marking this PR as stale due to lack of updates in the past few months. If after another couple of weeks the stale label has not been removed this PR will be closed. This stale marker and eventual auto close does not indicate a judgement of the PR just lack of reviewer bandwidth and helps us keep the PR queue more manageable. If you would like this PR re-opened you can do so and a committer can remove the stale tag. Or you can open a new PR. Try to help review other PRs to increase PR review bandwidth which in turn helps yours.