Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [Oracle-CDC] Can not find catalog table with factoryId [Oracle] #6680

Closed
3 tasks done
luzongzhu opened this issue Apr 10, 2024 · 9 comments
Closed
3 tasks done
Labels

Comments

@luzongzhu
Copy link
Contributor

luzongzhu commented Apr 10, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

Oracle-CDC sink Doris,
The following exception occurs

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:199)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'Oracle-CDC'.
	at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:100)
	at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:320)
	at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:181)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:161)
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:146)
	... 2 more
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: Can not find catalog table with factoryId [Oracle]
	at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.lambda$getCatalogTables$0(CatalogTableUtil.java:129)
	at java.util.Optional.map(Optional.java:215)
	at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.getCatalogTables(CatalogTableUtil.java:116)
	at org.apache.seatunnel.api.table.catalog.CatalogTableUtil.getCatalogTables(CatalogTableUtil.java:96)
	at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSourceFactory.lambda$createSource$1(OracleIncrementalSourceFactory.java:108)
	at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:112)
	at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:73)
	... 7 more

I checked the source code and found that there was an issue with the matching rule of TablePath. After modifying the source code and deploying it, a new error occurred,see details [Error Exception]

image

SeaTunnel Version

2.3.4

SeaTunnel Config

{
	    "env" : {
	        "job.name" : "1777942783487188993",
	        "job.mode" : "STREAMING",
	        "checkpoint.interval" : 5000,
	        "execution.parallelism" : 1
	    },
	    "source" : [
	        {
	            "base-url" : "jdbc:oracle:thin:@//xxxx:1531/test101",
	            "password" : "oracle",
	            "table-names" : [
	                "test101.CUX.CUX_CREDIT_FINANCE230207"
	            ],
	            "result_table_name" : "ST_GLOBAL_VIEW_NAME",
	            "database-names" : [
	                "test101"
	            ],
	            "schema-names" : [
	                "CUX"
	            ],
	            "source.reader.close.timeout" : 120000,
	            "plugin_name" : "Oracle-CDC",
	            "username" : "logminer_user"
	        }
	    ],
	    "transform" : [],
	    "sink" : [
	        {
	            "password" : "root123456",
	            "sink.enable-2pc" : "false",
	            "fenodes" : "xxx:8060",
	            "doris.config" : {
	                "format" : "json",
	                "read_json_by_line" : "true"
	            },
	            "table.identifier" : "cdc.CUX_CREDIT_FINANCE230207",
	            "source_table_name" : "ST_GLOBAL_VIEW_NAME",
	            "sink.enable-delete" : "true",
	            "plugin_name" : "Doris",
	            "sink.label-prefix" : "sink",
	            "username" : "root"
	        }
	    ]
	}

Running Command

$SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/oracle-cdc-demo.conf

Error Exception

[LOG-PATH]: /opt/dolphinscheduler/logs/20240410/13220904650080_60-309-388.log, [HOST]:  Host{address='172.20.0.23:1234', ip='172.20.0.23', port=1234}
[INFO] 2024-04-10 16:32:24.617 +0800 - Begin to pulling task
[INFO] 2024-04-10 16:32:24.619 +0800 - Begin to initialize task
[INFO] 2024-04-10 16:32:24.619 +0800 - Set task startTime: Wed Apr 10 16:32:24 CST 2024
[INFO] 2024-04-10 16:32:24.619 +0800 - Set task envFile: /opt/dolphinscheduler/conf/dolphinscheduler_env.sh
[INFO] 2024-04-10 16:32:24.619 +0800 - Set task appId: 309_388
[INFO] 2024-04-10 16:32:24.619 +0800 - End initialize task
[INFO] 2024-04-10 16:32:24.620 +0800 - Set task status to TaskExecutionStatus{code=1, desc='running'}
[INFO] 2024-04-10 16:32:24.620 +0800 - TenantCode:admin check success
[INFO] 2024-04-10 16:32:24.621 +0800 - ProcessExecDir:/tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388 check success

[INFO] 2024-04-10 16:32:24.670 +0800 - tenantCode :admin, task dir:/tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388
[INFO] 2024-04-10 16:32:24.670 +0800 - generate script file:/tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388/seatunnel_309_388.conf
[INFO] 2024-04-10 16:32:24.670 +0800 - SeaTunnel task command: ${SEATUNNEL_HOME}/bin/seatunnel.sh --config /tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388/seatunnel_309_388.conf
[INFO] 2024-04-10 16:32:24.671 +0800 - log path /opt/dolphinscheduler/logs/20240410/13220904650080_60-309-388.log,task log name taskAppId=TASK-20240410-13220904650080_60-309-388
[INFO] 2024-04-10 16:32:24.671 +0800 - Begin to create command file:/tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388/309_388.command
[INFO] 2024-04-10 16:32:24.671 +0800 - Success create command file, command: #!/bin/bash
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /opt/dolphinscheduler/conf/dolphinscheduler_env.sh
${SEATUNNEL_HOME}/bin/seatunnel.sh --config /tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388/seatunnel_309_388.conf
[INFO] 2024-04-10 16:32:24.674 +0800 - task run command: sudo -u admin -E bash /tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388/309_388.command
[INFO] 2024-04-10 16:32:24.676 +0800 - process start, process id is: 14149
[INFO] 2024-04-10 16:32:25.678 +0800 -  -> Apr 10, 2024 4:32:25 PM com.hazelcast.internal.config.AbstractConfigLocator
	INFO: Loading configuration '/opt/soft/seatunnel/config/seatunnel.yaml' from System property 'seatunnel.config'
	Apr 10, 2024 4:32:25 PM com.hazelcast.internal.config.AbstractConfigLocator
	INFO: Using configuration file at /opt/soft/seatunnel/config/seatunnel.yaml
	Apr 10, 2024 4:32:25 PM org.apache.seatunnel.engine.common.config.SeaTunnelConfig
	INFO: seatunnel.home is /tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388
	Apr 10, 2024 4:32:25 PM com.hazelcast.internal.config.AbstractConfigLocator
	INFO: Loading configuration '/opt/soft/seatunnel/config/hazelcast.yaml' from System property 'hazelcast.config'
	Apr 10, 2024 4:32:25 PM com.hazelcast.internal.config.AbstractConfigLocator
	INFO: Using configuration file at /opt/soft/seatunnel/config/hazelcast.yaml
[INFO] 2024-04-10 16:32:26.680 +0800 -  -> Apr 10, 2024 4:32:26 PM com.hazelcast.internal.config.AbstractConfigLocator
	INFO: Loading configuration '/opt/soft/seatunnel/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
	Apr 10, 2024 4:32:26 PM com.hazelcast.internal.config.AbstractConfigLocator
	INFO: Using configuration file at /opt/soft/seatunnel/config/hazelcast-client.yaml
	2024-04-10 16:32:26,582 INFO  [.c.i.s.ClientInvocationService] [main] - hz.client_1 [seatunnel] [5.1] Running with 2 response threads, dynamic=true
[INFO] 2024-04-10 16:32:27.681 +0800 -  -> 2024-04-10 16:32:26,682 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
	2024-04-10 16:32:26,683 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
	2024-04-10 16:32:26,720 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Trying to connect to cluster: seatunnel
	2024-04-10 16:32:26,724 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Trying to connect to [10.16.115.18]:5801
	2024-04-10 16:32:26,780 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
	2024-04-10 16:32:26,780 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Authenticated with server [10.16.115.18]:5801:d5193958-855f-4dac-b979-027fc961723b, server version: 5.1, local address: /172.20.0.23:48337
	2024-04-10 16:32:26,782 INFO  [c.h.i.d.Diagnostics           ] [main] - hz.client_1 [seatunnel] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
	2024-04-10 16:32:26,811 INFO  [c.h.c.i.s.ClientClusterService] [hz.client_1.event-5] - hz.client_1 [seatunnel] [5.1] 
	
	Members [1] {
		Member [10.16.115.18]:5801 - d5193958-855f-4dac-b979-027fc961723b
	}
	
	2024-04-10 16:32:26,845 INFO  [.c.i.s.ClientStatisticsService] [main] - Client statistics is enabled with period 5 seconds.
	2024-04-10 16:32:27,079 INFO  [.ClientJobExecutionEnvironment] [main] - add common jar in plugins :[]
	2024-04-10 16:32:27,095 INFO  [o.a.s.c.s.u.ConfigBuilder     ] [main] - Loading config file from path: /tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388/seatunnel_309_388.conf
	2024-04-10 16:32:27,202 INFO  [o.a.s.c.s.u.ConfigShadeUtils  ] [main] - Load config shade spi: [base64]
	2024-04-10 16:32:27,291 INFO  [o.a.s.c.s.u.ConfigBuilder     ] [main] - Parsed config file: 
	{
	    "env" : {
	        "job.name" : "1777977922728566785",
	        "job.mode" : "STREAMING",
	        "checkpoint.interval" : 5000,
	        "execution.parallelism" : 1
	    },
	    "source" : [
	        {
	            "base-url" : "jdbc:oracle:thin:@//xxxxx:1531/TEST101",
	            "password" : "oracle",
	            "exactly_once" : true,
	            "table-names" : [
	                "CUX.CUX_CREDIT_FINANCE230207"
	            ],
	            "result_table_name" : "ST_GLOBAL_VIEW_NAME",
	            "database-names" : [
	                "TEST101"
	            ],
	            "schema-names" : [
	                "CUX"
	            ],
	            "source.reader.close.timeout" : 120000,
	            "plugin_name" : "Oracle-CDC",
	            "table-names-config" : [
	                {
	                    "table" : "CUX.CUX_CREDIT_FINANCE230207",
	                    "primaryKeys" : [
	                        "FINANCE_ID"
	                    ]
	                }
	            ],
	            "username" : "logminer_user"
	        }
	    ],
	    "transform" : [],
	    "sink" : [
	        {
	            "password" : "root123456",
	            "sink.enable-2pc" : "false",
	            "fenodes" : "xxxxx:8060",
	            "doris.config" : {
	                "format" : "json",
	                "read_json_by_line" : "true"
	            },
	            "table.identifier" : "cdc.CUX_CREDIT_FINANCE230207",
	            "source_table_name" : "ST_GLOBAL_VIEW_NAME",
	            "sink.enable-delete" : "true",
	            "plugin_name" : "Doris",
	            "sink.label-prefix" : "sink",
	            "username" : "root"
	        }
	    ]
	}
	
	2024-04-10 16:32:27,308 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /opt/soft/seatunnel/connectors
	2024-04-10 16:32:27,317 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Oracle-CDC'} at: file:/opt/soft/seatunnel/connectors/connector-cdc-oracle-2.3.4.jar
	2024-04-10 16:32:27,318 INFO  [.s.p.d.AbstractPluginDiscovery] [main] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Doris'} at: file:/opt/soft/seatunnel/connectors/connector-doris-2.3.4.jar
	2024-04-10 16:32:27,322 WARN  [o.a.s.a.c.u.ConfigUtil        ] [main] - Option 'source_table_name' is a List, and it is recommended to configure it as ["string1","string2"]; we will only use ',' to split the String into a list.
	2024-04-10 16:32:27,327 INFO  [p.MultipleTableJobConfigParser] [main] - start generating all sources.
[INFO] 2024-04-10 16:32:28.682 +0800 -  -> 2024-04-10 16:32:28,334 INFO  [.s.c.s.j.c.AbstractJdbcCatalog] [main] - Catalog Oracle established connection to jdbc:oracle:thin:@//xxxxx:1531/TEST101
[INFO] 2024-04-10 16:32:30.682 +0800 -  -> 2024-04-10 16:32:30,147 INFO  [o.a.s.a.t.c.CatalogTableUtil  ] [main] - Get catalog tables, cost time: 2763
	2024-04-10 16:32:30,151 INFO  [.s.c.s.j.c.AbstractJdbcCatalog] [main] - Catalog Oracle closing
	2024-04-10 16:32:30,196 WARN  [.a.s.c.c.b.u.CatalogTableUtils] [main] - Table CUX.CUX_CREDIT_FINANCE230207 is not found in catalog tables, skip to merge config
	2024-04-10 16:32:30,409 INFO  [i.d.c.o.OracleConnection      ] [main] - Database Version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
[INFO] 2024-04-10 16:34:15.696 +0800 -  -> 2024-04-10 16:34:14,990 INFO  [.s.c.o.u.OracleConnectionUtils] [main] - 	 including 'TEST101.CUX.CUX_CREDIT_FINANCE230207' for further processing
	2024-04-10 16:34:15,026 INFO  [i.d.j.JdbcConnection          ] [pool-2-thread-1] - Connection gracefully closed
	2024-04-10 16:34:15,123 INFO  [i.d.c.o.OracleConnection      ] [main] - Database Version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
[INFO] 2024-04-10 16:34:17.779 +0800 - process has exited. execute path:/tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388, processId:14149 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1
[INFO] 2024-04-10 16:34:17.782 +0800 - Send task execute result to master, the current task status: TaskExecutionStatus{code=6, desc='failure'}
[INFO] 2024-04-10 16:34:17.782 +0800 - Remove the current task execute context from worker cache
[INFO] 2024-04-10 16:34:17.782 +0800 - The current execute mode isn't develop mode, will clear the task execute file: /tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388
[INFO] 2024-04-10 16:34:17.783 +0800 - Success clear the task execute file: /tmp/dolphinscheduler/exec/process/admin/10633993590912/13220904650080_60/309/388
[INFO] 2024-04-10 16:34:18.697 +0800 -  -> 2024-04-10 16:34:17,718 INFO  [i.d.j.JdbcConnection          ] [pool-3-thread-1] - Connection gracefully closed
	2024-04-10 16:34:17,721 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
	2024-04-10 16:34:17,726 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [10.16.115.18]:5801:d5193958-855f-4dac-b979-027fc961723b, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/172.20.0.23:48337->/10.16.115.18:5801}, remoteAddress=[10.16.115.18]:5801, lastReadTime=2024-04-10 16:34:16.851, lastWriteTime=2024-04-10 16:34:16.849, closedTime=2024-04-10 16:34:17.724, connected server version=5.1}
	2024-04-10 16:34:17,727 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
	2024-04-10 16:34:17,729 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
	2024-04-10 16:34:17,729 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
	2024-04-10 16:34:17,730 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - 
	
	===============================================================================
	
	
	2024-04-10 16:34:17,730 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Fatal Error, 
	
	2024-04-10 16:34:17,730 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues
	
	2024-04-10 16:34:17,730 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Reason:SeaTunnel job executed failed 
	
	2024-04-10 16:34:17,732 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
		at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:199)
		at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
		at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
	Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'Oracle-CDC'.
		at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:100)
		at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:320)
		at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:181)
		at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
		at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:161)
		at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:146)
		... 2 more
	Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: java.lang.NullPointerException
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSource.tableChanges(OracleIncrementalSource.java:155)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSource.createDebeziumDeserializationSchema(OracleIncrementalSource.java:104)
		at org.apache.seatunnel.connectors.cdc.base.source.IncrementalSource.<init>(IncrementalSource.java:114)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSource.<init>(OracleIncrementalSource.java:70)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSourceFactory.lambda$createSource$1(OracleIncrementalSourceFactory.java:119)
		at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:112)
		at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:73)
		... 7 more
	Caused by: java.lang.NullPointerException
		at org.apache.seatunnel.connectors.cdc.base.utils.CatalogTableUtils.mergeCatalogTableConfig(CatalogTableUtils.java:121)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.utils.OracleSchema.readTableSchema(OracleSchema.java:80)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.utils.OracleSchema.getTableSchema(OracleSchema.java:59)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleDialect.queryTableSchema(OracleDialect.java:116)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSource.lambda$tableChanges$0(OracleIncrementalSource.java:148)
		at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
		at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
		at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
		at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
		at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
		at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
		at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
		at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSource.tableChanges(OracleIncrementalSource.java:142)
		... 13 more
	 
	2024-04-10 16:34:17,732 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - 
	===============================================================================
	
	
	
	Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
		at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:199)
		at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
		at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
	Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'Oracle-CDC'.
		at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:100)
		at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:320)
		at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:181)
		at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
		at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:161)
		at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:146)
		... 2 more
	Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: java.lang.NullPointerException
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSource.tableChanges(OracleIncrementalSource.java:155)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSource.createDebeziumDeserializationSchema(OracleIncrementalSource.java:104)
		at org.apache.seatunnel.connectors.cdc.base.source.IncrementalSource.<init>(IncrementalSource.java:114)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSource.<init>(OracleIncrementalSource.java:70)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSourceFactory.lambda$createSource$1(OracleIncrementalSourceFactory.java:119)
		at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:112)
		at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:73)
		... 7 more
	Caused by: java.lang.NullPointerException
		at org.apache.seatunnel.connectors.cdc.base.utils.CatalogTableUtils.mergeCatalogTableConfig(CatalogTableUtils.java:121)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.utils.OracleSchema.readTableSchema(OracleSchema.java:80)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.utils.OracleSchema.getTableSchema(OracleSchema.java:59)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleDialect.queryTableSchema(OracleDialect.java:116)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSource.lambda$tableChanges$0(OracleIncrementalSource.java:148)
		at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
		at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
		at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
		at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
		at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
		at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
		at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
		at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
		at org.apache.seatunnel.connectors.seatunnel.cdc.oracle.source.OracleIncrementalSource.tableChanges(OracleIncrementalSource.java:142)
		... 13 more
[INFO] 2024-04-10 16:34:18.698 +0800 - FINALIZE_SESSION

Zeta or Flink or Spark Version

Zeta 2.3.4

Java or Scala Version

1.8

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@luzongzhu luzongzhu added the bug label Apr 10, 2024
@luzongzhu
Copy link
Contributor Author

luzongzhu commented Apr 10, 2024

The new configuration after modifying the source code ,It is a table without primary key @hailin0

{
    "env" : {
        "job.name" : "1777977922728566785",
        "job.mode" : "STREAMING",
        "checkpoint.interval" : 5000,
        "execution.parallelism" : 1
    },
    "source" : [
        {
            "base-url" : "jdbc:oracle:thin:@//xxxxx:1531/TEST101",
            "password" : "oracle",
            "exactly_once" : true,
            "table-names" : [
                "CUX.CUX_CREDIT_FINANCE230207"
            ],
            "result_table_name" : "ST_GLOBAL_VIEW_NAME",
            "database-names" : [
                "TEST101"
            ],
            "schema-names" : [
                "CUX"
            ],
            "source.reader.close.timeout" : 120000,
            "plugin_name" : "Oracle-CDC",
            "table-names-config" : [
                {
                    "table" : "CUX.CUX_CREDIT_FINANCE230207",
                    "primaryKeys" : [
                        "FINANCE_ID"
                    ]
                }
            ],
            "username" : "logminer_user"
        }
    ],
    "transform" : [],
    "sink" : [
        {
            "password" : "root123456",
            "sink.enable-2pc" : "false",
            "fenodes" : "xxxxx:8060",
            "doris.config" : {
                "format" : "json",
                "read_json_by_line" : "true"
            },
            "table.identifier" : "cdc.CUX_CREDIT_FINANCE230207",
            "source_table_name" : "ST_GLOBAL_VIEW_NAME",
            "sink.enable-delete" : "true",
            "plugin_name" : "Doris",
            "sink.label-prefix" : "sink",
            "username" : "root"
        }
    ]
}

@luzongzhu
Copy link
Contributor Author

Oracle version 12c,not have cdb

@Carl-Zhou-CN
Copy link
Member

I think it is the table name configuration that is wrong, not the code

@Carl-Zhou-CN
Copy link
Member

Would you mind referring to the configuration of oracle cdc e2e

@luzongzhu
Copy link
Contributor Author

oracle-cdc jdbc url 需要使用 sid的方式配置

@lililill
Copy link

遇到同样的问题,找到解决方案了吗

@luzongzhu
Copy link
Contributor Author

oracle-cdc jdbc url 需要使用 sid的方式配置

oracle-cdc jdbc url 需要使用 sid的方式配置

@wewelove
Copy link

wewelove commented May 20, 2024

2.3.5版本遇到同样问题, 配置如下

{
  "env": {
    "job.name": "演示任务 Oracle-CDC-openGauss",
    "job.mode": "STREAMING",
    "parallelism": 1,
    "checkpoint.interval": 10000
  },
  "source": [
    {
      "plugin_name": "Oracle-CDC",
      "base-url": "jdbc:oracle:thin:@192.168.1.113:1521:ORCL",
      "username": "system",
      "password": "oracle",
      "database-names": ["SAKILA"],
      "table-names": ["SAKILA.CITY"],
      "startup.mode": "initial",
      "result_table_name": "city"
    }
  ],
  "transform": [],
  "sink": [
    {
      "plugin_name": "Jdbc",
      "url": "jdbc:postgresql://192.168.1.113:5432/demo",
      "driver": "org.postgresql.Driver",
      "compatible_mode": "postgresLow",
      "user": "gaussdb",
      "password": "Qwer@1234",
      "generate_sink_sql": true,
      "field_ide": "LOWERCASE",
      "database": "demo",
      "table": "sakila.city",
      "schema_save_mode": "CREATE_SCHEMA_WHEN_NOT_EXIST",
      "data_save_mode": "DROP_DATA",
      "source_table_name": "city"
    }
  ]
}

@wewelove
Copy link

已经解决, 新的配置文件如下

{
  "env": {
    "job.name": "演示任务 Oracle-CDC-openGauss",
    "job.mode": "STREAMING",
    "parallelism": 1,
    "checkpoint.interval": 10000
  },
  "source": [
    {
      "plugin_name": "Oracle-CDC",
      "base-url": "jdbc:oracle:thin:@192.168.1.113:1521:ORCL",
      "username": "system",
      "password": "oracle",
      "database-names": ["ORCL"],
      "schema-names": ["SAKILA"],
      "table-names": ["ORCL.SAKILA.CITY"],
      "startup.mode": "initial",
      "result_table_name": "city"
    }
  ],
  "transform": [],
  "sink": [
    {
      "plugin_name": "Jdbc",
      "url": "jdbc:postgresql://192.168.1.113:5432/demo",
      "driver": "org.postgresql.Driver",
      "compatible_mode": "postgresLow",
      "user": "gaussdb",
      "password": "Qwer@1234",
      "generate_sink_sql": true,
      "field_ide": "LOWERCASE",
      "database": "demo",
      "table": "sakila.city",
      "schema_save_mode": "CREATE_SCHEMA_WHEN_NOT_EXIST",
      "data_save_mode": "DROP_DATA",
      "source_table_name": "city"
    }
  ]
}

关键是 source 中的以下几个字段, 必须按照以下模式配置:

"base-url": "jdbc:oracle:thin:@192.168.1.113:1521:ORCL",
"database-names": ["ORCL"],
"schema-names": ["SAKILA"],
"table-names": ["ORCL.SAKILA.CITY"],

@Carl-Zhou-CN Carl-Zhou-CN mentioned this issue Jun 25, 2024
3 tasks
NeoGitCrt1 pushed a commit to NeoGitCrt1/seatunnel that referenced this issue Jul 25, 2024
Oracle CDC : Can not find catalog table with factoryId [Oracle]
NeoGitCrt1 pushed a commit to NeoGitCrt1/seatunnel that referenced this issue Jul 25, 2024
Oracle CDC : Can not find catalog table with factoryId [Oracle]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants