Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [REST API] call submit job API resulted in an error #6581

Closed
2 of 3 tasks
ilsl1007 opened this issue Mar 26, 2024 · 8 comments · Fixed by #6596
Closed
2 of 3 tasks

[Bug] [REST API] call submit job API resulted in an error #6581

ilsl1007 opened this issue Mar 26, 2024 · 8 comments · Fixed by #6596
Labels

Comments

@ilsl1007
Copy link
Contributor

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

The second call to the submit job API resulted in an error, The error occurred during the second call, which only modified the query's time range.

SeaTunnel Version

2.3.4

SeaTunnel Config

# first call:
{
    "env": {
        "execution.parallelism": 1,
        "job.mode": "BATCH"
    },
    "source": [
        {
            "plugin_name": "TDengine",
            "url": "jdbc:TAOS-RS://192.168.100.1:6041/",
            "username": "root",
            "password": "taosdata",
            "database": "test",
            "stable": "meters",
            "lower_bound": "2017-07-14 02:40:00.000",
            "upper_bound": "2017-07-14 02:40:00.006",
            "result_table_name": "tdengine_result"
        }
    ],
    "sink": [
        {
            "plugin_name": "TDengine",
            "url": "jdbc:TAOS-RS://192.168.100.2:6041/",
            "username": "root",
            "password": "taosdata",
            "database": "test",
            "stable": "meters",
            "timezone": "UTC"
        }
    ]
}

# second call:
{
    "env": {
        "execution.parallelism": 1,
        "job.mode": "BATCH"
    },
    "source": [
        {
            "plugin_name": "TDengine",
            "url": "jdbc:TAOS-RS://192.168.100.1:6041/",
            "username": "root",
            "password": "taosdata",
            "database": "test",
            "stable": "meters",
            "lower_bound": "2017-07-14 02:40:00.006",
            "upper_bound": "2017-07-14 02:40:00.010",
            "result_table_name": "tdengine_result"
        }
    ],
    "sink": [
        {
            "plugin_name": "TDengine",
            "url": "jdbc:TAOS-RS://192.168.100.2:6041/",
            "username": "root",
            "password": "taosdata",
            "database": "test",
            "stable": "meters",
            "timezone": "UTC"
        }
    ]
}

Running Command

call the rest api
url is /hazelcast/rest/maps/submit-job

Error Exception

2024-03-26 17:45:37,052 INFO  [o.a.s.c.s.u.ConfigBuilder     ] [hz.main.cached.thread-1] - Loading config file from objectMap
2024-03-26 17:45:37,054 INFO  [o.a.s.c.s.u.ConfigBuilder     ] [hz.main.cached.thread-1] - Parsed config file: 
{
    "env" : {
        "execution.parallelism" : 1,
        "job.mode" : "BATCH"
    },
    "source" : [
        {
            "plugin_name" : "TDengine",
            "url" : "jdbc:TAOS-RS://192.168.100.1:6041/",
            "username" : "root",
            "password" : "taosdata",
            "database" : "test",
            "stable" : "meters",
            "lower_bound" : "2017-07-14 02:40:00.006",
            "upper_bound" : "2017-07-14 02:40:00.010",
            "result_table_name" : "tdengine_result"
        }
    ],
    "sink" : [
        {
            "plugin_name" : "TDengine",
            "url" : "jdbc:TAOS-RS://192.168.100.2:6041/",
            "username" : "root",
            "password" : "taosdata",
            "database" : "test",
            "stable" : "meters",
            "timezone" : "UTC"
        }
    ]
}

2024-03-26 17:45:37,055 INFO  [.r.RestJobExecutionEnvironment] [hz.main.cached.thread-1] - add common jar in plugins :[]
2024-03-26 17:45:37,057 INFO  [.s.p.d.AbstractPluginDiscovery] [hz.main.cached.thread-1] - Load SeaTunnelSink Plugin from /dataext/apache-seatunnel-2.3.4/connectors
2024-03-26 17:45:37,057 INFO  [.s.p.d.AbstractPluginDiscovery] [hz.main.cached.thread-1] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='TDengine'} at: file:/dataext/apache-seatunnel-2.3.4/connectors/connector-tdengine-2.3.4.jar
2024-03-26 17:45:37,058 INFO  [.s.p.d.AbstractPluginDiscovery] [hz.main.cached.thread-1] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='TDengine'} at: file:/dataext/apache-seatunnel-2.3.4/connectors/connector-tdengine-2.3.4.jar
2024-03-26 17:45:37,058 INFO  [o.a.s.e.c.p.ConfigParserUtil  ] [hz.main.cached.thread-1] - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations.
2024-03-26 17:45:37,058 WARN  [o.a.s.e.c.p.ConfigParserUtil  ] [hz.main.cached.thread-1] - This configuration is not recommended. A source/transform(TDengine) is configured with 'result_table_name' option value of 'tdengine_result', but subsequent transform/sink(TDengine) is not configured with 'source_table_name' option.
2024-03-26 17:45:37,059 INFO  [p.MultipleTableJobConfigParser] [hz.main.cached.thread-1] - start generating all sources.
2024-03-26 17:45:37,066 INFO  [.s.p.d.AbstractPluginDiscovery] [hz.main.cached.thread-1] - Load SeaTunnelSource Plugin from /dataext/apache-seatunnel-2.3.4/connectors
2024-03-26 17:45:37,066 INFO  [.s.p.d.AbstractPluginDiscovery] [hz.main.cached.thread-1] - Discovery plugin jar for: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='TDengine'} at: file:/dataext/apache-seatunnel-2.3.4/connectors/connector-tdengine-2.3.4.jar
2024-03-26 17:45:37,068 INFO  [.s.p.d.AbstractPluginDiscovery] [hz.main.cached.thread-1] - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='TDengine'} from classpath
2024-03-26 17:45:37,071 WARN  [Log4j2HttpPostCommandProcessor] [hz.main.cached.thread-1] - [localhost]:5801 [seatunnel] [5.1] An error occurred while handling request HttpCommand [HTTP_POST]{uri='/hazelcast/rest/maps/submit-job'}AbstractTextCommand[HTTP_POST]{requestId=1}
java.sql.SQLException: No suitable driver found for jdbc:TAOS-RS://192.168.100.1:6041/test?user=root&password=taosdata
        at java.sql.DriverManager.getConnection(DriverManager.java:689) ~[?:1.8.0_311]
        at java.sql.DriverManager.getConnection(DriverManager.java:270) ~[?:1.8.0_311]
        at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSource.getStableMetadata(TDengineSource.java:138) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.tdengine.source.TDengineSource.prepare(TDengineSource.java:90) ~[?:?]
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:84) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:315) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:181) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.server.rest.RestJobExecutionEnvironment.getLogicalDag(RestJobExecutionEnvironment.java:76) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.server.rest.RestJobExecutionEnvironment.build(RestJobExecutionEnvironment.java:99) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handleSubmitJob(RestHttpPostCommandProcessor.java:136) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handle(RestHttpPostCommandProcessor.java:82) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handle(RestHttpPostCommandProcessor.java:60) ~[seatunnel-starter.jar:2.3.4]
        at com.hazelcast.internal.ascii.TextCommandServiceImpl$CommandExecutor.run(TextCommandServiceImpl.java:402) ~[seatunnel-starter.jar:2.3.4]
        at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[seatunnel-starter.jar:2.3.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_311]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_311]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_311]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[seatunnel-starter.jar:2.3.4]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) ~[seatunnel-starter.jar:2.3.4]

Zeta or Flink or Spark Version

Zeta

Java or Scala Version

java version "1.8.0_311"
Java(TM) SE Runtime Environment (build 1.8.0_311-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.311-b11, mixed mode)

Screenshots

first call:
first call

second call:
second call

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@ilsl1007 ilsl1007 added the bug label Mar 26, 2024
@liunaijie
Copy link
Contributor

I also meet same issue, and this pr #6477 fixed this issue, you can try with this.

@ilsl1007
Copy link
Contributor Author

ilsl1007 commented Mar 27, 2024

I also meet same issue, and this pr #6477 fixed this issue, you can try with this.

@liunaijie hello,I built the installation package based on the latest code from the dev branch, and after testing, the same issue persists. Testing with MySQL was successful.

image

@liunaijie
Copy link
Contributor

hi @ilsl1007 Is your first submit can success, then all failed? Before this pr my temporary solution is load the driver every time.

@ilsl1007
Copy link
Contributor Author

hi @ilsl1007 Is your first submit can success, then all failed? Before this pr my temporary solution is load the driver every time.

@liunaijie Yes, only the first subit success. What modifications do I need to make for the temporary solution you mentioned about reloading the driver every time?

@liunaijie
Copy link
Contributor

hi @ilsl1007 Is your first submit can success, then all failed? Before this pr my temporary solution is load the driver every time.

@liunaijie Yes, only the first subit success. What modifications do I need to make for the temporary solution you mentioned about reloading the driver every time?

you can update the TDengineSource class, add this code Class.forName("your driver class");
maybe you also need update TDengineSourceReader class.
load the driver before you use it

@liunaijie
Copy link
Contributor

and maybe i know why this pr is not work for your case.
You use TDengineSource, this is different with JdbcSource. JdbcSource is create by JdbcSourceFactory. maybe this is not why this fix is not work for you.

@ilsl1007
Copy link
Contributor Author

and maybe i know why this pr is not work for your case. You use TDengineSource, this is different with JdbcSource. JdbcSource is create by JdbcSourceFactory. maybe this is not why this fix is not work for you.

@liunaijie Thank you. After using the temporary solution you mentioned, I'm now able to call the API successfully.

@liunaijie
Copy link
Contributor

and maybe i know why this pr is not work for your case. You use TDengineSource, this is different with JdbcSource. JdbcSource is create by JdbcSourceFactory. maybe this is not why this fix is not work for you.

@liunaijie Thank you. After using the temporary solution you mentioned, I'm now able to call the API successfully.

np~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants