Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

用户自定义函数udf,未显示的注册,是否可以直接使用? #30

Closed
nuaabuaa07 opened this issue Nov 7, 2018 · 10 comments

Comments

@nuaabuaa07
Copy link

No description provided.

@nuaabuaa07 nuaabuaa07 changed the title 用户自定义函数udf的使用 用户自定义函数udf,未显示的注册,是否可以直接使用? Nov 7, 2018
@nuaabuaa07
Copy link
Author

定义了用户自定义函数udf类,并且通过addjar参数指定了路径。是否可以在sql里直接使用?源代码中没有对自定义的函数做注册和声明,可以直接使用吗?

@yangsishu
Copy link
Contributor

用户自定义额函数CREATE (scala|table) FUNCTION CHARACTER_LENGTH WITH com.dtstack.Kun
在sql里面要加上这个sql,就能用了

@nuaabuaa07
Copy link
Author

这里的com.dtstack.Kun ,就是自己写的自定义函数类的绝对路径?

@zoudaokoulife
Copy link
Contributor

是的

@nuaabuaa07
Copy link
Author

在测试udf过程中出现如下现象:
1.初次通过addjar提交了自定义函数jar包,测试使用udf正常。再次提交新的jar包和新的udf函数,报错提示函数未注册。
2.原先第一次提交的jar包中的udf,在之后启动任务时,不用指定addjar路径参数,也可正常使用。
3.原先第一次提交的jar包中的udf,重新写入新的逻辑,指定addjar参数,提交任务,可以未报错,但是输出的是原先第一次的udf逻辑,没有使用新逻辑处理。也就是新处理逻辑未生效。
请问,提交的addjar文件时flink缓存在了某处吗?在哪里呢?

@yangsishu yangsishu reopened this Dec 6, 2018
@yangsishu
Copy link
Contributor

你是用什么模式执行的

@nuaabuaa07
Copy link
Author

flink是1.5.4 standalone模式
sh ./submit.sh -sql /home/jinzhiliang/test/flink/sqlTXT/sideSql.txt -name xctest -remoteSqlPluginPath /home/dl/projects/flink-1.5.4/plugins -localSqlPluginPath /home/dl/projects/flink-1.5.4/plugins -addjar ["/home/jinzhiliang/test/flink/addjar/flinkUDFtest-1.0-SNAPSHOT.jar"] -mode standalone -yarnconf /home/jinzhiliang/test/flink/yarnconf -flinkconf /home/dl/projects/flink-1.5.4/conf -confProp {"time.characteristic":"EventTime","sql.checkpoint.interval":10000}

@nuaabuaa07
Copy link
Author

我重启了flink1.5.4,不指定addjar,提交任务,原先第一次提交的udf函数任然可以使用。

@nuaabuaa07
Copy link
Author

你们测试有预计这个问题吗?

@nuaabuaa07
Copy link
Author

问题找到了,我第一次根本就没有读入自定义函数。sql里面写的有问题。现在改了一下,但是在加载自定义函数包的时候报类记载器错误:
Exception in thread "main" org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404)
at com.dtstack.flink.sql.launcher.LauncherMain.main(LauncherMain.java:93)
Caused by: java.lang.NoSuchMethodException: sun.misc.Launcher$AppClassLoader.addURL(java.net.URL)
at java.lang.Class.getDeclaredMethod(Class.java:2130)
at com.dtstack.flink.sql.util.FlinkUtil.urlClassLoaderAddUrl(FlinkUtil.java:271)
at com.dtstack.flink.sql.util.FlinkUtil.loadExtraJar(FlinkUtil.java:262)
at com.dtstack.flink.sql.Main.registerUDF(Main.java:246)
at com.dtstack.flink.sql.Main.main(Main.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants