New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support hive scalar function #16737
Support hive scalar function #16737
Conversation
5cf551a
to
ad879d7
Compare
b00f144
to
4769c3f
Compare
@prithvip @pettyjamesm @v-jizhang Hi, would you mind helping me review this feature? |
Have we thought about implementing this as a separate plugin rather than modifying presto-hive? That would give people more flexibility on enabling the feature. For example, at Facebook, we have multiple catalog using the Hive connector, but we might want to only have a single catalog for hive functions. |
It is feasible. I would love to revise it If it is necessary.
Another idea: Whether use the feature or not is decided by different hive.properties. |
presto-hive/src/main/java/com/facebook/presto/hive/function/type/PrestoTypes.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/function/type/ObjectInputDecoders.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/function/type/ObjectInputDecoders.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/function/type/ObjectInputDecoders.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/function/type/PrestoTypes.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/function/type/PrestoTypes.java
Outdated
Show resolved
Hide resolved
.../src/main/java/com/facebook/presto/spi/function/BuiltInScalarFunctionImplementationShim.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/function/HiveFunctionRegistry.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/function/HiveFunctionNamespaceManager.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/function/FunctionRegistry.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/function/FunctionRegistry.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/com/facebook/presto/hive/function/FunctionRegistry.java
Outdated
Show resolved
Hide resolved
c9c2272
to
183813f
Compare
3f72bf3
to
b13633b
Compare
presto-spi/src/main/java/com/facebook/presto/spi/function/FunctionNamespaceManager.java
Outdated
Show resolved
Hide resolved
8c09f1d
to
1f0cf93
Compare
presto-hive-functions/src/main/java/com/facebook/presto/hive/functions/HiveFunctionModule.java
Outdated
Show resolved
Hide resolved
presto-hive-functions/src/main/java/com/facebook/presto/hive/functions/HiveFunction.java
Outdated
Show resolved
Hide resolved
presto-hive-functions/src/main/java/com/facebook/presto/hive/functions/HiveFunctionConfig.java
Outdated
Show resolved
Hide resolved
...o-hive-functions/src/main/java/com/facebook/presto/hive/functions/HiveFunctionErrorCode.java
Outdated
Show resolved
Hide resolved
...functions/src/main/java/com/facebook/presto/hive/functions/HiveFunctionNamespaceManager.java
Outdated
Show resolved
Hide resolved
...functions/src/main/java/com/facebook/presto/hive/functions/HiveFunctionNamespaceManager.java
Outdated
Show resolved
Hide resolved
presto-hive-functions/src/main/java/com/facebook/presto/hive/functions/gen/CompilerUtils.java
Outdated
Show resolved
Hide resolved
presto-spi/src/main/java/com/facebook/presto/spi/function/FunctionNamespaceManager.java
Outdated
Show resolved
Hide resolved
63daaf2
to
e3b5c68
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great. Some final nits. Thanks for working on this!
...o-hive-function-namespace/src/main/java/com/facebook/presto/hive/functions/HiveFunction.java
Outdated
Show resolved
Hide resolved
...tion-namespace/src/main/java/com/facebook/presto/hive/functions/gen/ScalarMethodHandles.java
Outdated
Show resolved
Hide resolved
...on-namespace/src/main/java/com/facebook/presto/hive/functions/scalar/HiveScalarFunction.java
Outdated
Show resolved
Hide resolved
...namespace/src/main/java/com/facebook/presto/hive/functions/HiveFunctionNamespaceManager.java
Show resolved
Hide resolved
...namespace/src/main/java/com/facebook/presto/hive/functions/HiveFunctionNamespaceManager.java
Outdated
Show resolved
Hide resolved
...namespace/src/main/java/com/facebook/presto/hive/functions/HiveFunctionNamespaceManager.java
Outdated
Show resolved
Hide resolved
...namespace/src/main/java/com/facebook/presto/hive/functions/HiveFunctionNamespaceManager.java
Outdated
Show resolved
Hide resolved
897506c
to
16ee2ef
Compare
Introduce HiveFunctionRegistry interface to registry hive function. Implement SimpleHiveFunctionRegistry to get hive scalar function class. FunctionRegistry implements a mapping between function names and function classes like org.apache.hadoop.hive.ql.exec.FunctionRegistry. Co-authored-by: Todd Gao <todd.gao.2013@gmail.com> Co-authored-by: zybing <739551801@qq.com>
Co-authored-by: Todd Gao <todd.gao.2013@gmail.com> Co-authored-by: zybing <739551801@qq.com>
HiveScalarFunctionInvoker bridge Hive scalar function instance. Co-authored-by: Todd Gao <todd.gao.2013@gmail.com> Co-authored-by: zybing <739551801@qq.com>
2fbbff6
to
bc93078
Compare
...namespace/src/main/java/com/facebook/presto/hive/functions/HiveFunctionNamespaceManager.java
Outdated
Show resolved
Hide resolved
...namespace/src/main/java/com/facebook/presto/hive/functions/HiveFunctionNamespaceManager.java
Outdated
Show resolved
Hide resolved
375a412
to
df0784e
Compare
Co-authored-by: Todd Gao <todd.gao.2013@gmail.com> Co-authored-by: zybing <739551801@qq.com>
df0784e
to
b26f834
Compare
Hello @rongrong & @yulongfufu , I'm seeing the error below after this pr merged. Anything I missed?
|
nvm, I found I need |
@yulongfufu I tested the pmod method and an exception occurred
The error stack is as follows, do you know how to fix it?
|
ohh...I remember it because we met it before. The pmod fails to initialize because it extends the |
@yulongfufu
|
when i ran presto , i got this error
|
Did you configure the function namespace? I don't think it enabled by default. |
Support Hive built-in scalar function in Presto.
These functions are already in Presto dependency(com.facebook.presto.hive:hive-apache). There are several steps to reuse Hive function :
The overall changes are very less intrusive to the original code and when users don't use hive function namespace i.e.
hive.default.function_name
, no behavior changes.Related issue: #16478
Proposal: #16592
Design doc: https://bytedance.feishu.cn/docs/doccnA5uuqTrBFIc1SiwrahulGh
Framework
Description of Changes
Add a new plugin
presto-hive-functions
Introduce
HiveFunctionRegistry
interface to registry hive function. We can implement different FunctionRegistries to get different function classes (e.g. hive built-in function / user-defined hive function which stored in external storage).Implement
SimpleHiveFunctionRegistry
to get hive scalar function class through FunctionRegistry which likesorg.apache.hadoop.hive.ql.exec.FunctionRegistry
.Introduce encoder / decoder to convert type between Presto and Hive.
Introduce
HiveScalarFunctionInvoker
to create、initialize and bridge Hive scalar function instance. At last the real execution of Hive Function will be called through reflection i.e.Invoker.evaluate.Introduce
HiveFunctionNamespaceManager
to load HiveFunctionNamespace.Usage
It will load the hive built-in function at runtime when using the hive function namespace i.e.
hive.default.function_name
, but when users don't use hive catalog function namespace, no behavior changes.For example:
Similar function which have the same function name:
Hive function:
Roadmap
It's the first PR which supports hive built-in scalar function. And then we will support hive built-in aggregation function and user-defined hive function which stored in external storage in later Prs.
Test plan
Added unit tests
Verified in online cluster for one year(300k ~ 500k sql / day)