Skip to content

HIVE-28139: Upgrade to datasketches 1.2.0 - #5144

Merged
zhangbutao merged 1 commit into
apache:masterfrom
zhangbutao:HIVE-28139
Mar 25, 2024
Merged

zhangbutao merged 1 commit into
apache:masterfrom
zhangbutao:HIVE-28139

Conversation

@zhangbutao

@zhangbutao zhangbutao commented Mar 20, 2024 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

https://datasketches.apache.org/docs/Community/Downloads.html shows that the latest version of datasketches-hive(Java Hive Adaptor) is 1.2.0, so we can upgrade it to from 1.1.0 to 1.2.0.

Why are the changes needed?

Does this PR introduce any user-facing change?

No

Is the change a dependency upgrade?

Yes
mvn_den_log.txt

How was this patch tested?

Existing UTs.

@okumin

okumin commented Mar 22, 2024 •

Copy link
Copy Markdown
Contributor

We may ask the DataSketches community to cut a new tag someday. I recalled making changes to avoid WARNING when registering sketch UDFs.
https://github.com/apache/datasketches-hive/commits/master/

In another thought, we may port UDFs in datasketches-hive to hive in the future to maintain them on our own.

@zhangbutao

zhangbutao commented Mar 22, 2024 •

Copy link
Copy Markdown
Contributor Author

We may ask the DataSketches community to cut a new tag someday. I recalled making changes to avoid WARNING when registering sketch UDFs. https://github.com/apache/datasketches-hive/commits/master/

Good to know you fix this issue!

In another thought, we may port UDFs in datasketches-hive to hive in the future to maintain them on our own.

I see datasketches-hive is still under development, though there is no commit for long time. I guess maybe it is stable.
BTW, we have inclued datasketches-hive jar in Hive's binary install package, so i think we can use these udf directly instead of porting them? But i am not sure if we can use these udf in Hive4 as datasketches-hive was built on Hive3 and Hive4 may have some compatibility issue with Hive3 on udf initialize.

But if we Hive community has strong usage about datasketches-hive udf, i think it is good to port them into Hive's repository. I would hear more folk's opinion about this and i think this part can be discussed more through Hive dev email.

I just did take a look at the Hive4 code. I found https://issues.apache.org/jira/browse/HIVE-22940 has registered some datasketches-hive udf, could you please help to double check this? @okumin

@okumin

okumin commented Mar 22, 2024

Copy link
Copy Markdown
Contributor

But if we Hive community has strong usage about datasketches-hive udf, i think it is good to port them into Hive's repository. I would hear more folk's opinion about this and i think this part can be discussed more through Hive dev email.

Just a random thought but I agree as a future option.

I just did take a look at the Hive4 code. I found https://issues.apache.org/jira/browse/HIVE-22940 has registered some datasketches-hive udf, could you please help to double check this? @okumin

That's true, and I remember I modified UDFs which are registered by FunctionRegistry so that I don't see WARNINGs. As far as I know, it is just a warning and no critical errors happen.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

@zhangbutao
zhangbutao merged commit d73aef5 into apache:master Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants