Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-7013] Update BigQueryHllSketchCompatibilityIT to cover empty sketch cases #9778

Merged
merged 1 commit into from
Nov 11, 2019

Conversation

robinyqiu
Copy link
Contributor

r: @zfraa


Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status Build Status
Build Status
Build Status
Build Status Build Status Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
--- Build Status
Build Status
Build Status
Build Status
--- --- Build Status
XLang --- --- --- Build Status --- --- ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status
Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@robinyqiu
Copy link
Contributor Author

Run Java PostCommit

@robinyqiu
Copy link
Contributor Author

Copy link

@zfraa zfraa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice that you got this to work and that it's now available to folks as an example how to do this!
Main question is double-checking that this is testing what we think it is testing.
Beyond this PR, we should make sure that this gets covered in a pattern. But also: is there any way we could make the conversion functors (parseQueryResultToByteArray, and the inlined one for the other direction) available to users as utility objects or methods? It seems suboptimal if every person who wants to have Beam and BQ interoperate on HLL++ would need to copy out the code from a pattern. E.g., if there's ever a new edge case to be handled in this conversion, it's easier to update that centrally.

@robinyqiu
Copy link
Contributor Author

is there any way we could make the conversion functors (parseQueryResultToByteArray, and the inlined one for the other direction) available to users as utility objects or methods?

I have considered this but unfortunately we cannot do that, because in the same function users might want to parse other fields. However I do find a way to extract part of the logic into its own function. See the getSketchFromByteBuffer function.

Copy link

@zfraa zfraa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay, thank you for adding the utility function to HllCount.java! It's a pity we can't do it for both directions...

@robinyqiu
Copy link
Contributor Author

Run Java PreCommit

@robinyqiu
Copy link
Contributor Author

Run Java_Examples_Dataflow PreCommit

@robinyqiu
Copy link
Contributor Author

Run Java PostCommit

@boyuanzz boyuanzz merged commit 47d5ee9 into apache:master Nov 11, 2019
@robinyqiu robinyqiu deleted the hll branch December 23, 2019 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants