Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat:Support Bytes to UUID UDF Function #9640

Merged
merged 10 commits into from
Oct 20, 2022

Conversation

pkgonan
Copy link
Contributor

@pkgonan pkgonan commented Oct 12, 2022

@pkgonan pkgonan requested a review from a team as a code owner October 12, 2022 08:14
@CLAassistant
Copy link

CLAassistant commented Oct 12, 2022

CLA assistant check
All committers have signed the CLA.

@pkgonan
Copy link
Contributor Author

pkgonan commented Oct 12, 2022

Hi. It does not work well during the build process. Please check below.

Link

CI Error

[2022-10-12T08:51:46.003Z] [FATAL] Non-resolvable parent POM for io.confluent.ksql:ksqldb-parent:7.4.0-0: Could not find artifact io.confluent:common:pom:7.4.0-397 in confluent-snapshots (https://s3-us-west-2.amazonaws.com/confluent-snapshots/) and 'parent.relativePath' points at wrong local POM @ line 29, column 13

@pkgonan
Copy link
Contributor Author

pkgonan commented Oct 14, 2022

@jaceklaskowski @jnh5y @JimGalasyn @cprasad1 @nateab @pgaref
Hi. May I ask for a review on this Pull Request ?

Copy link
Member

@nateab nateab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks for the contribution! Once you get a green build on CI feel free to merge

@nateab
Copy link
Member

nateab commented Oct 14, 2022

hmm the CI failure might be an issue with our build process, i will raise it to our tools team

@nateab
Copy link
Member

nateab commented Oct 14, 2022

@pkgonan while we wait for tools to help us resolve, i ran your branch locally but i ran into issues with checkstyle, i would reccomend running checkstyle, spotbugs, and the tests locally as well. You can use something like ./mvnw clean package -T 2C to speed it up

@pkgonan
Copy link
Contributor Author

pkgonan commented Oct 16, 2022

@nateab
Hi. Thanks for the review.
Your advice has been reflected and refactoring has been completed. Please check. If there are no problems, merge please. :)

@pkgonan
Copy link
Contributor Author

pkgonan commented Oct 16, 2022

@nateab
CI Test failed, and when I check it, it is confirmed that the test has failed in ksqldb-engine.

Upon closer inspection, it appears to be a problem with a specific test within ksqldb-engine. However, this seems to be a problem unrelated to this UUID feature PR.

I checked out the master branch and found the same error as a result of running the corresponding test codes.

CI Test Result

22:09:17  [INFO] Tests run: 207, Failures: 0, Errors: 0, Skipped: 0
22:09:17  [INFO] 
22:09:17  [INFO] ------------------------------------------------------------------------
22:09:17  [INFO] Reactor Summary for ksqldb-parent 7.4.0-0:
22:09:17  [INFO] 
22:09:17  [INFO] ksqldb-parent ...................................... SUCCESS [04:27 min]
22:09:17  [INFO] ksqldb-test-util ................................... SUCCESS [02:00 min]
22:09:17  [INFO] ksqldb-udf ......................................... SUCCESS [01:43 min]
22:09:17  [INFO] ksqldb-common ...................................... SUCCESS [02:29 min]
22:09:17  [INFO] ksqlDB UDF / UDAF :: Quickstart .................... SUCCESS [01:24 min]
22:09:17  [INFO] ksqldb-serde ....................................... SUCCESS [02:16 min]
22:09:17  [INFO] ksqldb-engine-common ............................... SUCCESS [01:58 min]
22:09:17  [INFO] ksqldb-rest-model .................................. SUCCESS [02:00 min]
22:09:17  [INFO] ksqldb-execution ................................... SUCCESS [02:37 min]
22:09:17  [INFO] ksqldb-metastore ................................... SUCCESS [01:51 min]
22:09:17  [INFO] ksqldb-parser ...................................... SUCCESS [01:53 min]
22:09:17  [INFO] ksqldb-streams ..................................... SUCCESS [02:29 min]
22:09:17  [INFO] ksqldb-engine ...................................... FAILURE [  02:32 h]
22:09:17  [INFO] ksqldb-rest-client ................................. SKIPPED
22:09:17  [INFO] ksqldb-version-metrics-client ...................... SKIPPED
22:09:17  [INFO] ksqldb-rest-app .................................... SKIPPED
22:09:17  [INFO] ksqldb-api-client .................................. SKIPPED
22:09:17  [INFO] ksqldb-tools ....................................... SKIPPED
22:09:17  [INFO] ksqldb-cli ......................................... SKIPPED
22:09:17  [INFO] ksqldb-examples .................................... SKIPPED
22:09:17  [INFO] ksqldb-console-scripts ............................. SKIPPED
22:09:17  [INFO] ksqldb-etc ......................................... SKIPPED
22:09:17  [INFO] ksqldb-testing-tool ................................ SKIPPED
22:09:17  [INFO] ksqldb-functional-tests ............................ SKIPPED
22:09:17  [INFO] ksqldb-package ..................................... SKIPPED
22:09:17  [INFO] ksqldb-benchmark ................................... SKIPPED
22:09:17  [INFO] ksqlDB RocksDB Config Setter ....................... SKIPPED
22:09:17  [INFO] ksqldb-docker ...................................... SKIPPED
22:09:17  [INFO] ksqldb-api-reactive-streams-tck .................... SKIPPED
22:09:17  [INFO] ------------------------------------------------------------------------
22:09:17  [INFO] BUILD FAILURE
22:09:17  [INFO] ------------------------------------------------------------------------
22:09:17  [INFO] Total time:  02:59 h
22:09:17  [INFO] Finished at: 2022-10-16T13:09:13Z
22:09:17  [INFO] ------------------------------------------------------------------------
22:09:17  [INFO] [jenkins-event-spy] Generated /home/jenkins/workspace/ksql_PR-9640@tmp/withMavene348286d/maven-spy-20221016-100948-3623890007551158498063.log

ksqldb-engine Test

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ExpTest.shouldHandlePositive:56 
Expected: is <2.718281828459045>
     but: was <2.7182818284590455>
[ERROR] Errors: 
[ERROR]   ParseDateTest.shouldConvertCaseInsensitiveStringToDate:70 » KsqlFunction Failed to parse date '01-dec-2021' with formatter 'dd-MMM-yyyy': Unparseable date: 01-dec-2021
[ERROR]   ParseTimeTest.shouldConvertCaseInsensitiveStringToDate:52 » KsqlFunction Failed to parse time '12:01:05 aM' with formatter 'hh:mm:ss a': Text '12:01:05 aM' could not be parsed at index 9
[ERROR]   StringToDateTest.shouldConvertCaseInsensitiveStringToDate:51 » KsqlFunction Failed to parse date '01-dec-2021' with formatter 'dd-MMM-yyyy': Text '01-dec-2021' could not be parsed at index 3
[INFO] 
[ERROR] Tests run: 3447, Failures: 1, Errors: 3, Skipped: 0
[INFO] 

@nateab
Copy link
Member

nateab commented Oct 17, 2022

hey @pkgonan looking at the latest ci run https://jenkins.public.confluent.io/job/ksql/job/PR-9640/10/execution/node/33/log/, i see this failure now instead

15:11:00 [INFO] Reactor Summary for ksqldb-parent 7.4.0-0: 15:11:00 [INFO] 15:11:00 [INFO] ksqldb-parent ...................................... SUCCESS [04:46 min] 15:11:00 [INFO] ksqldb-test-util ................................... SUCCESS [01:59 min] 15:11:00 [INFO] ksqldb-udf ......................................... SUCCESS [01:41 min] 15:11:00 [INFO] ksqldb-common ...................................... SUCCESS [02:25 min] 15:11:00 [INFO] ksqlDB UDF / UDAF :: Quickstart .................... SUCCESS [01:27 min] 15:11:00 [INFO] ksqldb-serde ....................................... SUCCESS [02:13 min] 15:11:00 [INFO] ksqldb-engine-common ............................... SUCCESS [01:53 min] 15:11:00 [INFO] ksqldb-rest-model .................................. SUCCESS [01:59 min] 15:11:00 [INFO] ksqldb-execution ................................... SUCCESS [02:37 min] 15:11:00 [INFO] ksqldb-metastore ................................... SUCCESS [01:50 min] 15:11:00 [INFO] ksqldb-parser ...................................... SUCCESS [01:49 min] 15:11:00 [INFO] ksqldb-streams ..................................... SUCCESS [02:26 min] 15:11:00 [INFO] ksqldb-engine ...................................... SUCCESS [23:32 min] 15:11:00 [INFO] ksqldb-rest-client ................................. SUCCESS [01:59 min] 15:11:00 [INFO] ksqldb-version-metrics-client ...................... SUCCESS [01:57 min] 15:11:00 [INFO] ksqldb-rest-app .................................... SUCCESS [ 01:20 h] 15:11:00 [INFO] ksqldb-api-client .................................. SUCCESS [04:03 min] 15:11:00 [INFO] ksqldb-tools ....................................... SUCCESS [03:40 min] 15:11:00 [INFO] ksqldb-cli ......................................... SUCCESS [03:51 min] 15:11:00 [INFO] ksqldb-examples .................................... SUCCESS [02:46 min] 15:11:00 [INFO] ksqldb-console-scripts ............................. SUCCESS [ 4.111 s] 15:11:00 [INFO] ksqldb-etc ......................................... SUCCESS [ 4.129 s] 15:11:00 [INFO] ksqldb-testing-tool ................................ SUCCESS [02:48 min] 15:11:00 [INFO] ksqldb-functional-tests ............................ FAILURE [27:09 min] 15:11:00 [INFO] ksqldb-package ..................................... SKIPPED 15:11:00 [INFO] ksqldb-benchmark ................................... SKIPPED 15:11:00 [INFO] ksqlDB RocksDB Config Setter ....................... SKIPPED 15:11:00 [INFO] ksqldb-docker ...................................... SKIPPED 15:11:00 [INFO] ksqldb-api-reactive-streams-tck .................... SKIPPED 15:11:00 [INFO] ------------------------------------------------------------------------ 15:11:00 [INFO] BUILD FAILURE 15:11:00 [INFO] ------------------------------------------------------------------------ 15:11:00 [INFO] Total time: 02:59 h 15:11:00 [INFO] Finished at: 2022-10-16T22:10:57Z 15:11:00 [INFO] ------------------------------------------------------------------------ 15:11:00 [INFO] [jenkins-event-spy] Generated /home/jenkins/workspace/ksql_PR-9640@tmp/withMavena2d760f4/maven-spy-20221016-191130-1351536012141309772159.log 15:11:00 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M7:verify (verify) on project ksqldb-functional-tests: 15:11:00 [ERROR] 15:11:00 [ERROR] Please refer to /home/jenkins/workspace/ksql_PR-9640/ksqldb-functional-tests/target/failsafe-reports for the individual test results. 15:11:00 [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. 15:11:00 [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? 15:11:00 [ERROR] Command was /bin/sh -c cd '/home/jenkins/workspace/ksql_PR-9640/ksqldb-functional-tests' && '/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/jre/bin/java' '-javaagent:/home/jenkins/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=/home/jenkins/workspace/ksql_PR-9640/ksqldb-functional-tests/target/jacoco.exec' '-Djava.awt.headless=true' '-jar' '/home/jenkins/workspace/ksql_PR-9640/ksqldb-functional-tests/target/surefire/surefirebooter-20221016220142178_559.jar' '/home/jenkins/workspace/ksql_PR-9640/ksqldb-functional-tests/target/surefire' '2022-10-16T19-12-45_058-jvmRun2' 'surefire-20221016220142178_553tmp' 'surefire_43-20221016220142178_556tmp' 15:11:00 [ERROR] Error occurred in starting fork, check output in log 15:11:00 [ERROR] Process Exit Code: 143 15:11:00 [ERROR] Crashed tests: 15:11:00 [ERROR] RestQueryTranslationTestBatch1 15:11:00 [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? 15:11:00 [ERROR] Command was /bin/sh -c cd '/home/jenkins/workspace/ksql_PR-9640/ksqldb-functional-tests' && '/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/jre/bin/java' '-javaagent:/home/jenkins/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=/home/jenkins/workspace/ksql_PR-9640/ksqldb-functional-tests/target/jacoco.exec' '-Djava.awt.headless=true' '-jar' '/home/jenkins/workspace/ksql_PR-9640/ksqldb-functional-tests/target/surefire/surefirebooter-20221016220142178_558.jar' '/home/jenkins/workspace/ksql_PR-9640/ksqldb-functional-tests/target/surefire' '2022-10-16T19-12-45_058-jvmRun3' 'surefire-20221016220142178_551tmp' 'surefire_41-20221016220142178_554tmp' 15:11:00 [ERROR] Error occurred in starting fork, check output in log 15:11:00 [ERROR] Process Exit Code: 143 15:11:00 [ERROR] Crashed tests: 15:11:00 [ERROR] RestQueryTranslationTestBatch2 15:11:00 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:513) 15:11:00 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:460) 15:11:00 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:327) 15:11:00 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:269) 15:11:00 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1334) 15:11:00 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1167) 15:11:00 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:931) 15:11:00 [ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137) 15:11:00 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210) 15:11:00 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156) 15:11:00 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148) 15:11:00 [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117) 15:11:00 [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81) 15:11:00 [ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56) 15:11:00 [ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) 15:11:00 [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305) 15:11:00 [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192) 15:11:00 [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105) 15:11:00 [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957) 15:11:00 [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289) 15:11:00 [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:193) 15:11:00 [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 15:11:00 [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 15:11:00 [ERROR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 15:11:00 [ERROR] at java.lang.reflect.Method.invoke(Method.java:498) 15:11:00 [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282) 15:11:00 [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225) 15:11:00 [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406) 15:11:00 [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

@pkgonan
Copy link
Contributor Author

pkgonan commented Oct 18, 2022

@nateab I think that CI is not stable and broken.

The version of the io.confluent.common project in pom.xml committed to the master branch is 7.4.0-430 (

ksql/pom.xml

Line 32 in 3f96077

<version>7.4.0-430</version>
). But when I look for the actual snapshot storage, it doesn't find that version (https://s3-us-west-2.amazonaws.com/confluent-snapshots). So, like the one mentioned at the top of this PR, these problems continue to occur.

CI Error

10:03:14  [ERROR] [ERROR] Some problems were encountered while processing the POMs:
10:03:14  [FATAL] Non-resolvable parent POM for io.confluent.ksql:ksqldb-parent:7.4.0-0: Could not find artifact io.confluent:common:pom:7.4.0-430 in confluent-snapshots (https://s3-us-west-2.amazonaws.com/confluent-snapshots/) and 'parent.relativePath' points at wrong local POM @ line 29, column 13
10:03:14   @ 
10:03:14  [INFO] [jenkins-event-spy] Generated /home/jenkins/workspace/ksql_PR-9640@tmp/withMavenb75c63ba/maven-spy-20221018-010313-9173286833380177714289.log
10:03:14  [ERROR] The build could not read 1 project -> [Help 1]
10:03:14  [ERROR]   
10:03:14  [ERROR]   The project io.confluent.ksql:ksqldb-parent:7.4.0-0 (/home/jenkins/workspace/ksql_PR-9640/pom.xml) has 1 error
10:03:14  [ERROR]     Non-resolvable parent POM for io.confluent.ksql:ksqldb-parent:7.4.0-0: Could not find artifact io.confluent:common:pom:7.4.0-430 in confluent-snapshots (https://s3-us-west-2.amazonaws.com/confluent-snapshots/) and 'parent.relativePath' points at wrong local POM @ line 29, column 13 -> [Help 2]
10:03:14  [ERROR] 
10:03:14  [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
10:03:14  [ERROR] Re-run Maven using the -X switch to enable full debug logging.
10:03:14  [ERROR] 
10:03:14  [ERROR] For more information about the errors and possible solutions, please read the following articles:
10:03:14  [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
10:03:14  [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

https://jenkins.public.confluent.io/job/ksql/job/PR-9640/12/console

@pkgonan
Copy link
Contributor Author

pkgonan commented Oct 18, 2022

@nateab
Should I have chosen a branch other than master as the base branch when creating the PR?

It seems that the master branch is not stable. It looks like a non-existent snapshot version is being committed to master branch.

The source code hasn't changed, but the results of the tests keep changing.

@nateab
Copy link
Member

nateab commented Oct 19, 2022

@pkgonan Yes there are some currently known issues around our build system right now, but choosing master as the base branch is still correct. I will follow up with tools again to try to resolve the build issue. In the meanwhile does your PR pass locally at least? If so, I can also just build your PR locally to verify and then merge if we cant sort out the build issues before then

@pkgonan
Copy link
Contributor Author

pkgonan commented Oct 20, 2022

@nateab
Hi.

To build in your local environment, I have to download the dependencies in the Confluent Snapshot Repository. So of course a local build cannot succeed.

스크린샷 2022-10-20 오전 10 47 28

But the test for the UUID feature I added works fine.
Feature : https://github.com/confluentinc/ksql/pull/9640/files#diff-5df68425b4dacea27d856c001bb7971a7f1522e6ec10318ca7ac430df46416d3

Test Code : https://github.com/confluentinc/ksql/pull/9640/files#diff-ea90fe8c1789cc2e2943fd22dad1a00e600aa86b374105df17cac45212f095d4

스크린샷 2022-10-20 오전 10 46 01

@nateab
Copy link
Member

nateab commented Oct 20, 2022

@pkgonan okay ive verified that it passes all checks locally so ill go ahead and merge it, thanks for your first contribution to ksqlDB! 🥳

@nateab
Copy link
Member

nateab commented Oct 20, 2022

@pkgonan i think if you add --settings maven-settings.xml to your maven command, it might help with building locally?

@nateab nateab merged commit c3bc367 into confluentinc:master Oct 20, 2022
@nateab
Copy link
Member

nateab commented Oct 20, 2022

@pkgonan hey it would be good to also add some documentation on this new UDF here please in a follow up PR https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/scalar-functions/

@pkgonan
Copy link
Contributor Author

pkgonan commented Oct 21, 2022

@nateab Hi. Thank you very much. I'll add the docs via a new PR.

@pkgonan
Copy link
Contributor Author

pkgonan commented Oct 21, 2022

@nateab
Thanks for your help.

Finally, i added docs. Would you please check that?

Docs PR

@pkgonan pkgonan mentioned this pull request May 3, 2023
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants