Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-6493] Convert the WordCount samples to Kotlin #8034

Closed
wants to merge 501 commits into from

Conversation

harshithdwivedi
Copy link
Contributor

@harshithdwivedi harshithdwivedi commented Mar 12, 2019

This PR converts the WordCounts sample from Java to Kotlin.
Please have a look and let me know if any changes are required.
@kennknowles @pabloem

I made this new PR since there have been some 2FA issues with my earlier GitHub Account (@the-dagger) and I'm locked out of it for the time being.

P.S. I haven't converted all the tests to Kotlin yet; I'll do that once all the source classes are migrated

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- --- --- --- ---
Java Build Status Build Status Build Status Build Status
Build Status
Build Status
Build Status Build Status Build Status
Python Build Status
Build Status
--- Build Status
Build Status
Build Status --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@harshithdwivedi
Copy link
Contributor Author

@kennknowles The Java Precommit test is failing here, can you help me debug why that's the case?

@pabloem
Copy link
Member

pabloem commented Mar 15, 2019

If you open the "details" of the precommit run, you can see what Jenkins says. Here is some stuff:

By the look of it, some Java warnings. You can also look at the Gradle Build Scan. Where you'll see some kotlin-related tasks failed

Here are the errors from the compile /test task for the examples: https://scans.gradle.com/s/sklwyla23gpbq/console-log?task=:beam-examples-kotlin:compileTestJava#L15

Gradle builds ask for emails, but they provide a lot of value : ) I recommend you check it out.

@pabloem
Copy link
Member

pabloem commented Mar 25, 2019

@harshithdwivedi lmk if I can help

@harshithdwivedi
Copy link
Contributor Author

Hi @pabloem I was out travelling for the last week so sorry for the inactivity here.
I'm working on this as we speak and should send out a fix that solves the build errors by eod.

I'll let you know in case I stumble upon any issue; thanks!

@pabloem
Copy link
Member

pabloem commented Mar 25, 2019

No need to apologize : ) I just didn't want you to forget about thisw, because we're grateful for the contribution.
Sorry about the spotless trouble. You can run ./gradlew :beam-examples-kotlin:spotlessApply to do the autoformatting.

@harshithdwivedi
Copy link
Contributor Author

@pabloem Looks like all the tests have finally passed 😅
LMK if you need me to squash the commits.

apilloud and others added 23 commits March 26, 2019 13:46
[BEAM-6401] Upgrade to Calcite 1.19
…ner/revert_pr6467"

This reverts commit 2bf5ead, reversing
changes made to 6a59667.
Adding links to beam's github page from the contribution guide for easy discovery of beam's github pages.
1. git clone git@github.com:apache/beam.git requires the user to be logged in, while https://github.com/apache/beam does not.
2. Spaces in the example link need to be escaped.
[BEAM-6918] Changed github link to not require login and fixed broken example link
[BEAM-6892] Schemas and destinations are provided to WriteToBigQuery separately (apache#8093)
Add link to dashboard to doc.
The buffer timeout can overflow when it is Long.MAX_VALUE.
Update contribution guide with links
Correct the flag name of enable_streaming_engine in the warning text.
[BEAM-6927] Fix ./gradlew :beam-test-tools:goTest
- Added support to limit results.
- Abstracted MongoDB query builder into a separate class.
- Cleaned the BoundedMongoDbReader start method.
- Added support to pass ObjectId as string.
- Separated MongoDB's find and aggregation into two separate query classes.
- Utilized SerializableFunction in both query builders.
- [BEAM-4567] Fix splitting with bucket auto (Fixes use of Atlas MongoDB
@kennknowles
Copy link
Member

You should find a link to the Gradle scan on the Jenkins results page for the build.

@kennknowles
Copy link
Member

The errors are caused by a recent change that moved things around inside the sdk.util package into sdk.util.gcp. If you rebase your PR again master you will get the errors and they will be easy to fix, just an import change.

:beam-examples-kotlin:compileKotlin FAILED
--
e: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt: (38, 33): Unresolved reference: RetryHttpRequestInitializer
e: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt: (40, 33): Unresolved reference: Transport
e: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt: (351, 21): Unresolved reference: Transport
e: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt: (352, 21): Unresolved reference: Transport
e: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt: (357, 29): Unresolved reference: RetryHttpRequestInitializer
e: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt: (365, 21): Unresolved reference: Transport
e: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt: (366, 21): Unresolved reference: Transport
e: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt: (371, 29): Unresolved reference: RetryHttpRequestInitializer

@kennknowles
Copy link
Member

You can see the changes in #8217 and also the changes to the Java WordCount. Sorry for this mix up!

ibzib and others added 24 commits April 11, 2019 14:30
[BEAM-7053] prevent errors in Spark options
[BEAM-5663] Add Python 3.6 and Python 3.7 precommit test suites
[BEAM-7046] Restore os.environ in HttpClientTest
Signed-off-by: harshithdwivedi <harshit@pitech.app>
Signed-off-by: harshithdwivedi <noreply@harshit.app>

feat : kotlinize DebuggingWordcount

Signed-off-by: harshithdwivedi <noreply@harshit.app>

feat : kotlinize MinimalWordcount

Signed-off-by: harshithdwivedi <noreply@harshit.app>

fix : failing tests

Signed-off-by: harshithdwivedi <noreply@harshit.app>

feat : kotlinize common files

Signed-off-by: harshithdwivedi <noreply@harshit.app>
Signed-off-by: harshithdwivedi <harshit@pitech.app>
Signed-off-by: harshithdwivedi <harshit@pitech.app>
Signed-off-by: Harshit Dwivedi <harshithdwivedi@gmail.com>
Signed-off-by: Harshit Dwivedi <harshit@pitech.app>
Signed-off-by: Harshit Dwivedi <harshit@pitech.app>
Signed-off-by: Harshit Dwivedi <harshit@pitech.app>
Signed-off-by: Harshit Dwivedi <harshit@pitech.app>
Signed-off-by: Harshit Dwivedi <harshit@pitech.app>
Signed-off-by: Harshit Dwivedi <harshit@pitech.app>
Signed-off-by: Harshit Dwivedi <harshit@pitech.app>
Signed-off-by: Harshit Dwivedi <harshit@pitech.app>
Signed-off-by: Harshit Dwivedi <harshit@pitech.app>
@harshithdwivedi
Copy link
Contributor Author

Looks like I messed up on my end while rebasing the PR.
I've created a fresh one incorporating the changes relevent to the Issue.
#8291

@harshithdwivedi harshithdwivedi deleted the wordcountKotlin branch April 12, 2019 10:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet