Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-5] Add Flink Runner #12

Closed
wants to merge 149 commits into from
Closed

[BEAM-5] Add Flink Runner #12

wants to merge 149 commits into from

Conversation

@mxm
Copy link
Contributor

@mxm mxm commented Mar 2, 2016

As of BEAM-5 and the previous mailing list discussion, I would like to contribute the Flink Runner to Beam.

According to the repository layout document, I have created a Beam Runners project which will host the runners. Therein, I have added the Flink runner with a few modifications mostly related to code formatting and licensing. These changes you can see in the individual commits. Note, that everything builds and the tests run fine.

The Runner is in sync with the latest Beam SDK version which is currently 1.5.0-SNAPSHOT. I would like to point out that the runner also supports the stable 1.0.0 version (in the 1.0.0 branch at https://github.com/dataArtisans/flink-dataflow) but I've added the latest version here to keep up with the latest changes in Beam.

Please let me know what you think about the pull request and when you would like to merge it.

@davorbonaci
Copy link
Member

@davorbonaci davorbonaci commented Mar 3, 2016

Awesome, Max! This sounds great.

Let me take this code review. I'm also looping in Mark, who may be able to provide additional insight. I'd expect we'd be able to provide a few thoughts and get it merged quickly.

R: @davorbonaci
CC: @mshields822

@mxm
Copy link
Contributor Author

@mxm mxm commented Mar 3, 2016

Hi @davorbonaci! Thanks for the feedback. I would like to merge this pull request tomorrow if you don't have any objections. Let me know if you need anything more. Surely, we will continue to work on the Flink Runner once it is merged. We can then also transfer any open issues to the Apache Beam Jira.

@mshields822
Copy link
Contributor

@mshields822 mshields822 commented Mar 3, 2016

Agree - plenty yet to be done, but at least we are buildable with unit tests passing.

@davorbonaci
Copy link
Member

@davorbonaci davorbonaci commented Mar 4, 2016

I have just 2 ideas for consideration, none of them is blocking:

  • What do you think about changing the directory name from runners/flink-runner to runners/flink? Sounds simpler, and perhaps less unnecessary duplication. I'm sorry if I'm now giving a different recommendation than what the document initially said.
  • When we did the Dataflow code drop, we preserved all commit history of the old repository under the pretense that it may help other better understand why certain things were done. Perhaps you can consider doing the same. An extra benefit is that it gives an appropriate credit to all authors of this codebase and makes the graphs more accurate.

Let me know what you think. Either way, I think it is great. (And I'm looking forward in working with you going forward!)

@mxm
Copy link
Contributor Author

@mxm mxm commented Mar 4, 2016

  • Sure we can change the directory name from runners/flink-runner to runners/flink. Looks much nicer.
  • I didn't want to import all commits into this pull request because I thought it would discourage merging. However, if we can preserve commit history, I'm all for it. I'll push a new version with the entire history.

I'm also looking forward to working with you @davorbonaci!

@mxm
Copy link
Contributor Author

@mxm mxm commented Mar 4, 2016

Two options for preserving the commits:

  1. Make a merge commit. This merges and interleaves the whole history of the Dataflow SDK. Probably not something we want.

  2. Rebase. Unfortunately, only possible if we rewrite the Flink Runner history to be relative to the runners/flink directory. Otherwise, we run into conflicts with shared files like /pom.xml and we would loose proper history there if we changed these files during rebasing.

So I would go for option 2).

mxm and others added 5 commits Mar 4, 2016
- fix AvroITCase

- timestamp set correctly

- fix advancing of reader in UnboundedSourceWrapper

- fix final watermark

- fix pom file

- cleanup

- re-add JavaDoc
@mxm mxm force-pushed the master branch 2 times, most recently from aa4601d to 85a6568 Mar 4, 2016
@mxm
Copy link
Contributor Author

@mxm mxm commented Mar 4, 2016

There you go. I feel like a real Git magician now. For your information, I rewrote the history using

git filter-branch -f --index-filter 'git ls-files -s | perl -pe "s/\t/\trunners\/flink/g" |
   GIT_INDEX_FILE=${GIT_INDEX_FILE}.new git update-index --index-info &&
   mv "${GIT_INDEX_FILE}.new" "$GIT_INDEX_FILE"' HEAD

And then put the other changes on top again. Everything integrates nicely and should be good to merge now :)

@asfgit asfgit closed this in 4da935b Mar 4, 2016
@davorbonaci
Copy link
Member

@davorbonaci davorbonaci commented Mar 4, 2016

This looks great. An awesome milestone for the entire Beam project!

( I went ahead and merged this for you while we sort out your permissions issues. )

@mxm
Copy link
Contributor Author

@mxm mxm commented Mar 4, 2016

Thanks for the merge! Let's take Beam to the next level :)

@tomwhite tomwhite mentioned this pull request Mar 10, 2016
aljoscha pushed a commit to aljoscha/beam that referenced this issue Mar 14, 2018
Only access output coder if available
tvalentyn pushed a commit to tvalentyn/beam that referenced this issue May 15, 2018
pabloem pushed a commit to pabloem/beam that referenced this issue Feb 13, 2021
* New DebeziumIO class.

* Merge connector code

* DebeziumIO and MySqlConnector integrated.

* Added FormatFuntion param to Read builder on DebeziumIO.

* Added arguments checker to DebeziumIO.

* Add simple JSON mapper object (#1)

* Add simple JSON mapper object

* Fixed Mapper.

* Add SqlServer connector test

* Added PostgreSql Connector Test

PostgreSql now works with Json mapper

* Added PostgreSql Connector Test

PostgreSql now works with Json mapper

* Fixing MySQL schema DataException

Using file instead of schema should fix it

* MySQL Connector updated from 1.3.0 to 1.3.1

Co-authored-by: osvaldo-salinas <osvaldo.salinas@wizeline.com>
Co-authored-by: Carlos Dominguez <carlos.dominguez@carlos.dominguez>
Co-authored-by: Carlos Domínguez <carlos.dominguez@wizeline.com>

* Add debeziumio tests

* Debeziumio testing json mapper (#3)

* Some code refactors. Use a default DBHistory if not provided

* Add basic tests for Json mapper

* Debeziumio time restriction (apache#5)

* Add simple JSON mapper object

* Fixed Mapper.

* Add SqlServer connector test

* Added PostgreSql Connector Test

PostgreSql now works with Json mapper

* Added PostgreSql Connector Test

PostgreSql now works with Json mapper

* Fixing MySQL schema DataException

Using file instead of schema should fix it

* MySQL Connector updated from 1.3.0 to 1.3.1

* Some code refactors. Use a default DBHistory if not provided

* Adding based-time restriction

Stop polling after specified amount of time

* Add basic tests for Json mapper

* Adding new restriction

Uses a time-based restriction

* Adding optional restrcition

Uses an optional time-based restriction

Co-authored-by: juanitodread <juanitodread@gmail.com>
Co-authored-by: osvaldo-salinas <osvaldo.salinas@wizeline.com>

* Upgrade DebeziumIO connector (apache#4)

* Address comments (Change dependencies to testCompile, Set JsonMapper/Coder as default, refactors) (apache#8)

* Revert file

* Change dependencies to testCompile
* Move Counter sample to unit test

* Set JsonMapper as default mapper function
* Set String Coder as default coder when using JsonMapper
* Change logs from info to debug

* Debeziumio javadoc (apache#9)

* Adding javadoc

* Added some titles and examples

* Added SourceRecordJson doc

* Added Basic Connector doc

* Added KafkaSourceConsumer doc

* Javadoc cleanup

* Removing BasicConnector

No usages of this class were found overall

* Editing documentation

* Debeziumio fetched records restriction (apache#10)

* Adding javadoc

* Adding restriction by number of fetched records

Also adding a quick-fix for null value within SourceRecords
Minor fix on both MySQL and PostgreSQL Connectors Tests

* Run either by time or by number of records

* Added DebeziumOffsetTrackerTest

Tests both restrictions: By amount of time and by Number of records

* Removing comment

* DebeziumIO test for DB2. (apache#11)

* DebeziumIO test for DB2.

* DebeziumIO javadoc.

* Clean code:removed commented code lines on DebeziumIOConnectorTest.java

* Clean code:removing unused imports and using readAsJson().

Co-authored-by: Carlos Domínguez <74681048+carlosdominguezwl@users.noreply.github.com>

* Debezium limit records (now configurable) (apache#12)

* Adding javadoc

* Records Limit is now configurable

(It was fixed before)

* Debeziumio dockerize (apache#13)

* Add mysql docker container to tests

* Move debezium mysql integration test to its own file

* Add assertion to verify that the results contains a record.

* Debeziumio readme (apache#15)

* Adding javadoc

* Adding README file

* Add number of records configuration to the DebeziumIO component (apache#16)

* Code refactors (apache#17)

* Remove/ignore null warnings

* Remove DB2 code

* Remove docker dependency in DebeziumIO unit test and max number of recods to MySql integration test

* Change access modifiers accordingly

* Remove incomplete integration tests (Postgres and SqlServer)

* Add experimenal tag

* Debezium testing stoppable consumer (apache#18)

* Add try-catch-finally, stop SourceTask at finally.

* Fix warnings

* stopConsumer and processedRecords local variables removed. UT for task stop use case added

* Fix minor code style issue

Co-authored-by: juanitodread <juanitodread@gmail.com>

* Fix style issues (check, spotlessApply) (apache#19)

Co-authored-by: Osvaldo Salinas <osvaldo.salinas@osvaldo.salinas>
Co-authored-by: alejandro.maguey <alejandro.maguey@wizeline.com>
Co-authored-by: osvaldo-salinas <osvaldo.salinas@wizeline.com>
Co-authored-by: Carlos Dominguez <carlos.dominguez@carlos.dominguez>
Co-authored-by: Carlos Domínguez <carlos.dominguez@wizeline.com>
Co-authored-by: Carlos Domínguez <74681048+carlosdominguezwl@users.noreply.github.com>
Co-authored-by: Alejandro Maguey <alexmaguey1@gmail.com>
Co-authored-by: Hassan Reyes <hassanreyes@users.noreply.github.com>
pabloem pushed a commit that referenced this issue Feb 17, 2021
Debeziumio PoC (#7)

* New DebeziumIO class.

* Merge connector code

* DebeziumIO and MySqlConnector integrated.

* Added FormatFuntion param to Read builder on DebeziumIO.

* Added arguments checker to DebeziumIO.

* Add simple JSON mapper object (#1)

* Add simple JSON mapper object

* Fixed Mapper.

* Add SqlServer connector test

* Added PostgreSql Connector Test

PostgreSql now works with Json mapper

* Added PostgreSql Connector Test

PostgreSql now works with Json mapper

* Fixing MySQL schema DataException

Using file instead of schema should fix it

* MySQL Connector updated from 1.3.0 to 1.3.1

Co-authored-by: osvaldo-salinas <osvaldo.salinas@wizeline.com>
Co-authored-by: Carlos Dominguez <carlos.dominguez@carlos.dominguez>
Co-authored-by: Carlos Domínguez <carlos.dominguez@wizeline.com>

* Add debeziumio tests

* Debeziumio testing json mapper (#3)

* Some code refactors. Use a default DBHistory if not provided

* Add basic tests for Json mapper

* Debeziumio time restriction (#5)

* Add simple JSON mapper object

* Fixed Mapper.

* Add SqlServer connector test

* Added PostgreSql Connector Test

PostgreSql now works with Json mapper

* Added PostgreSql Connector Test

PostgreSql now works with Json mapper

* Fixing MySQL schema DataException

Using file instead of schema should fix it

* MySQL Connector updated from 1.3.0 to 1.3.1

* Some code refactors. Use a default DBHistory if not provided

* Adding based-time restriction

Stop polling after specified amount of time

* Add basic tests for Json mapper

* Adding new restriction

Uses a time-based restriction

* Adding optional restrcition

Uses an optional time-based restriction

Co-authored-by: juanitodread <juanitodread@gmail.com>
Co-authored-by: osvaldo-salinas <osvaldo.salinas@wizeline.com>

* Upgrade DebeziumIO connector (#4)

* Address comments (Change dependencies to testCompile, Set JsonMapper/Coder as default, refactors) (#8)

* Revert file

* Change dependencies to testCompile
* Move Counter sample to unit test

* Set JsonMapper as default mapper function
* Set String Coder as default coder when using JsonMapper
* Change logs from info to debug

* Debeziumio javadoc (#9)

* Adding javadoc

* Added some titles and examples

* Added SourceRecordJson doc

* Added Basic Connector doc

* Added KafkaSourceConsumer doc

* Javadoc cleanup

* Removing BasicConnector

No usages of this class were found overall

* Editing documentation

* Debeziumio fetched records restriction (#10)

* Adding javadoc

* Adding restriction by number of fetched records

Also adding a quick-fix for null value within SourceRecords
Minor fix on both MySQL and PostgreSQL Connectors Tests

* Run either by time or by number of records

* Added DebeziumOffsetTrackerTest

Tests both restrictions: By amount of time and by Number of records

* Removing comment

* DebeziumIO test for DB2. (#11)

* DebeziumIO test for DB2.

* DebeziumIO javadoc.

* Clean code:removed commented code lines on DebeziumIOConnectorTest.java

* Clean code:removing unused imports and using readAsJson().

Co-authored-by: Carlos Domínguez <74681048+carlosdominguezwl@users.noreply.github.com>

* Debezium limit records (now configurable) (#12)

* Adding javadoc

* Records Limit is now configurable

(It was fixed before)

* Debeziumio dockerize (#13)

* Add mysql docker container to tests

* Move debezium mysql integration test to its own file

* Add assertion to verify that the results contains a record.

* Debeziumio readme (#15)

* Adding javadoc

* Adding README file

* Add number of records configuration to the DebeziumIO component (#16)

* Code refactors (#17)

* Remove/ignore null warnings

* Remove DB2 code

* Remove docker dependency in DebeziumIO unit test and max number of recods to MySql integration test

* Change access modifiers accordingly

* Remove incomplete integration tests (Postgres and SqlServer)

* Add experimenal tag

* Debezium testing stoppable consumer (#18)

* Add try-catch-finally, stop SourceTask at finally.

* Fix warnings

* stopConsumer and processedRecords local variables removed. UT for task stop use case added

* Fix minor code style issue

Co-authored-by: juanitodread <juanitodread@gmail.com>

* Fix style issues (check, spotlessApply) (#19)

Co-authored-by: Osvaldo Salinas <osvaldo.salinas@osvaldo.salinas>
Co-authored-by: alejandro.maguey <alejandro.maguey@wizeline.com>
Co-authored-by: osvaldo-salinas <osvaldo.salinas@wizeline.com>
Co-authored-by: Carlos Dominguez <carlos.dominguez@carlos.dominguez>
Co-authored-by: Carlos Domínguez <carlos.dominguez@wizeline.com>
Co-authored-by: Carlos Domínguez <74681048+carlosdominguezwl@users.noreply.github.com>
Co-authored-by: Alejandro Maguey <alexmaguey1@gmail.com>
Co-authored-by: Hassan Reyes <hassanreyes@users.noreply.github.com>

Add missing apache license to README.md

Enabling integration test for DebeziumIO (#20)

Rename connector package cdc=>debezium. Update doc references (#21)

Fix code style on DebeziumIOMySqlConnectorIT
usingh83 added a commit to usingh83/beam that referenced this issue May 7, 2021
# This is the 1st commit message:

Java PreCommit failure fix

spotless failure fix

 Java PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit refix

Java_Examples_Dataflow PreCommit fix

build failure corrected

Spotless check

Spotless check

reorganizing pipeline

delete the unused folder

Revert "Delete build.gradle"

This reverts commit c39a4e4

Delete build.gradle

don't need this file

adding comments and java docs, and removing unneeded dependencies.

Linting the project and making some stuff private

Reorganized and redefined to logic as per standard beam IO structure.

Lint the files.

Added changes for making the implementation more streamlined and understandable

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

# This is a combination of 15 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message apache#3:

Lint the files.

# This is the commit message apache#4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message apache#5:

Linting the project and making some stuff private

# This is the commit message apache#6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message apache#7:

delete the unused folder

# This is the commit message apache#8:

reorganizing pipeline

# This is the commit message apache#9:

Spotless check

# This is the commit message apache#10:

Spotless check

# This is the commit message apache#11:

build failure corrected

# This is the commit message apache#12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message apache#13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message apache#14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#15:

Java_Examples_Dataflow PreCommit assign nullable correctly
usingh83 added a commit to usingh83/beam that referenced this issue May 7, 2021
# This is the 1st commit message:

Java PreCommit failure fix

spotless failure fix

 Java PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit refix

Java_Examples_Dataflow PreCommit fix

build failure corrected

Spotless check

Spotless check

reorganizing pipeline

delete the unused folder

Revert "Delete build.gradle"

This reverts commit c39a4e4

Delete build.gradle

don't need this file

adding comments and java docs, and removing unneeded dependencies.

Linting the project and making some stuff private

Reorganized and redefined to logic as per standard beam IO structure.

Lint the files.

Added changes for making the implementation more streamlined and understandable

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

# This is a combination of 15 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message apache#3:

Lint the files.

# This is the commit message apache#4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message apache#5:

Linting the project and making some stuff private

# This is the commit message apache#6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message apache#7:

delete the unused folder

# This is the commit message apache#8:

reorganizing pipeline

# This is the commit message apache#9:

Spotless check

# This is the commit message apache#10:

Spotless check

# This is the commit message apache#11:

build failure corrected

# This is the commit message apache#12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message apache#13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message apache#14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#3:

# This is a combination of 16 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message apache#3:

Lint the files.

# This is the commit message apache#4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message apache#5:

Linting the project and making some stuff private

# This is the commit message apache#6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message apache#7:

delete the unused folder

# This is the commit message apache#8:

reorganizing pipeline

# This is the commit message apache#9:

Spotless check

# This is the commit message apache#10:

Spotless check

# This is the commit message apache#11:

build failure corrected

# This is the commit message apache#12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message apache#13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message apache#14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#16:

 Java PreCommit assign nullable correctly

 Java PreCommit assign nullable correctly

spotless failure fix

Java PreCommit failure fix

correcting the if checks

cleaning up and adding readme

spotless fixed

readme fixed and compileJava
 fix

compileJava fix

compileJava fix now

spotless fix now

Java PreCommi fix

Java PreCommit fix

# This is a combination of 16 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message apache#3:

Lint the files.

# This is the commit message apache#4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message apache#5:

Linting the project and making some stuff private

# This is the commit message apache#6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message apache#7:

delete the unused folder

# This is the commit message apache#8:

reorganizing pipeline

# This is the commit message apache#9:

Spotless check

# This is the commit message apache#10:

Spotless check

# This is the commit message apache#11:

build failure corrected

# This is the commit message apache#12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message apache#13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message apache#14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#16:

 Java PreCommit assign nullable correctly

 Java PreCommit assign nullable correctly

spotless failure fix

Java PreCommit failure fix

correcting the if checks

cleaning up and adding readme

spotless fixed

readme fixed and compileJava
 fix

compileJava fix

compileJava fix now

spotless fix now

Java PreCommi fix

Java PreCommit fix
usingh83 added a commit to usingh83/beam that referenced this issue May 13, 2021
# This is the 1st commit message:

# This is a combination of 2 commits.
# This is the 1st commit message:

Java PreCommit failure fix

spotless failure fix

 Java PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit refix

Java_Examples_Dataflow PreCommit fix

build failure corrected

Spotless check

Spotless check

reorganizing pipeline

delete the unused folder

Revert "Delete build.gradle"

This reverts commit c39a4e4

Delete build.gradle

don't need this file

adding comments and java docs, and removing unneeded dependencies.

Linting the project and making some stuff private

Reorganized and redefined to logic as per standard beam IO structure.

Lint the files.

Added changes for making the implementation more streamlined and understandable

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

# This is a combination of 15 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message apache#3:

Lint the files.

# This is the commit message apache#4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message apache#5:

Linting the project and making some stuff private

# This is the commit message apache#6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message apache#7:

delete the unused folder

# This is the commit message apache#8:

reorganizing pipeline

# This is the commit message apache#9:

Spotless check

# This is the commit message apache#10:

Spotless check

# This is the commit message apache#11:

build failure corrected

# This is the commit message apache#12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message apache#13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message apache#14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#2:

# This is a combination of 3 commits.
# This is the 1st commit message:

Java PreCommit failure fix

spotless failure fix

 Java PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit refix

Java_Examples_Dataflow PreCommit fix

build failure corrected

Spotless check

Spotless check

reorganizing pipeline

delete the unused folder

Revert "Delete build.gradle"

This reverts commit c39a4e4

Delete build.gradle

don't need this file

adding comments and java docs, and removing unneeded dependencies.

Linting the project and making some stuff private

Reorganized and redefined to logic as per standard beam IO structure.

Lint the files.

Added changes for making the implementation more streamlined and understandable

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

# This is a combination of 15 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message apache#3:

Lint the files.

# This is the commit message apache#4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message apache#5:

Linting the project and making some stuff private

# This is the commit message apache#6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message apache#7:

delete the unused folder

# This is the commit message apache#8:

reorganizing pipeline

# This is the commit message apache#9:

Spotless check

# This is the commit message apache#10:

Spotless check

# This is the commit message apache#11:

build failure corrected

# This is the commit message apache#12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message apache#13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message apache#14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#3:

# This is a combination of 16 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message apache#3:

Lint the files.

# This is the commit message apache#4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message apache#5:

Linting the project and making some stuff private

# This is the commit message apache#6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message apache#7:

delete the unused folder

# This is the commit message apache#8:

reorganizing pipeline

# This is the commit message apache#9:

Spotless check

# This is the commit message apache#10:

Spotless check

# This is the commit message apache#11:

build failure corrected

# This is the commit message apache#12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message apache#13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message apache#14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#16:

 Java PreCommit assign nullable correctly

 Java PreCommit assign nullable correctly

spotless failure fix

Java PreCommit failure fix

correcting the if checks

cleaning up and adding readme

spotless fixed

readme fixed and compileJava
 fix

compileJava fix

compileJava fix now

spotless fix now

Java PreCommi fix

Java PreCommit fix

# This is a combination of 16 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message apache#3:

Lint the files.

# This is the commit message apache#4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message apache#5:

Linting the project and making some stuff private

# This is the commit message apache#6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message apache#7:

delete the unused folder

# This is the commit message apache#8:

reorganizing pipeline

# This is the commit message apache#9:

Spotless check

# This is the commit message apache#10:

Spotless check

# This is the commit message apache#11:

build failure corrected

# This is the commit message apache#12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message apache#13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message apache#14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#16:

 Java PreCommit assign nullable correctly

 Java PreCommit assign nullable correctly

spotless failure fix

Java PreCommit failure fix

correcting the if checks

cleaning up and adding readme

spotless fixed

readme fixed and compileJava
 fix

compileJava fix

compileJava fix now

spotless fix now

Java PreCommi fix

Java PreCommit fix

# This is a combination of 3 commits.
# This is the 1st commit message:

Java PreCommit failure fix

spotless failure fix

 Java PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit refix

Java_Examples_Dataflow PreCommit fix

build failure corrected

Spotless check

Spotless check

reorganizing pipeline

delete the unused folder

Revert "Delete build.gradle"

This reverts commit c39a4e4

Delete build.gradle

don't need this file

adding comments and java docs, and removing unneeded dependencies.

Linting the project and making some stuff private

Reorganized and redefined to logic as per standard beam IO structure.

Lint the files.

Added changes for making the implementation more streamlined and understandable

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

# This is a combination of 15 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message apache#3:

Lint the files.

# This is the commit message apache#4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message apache#5:

Linting the project and making some stuff private

# This is the commit message apache#6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message apache#7:

delete the unused folder

# This is the commit message apache#8:

reorganizing pipeline

# This is the commit message apache#9:

Spotless check

# This is the commit message apache#10:

Spotless check

# This is the commit message apache#11:

build failure corrected

# This is the commit message apache#12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message apache#13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message apache#14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#3:

# This is a combination of 16 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message apache#3:

Lint the files.

# This is the commit message apache#4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message apache#5:

Linting the project and making some stuff private

# This is the commit message apache#6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message apache#7:

delete the unused folder

# This is the commit message apache#8:

reorganizing pipeline

# This is the commit message apache#9:

Spotless check

# This is the commit message apache#10:

Spotless check

# This is the commit message apache#11:

build failure corrected

# This is the commit message apache#12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message apache#13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message apache#14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#16:

 Java PreCommit assign nullable correctly

 Java PreCommit assign nullable correctly

spotless failure fix

Java PreCommit failure fix

correcting the if checks

cleaning up and adding readme

spotless fixed

readme fixed and compileJava
 fix

compileJava fix

compileJava fix now

spotless fix now

Java PreCommi fix

Java PreCommit fix

# This is a combination of 16 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message apache#2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message apache#3:

Lint the files.

# This is the commit message apache#4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message apache#5:

Linting the project and making some stuff private

# This is the commit message apache#6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message apache#7:

delete the unused folder

# This is the commit message apache#8:

reorganizing pipeline

# This is the commit message apache#9:

Spotless check

# This is the commit message apache#10:

Spotless check

# This is the commit message apache#11:

build failure corrected

# This is the commit message apache#12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message apache#13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message apache#14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message apache#16:

 Java PreCommit assign nullable correctly

 Java PreCommit assign nullable correctly

spotless failure fix

Java PreCommit failure fix

correcting the if checks

cleaning up and adding readme

spotless fixed

readme fixed and compileJava
 fix

compileJava fix

compileJava fix now

spotless fix now

Java PreCommi fix

Java PreCommit fix

Final Commit with all changes

Added unit test

adding examples for usage

usage for TwitterIO added and Java PreCommit failure fix

Spotless PreCommit failure fix
pabloem pushed a commit that referenced this issue May 18, 2021
…eams data from twitter

* # This is a combination of 2 commits.
# This is the 1st commit message:

Java PreCommit failure fix

spotless failure fix

 Java PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit refix

Java_Examples_Dataflow PreCommit fix

build failure corrected

Spotless check

Spotless check

reorganizing pipeline

delete the unused folder

Revert "Delete build.gradle"

This reverts commit c39a4e4

Delete build.gradle

don't need this file

adding comments and java docs, and removing unneeded dependencies.

Linting the project and making some stuff private

Reorganized and redefined to logic as per standard beam IO structure.

Lint the files.

Added changes for making the implementation more streamlined and understandable

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

# This is a combination of 15 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message #3:

Lint the files.

# This is the commit message #4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message #5:

Linting the project and making some stuff private

# This is the commit message #6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message #7:

delete the unused folder

# This is the commit message #8:

reorganizing pipeline

# This is the commit message #9:

Spotless check

# This is the commit message #10:

Spotless check

# This is the commit message #11:

build failure corrected

# This is the commit message #12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message #13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message #14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #15:

Java_Examples_Dataflow PreCommit assign nullable correctly

* # This is a combination of 2 commits.
# This is the 1st commit message:

# This is a combination of 2 commits.
# This is the 1st commit message:

Java PreCommit failure fix

spotless failure fix

 Java PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit refix

Java_Examples_Dataflow PreCommit fix

build failure corrected

Spotless check

Spotless check

reorganizing pipeline

delete the unused folder

Revert "Delete build.gradle"

This reverts commit c39a4e4

Delete build.gradle

don't need this file

adding comments and java docs, and removing unneeded dependencies.

Linting the project and making some stuff private

Reorganized and redefined to logic as per standard beam IO structure.

Lint the files.

Added changes for making the implementation more streamlined and understandable

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

# This is a combination of 15 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message #3:

Lint the files.

# This is the commit message #4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message #5:

Linting the project and making some stuff private

# This is the commit message #6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message #7:

delete the unused folder

# This is the commit message #8:

reorganizing pipeline

# This is the commit message #9:

Spotless check

# This is the commit message #10:

Spotless check

# This is the commit message #11:

build failure corrected

# This is the commit message #12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message #13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message #14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #2:

# This is a combination of 3 commits.
# This is the 1st commit message:

Java PreCommit failure fix

spotless failure fix

 Java PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit refix

Java_Examples_Dataflow PreCommit fix

build failure corrected

Spotless check

Spotless check

reorganizing pipeline

delete the unused folder

Revert "Delete build.gradle"

This reverts commit c39a4e4

Delete build.gradle

don't need this file

adding comments and java docs, and removing unneeded dependencies.

Linting the project and making some stuff private

Reorganized and redefined to logic as per standard beam IO structure.

Lint the files.

Added changes for making the implementation more streamlined and understandable

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

# This is a combination of 15 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message #3:

Lint the files.

# This is the commit message #4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message #5:

Linting the project and making some stuff private

# This is the commit message #6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message #7:

delete the unused folder

# This is the commit message #8:

reorganizing pipeline

# This is the commit message #9:

Spotless check

# This is the commit message #10:

Spotless check

# This is the commit message #11:

build failure corrected

# This is the commit message #12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message #13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message #14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #3:

# This is a combination of 16 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message #3:

Lint the files.

# This is the commit message #4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message #5:

Linting the project and making some stuff private

# This is the commit message #6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message #7:

delete the unused folder

# This is the commit message #8:

reorganizing pipeline

# This is the commit message #9:

Spotless check

# This is the commit message #10:

Spotless check

# This is the commit message #11:

build failure corrected

# This is the commit message #12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message #13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message #14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #16:

 Java PreCommit assign nullable correctly

 Java PreCommit assign nullable correctly

spotless failure fix

Java PreCommit failure fix

correcting the if checks

cleaning up and adding readme

spotless fixed

readme fixed and compileJava
 fix

compileJava fix

compileJava fix now

spotless fix now

Java PreCommi fix

Java PreCommit fix

# This is a combination of 16 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message #3:

Lint the files.

# This is the commit message #4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message #5:

Linting the project and making some stuff private

# This is the commit message #6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message #7:

delete the unused folder

# This is the commit message #8:

reorganizing pipeline

# This is the commit message #9:

Spotless check

# This is the commit message #10:

Spotless check

# This is the commit message #11:

build failure corrected

# This is the commit message #12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message #13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message #14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #16:

 Java PreCommit assign nullable correctly

 Java PreCommit assign nullable correctly

spotless failure fix

Java PreCommit failure fix

correcting the if checks

cleaning up and adding readme

spotless fixed

readme fixed and compileJava
 fix

compileJava fix

compileJava fix now

spotless fix now

Java PreCommi fix

Java PreCommit fix

# This is a combination of 3 commits.
# This is the 1st commit message:

Java PreCommit failure fix

spotless failure fix

 Java PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit assign nullable correctly

Java_Examples_Dataflow PreCommit refix

Java_Examples_Dataflow PreCommit fix

build failure corrected

Spotless check

Spotless check

reorganizing pipeline

delete the unused folder

Revert "Delete build.gradle"

This reverts commit c39a4e4

Delete build.gradle

don't need this file

adding comments and java docs, and removing unneeded dependencies.

Linting the project and making some stuff private

Reorganized and redefined to logic as per standard beam IO structure.

Lint the files.

Added changes for making the implementation more streamlined and understandable

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

# This is a combination of 15 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message #3:

Lint the files.

# This is the commit message #4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message #5:

Linting the project and making some stuff private

# This is the commit message #6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message #7:

delete the unused folder

# This is the commit message #8:

reorganizing pipeline

# This is the commit message #9:

Spotless check

# This is the commit message #10:

Spotless check

# This is the commit message #11:

build failure corrected

# This is the commit message #12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message #13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message #14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #3:

# This is a combination of 16 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message #3:

Lint the files.

# This is the commit message #4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message #5:

Linting the project and making some stuff private

# This is the commit message #6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message #7:

delete the unused folder

# This is the commit message #8:

reorganizing pipeline

# This is the commit message #9:

Spotless check

# This is the commit message #10:

Spotless check

# This is the commit message #11:

build failure corrected

# This is the commit message #12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message #13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message #14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #16:

 Java PreCommit assign nullable correctly

 Java PreCommit assign nullable correctly

spotless failure fix

Java PreCommit failure fix

correcting the if checks

cleaning up and adding readme

spotless fixed

readme fixed and compileJava
 fix

compileJava fix

compileJava fix now

spotless fix now

Java PreCommi fix

Java PreCommit fix

# This is a combination of 16 commits.
# This is the 1st commit message:

Added a connector that streams data from twitter using a Standard Twitter app.

# This is the commit message #2:

Added changes for making the implementation more streamlined and understandable

# This is the commit message #3:

Lint the files.

# This is the commit message #4:

Reorganized and redefined to logic as per standard beam IO structure.

# This is the commit message #5:

Linting the project and making some stuff private

# This is the commit message #6:

adding comments and java docs, and removing unneeded dependencies.

# This is the commit message #7:

delete the unused folder

# This is the commit message #8:

reorganizing pipeline

# This is the commit message #9:

Spotless check

# This is the commit message #10:

Spotless check

# This is the commit message #11:

build failure corrected

# This is the commit message #12:

Java_Examples_Dataflow PreCommit fix

# This is the commit message #13:

Java_Examples_Dataflow PreCommit refix

# This is the commit message #14:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #15:

Java_Examples_Dataflow PreCommit assign nullable correctly

# This is the commit message #16:

 Java PreCommit assign nullable correctly

 Java PreCommit assign nullable correctly

spotless failure fix

Java PreCommit failure fix

correcting the if checks

cleaning up and adding readme

spotless fixed

readme fixed and compileJava
 fix

compileJava fix

compileJava fix now

spotless fix now

Java PreCommi fix

Java PreCommit fix

Final Commit with all changes

Added unit test

adding examples for usage

usage for TwitterIO added and Java PreCommit failure fix

Spotless PreCommit failure fix

* Unit test for multiple config added, and beautification

* Spotless apply fixed

* Removing redundant comments

* Removing newly added test

* adding newly added test back
MabelYC pushed a commit to MabelYC/beam that referenced this issue Sep 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
8 participants