Contrib module for natsio #61

tyagihas · 2015-09-29T09:10:10Z

No description provided.

googlebot · 2015-09-29T09:10:11Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project, in which case you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please let us know the company's name.

tyagihas · 2015-09-29T09:49:07Z

I signed it!

googlebot · 2015-09-29T09:49:10Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for the commit author(s). If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.

googlebot · 2015-09-29T10:29:19Z

CLAs look good, thanks!

dhalperi · 2015-09-29T15:23:53Z

contrib/natsio/README.md

+Run tests with Maven. The following command will launch a producer and consumer job respectively.<BR>
+The producer publishes messages and the consumer will receive them via NATS server.
+```bash
+<<<<<<< HEAD


merge conflicts?

dhalperi · 2015-09-29T17:40:57Z

contrib/natsio/pom.xml

@@ -0,0 +1,58 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">


It would be good to follow the model set out in the contrib join library. Specifically

Artifact naming

License information

Compiler source and target versions (1.7, probably)

optionally, Checkstyle

Also,

review dependencies

dhalperi · 2015-10-09T18:49:44Z

contrib/natsio/src/main/java/com/google/cloud/dataflow/contrib/natsio/NatsIO.java

+						} catch (InterruptedException e) {
+							e.printStackTrace();
+						}
+					}


It looks like a NullPointerException can happen here when the thread is interrupted. This seems like the wrong handling of the InterruptedException?

Agree. I will change to throw an IOException.

Since this is for a source/sink, what you want to do is effectively tell the caller above that you were closed by an interrupt.

Throwing something like ClosedByInterruptException seems to make sense after cleaning up any resources.

We will fail the task, and restart the write operation again (on this or another worker).

Yes, that makes more sense. I will change it to throw ClosedByInterruptException.

dhalperi · 2015-10-19T21:29:38Z

Hi @tyagihas:

I am encountering the following error installing this module:

[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /home/dhalperi/tyagihas/DataflowJavaSDK/contrib/natsio/src/main/java/com/google/cloud/dataflow/contrib/nats
io/example/NatsIOBench.java:[14,48] cannot find symbol
  symbol:   class NatsIOTest
  location: package com.google.cloud.dataflow.contrib.natsio
[ERROR] /home/dhalperi/tyagihas/DataflowJavaSDK/contrib/natsio/src/main/java/com/google/cloud/dataflow/contrib/nats
io/example/NatsIOBench.java:[26,67] cannot find symbol
  symbol:   class NatsIOTest
  location: class com.google.cloud.dataflow.contrib.natsio.example.NatsIOBench
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.870s
[INFO] Finished at: Mon Oct 19 21:13:44 UTC 2015
[INFO] Final Memory: 19M/56M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on proj
ect google-cloud-dataflow-java-contrib-natsio: Compilation failure: Compilation failure:
[ERROR] /home/dhalperi/tyagihas/DataflowJavaSDK/contrib/natsio/src/main/java/com/google/cloud/dataflow/contrib/nats
io/example/NatsIOBench.java:[14,48] cannot find symbol
[ERROR] symbol:   class NatsIOTest
[ERROR] location: package com.google.cloud.dataflow.contrib.natsio
[ERROR] /home/dhalperi/tyagihas/DataflowJavaSDK/contrib/natsio/src/main/java/com/google/cloud/dataflow/contrib/nats
io/example/NatsIOBench.java:[26,67] cannot find symbol
[ERROR] symbol:   class NatsIOTest
[ERROR] location: class com.google.cloud.dataflow.contrib.natsio.example.NatsIOBench
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Here's a log of the important commands I have run, on a fresh Ubuntu 15.04 VM:

sudo apt-get update && sudo apt-get install -y git maven openjdk-8-jdk
git clone https://github.com/GoogleCloudPlatform/DataflowJavaSDK.git && cd DataflowJavaSDK && mvn install && cd ~
git clone -b natsio --single-branch https://github.com/tyagihas/DataflowJavaSDK.git natsio
cd natsio/contrib/natsio && mvn install

I suspect you want the NatsIOBench class to be in the test rather than in main. Normally, code in main should not depend on code in test.

dhalperi · 2015-10-19T21:32:52Z

Assigning back to you for now, please reassign to me once fixed :).

dhalperi · 2015-10-19T21:33:29Z

Hmm, can't re-assign to you, so please just bump the request once fixed. Thanks.

tyagihas · 2015-10-20T07:33:16Z

Fixed the class reference. NatsIOBench shouldn't include reference to NatsIOTest.
I also added some default test properties for maven test and only three parameters are mandatory for minimum testing.

mvn install -Dproject="project id" -DstagingLocation="bucket" -Dnats.servers=nats://"GCE private ip":4222

dhalperi · 2015-10-20T23:54:46Z

I ran the job as follows: mvn -Dproject="$PROJECT" -DstagingLocation="$STAGING" -D"nats.servers=nats://0.0.0.0:4222" test

Two Dataflow jobs start in my project, but they never make any progress and have not stopped after at least 10 minutes of running. In Cloud Logging, I see some very confusing logs, attached below.

It looks like some of the code needs to use a org.slf4h.Logger instead of System.err/out to report errors. Maybe this is coming from org.nats.Connection in this stack trace? https://github.com/tyagihas/java_nats/blob/master/src/main/java/org/nats/Connection.java#L215

This failure is somehow not failing the job, and I think it should.

dhalperi · 2015-10-21T00:01:48Z

Both jobs did fail after 15 minutes, I guess because it was retried 4 times.

dhalperi · 2015-10-21T00:05:44Z

contrib/natsio/README.md

+```bash
+% mvn test -Dtest=NatsIOTest#publishSubscribe -DstagingLocation=gs://<bucket> ¥
+-Dproject=<project id> -Dnats.servers=nats://<server>:4222 -Dnats.queue=queue1 ¥
+-Dloop=30000 -Dinterval=0 -Dsubjects=test1 -Dconsumers=1 -Dproducers=1 -Dnats.maxRecords=20000 ¥


This symbol that is rendering as a Yen sign for me -- is this a Japanese keyboard version of \ to mean "continues on next line"? (if so, cool!)

Yes, it's same meaning as backslash.

dhalperi · 2015-11-03T05:41:39Z

GitHub notifications don't come when patches are pushed; please comment once code is updated next.

Thanks!

tyagihas · 2015-11-05T16:10:34Z

Thank you for all the feedbacks. I've just pushed the latest code.

tyagihas · 2015-11-10T10:32:08Z

Thank you for the comments. I just push a new commit.

tyagihas · 2016-01-27T07:23:22Z

Hi @dhalperi

It's been a while since last update. Did you have time to look at the latest commit?

dhalperi · 2016-01-27T07:49:05Z

contrib/natsio/src/main/java/com/google/cloud/dataflow/contrib/natsio/NatsIO.java

+
+			public NatsSource(String subject, Properties props) {
+				if (subject == null) {
+					throw new NullPointerException("subject");


would be simpler if you use Guava Preconditions:

import static com.google.common.base.Preconditions.checkNotNull; .... checkNotNull(subject, "subject"); ...

dhalperi · 2016-03-30T20:45:40Z

No movement for 60 days since last review, so I'm closing.

Note that we are moving active development to Apache Beam (incubating). If you wish to resubmit, please read the Beam contribution guide and submit there.

teppeiy and others added 2 commits September 29, 2015 17:52

natsio Connector

6b63724

Update README.md

56ea0c2

googlebot added the cla: no label Sep 29, 2015

tyagihas added 2 commits September 29, 2015 19:23

natsio Connector

4ef586c

natsio Connector

db86cfc

googlebot added cla: yes and removed cla: no labels Sep 29, 2015

dhalperi reviewed Sep 29, 2015
View reviewed changes

dhalperi self-assigned this Sep 29, 2015

dhalperi reviewed Sep 29, 2015
View reviewed changes

tyagihas added 3 commits September 30, 2015 14:26

Fix pom.xml and remove redunant import

cf6dad2

Fixing imports

b39e8fa

Fixing arg names

1b2d269

dhalperi reviewed Oct 9, 2015
View reviewed changes

Modifying exception handling

d832453

dhalperi removed their assignment Oct 19, 2015

dhalperi self-assigned this Oct 19, 2015

Fixing class reference and adding default test properties

3c352b2

dhalperi reviewed Oct 21, 2015
View reviewed changes

Modifying test apps

49ac21e

Changing test scenario

8fa1ad4

dhalperi reviewed Jan 27, 2016
View reviewed changes

dhalperi closed this Mar 30, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Contrib module for natsio #61

Contrib module for natsio #61

tyagihas commented Sep 29, 2015

googlebot commented Sep 29, 2015

tyagihas commented Sep 29, 2015

googlebot commented Sep 29, 2015

googlebot commented Sep 29, 2015

dhalperi Sep 29, 2015

tyagihas Sep 30, 2015

dhalperi Sep 29, 2015

dhalperi Oct 9, 2015

tyagihas Oct 13, 2015

dhalperi Oct 13, 2015

tyagihas Oct 14, 2015

dhalperi commented Oct 19, 2015

dhalperi commented Oct 19, 2015

dhalperi commented Oct 19, 2015

tyagihas commented Oct 20, 2015

dhalperi commented Oct 20, 2015

dhalperi commented Oct 21, 2015

dhalperi Oct 21, 2015

tyagihas Oct 30, 2015

dhalperi commented Nov 3, 2015

tyagihas commented Nov 5, 2015

tyagihas commented Nov 10, 2015

tyagihas commented Jan 27, 2016

dhalperi Jan 27, 2016

dhalperi commented Mar 30, 2016

		@@ -0,0 +1,58 @@
		<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

Contrib module for natsio #61

Contrib module for natsio #61

Conversation

tyagihas commented Sep 29, 2015

googlebot commented Sep 29, 2015

tyagihas commented Sep 29, 2015

googlebot commented Sep 29, 2015

googlebot commented Sep 29, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhalperi commented Oct 19, 2015

dhalperi commented Oct 19, 2015

dhalperi commented Oct 19, 2015

tyagihas commented Oct 20, 2015

dhalperi commented Oct 20, 2015

dhalperi commented Oct 21, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhalperi commented Nov 3, 2015

tyagihas commented Nov 5, 2015

tyagihas commented Nov 10, 2015

tyagihas commented Jan 27, 2016

Choose a reason for hiding this comment

dhalperi commented Mar 30, 2016