[BEAM-1237] Create AmqpIO #1725

jbonofre · 2017-01-02T10:46:06Z

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
If this contribution is large, please file an Apache
Individual Contributor License Agreement.

asfbot · 2017-01-02T11:12:19Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/6360/
--none--

davorbonaci · 2017-02-27T18:07:45Z

Is this ready for review? Perhaps assign a reviewer?

jbonofre · 2017-03-28T06:39:32Z

I'm updating/polishing this one. It should be ready for review just after.

jbonofre · 2017-06-06T11:47:05Z

Resuming/finalizing my work on this one.

coveralls · 2017-06-06T12:27:51Z

Coverage decreased (-0.4%) to 69.401% when pulling f446eed on jbonofre:BEAM-1237-AMQPIO into 810db7f on apache:master.

jbonofre · 2017-06-06T14:35:17Z

R: @jkff
CC: @dhalperi

coveralls · 2017-06-06T15:45:26Z

Coverage decreased (-0.02%) to 70.62% when pulling eb3afd6 on jbonofre:BEAM-1237-AMQPIO into aebd3a4 on apache:master.

jkff

Thanks, here's a first round.

jkff · 2017-06-06T21:11:13Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java

+ * AmqpIO supports AMQP 1.0 protocol using the Apache QPid Proton-J library.
+ *
+ * <p>If you want to use AMQP 0.9 protocol, you might also be interested in the Apache Beam
+ * RabbitMqIO.


Which does not exist, right? So better not mention it here

It's in another pull request. Ok to remove this note.

jkff · 2017-06-06T21:12:46Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java

+      for (String address : addresses()) {
+        stringBuilder.append(address).append(" ");
+      }
+      builder.add(DisplayData.item("addresses", stringBuilder.toString()));


Joiner.on(" ").join(addresses())

jkff · 2017-06-06T21:15:40Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java

+        Tracker tracker = messenger.incomingTracker();
+        checkpointMark.trackers.add(tracker);
+        currentTimestamp = new Instant(message.getCreationTime());
+        if (currentTimestamp.isBefore(checkpointMark.watermark)) {


Hm, why? This will set the watermark to the lowest message timestamp ever observed, right?

Yes, the watermark is the lowest message timestamp in the checkpoint (waiting finalize). Any other suggestion ?

jkff · 2017-06-06T21:17:23Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java

+   * A {@link PTransform} to read/receive messages using AMQP 1.0 protocol.
+   */
+  @AutoValue
+  public abstract static class Read extends PTransform<PBegin, PCollection<String>> {


Can this return a PCollection<Message>, just like Pubsub and Kafka do? Likewise for write.

jkff · 2017-06-06T21:18:16Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java

+    @Override
+    public boolean advance() {
+      messenger.recv();
+      if (messenger.incoming() > 0) {


Invert this to reduce nesting

jkff · 2017-06-06T21:19:18Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java

+      for (String address : addresses()) {
+        stringBuilder.append(address).append(" ");
+      }
+      builder.add(DisplayData.item("addresses", stringBuilder.toString()));


jkff · 2017-06-06T21:20:12Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java

+          Message message = Message.Factory.create();
+          message.setAddress(address);
+          if (spec.subject() != null) {
+            message.setSubject(spec.subject());


Seems a bit weird that this is the only field we allow to set except body. Would be less weird if the IO took Message's rather than String :)

jkff · 2017-06-06T21:21:01Z

sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpIOTest.java

+  public void testRead() throws Exception {
+    PCollection<String> output = pipeline.apply(AmqpIO.read()
+        .withMaxNumRecords(10)
+        .withAddresses(Collections.singletonList("amqp://~localhost:" + port)));


Does this work under ipv6 too? I think I've seen trouble before with explicitly using localhost...

It should work but I will double check anyway.

jkff · 2017-06-06T21:21:28Z

sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpIOTest.java

+    };
+    sender.start();
+    pipeline.run();
+    sender.join();


Should be in try/finally w.r.t. pipeline.run()

Still relevant.

jkff · 2017-06-06T21:22:03Z

sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpIOTest.java

+    receiver.start();
+
+    List<String> data = new ArrayList<>();
+    for (int i = 0; i < 1000; i++) {


Use fewer messages and make the test faster and less log-spammy?

jbonofre · 2017-06-12T12:10:07Z

@jkff I implemented what you suggested: dealing directly with PCollection<Message> and providing the corresponding Coder.
NB: sorry, I squashed my changes by mistake.

coveralls · 2017-06-12T13:19:45Z

Coverage increased (+0.02%) to 70.675% when pulling 694ff7e on jbonofre:BEAM-1237-AMQPIO into 1597f3c on apache:master.

jbonofre · 2017-06-13T19:24:25Z

I'm working on:

the coder will encode/decode the entire message (not only the body)
registration of the coder.
I will update the pull request accordingly.

coveralls · 2017-06-14T21:41:59Z

Coverage remained the same at 70.681% when pulling e58dda8 on jbonofre:BEAM-1237-AMQPIO into dd9abc3 on apache:master.

jbonofre · 2017-06-20T10:03:31Z

@jkff I updated the PR with a AmqpMessage which is a wrapper of Message in order to provide equals() method (required by the MutationDetector to check if there's no mutation in the elements of a PCollection).

jbonofre · 2017-06-20T10:03:47Z

@jkff I think the PR is ready for a new review round.

coveralls · 2017-06-20T11:33:46Z

Changes Unknown when pulling dd52740 on jbonofre:BEAM-1237-AMQPIO into ** on apache:master**.

jkff

Thanks. For the coder problem - is it possible that Message.encode/decode is non-deterministic? Are you familiar with the encoding format?

jkff · 2017-06-20T22:47:10Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java

+      Tracker tracker = messenger.incomingTracker();
+      checkpointMark.trackers.add(tracker);
+      currentTimestamp = new Instant(message.getCreationTime());
+      if (currentTimestamp.isBefore(checkpointMark.watermark)) {


This doesn't seem like a very useful definition of watermark - "min(all seen messages)" will only go backwards; watermarks should not go backwards.

jkff · 2017-06-20T22:48:24Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java

+    }
+
+    /**
+     * Define the AMQP messages subject.


Why do you need this? The user can already set the subject on the Message in the PCollection.

jkff · 2017-06-20T22:49:00Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java

+      public void processElement(ProcessContext processContext) throws Exception {
+        AmqpMessage message = processContext.element();
+        for (String address : spec.addresses()) {
+          message.getMessage().setAddress(address);


Mutating the current element is forbidden, you need to make a copy

jkff · 2017-06-20T22:49:28Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessage.java

+import org.apache.qpid.proton.message.Message;
+
+/**
+ * Extend AMQ ProtonJ Message to override the equals() method.


This is unnecessary per discussion on beam-dev - please remove this class and let's figure out why coder-based equality is not helping.

jkff · 2017-06-20T22:49:55Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoder.java

+    return new AmqpMessageCoder();
+  }
+
+  // private static final int[] MESSAGE_SIZES = [1 << 14 /* 16 KiB */,1 << 20 /* 1 MiB */, 1 << 26


Seems unfinished.

jkff · 2017-06-20T22:55:17Z

...ava/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderProviderRegistrar.java

+  @Override
+  public List<CoderProvider> getCoderProviders() {
+    return ImmutableList.of(
+        CoderProviders.forCoder(TypeDescriptor.of(Message.class),


This provides a coder for Message, but the API is now in terms of AmqpMessage. I suppose this comment can be resolved once you remove AmqpMessage.

jbonofre · 2017-06-21T11:51:43Z

@jkff I updated according to your comments. Especially, I changed:

directly use ProtonJ Message in PCollection
simplify WriteFn to use address and subject defined in the message elements of the PCollection
change the watermark to the message creation date
This PR is ready for a new review round. Thanks !

coveralls · 2017-06-21T13:19:53Z

Coverage increased (+0.1%) to 70.839% when pulling 6cc24e5 on jbonofre:BEAM-1237-AMQPIO into 50acc6c on apache:master.

jkff

Thanks!

jkff · 2017-06-21T18:15:39Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoder.java

+  @Override
+  public void encode(Message value, OutputStream outStream) throws CoderException, IOException {
+    byte[] data = new byte[4096];
+    int bytesWritten = value.encode(data, 0, data.length);


What will this do if the message is bigger than 4096 bytes?

It will fail. I will add a test and a set of messages size that I will test.

jkff · 2017-06-21T18:16:47Z

sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpIOTest.java

+
+  private static final Logger LOG = LoggerFactory.getLogger(AmqpIOTest.class);
+
+  private static int port;


Shouldn't be static, I think

jkff · 2017-06-21T18:17:34Z

sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpIOTest.java

+/**
+ * Tests on {@link AmqpIO}.
+ */
+public class AmqpIOTest {


@RunWith(JUnit4.class)

jkff · 2017-06-21T18:18:29Z

sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpIOTest.java

+  public void testRead() throws Exception {
+    PCollection<Message> output = pipeline.apply(AmqpIO.read()
+        .withMaxNumRecords(100)
+        .withAddresses(Collections.singletonList("amqp://~localhost:" + port)));


Where does this test start an AMQP broker?

AMQP broker can be optional: as the Reader create a Messenger, the Messenger is able to directly receive messages (point to point).

coveralls · 2017-06-21T20:27:39Z

Coverage increased (+0.1%) to 70.828% when pulling 3dba8ee on jbonofre:BEAM-1237-AMQPIO into 50acc6c on apache:master.

jbonofre · 2017-06-26T19:57:59Z

@jkff I updated the Coder to support different message sizes. Do you want me to add some AmqpIO tests using ActiveMQ as AMQP broker ?

jkff

Thanks! OK to self-merge after addressing remaining minor comments.

jkff · 2017-06-27T03:36:36Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java

+      // they bind the listener
+      List<UnboundedAmqpSource> sources = new ArrayList<>();
+      if (desiredNumSplits > 0) {
+        for (int i = 0; i < desiredNumSplits; i++) {


How about for (int i = 0; i < Math.max(1, desiredNumSplits); ++i)

jkff · 2017-06-27T03:42:33Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoder.java

+
+  private static final int[] MESSAGE_SIZES = new int[]{
+      8 * 1024, // 8 KiB
+      64 * 1024, // 62 KiB


Why not "64KB" ? :) I think actually these comments are unnecessary anyway.

jkff · 2017-06-27T03:43:37Z

sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoder.java

+        encode(value, outStream, maxMessageSize);
+        return;
+      } catch (Exception e) {
+        if (maxMessageSize == MESSAGE_SIZES[MESSAGE_SIZES.length - 1]) {


You can slightly simplify:
for (maxMessageSize) {try {encode; return;} catch {continue;}}
throw new CoderException("...");

(gets rid of the if)

jkff · 2017-06-27T03:44:32Z

sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpIOTest.java

+    };
+    sender.start();
+    pipeline.run();
+    sender.join();


Still relevant.

jkff · 2017-06-27T03:44:57Z

sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpIOTest.java

+    LOG.info("Starting pipeline");
+    pipeline.run();
+    LOG.info("Join receiver thread");
+    receiver.join();


Also should be in try/finally

jkff · 2017-06-27T03:45:13Z

sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpIOTest.java

+      message.setSubject("test");
+      data.add(message);
+    }
+    pipeline.apply(Create.of(data).withCoder(AmqpMessageCoder.of())).apply(AmqpIO.write());


Is withCoder still necessary here?

Yes, it's still required on the Create().

jkff · 2017-06-27T03:45:59Z

sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderTest.java

+    message.setSubject("test");
+    AmqpMessageCoder coder = AmqpMessageCoder.of();
+
+    byte[] encoded = CoderUtils.encodeToByteArray(coder, message);


Message clone = CoderUtils.clone(coder, message)

jkff · 2017-06-27T03:47:11Z

sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderTest.java

+
+  @Test
+  public void encodeDecodeLargeMessage() throws Exception {
+    thrown.expect(CoderException.class);


I don't think that "cannot encode more than 64MB" is part of what the user wants from this coder. I'd prefer if you had a test that it can encode a moderately sized message, say a few MB.

…r deals the whole message (not only the body) TODO: fix the Coder

… the Write, change the watermark

…ows an exception is the message is larger than supported

coveralls · 2017-06-27T09:23:50Z

Coverage remained the same at 70.875% when pulling f8a14b3 on jbonofre:BEAM-1237-AMQPIO into 8036001 on apache:master.

coveralls · 2017-06-27T14:32:41Z

Coverage increased (+0.02%) to 70.893% when pulling ac944e1 on jbonofre:BEAM-1237-AMQPIO into 8036001 on apache:master.

jkff · 2017-06-28T03:17:52Z

Seems like AmqpIOTest sometimes hangs on my machine, hanging mvn clean verify:

...
Jun 27, 2017 7:16:08 PM org.apache.beam.sdk.io.amqp.AmqpIOTest$2 run
INFO: Received: AmqpValue{Test 93}
Jun 27, 2017 7:16:08 PM org.apache.beam.sdk.io.amqp.AmqpIOTest$2 run
INFO: Received: AmqpValue{Test 94}
Jun 27, 2017 7:16:08 PM org.apache.beam.sdk.io.amqp.AmqpIOTest$2 run
INFO: Received: AmqpValue{Test 95}
Jun 27, 2017 7:16:08 PM org.apache.beam.sdk.io.amqp.AmqpIOTest findFreeNetworkPort
INFO: Finding free network port
Jun 27, 2017 7:16:10 PM org.apache.qpid.proton.messenger.impl.MessengerImpl processAllConnectors
SEVERE: Error processing connection
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.qpid.proton.driver.impl.ConnectorImpl.process(ConnectorImpl.java:89)
at org.apache.qpid.proton.messenger.impl.MessengerImpl.processAllConnectors(MessengerImpl.java:687)
at org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:863)
at org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:844)
at org.apache.qpid.proton.messenger.impl.MessengerImpl.send(MessengerImpl.java:417)
at org.apache.qpid.proton.messenger.impl.MessengerImpl.send(MessengerImpl.java:394)
at org.apache.beam.sdk.io.amqp.AmqpIOTest$1.run(AmqpIOTest.java:82)

Would you mind rolling this back / disabling the test, investigating the failure, and rolling forward?

jbonofre · 2017-06-28T05:24:17Z

Let me disable the test for now. Does it occur only on your machine (I didn't see that on Jenkins) ?

jkff · 2017-06-28T05:27:06Z

It occurs pretty reliably on my machine with mvn clean verify -Prelease -T 1C -fae -pl '!sdks/python' (the command I usually use to test changes before merging)

jbonofre · 2017-06-28T05:28:31Z

OK, thanks, I'm disabling the tests and investigate.

jbonofre · 2017-06-28T05:41:26Z

Hmmm, I don't reproduce on my machine. Let me investigate deeper.

jbonofre · 2017-06-28T10:20:02Z

Can you try mvn clean install from root first, and then mvn clean verify in the amqp module ?

jbonofre force-pushed the BEAM-1237-AMQPIO branch from f446eed to eb3afd6 Compare June 6, 2017 14:34

jkff requested changes Jun 6, 2017

View reviewed changes

jbonofre force-pushed the BEAM-1237-AMQPIO branch 2 times, most recently from cb1cf7e to 694ff7e Compare June 12, 2017 12:09

jbonofre force-pushed the BEAM-1237-AMQPIO branch from 694ff7e to e58dda8 Compare June 14, 2017 20:11

jbonofre force-pushed the BEAM-1237-AMQPIO branch from ffc6517 to dd52740 Compare June 20, 2017 10:02

jkff requested changes Jun 20, 2017

View reviewed changes

jbonofre force-pushed the BEAM-1237-AMQPIO branch from dd52740 to 6cc24e5 Compare June 21, 2017 11:49

jkff requested changes Jun 21, 2017

View reviewed changes

jbonofre force-pushed the BEAM-1237-AMQPIO branch from 3dba8ee to 5b250f4 Compare June 26, 2017 19:55

jkff approved these changes Jun 27, 2017

View reviewed changes

[BEAM-1237] Create AmqpIO

c22b22e

jbonofre added 7 commits June 27, 2017 08:38

[BEAM-1237] Add AmqpMessageCoderProviderRegistrar and AmqpMessageCode…

a82b440

…r deals the whole message (not only the body) TODO: fix the Coder

[BEAM-1237] Improve the AmqpMessageCoder and add a test on the coder

5674be8

[BEAM-1237] Create a AmqpMessage wrapper to provide equals()

83a4865

[BEAM-1237] Directly use ProtonJ Message in the PCollection, simplify…

b4c4d06

… the Write, change the watermark

[BEAM-1237] Cleanup on the test

1d2c3d5

[BEAM-1237] AmqpMessageCoder supports different message sizes and thr…

97d7fdf

…ows an exception is the message is larger than supported

[BEAM-1237] Improve code style and tests

7b63c94

jbonofre force-pushed the BEAM-1237-AMQPIO branch from 5b250f4 to 7b63c94 Compare June 27, 2017 06:38

[BEAM-1237] Use try/finally in test

f8a14b3

[BEAM-1237] Sort IO modules by alphanumeric order

ac944e1

asfgit closed this in 9df865a Jun 27, 2017

jbonofre deleted the BEAM-1237-AMQPIO branch June 27, 2017 13:34


		private static final Logger LOG = LoggerFactory.getLogger(AmqpIOTest.class);

		private static int port;

[BEAM-1237] Create AmqpIO #1725

[BEAM-1237] Create AmqpIO #1725

Conversation

jbonofre commented Jan 2, 2017

asfbot commented Jan 2, 2017

davorbonaci commented Feb 27, 2017

jbonofre commented Mar 28, 2017

jbonofre commented Jun 6, 2017

coveralls commented Jun 6, 2017

jbonofre commented Jun 6, 2017

coveralls commented Jun 6, 2017

jkff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbonofre commented Jun 12, 2017

coveralls commented Jun 12, 2017

jbonofre commented Jun 13, 2017

coveralls commented Jun 14, 2017

jbonofre commented Jun 20, 2017

jbonofre commented Jun 20, 2017

coveralls commented Jun 20, 2017

jkff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbonofre commented Jun 21, 2017

coveralls commented Jun 21, 2017

jkff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Jun 21, 2017

jbonofre commented Jun 26, 2017

jkff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Jun 27, 2017

coveralls commented Jun 27, 2017

jkff commented Jun 28, 2017

jbonofre commented Jun 28, 2017

jkff commented Jun 28, 2017

jbonofre commented Jun 28, 2017

jbonofre commented Jun 28, 2017

jbonofre commented Jun 28, 2017