[BEAM-1240] Create RabbitMqIO #1729

jbonofre · 2017-01-03T17:09:57Z

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
If this contribution is large, please file an Apache
Individual Contributor License Agreement.

asfbot · 2017-01-03T17:34:38Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/6376/
--none--

davorbonaci · 2017-02-27T18:08:50Z

Is this ready for review? Perhaps assign a reviewer?

jbonofre · 2017-02-27T18:16:49Z

Yes. Just pinged Dan about that ;)

reuvenlax · 2017-08-29T22:14:46Z

@jbonofre is this PR still useful? If so, could you suggest a reviewer so this makes progress?

coveralls · 2017-08-29T23:35:09Z

Coverage decreased (-0.4%) to 69.394% when pulling 63f5a3e on jbonofre:BEAM-1240-RABBITMQIO into 810db7f on apache:master.

jbonofre · 2017-08-30T05:39:26Z

@reuvenlax yes it is in complement of the AMQPIO. I will link with the PR I have on my own github (as requested by Eugene).

aaltay · 2017-10-20T18:54:08Z

Hey @jbonofre, would @jkff be a reviewer for this?

jkff

Thanks, a couple of questions.

What benefit does this provide over using AmqpIO? RabbitMQ already implements the AMQP protocol, so AmqpIO can be used to talk to RabbitMQ, right?
Both unit tests are currently marked "ignored", i.e. this code effectively doesn't have tests, and it will be easy for somebody to accidentally break it. What's the plan for dealing with this? I think this needs to be fixed before the PR can go in.

jbonofre · 2017-10-21T05:28:42Z

I will rebase and update this PR.

About the questions:

RabbitMQ implements AMPQ 1.0 whereas AMPQIO (based on QPid) implements AMQP 0.9
I will enable and update the tests.

Thanks !

johanhaleby · 2017-11-04T17:49:50Z

@jbonofre It says in the AqmpIO javadoc that it supports AMQP 1.0, don't you mean the other way around (i.e. that RabbitMQ uses 0.9.1 by default)?

Anyway I'm really looking forward for this PR to be merged and released.

johanhaleby · 2017-11-05T13:20:00Z

Are you interested in getting some help to get the tests fixed? If so I might be able to help out. Writing embedded RabbitMQ tests can be quite tricky, but there might be a couple of workarounds:

Use apache qpid embedded broker and make sure that it's configured to use AMQP 0.9.x. See this stackoverflow answer.
Use something like TestContainers to start RabbitMQ as a Docker container. This requires that Docker is installed on the build server though and I have no insight into this? Also the test suite will be slower since the Docker image needs to be pulled.
Use embedded rabbitmq which downloads an entire RabbitMQ installation when starting the test case. If Docker is not available on the build server this might be a better alternative but it'll also slow down the test significantly (at least when building from scratch) due do downloading of RabbitMQ.

WDYT?

jkff · 2017-11-05T17:44:10Z

Sorry, I still don't understand why a RabbitMqIO is necessary in general, as opposed to using AmqpIO pointed at the RabbitMQ endpoint? Regardless of whether RabbitMQ implements AMQP 0.9.1 or 1.0, it seems like the way to support a particular version of AMQP should be to upgrade AmqpIO? @johanhaleby are you currently blocked by something that AmqpIO can't do?

johanhaleby · 2017-11-05T18:04:10Z

it seems like the way to support a particular version of AMQP should be to upgrade AmqpIO?

Well that sounds reasonable indeed, unless there're some specific features of RabbitMQ that it cannot support?

are you currently blocked by something that AmqpIO can't do?

To be honest I haven't even tried it, I just googled for Beam and rabbitmq support which lead me to this issue. Reading the amqp javadoc suggests that it only supports AMQP 1.0 (and for some reason I thought that was intentional), please correct me if I'm wrong. If it only supports AMQP 1.0, then I'm indeed blocked since we're using 0.9.1 in RabbitMQ. But then again, if we could make the AmqpIO support 0.9.1 then that sounds better to me.

jbonofre · 2017-11-07T05:23:47Z

I fixed and updated RabbitMQIO.

Actually, it's the opposite in term of version: AMQPIO is AMQP 1.0/1.1 version whereas RabbitMQIO is 0.9.

jbonofre · 2017-11-07T05:29:30Z

@jkff I fixed the IO and the tests. As said, AMQP IO (with usage of proton/qpid) only supports AMQP 1.0/1.1. In order to support AMQP 0.9.x (version used by RabbitMQ), we need to use the RabbitMQ client. That's the purpose of this PR (AMQP 0.9.x is very RabbitMQ specific).

@johanhaleby the tests are now OK and the IO is ready to be tested.

johanhaleby · 2017-11-07T11:14:25Z

Is there no single AMQP client library for Java that supports both AMQP 1.0/1.1 and 0.9.x so that we could unify AmqpIO and this PR? tis 7 nov. 2017 kl. 06:29 skrev Jean-Baptiste Onofré < notifications@github.com>:

@jkff <https://github.com/jkff> I fixed the IO and the tests. As said, AMQP IO (with usage of proton/qpid) only supports AMQP 1.0/1.1. In order to support AMQP 0.9.x (version used by RabbitMQ), we need to use the RabbitMQ client. That's the purpose of this PR (AMQP 0.9.x is very RabbitMQ specific). @johanhaleby <https://github.com/johanhaleby> the tests are now OK and the IO is ready to be tested. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1729 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALalUVqMpXzuGKLuS6KlmajEsxkVdjnks5sz-q_gaJpZM4LZ0e4> .

-- Sent from my phone

jbonofre · 2017-11-07T13:38:15Z

That's another option: I can gather both lib in AmqpIO and use a configuration to let the user decide the AMQP protocol version.

As AMQP 0.9 is "only" RabbitMQ, I think it gives more visibility to have a dedicated IO.

@jkff thoughts ?

jkff · 2017-11-07T22:50:47Z

OK, I think I agree that RabbitMqIO makes sense for AMQP 0-9-1 because AFAIK RabbitMQ is the only system that implements AMQP 0-9-1 (all other "generic AMQP implementations" implement AMQP 1.0) so if people need it, it's probably because they are using RabbitMQ, and in that case they'll find it easier to locate it by RabbitMqIO. I'll take a look at the PR contents soon.

jkff · 2017-11-07T22:54:33Z

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java

+
+    @Override
+    public void validate(PipelineOptions pipelineOptions) {
+      checkState(connectionConfig() != null, "RabbitMqIO.read() requires a connection config "


this should be in expand() because it doesn't use the PipelineOptions. Also should be rephrased:

checkState(..., "withConnectionConfig() is required.")

My bad, it's an "old" PR and I have to update to match the IO style guide. I'm doing it now, thanks !

Actually, the checkState doesn't make sense anymore as a default ConnectionConfig is created in the constructor method. I remove this.

jkff · 2017-11-07T22:54:51Z

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java

+    }
+
+    @Override
+    public Coder<byte[]> getDefaultOutputCoder() {


This method is deprecated, override getOutputCoder instead

jkff · 2017-11-07T22:55:25Z

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java

+    }
+
+    @Override
+    public void validate() {


This is a no-op, so this method can be omitted

jkff · 2017-11-07T22:55:39Z

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java

+      // RabbitMQ uses queue, so, we can have several concurrent consumers as source
+      List<RabbitMQSource> sources = new ArrayList<>();
+      for (int i = 0; i < desiredNumSplits; i++) {
+        sources.add(new RabbitMQSource(spec));


This can be simply sources.add(this)

Is it not a new source for each split (to have concurrent consumers) ?

RabbitMQSource is simply an immutable java object, there's no difference between returning one object or returning another object that is completely equivalent to it. A Beam runner will anyway serialize them at construction time and deserialize each split on the respective worker.

jkff · 2017-11-07T22:58:44Z

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java

+        if (channel == null) {
+          throw new IOException("No RabbitMQ channel available");
+        }
+        channel.queueDeclare(source.spec.connectionConfig().queue(), false, false, false, null);


Please annotate the false and null parameters with names for readability

jkff · 2017-11-07T23:02:44Z

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java

+        if (channel == null) {
+          throw new IOException("No RabbitMQ channel available");
+        }
+        channel.queueDeclare(spec.connectionConfig().queue(), false, false, false, null);


I'm not sure what this code is trying to do. I believe RabbitMQ producers should publish to an exchange rather than a queue, and they shouldn't even care whether any queues bound to this exchange currently exist:

https://www.rabbitmq.com/tutorials/tutorial-three-python.html

The core idea in the messaging model in RabbitMQ is that the producer never sends any messages directly to a queue. Actually, quite often the producer doesn't even know if a message will be delivered to any queue at all. Instead, the producer can only send messages to an exchange....

Yes, forgot to merge that change. Merging.

jkff · 2017-11-07T23:05:07Z

sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java

+    broker = new Broker();
+    BrokerOptions options = new BrokerOptions();
+    options.setConfigProperty("qpid.amqp_port", String.valueOf(PORT));
+    options.setConfigProperty("qpid.work_dir", Files.createTempDirectory("qpid").toString());


Use a @Rule TemporaryFolder

jkff · 2017-11-07T23:05:34Z

sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java

+ */
+public class RabbitMqIOTest {
+
+  public static final int PORT = 5672;


What if this port is taken?

jkff · 2017-11-07T23:06:10Z

sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java

+    connectionFactory.setUri("amqp://guest:guest@localhost:" + PORT);
+    Connection connection = connectionFactory.newConnection();
+    Channel channel = connection.createChannel();
+    channel.queueDeclare("READ", false, false, false, null);


Here too, the publisher should not be declaring queues, and rather it should be publishing to an exchange.

jkff · 2017-11-07T23:07:12Z

sdks/java/io/rabbitmq/src/test/qpid/etc/passwd

+# specific language governing permissions and limitations
+# under the License.
+#
+guest:guest


Is it possible to specify this programmatically rather than via a configuration file?

Yes, it should be do-able. I'm doing this.

Any progress?

jkff · 2017-11-09T00:56:47Z

OK, seems that the changes are not done yet. Please ping this thread when ready for another round.

asfgit · 2017-11-12T16:11:22Z

SUCCESS

--none--

jbonofre · 2017-11-12T16:17:24Z

@jkff I addressed most of your comments. Just checking to configuration the plain authentication provider programmatically in order to avoid the passwd file.

asfgit · 2017-11-12T16:33:26Z

FAILURE

--none--

asfgit · 2017-11-12T17:18:41Z

SUCCESS

--none--

jbonofre · 2017-11-13T05:06:16Z

retest this please

asfgit · 2017-11-13T05:09:57Z

SUCCESS

--none--

jbonofre · 2018-07-19T09:17:10Z

@echauchot great thanks !

jkff

Thanks! (my last review on this one - deferring to other reviewers afterwards)

jkff · 2018-07-19T18:54:27Z

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java

+ *
+ * <pre>{@code
+ * pipeline.apply(
+ *   RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")


Missing ), and missing "PCollection messages = ..."

jkff · 2018-07-19T18:54:49Z

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java

+ *
+ * <h3>Publishing messages to RabbitMQ server</h3>
+ *
+ * <p>{@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ server queue or


It's actually RabbitMqMessage

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java

jkff · 2018-07-19T18:57:21Z

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java

+    @Override
+    public boolean advance() throws IOException {
+      try {
+        QueueingConsumer.Delivery delivery = consumer.nextDelivery();


Still relevant

sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java

jkff · 2018-07-19T19:02:17Z

sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java

+        new Thread(
+            () -> {
+              try {
+                Thread.sleep(5000);


Maybe a simple while with timeout to check the Qpid status to avoid flaky tests in case 5 sec is not enough for the QPid init time on an overloaded jenkins server.

sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java

jkff · 2018-07-19T19:03:17Z

sdks/java/io/rabbitmq/src/test/qpid/etc/passwd

+# specific language governing permissions and limitations
+# under the License.
+#
+guest:guest


jbonofre · 2018-07-19T19:48:32Z

@jkff thanks for the update, I will address your comments !

akankshajain18 · 2018-07-20T07:37:40Z

Hi All

Thanks for RmqIO :)
But It will be great if setDurable(boolean durable) API get exposed to end user
As of now, non-durable queue get created everytime.
[
// channel.queueDeclare(queueName, durable, exclusive, autoDelete, arguments);
channel.queueDeclare(queueName, false, false, false, null)
]

echauchot

@jbonofre, this IO seems to be in very good shape, thanks !
Since @jkff resumed his review I did just a quick review focusing on tests. I also have a comment on the watermark management.

sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java

jbonofre · 2018-07-21T15:40:55Z

I exposed durable to end users for queue declaration (as requested by @akankshajain18).

jbonofre · 2018-07-21T18:53:08Z

retest this please

akankshajain18 · 2018-07-23T06:49:33Z

Thanks @jbonofre

akankshajain18 · 2018-08-03T13:07:49Z

Hello Folks

While working with RmqIO and Flink local runner of the beam, I observed, Even if Rmq has data to consume, Beam pipeline Job execution switched to status FINISHED without any exception.
In logs, RabbitMqIO$UnboundedRabbitMqReader close() method get called, which led to stopping the connection between RMQ and my beam job(running on Flink runner).

Code :
[@OverRide
public void close() throws IOException {
if (connectionHandler != null) {
connectionHandler.stop();
}
}]

It will be great if someone can help me out about this random behavior.

jbonofre · 2018-09-25T15:50:54Z

@akankshajain18 that's probably related to the watermark behavior which is not correct for now. I'm fixing that and other comments. I'm moving forward on this PR. Sorry for the delay.

echauchot

Thanks JB !
minor changes on historical comments + my first round comments and we are done !

echauchot

Thanks for addressing my comments JB! You just missed the one on equals impl and the one on Thread.sleep for which I added 2 suggestions. Also suggested to use maxNumRecords in place of absolute 10 in one of the tests.

sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java

echauchot · 2018-10-17T08:49:07Z

sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java

+      p.run();
+
+      while (received.size() < maxNumRecords) {
+        Thread.sleep(500);


sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java

echauchot · 2018-10-17T09:51:20Z

sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java

+      p.run();
+
+      while (received.size() < maxNumRecords) {
+        Thread.sleep(500);


There is no problem with this sleep because it is not a wait for an absolute period of time but more a way to temporise the check of the receive queue depth. As a very minor suggestion (only if you feel the envy :) ), I would implement it with a new CountDownLatch(maxNumRecords); countdown.await() and countdown.countDown in the TestConsumer when it receives a message.
But once again it is very minor way to reduce test time a tiny bit but I'm ok with current impl

echauchot

LGTM ! Thanks for your work JB ! Please squash the commits into the first one and ping me when it's done so that I can merge the PR

jbonofre · 2018-10-25T03:27:37Z

@echauchot Squash & rebase done.

alafanechere · 2019-01-14T15:31:45Z

I tried to use the RabbitMqIO with the direct runner to generate an unbounded PCollection from a queue. I encounter a NPE :

java.lang.NullPointerException
    at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement (UnboundedReadEvaluatorFactory.java:169)

...

After investigation it looks like it's caused by the fact that no default is given to
checkpointMark.oldestTimestamp. getWatermark() is called before the mutation of the currentTimestamp variable, raising a NPE. I fixed the problem on my side, reimplementing the class and overriding getWatermark to return Instant.now() if checkpointMark.oldestTimestamp is null :

@Override
publicInstantgetWatermark() {
  if (checkpointMark.oldestTimestamp == null) {
    returnInstant.now();
  }
  return checkpointMark.oldestTimestamp;
}

It looks likes this bug as already been raised on Jira and on Stack Overflow

Do you think it's a problem that needs to be corrected ? Is there any workaround you might think of ?

jbonofre · 2019-01-14T15:33:06Z

@alafanechere agree, can you please create a dedicated Jira about that ? I will fix it.

jbonofre · 2019-01-14T15:34:24Z

@alafanechere by the way, I can create the Jira for you (and I will assign the Jira to me). Please let me know.

alafanechere · 2019-01-14T15:44:09Z

Thanks ! I juste created the JIRA : https://jira.apache.org/jira/browse/BEAM-6424

alafanechere · 2019-03-15T11:45:47Z

Hello @jbonofre , I still have this issue with the IO. Is there a specific configuration of the queue to make the consumption / ack of message work seamless with the Beam ? I tried to bypass the problem by patching the IO and using autoack on arrival but it looks like bad design...

jkff requested changes Oct 20, 2017

View reviewed changes

jbonofre force-pushed the BEAM-1240-RABBITMQIO branch 2 times, most recently from 29c3b65 to 4b127a4 Compare November 7, 2017 05:22

jkff requested changes Nov 7, 2017

View reviewed changes

jbonofre force-pushed the BEAM-1240-RABBITMQIO branch from 219b390 to 76401c1 Compare November 8, 2017 07:59

jbonofre force-pushed the BEAM-1240-RABBITMQIO branch from 76401c1 to 817c93d Compare November 12, 2017 16:07

jkff requested changes Jul 19, 2018

View reviewed changes

echauchot reviewed Jul 20, 2018

View reviewed changes

jbonofre force-pushed the BEAM-1240-RABBITMQIO branch from 5cac406 to f1665f4 Compare July 21, 2018 15:38

jbonofre force-pushed the BEAM-1240-RABBITMQIO branch from f1665f4 to ca90832 Compare September 24, 2018 14:40

jbonofre force-pushed the BEAM-1240-RABBITMQIO branch 2 times, most recently from 424bea6 to 84c948e Compare October 11, 2018 06:58

echauchot requested changes Oct 11, 2018

View reviewed changes

jbonofre force-pushed the BEAM-1240-RABBITMQIO branch 2 times, most recently from bfa77c6 to a4d9e5b Compare October 14, 2018 16:16

echauchot reviewed Oct 17, 2018

View reviewed changes

jbonofre force-pushed the BEAM-1240-RABBITMQIO branch from 8a205ac to ebfe51b Compare October 21, 2018 06:03

echauchot approved these changes Oct 24, 2018

View reviewed changes

[BEAM-1240] Create RabbitMqIO

119d0bc

jbonofre force-pushed the BEAM-1240-RABBITMQIO branch from 0577b86 to 119d0bc Compare October 25, 2018 03:27

echauchot merged commit 6c1d92a into apache:master Oct 25, 2018

jbonofre deleted the BEAM-1240-RABBITMQIO branch January 14, 2019 15:32

[BEAM-1240] Create RabbitMqIO #1729

[BEAM-1240] Create RabbitMqIO #1729

Conversation

jbonofre commented Jan 3, 2017

asfbot commented Jan 3, 2017

davorbonaci commented Feb 27, 2017

jbonofre commented Feb 27, 2017

reuvenlax commented Aug 29, 2017

coveralls commented Aug 29, 2017

jbonofre commented Aug 30, 2017

aaltay commented Oct 20, 2017

jkff left a comment

Choose a reason for hiding this comment

jbonofre commented Oct 21, 2017

johanhaleby commented Nov 4, 2017

johanhaleby commented Nov 5, 2017 • edited Loading

jkff commented Nov 5, 2017

johanhaleby commented Nov 5, 2017 • edited Loading

jbonofre commented Nov 7, 2017

jbonofre commented Nov 7, 2017

johanhaleby commented Nov 7, 2017 via email

jbonofre commented Nov 7, 2017

jkff commented Nov 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkff commented Nov 9, 2017

asfgit commented Nov 12, 2017

jbonofre commented Nov 12, 2017

asfgit commented Nov 12, 2017

asfgit commented Nov 12, 2017

jbonofre commented Nov 13, 2017

asfgit commented Nov 13, 2017

jbonofre commented Jul 19, 2018

jkff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbonofre commented Jul 19, 2018

akankshajain18 commented Jul 20, 2018 • edited Loading

echauchot left a comment

Choose a reason for hiding this comment

jbonofre commented Jul 21, 2018 • edited Loading

jbonofre commented Jul 21, 2018

akankshajain18 commented Jul 23, 2018

akankshajain18 commented Aug 3, 2018

jbonofre commented Sep 25, 2018

echauchot left a comment

Choose a reason for hiding this comment

echauchot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

echauchot left a comment

Choose a reason for hiding this comment

jbonofre commented Oct 25, 2018

alafanechere commented Jan 14, 2019

jbonofre commented Jan 14, 2019

jbonofre commented Jan 14, 2019

alafanechere commented Jan 14, 2019 • edited Loading

johanhaleby commented Nov 5, 2017 •

edited

Loading

johanhaleby commented Nov 5, 2017 •

edited

Loading

akankshajain18 commented Jul 20, 2018 •

edited

Loading

jbonofre commented Jul 21, 2018 •

edited

Loading

alafanechere commented Jan 14, 2019 •

edited

Loading

alafanechere commented Mar 15, 2019 •

edited

Loading