JBTM-3212 Respect LRA deadlines during recovery. #1530

mmusgrov · 2019-11-26T19:39:02Z

https://issues.jboss.org/browse/JBTM-3212
https://issues.jboss.org/browse/JBTM-3222

!QA_JTA !QA_JTS_JDKORB !QA_JTS_OPENJDKORB !QA_JTS_JACORB !BLACKTIE !XTS !PERF NO_WIN !AS_TESTS !TOMCAT !JACOCO !mysql !postgres !db2 !oracle !RTS !MAIN !LRA

This PR adds recovery testing for timed out LRAs

jbosstm-bot · 2019-11-26T19:42:44Z

Started testing this pull request with LRA profile: http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/803/

jbosstm-bot · 2019-11-26T20:01:48Z

LRA profile tests failed (http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/803/): narayana build failed

jbosstm-bot · 2019-11-27T00:52:42Z

Started testing this pull request with LRA profile: http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/805/

jbosstm-bot · 2019-11-27T01:24:05Z

LRA profile tests failed (http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/805/): narayana build failed

jbosstm-bot · 2019-11-27T08:04:59Z

Started testing this pull request with LRA profile: http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/806/

jbosstm-bot · 2019-11-27T08:13:19Z

Started testing this pull request with LRA profile: http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/807/

jbosstm-bot · 2019-11-27T08:43:24Z

LRA profile tests failed (http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/806/): LRA Test failed

jbosstm-bot · 2019-11-27T08:50:49Z

LRA profile tests failed (http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/807/): LRA Test failed

jbosstm-bot · 2019-11-27T10:22:05Z

Started testing this pull request with LRA profile: http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/808/

jbosstm-bot · 2019-11-27T10:32:36Z

Started testing this pull request with LRA profile: http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/809/

jbosstm-bot · 2019-11-27T11:00:19Z

LRA profile tests failed (http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/808/): LRA Test failed

jbosstm-bot · 2019-11-27T11:34:15Z

LRA profile tests failed (http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/809/): LRA Test failed

jbosstm-bot · 2019-11-27T15:29:48Z

Started testing this pull request with LRA profile: http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/810/

jbosstm-bot · 2019-11-27T16:41:59Z

LRA profile tests failed (http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/810/): LRA Test failed

jbosstm-bot · 2019-11-28T15:17:26Z

Started testing this pull request with LRA profile: http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/811/

jbosstm-bot · 2019-11-28T16:23:21Z

LRA profile tests passed - Job complete http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/811/

xstefank

Just a note, but shouldn't we focus all LRA work on the LRA branch? In this case this a pretty big PR and potentially a lot of conflicts with the work done in the LRA branch.

rts/lra/lra-client/src/main/java/io/narayana/lra/client/NarayanaLRAClient.java

rts/lra/lra-coordinator-jar/pom.xml

xstefank · 2019-11-29T08:05:46Z

rts/lra/lra-coordinator-jar/pom.xml

+        <dependency>
+            <groupId>org.jboss.narayana.rts</groupId>
+            <artifactId>narayana-lra</artifactId>
+            <version>5.10.1.Final-SNAPSHOT</version>


${project.version}?

xstefank · 2019-11-29T08:12:05Z

.../lra-coordinator-jar/src/main/java/io/narayana/lra/coordinator/domain/model/Transaction.java

+                long ttl = ChronoUnit.MILLIS.between(LocalDateTime.now(ZoneOffset.UTC), finishTime);
+
+                if (ttl <= 0) {
+                    System.out.printf("TIMER has expired since last reload%n");


Should it be better to log these statements instead of system.out?

Good catch, this should be using the logger framework

+1 (and to the other occurrences of printf in this PR)

xstefank · 2019-11-29T08:23:23Z

...ra/lra-coordinator-jar/src/test/java/io/narayana/lra/coordinator/LRACoordinatorTestCase.java

+    /**
+     * Test that an LRA which times out while there is no running coordinator is cancelled
+     * when a coordinator is restarted
+     * @throws URISyntaxException if the LRA or revovery URIs are invalid (should never happen)


Suggested change

* @throws URISyntaxException if the LRA or revovery URIs are invalid (should never happen)

* @throws URISyntaxException if the LRA or recovery URIs are invalid (should never happen)

...ra/lra-coordinator-jar/src/test/java/io/narayana/lra/coordinator/LRACoordinatorTestCase.java

xstefank · 2019-11-29T08:29:10Z

...ra/lra-coordinator-jar/src/test/java/io/narayana/lra/coordinator/LRACoordinatorTestCase.java

+        // start an LRA with a long timeout to validate that timed LRAs do not finish early during recovery
+        URI longLRA = lraClient.startLRA(null, "Long Timeout Recovery Test", LONG_TIMEOUT, ChronoUnit.MILLIS);
+        // start an LRA with a short timeout to validate that timed LRAs that time out when the coordinator is unavailable are cancelled
+        URI shortLRA = lraClient.startLRA(null, "Long Timeout Recovery Test", SHORT_TIMEOUT, ChronoUnit.MILLIS);


different clientId can help with debugging in future

xstefank · 2019-11-29T08:36:33Z

rts/lra/lra-coordinator-jar/src/test/java/io/narayana/lra/coordinator/TestBase.java

+
+    void stopContainer() {
+        if (containerController.isStarted(COORDINATOR_CONTAINER)) {
+            LRALogger.logger.info("STOPPING CONTAINER%n");


I was going to mention that printf are probably OK in the tests. But I think it is a bit confusing that there is a mix of printf and the logger. I think that using the LRA internal logger could be confusing in the output

+1 I removed all system printf's and fixed this format string on the logger

xstefank · 2019-11-29T08:37:56Z

rts/lra/lra-coordinator-jar/src/test/java/io/narayana/lra/coordinator/TestBase.java

+    }
+
+    int recover() throws URISyntaxException {
+        try (Response response = ClientBuilder.newClient().target(new URI(recoveryUrl))


Is the created client also closed in try-with-resources?

I am not even sure if Client is AutoCloseable. If it was I would guess it could need something like the following (assuming it would compile) to work:
try (Client client = ClientBuilder.newClient(); Response resource = client.target(...

but I still close it usually when I type rest client invocations. But you are right it's not AutoClosable. Not sure if this it is necessary to close it.

Agreed, I was not suggesting not to close it - just trying to develop the thread. If it's not autocloseable then I think we just need the try-with-resources wrapped with a try/finally block

The call that closes the underlying connections/resources is either when the response is closed or when the response entity is fully read (which is what the above code does in its try with resources code block).

The spec has this to say about the JAX-RS client API close method:

Calling this method effectively invalidates all resource targets produced by the client instance. Invoking any method on such targets once the client is closed would result in an IllegalStateException being thrown.

So since I am not using any of those targets after the call finishes (because the client is implicitly created) I don't need to call close on it.

How about this statement "Client instances must be properly closed before being disposed to avoid leaking resources." from https://docs.oracle.com/javaee/7/api/javax/ws/rs/client/Client.html

Maybe later on in the project we can do some resource utilisation testing and performance optimisations but let's do it as part of a more comprehensive analysis of the non functional behaviour of the system.

Okay, it is just a test but fair enough. If that is the only pending issue then I will push a fix for this comment?

tomjenkinson · 2019-11-29T09:58:33Z

@xstefank I thought the strategy was that the LRA branch was just for the features that rely on LRA GA or RC2? We can still make changes on the main branch when the change would not rely on the temporary artifacts.

That said I think we could request an RC2 if there is divergence going on as we don't want to spend a lot of time reconciling the changes.

tomjenkinson · 2019-11-29T10:02:01Z

rts/lra/lra-coordinator-jar/src/main/java/io/narayana/lra/coordinator/api/Coordinator.java

@@ -275,6 +275,7 @@ public Response startLRA(
        }

        String coordinatorUrl = String.format("%s%s", context.getBaseUri(), COORDINATOR_PATH_NAME);
+


formatting only

tomjenkinson · 2019-11-29T10:03:10Z

rts/lra/lra-coordinator-jar/src/main/java/io/narayana/lra/coordinator/api/JaxRsActivator.java

@@ -10,7 +10,7 @@
 // mark the war as a JAX-RS archive
 @ApplicationPath("/")
 @OpenAPIDefinition(
-    info = @Info(title = "LRA Coordinator", version = "1.0"),
+    info = @Info(title = "LRA Coordinator", version = "1.0-RC1"),


This looks like something we will have to watch out for. I wonder if how we can make this more prominent that it needs to be updated with LRA coordinator versions?

I appreciate this concern is not introduced by this PR but is something to follow up on

I created a final constant (in the same file) to give some degree of prominence to it. Now that you have mentioned it I thing I will remember to update it as we produce new LRA releases.

...lra-coordinator-jar/src/main/java/io/narayana/lra/coordinator/domain/service/LRAService.java

tomjenkinson · 2019-11-29T10:16:36Z

rts/lra/lra-test/lra-test-tck/pom.xml

@@ -126,6 +110,5 @@
                <lra.tck.suite.thorntail.trace.params>-Dthorntail.logging=TRACE</lra.tck.suite.thorntail.trace.params>
            </properties>
        </profile>
-


I missed this one and adding it back in again will trigger a full rebuild.
But I would also say that since this is pre release code removing the empty line does not obfuscate the intent of the PR and does not impact anyone wanting to diff the code.

tomjenkinson · 2019-11-29T10:22:11Z

@mmusgrov I have added some comments

xstefank · 2019-11-29T10:26:48Z

@xstefank I thought the strategy was that the LRA branch was just for the features that rely on LRA GA or RC2? We can still make changes on the main branch when the change would not rely on the temporary artifacts.

That said I think we could request an RC2 if there is divergence going on as we don't want to spend a lot of time reconciling the changes.

Yes, but if we are changing LRA subsystem in any way there will be conflicts. Changes are made for different reasons but over the same codebase.

tomjenkinson · 2019-11-29T12:00:28Z

@xstefank I thought the strategy was that the LRA branch was just for the features that rely on LRA GA or RC2? We can still make changes on the main branch when the change would not rely on the temporary artifacts.
That said I think we could request an RC2 if there is divergence going on as we don't want to spend a lot of time reconciling the changes.

Yes, but if we are changing LRA subsystem in any way there will be conflicts. Changes are made for different reasons but over the same codebase.

As this is a bug we should try to get it in the master branch where the feature is already present. But I do agree that we should expand the use of the LRA branch to any new features to save on this reconciliation.

mmusgrov · 2019-11-29T15:22:48Z

Just a note, but shouldn't we focus all LRA work on the LRA branch? In this case this a pretty big PR and potentially a lot of conflicts with the work done in the LRA branch.

No I don't think so. The LRA branch was created in order to avoid dependencies on snapshot builds and for tracking changes to the MP-LRA spec and TCK. My change affects the LRA implementation only which is more than just an MP-LRA implementation. Also since this change works fine with the spec version RC1 it is fine to apply the fix to the master branch (in fact that is the best place for it since users will use our master branch (not the LRA branch which is a moving target) so they will get the fix.

mmusgrov · 2019-11-29T15:23:14Z

@tomjenkinson and @xstefank can you both take another look please.

jbosstm-bot · 2019-11-29T15:23:22Z

Started testing this pull request with LRA profile: http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/813/

tomjenkinson · 2019-11-29T15:20:15Z

rts/lra/lra-coordinator-jar/pom.xml

-                     <artifactId>jconsole</artifactId>
-                 </exclusion>
-             </exclusions>
+                <exclusion>


formatting?

Since this is pre release code this change does not obfuscate the intent of the PR and does not impact anyone wanting to diff the code. After release one it would make sense to minimise the occasion reformatting of blocks.

tomjenkinson · 2019-11-29T15:22:34Z

rts/lra/lra-coordinator-jar/src/test/java/io/narayana/lra/coordinator/TestBase.java

+    }
+
+    int recover() throws URISyntaxException {
+        try (Response response = ClientBuilder.newClient().target(new URI(recoveryUrl))


How about this statement "Client instances must be properly closed before being disposed to avoid leaking resources." from https://docs.oracle.com/javaee/7/api/javax/ws/rs/client/Client.html

rts/lra/lra-coordinator-jar/pom.xml

mmusgrov · 2019-11-29T15:43:52Z

Just a note, but shouldn't we focus all LRA work on the LRA branch? In this case this a pretty big PR and potentially a lot of conflicts with the work done in the LRA branch.

No I don't think so. The LRA branch was created in order to avoid dependencies on snapshot builds and for tracking changes to the MP-LRA spec and TCK. My change affects the LRA implementation only which is more than just an MP-LRA implementation. Also since this change works fine with the spec version RC1 it is fine to apply the fix to the master branch (in fact that is the best place for it since users will use our master branch (not the LRA branch which is a moving target) so they will get the fix.

I just checked and the LRA branch only has two small commits on it which I doubt will cause any conflicts - I don't mind doing the merge when we have an RC2. But I still think we are at least a couple of weeks away from RC2 so if we can get more (all even) of the MP-LRA issues resolved then we could do an RC2.

jbosstm-bot · 2019-11-29T16:29:35Z

LRA profile tests passed - Job complete http://narayanaci1.eng.hst.ams2.redhat.com/job/btny-pulls-narayana/PROFILE=LRA,jdk=jdk8.latest,label=linux/813/

xstefank

There are still a few JAX-RS Client not closed in LRACoordinatorTestCase but if that's a design decision than ignore this message.

rts/lra/lra-coordinator-jar/pom.xml

xstefank · 2019-12-02T07:11:20Z

.../lra-coordinator-jar/src/main/java/io/narayana/lra/coordinator/domain/model/Transaction.java

-                LocalDateTime ft = LocalDateTime.now().plusNanos(timeLimit * 1000000);
+        if (finishTime != null) {
+            // check whether the new time limit is less than the current one
+            LocalDateTime ft = LocalDateTime.now().plusNanos(timeLimit * 1000000);


sorry I missed this previously. This is missing ZoneOffset.UTC.

xstefank · 2019-12-02T07:26:31Z

rts/lra/lra-coordinator-jar/src/test/java/io/narayana/lra/coordinator/TestBase.java

+
+    @After
+    public void after() {
+        LRALogger.logger.debugf("Finised test %s", testName);


Suggested change

LRALogger.logger.debugf("Finised test %s", testName);

LRALogger.logger.debugf("Finished test %s", testName);

xstefank · 2019-12-02T07:29:37Z

rts/lra/lra-coordinator-jar/src/test/java/io/narayana/lra/coordinator/TestBase.java

+
+            Assert.assertEquals("Unexpected status from reovery call:", 200, response.getStatus());
+
+            // the result wil be a List<LRAStatusHolder> of recovering LRAs but we just need the count


Suggested change

// the result wil be a List<LRAStatusHolder> of recovering LRAs but we just need the count

// the result will be a List<LRAStatusHolder> of recovering LRAs but we just need the count

tomjenkinson · 2019-12-02T10:02:33Z

rts/lra/lra-coordinator-jar/pom.xml

@@ -1,5 +1,7 @@
 <?xml version="1.0" encoding="UTF-8" standalone="no"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+


tomjenkinson · 2019-12-02T10:02:39Z

rts/lra/lra-coordinator-jar/pom.xml

    <modelVersion>4.0.0</modelVersion>
+


mmusgrov · 2019-12-02T11:03:14Z

There are still a few JAX-RS Client not closed in LRACoordinatorTestCase but if that's a design decision than ignore this message.

It was deliberate (though not a "design decision"). This is a surefire test case and the JVM is torn down after each test so even if there were consequences to not closing the client it would have no effect on the test. Test cases are different from production code and the goal is coverage and simplicity (I do agree that closing responses is important since network resources live in the kernel).

But that said, I have made the changes anyway.

tomjenkinson

There are some formatting changes but as LRA is still in active development it does not block the merge and the specific formatting changes do not inhibit debugging as they are in poms.

…hen comparing LRA ids

mmusgrov force-pushed the JBTM-3112 branch from 38e4778 to 250681e Compare November 26, 2019 23:26

mmusgrov force-pushed the JBTM-3112 branch 2 times, most recently from a066a56 to c09e28f Compare November 27, 2019 07:57

mmusgrov force-pushed the JBTM-3112 branch from c09e28f to 4d8d9f1 Compare November 27, 2019 08:07

mmusgrov force-pushed the JBTM-3112 branch from 4d8d9f1 to ec712d3 Compare November 27, 2019 10:24

mmusgrov requested a review from tomjenkinson November 28, 2019 16:25

xstefank reviewed Nov 29, 2019

View reviewed changes

tomjenkinson reviewed Nov 29, 2019

View reviewed changes

...lra-coordinator-jar/src/main/java/io/narayana/lra/coordinator/domain/service/LRAService.java Show resolved Hide resolved

tomjenkinson reviewed Nov 29, 2019

View reviewed changes

xstefank mentioned this pull request Nov 29, 2019

JBTM-3221 Refactor NarayanaLRARecovery according to the specification… #1531

Merged

mmusgrov force-pushed the JBTM-3112 branch from 0f77a82 to 5d7d231 Compare November 29, 2019 15:17

tomjenkinson reviewed Nov 29, 2019

View reviewed changes

xstefank reviewed Dec 2, 2019

View reviewed changes

tomjenkinson reviewed Dec 2, 2019

View reviewed changes

mmusgrov force-pushed the JBTM-3112 branch from f6f18d8 to 7554137 Compare December 2, 2019 10:39

xstefank approved these changes Dec 2, 2019

View reviewed changes

mmusgrov force-pushed the JBTM-3112 branch from 7554137 to 3cecb6b Compare December 2, 2019 11:03

tomjenkinson approved these changes Dec 2, 2019

View reviewed changes

mmusgrov added 2 commits December 2, 2019 12:29

JBTM-3112 Respect LRA deadlines during recovery.

0a412b0

JBTM-3222 Handle different versions of localhost (127.0.0.1 and :1) w…

6efd0d0

…hen comparing LRA ids

mmusgrov force-pushed the JBTM-3112 branch from 3cecb6b to 6efd0d0 Compare December 2, 2019 12:33

mmusgrov merged commit cc46def into jbosstm:master Dec 2, 2019

mmusgrov deleted the JBTM-3112 branch December 2, 2019 12:35

mmusgrov changed the title ~~JBTM-3112 Respect LRA deadlines during recovery.~~ JBTM-3212 Respect LRA deadlines during recovery. Dec 2, 2019

mmusgrov mentioned this pull request Dec 3, 2019

JBTM-3226 Downgrade byteman version when using multi rule scripts #1533

Merged

	* @throws URISyntaxException if the LRA or revovery URIs are invalid (should never happen)
	* @throws URISyntaxException if the LRA or recovery URIs are invalid (should never happen)

		@@ -275,6 +275,7 @@ public Response startLRA(
		}

		String coordinatorUrl = String.format("%s%s", context.getBaseUri(), COORDINATOR_PATH_NAME);

	LRALogger.logger.debugf("Finised test %s", testName);
	LRALogger.logger.debugf("Finished test %s", testName);


		Assert.assertEquals("Unexpected status from reovery call:", 200, response.getStatus());

		// the result wil be a List<LRAStatusHolder> of recovering LRAs but we just need the count

	// the result wil be a List<LRAStatusHolder> of recovering LRAs but we just need the count
	// the result will be a List<LRAStatusHolder> of recovering LRAs but we just need the count

		@@ -1,5 +1,7 @@
		<?xml version="1.0" encoding="UTF-8" standalone="no"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

JBTM-3212 Respect LRA deadlines during recovery. #1530

JBTM-3212 Respect LRA deadlines during recovery. #1530

Conversation

mmusgrov commented Nov 26, 2019 • edited Loading

jbosstm-bot commented Nov 26, 2019

jbosstm-bot commented Nov 26, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 27, 2019

jbosstm-bot commented Nov 28, 2019

jbosstm-bot commented Nov 28, 2019

xstefank left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomjenkinson commented Nov 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomjenkinson commented Nov 29, 2019

xstefank commented Nov 29, 2019

tomjenkinson commented Nov 29, 2019

mmusgrov commented Nov 29, 2019

mmusgrov commented Nov 29, 2019

jbosstm-bot commented Nov 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmusgrov commented Nov 29, 2019

jbosstm-bot commented Nov 29, 2019

xstefank left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmusgrov commented Dec 2, 2019

tomjenkinson left a comment

Choose a reason for hiding this comment

mmusgrov commented Nov 26, 2019 •

edited

Loading