rh-che #557: Adding property and handler for stopping k8s / openshift workspace if unrecoverable event occurs during workspace startup #9426

ibuziuk · 2018-04-13T17:21:12Z

What does this PR do?

Adding property and handler for stopping k8s / openshift workspace if unrecoverable event occurs during workspace startup

Demo - https://youtu.be/J_DfNZXidE0

What issues does this PR fix or reference?

redhat-developer/rh-che#557

Adding property che.infra.kubernetes.workspace_unrecoverable_events

if during workspace startup an unrecoverable event occurs (e.g. "Failed Mount" "Failed Scheduling") terminate workspace immediately instead of waiting util timeout

Release Notes

Adding property che.infra.kubernetes.workspace_unrecoverable_events

if during workspace startup an unrecoverable event occurs (e.g. "Failed Mount" "Failed Scheduling") terminate workspace immediately instead of waiting util timeout

Docs PR

eclipse-che/che-docs#395

codenvy-ci · 2018-04-13T17:24:02Z

Can one of the admins verify this patch?

codenvy-ci · 2018-04-13T17:24:04Z

Can one of the admins verify this patch?

codenvy-ci · 2018-04-13T17:26:01Z

Can one of the admins verify this patch?

sleshchenko · 2018-04-16T11:00:56Z

...main/java/org/eclipse/che/workspace/infrastructure/kubernetes/KubernetesInternalRuntime.java

@@ -420,6 +426,10 @@ protected void startMachines() throws InfrastructureException {
    // namespace.pods().watch(new AbnormalStopHandler());
    // namespace.pods().watchContainers(new MachineLogsPublisher());

+    if (failFastOnUnrecoverableEvents) {
+      namespace.pods().watchContainers(new UnrecoverableEventHanler());


I think it could be useful add MachineLogsPublisher here. WDYT?

@sleshchenko can you elaborate on what exactly to do?

I think it should be something like the following:

--> start try { namespace.pods().watchContainers(new MachineLogsPublisher()); if (failFastOnUnrecoverableEvents) { namespace.pods().watchContainers(new UnrecoverableEventHanler()); } //doStart here } catch .... ... } finally { namespace.pods().stopWatch(); }

In this case clients would receive K8s/OS events text as machine's output, and it would be easier to understand why workspace start fails.

@sleshchenko Adding MachineLogsPublisher seems to be unrelated to this issue but rather to #7653 IMO. Could this be addressed in another PR ?

@ibuziuk For me, it looks like it's easy to add it here (I guess described changes should be enough) and it will be useful (machine logs panel will have all events). It's why I write it here.
But I agree that it is not related to a solved issue and it can be added in another PR.

@sleshchenko ok, sounds relevant - let me add it to this PR

sleshchenko · 2018-04-16T11:01:29Z

...main/java/org/eclipse/che/workspace/infrastructure/kubernetes/KubernetesInternalRuntime.java

+    }
+
+    private boolean isUnrecoverable(String reason) {
+      return reason.equals("Failed Mount") || reason.equals("Failed Scheduling");


Maybe it makes sense to make these values configurable?

@sleshchenko do you mean property with comma separated reasons ?

I think yes.

garagatyi · 2018-04-16T09:55:21Z

...java/org/eclipse/che/workspace/infrastructure/kubernetes/namespace/event/ContainerEvent.java

@@ -80,6 +88,9 @@ public String toString() {
        + ", containerName='"
        + containerName
        + '\''
+        + ", reason='"


A new field should be used in equals/hashcode methods too

yeah, good point

garagatyi · 2018-04-16T10:52:14Z

...main/java/org/eclipse/che/workspace/infrastructure/kubernetes/KubernetesInternalRuntime.java

@@ -240,6 +244,8 @@ private void waitMachines(
      failure.completeExceptionally(ex);
      cancelAll(toCancelFutures);
      wrapAndRethrow(ex.getCause());
+    } finally {
+      namespace.pods().stopWatch();


Have you considered moving canceling of watching to internalStart method? I think that there might be a situation in future when newly introduced bug prevents the execution of method waitMachines which would lead to a leak of the watch connections.

sounds good, let me fix

moved to internalStart

garagatyi · 2018-04-16T10:56:58Z

...main/java/org/eclipse/che/workspace/infrastructure/kubernetes/KubernetesInternalRuntime.java

+      String reason = event.getReason();
+      String message = event.getMessage();
+      if (isUnrecoverable(reason)) {
+        LOG.error("Unrecoverable event occured {}, {}", reason, message);


Do you think dev/ops team would be able to address logged problem somehow? Maybe it is worth to add more info to be able to debug the issue or remove logging at all?

@garagatyi what info do you will be relevant here ? req_id / identity id would be added to the logs automatically I assume, no ?

Maybe workspace ID, maybe some pod/container info if any

have added pod info

garagatyi · 2018-04-16T10:59:55Z

...main/java/org/eclipse/che/workspace/infrastructure/kubernetes/KubernetesInternalRuntime.java

@@ -600,6 +610,26 @@ public void accept(ProbeResult probeResult) {
    }
  }

+  /** Listens container's events and terminates runtime if unrecoverable event occurs. */
+  public class UnrecoverableEventHanler implements ContainerEventHandler {


Can you add tests for this case?

garagatyi · 2018-04-16T11:00:36Z

...st/java/org/eclipse/che/workspace/infrastructure/openshift/OpenShiftInternalRuntimeTest.java

@@ -193,6 +195,7 @@ public void shouldStartMachines() throws Exception {
    verify(routes).create(any());
    verify(services).create(any());

+    verify(project.pods(), times(1)).watchContainers(any());


Can you also check for the cancellation of watching?

garagatyi · 2018-04-16T11:00:54Z

...st/java/org/eclipse/che/workspace/infrastructure/openshift/OpenShiftInternalRuntimeTest.java

@@ -137,6 +138,7 @@ public void setup() throws Exception {
        new OpenShiftInternalRuntime(
            13,
            5,
+            FAIL_ON_UNRECOVERABLE_EVENT,


Can you add a case when the value is false?

@garagatyi do you think it is ok to check one infra with false & other with true, or you think there should be both cases handled in both infras ?

Since they might have their own bugs we should have tests for both

garagatyi · 2018-04-16T11:01:05Z

.../java/org/eclipse/che/workspace/infrastructure/kubernetes/KubernetesInternalRuntimeTest.java

@@ -180,6 +182,7 @@ public void setup() throws Exception {
        new KubernetesInternalRuntime<>(
            13,
            5,
+            FAIL_ON_UNRECOVERABLE_EVENT,


Can you add a case when the value is false?

garagatyi · 2018-04-16T11:07:09Z

...main/java/org/eclipse/che/workspace/infrastructure/kubernetes/KubernetesInternalRuntime.java

@@ -600,6 +610,26 @@ public void accept(ProbeResult probeResult) {
    }
  }

+  /** Listens container's events and terminates runtime if unrecoverable event occurs. */


AFAIK we don't have runtime interruption in place for k8s runtime yet. Can you elaborate how this handler stops start of a runtime?

s/runtime/workspace/

sleshchenko

Your changes will make life easier during an investigation of workspace start failing 👍

sleshchenko · 2018-04-17T07:16:38Z

assembly/assembly-wsmaster-war/src/main/webapp/WEB-INF/classes/che/che.properties

@@ -347,6 +347,11 @@ che.infra.kubernetes.workspace_start_timeout_min=8
 # Defines the timeout in minutes that limits the period for which Kubernetes Ingress become ready
 che.infra.kubernetes.ingress_start_timeout_min=5

+# If during workspace startup an unrecoverable event defined in 'che.infra.kubernetes.workspace_unrecoverable_events' occurs
+# terminate workspace immediately instead of waiting util timeout


Should util be until?

sleshchenko · 2018-04-17T07:30:10Z

...main/java/org/eclipse/che/workspace/infrastructure/kubernetes/KubernetesInternalRuntime.java

+        String message = event.getMessage();
+        String podName = event.getPodName();
+        try {
+          internalStop(emptyMap());


I think it doesn't work correctly until k8s runtime start interruption will be implemented.
I guess client will receive two workspace status events:

one that workspace failed to start because of Unrecoverable;

a second one that start timeout exceeded.

Let me check that. BTW I think we can merge these changes with the described issue and it will be fixed with implemented interruption of k8s runtime start.

I've checked. So here is the Che Server logs

| 2018-04-17 08:02:45,912[io-8080-exec-10] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 236] - Starting workspace 'che/qwe' with id 'workspacea75oxtgryzdjq5iq' by user 'che' /////////////////// Here Client received that workspace failed to start but Che Server doesn't contain the corresponding message /////////////////////////// /////////////////// ~3 minutes later reached workspace start timeout /////////////////////////// | 2018-04-17 08:05:18,006[aceSharedPool-0] [WARN ] [.i.k.KubernetesInternalRuntime 177] - Failed to start Kubernetes runtime of workspace workspacelhu15ifhded6jrxf. Cause: Server 'terminal' in machine 'dev-machine' not available. | 2018-04-17 08:05:18,006[ServersChecker] [ERROR] [.i.k.KubernetesInternalRuntime 335] - Unable to update status of the machine '%s:%s'. Cause: %s | 2018-04-17 08:05:18,066[aceSharedPool-0] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 292] - Workspace 'che:wksp-g9f7' with id 'workspacelhu15ifhded6jrxf' start failed | 2018-04-17 08:05:18,070[aceSharedPool-0] [WARN ] [o.e.c.a.w.s.WorkspaceManager 423] - Cannot set error status of the workspace workspacelhu15ifhded6jrxf. Error is: Workspace with id 'workspacelhu15ifhded6jrxf' doesn't exist

sleshchenko · 2018-04-17T07:34:35Z

assembly/assembly-wsmaster-war/src/main/webapp/WEB-INF/classes/che/che.properties

@@ -347,6 +347,11 @@ che.infra.kubernetes.workspace_start_timeout_min=8
 # Defines the timeout in minutes that limits the period for which Kubernetes Ingress become ready
 che.infra.kubernetes.ingress_start_timeout_min=5

+# If during workspace startup an unrecoverable event defined in 'che.infra.kubernetes.workspace_unrecoverable_events' occurs
+# terminate workspace immediately instead of waiting util timeout
+che.infra.kubernetes.workspace_fail_on_unrecoverable_events=true


Can you describe why it is necessary to have separate configuration property for disabling listening to unrecoverable events. Maybe it makes sense to configure empty che.infra.kubernetes.workspace_unrecoverable_events for disabling instead of having a separate one?

good point, less config we have the better

skabashnyuk · 2018-04-17T08:38:42Z

infrastructures/kubernetes/pom.xml

@@ -66,6 +66,10 @@
            <groupId>javax.inject</groupId>
            <artifactId>javax.inject</artifactId>
        </dependency>
+        <dependency>
+            <groupId>org.codehaus.plexus</groupId>


I guess we don't have CQ for this dependency. Can you use some alternatives?

@skabashnyuk seems to be defined in parent https://github.com/eclipse/che-parent/blob/master/pom.xml#L252

It's not an issue where it's defined. the problem is that it becomes compile level and we have no cq for it. https://dev.eclipse.org/ipzilla/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&component=ecd.che&long_desc_type=substring&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&keywords_type=allwords&keywords=&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0=

AFAIK you can find the same helper method on other artifacts like guava

@skabashnyuk ok, good to know - changed to com.google.common.base.Strings

l0rd · 2018-04-17T10:09:15Z

assembly/assembly-wsmaster-war/src/main/webapp/WEB-INF/classes/che/che.properties

@@ -347,6 +347,10 @@ che.infra.kubernetes.workspace_start_timeout_min=8
 # Defines the timeout in minutes that limits the period for which Kubernetes Ingress become ready
 che.infra.kubernetes.ingress_start_timeout_min=5

+# If during workspace startup an unrecoverable event defined in the property occurs,
+# terminate workspace immediately instead of waiting until timeout
+che.infra.kubernetes.workspace_unrecoverable_events=Failed Mount,Failed Scheduling,Failed to pull image


For users it may be interesting to have a list (or better a link) of valid kubernetes events.

@l0rd could you please clarify a bit ? is it about changing the property format

I think Mario means a link to docs where all the event types are listed.
And 👍 to add it here.

If I am a Che user and I am editing che.properties I don't know what are the valid values for this new property. You may want to say in the comment that this is a comma separated list of kubernetes events. And it would be great if we could add a link to a list of valid kubernetes events (but I am not sure we can easily find one so don't lose too much time on this)

garagatyi · 2018-04-17T11:53:53Z

...main/java/org/eclipse/che/workspace/infrastructure/kubernetes/KubernetesInternalRuntime.java

+        try {
+          internalStop(emptyMap());
+        } catch (InfrastructureException e) {
+          LOG.error("Unrecoverable event occured {}, {}, {}", reason, message, podName);


It seems that without workspace ID it won't be possible to match the incident with a workspace/user. I think we need WSID her.

ok, will also add workspace id

garagatyi · 2018-04-17T11:55:48Z

...java/org/eclipse/che/workspace/infrastructure/kubernetes/namespace/event/ContainerEvent.java

@@ -62,6 +70,7 @@ public boolean equals(Object o) {
    ContainerEvent that = (ContainerEvent) o;
    return Objects.equals(podName, that.podName)
        && Objects.equals(containerName, that.containerName)
+        && Objects.equals(reason, that.reason)


Can you also add the reason to the hashcode method? It is common to have them aligned.

done, thanks for the pointer

garagatyi · 2018-04-17T12:02:42Z

...c/main/java/org/eclipse/che/workspace/infrastructure/openshift/OpenShiftInternalRuntime.java

@@ -92,7 +97,11 @@ protected void startMachines() throws InfrastructureException {
    }
    // TODO https://github.com/eclipse/che/issues/7653
    // project.pods().watch(new AbnormalStopHandler());
-    // project.pods().watchContainers(new MachineLogsPublisher());
+
+    project.pods().watchContainers(new MachineLogsPublisher());


TLDR please, add finally here as well
Since we set watchers here we should also ensure that all the watchers are properly deleted. We can not guarantee that there is no bug which prevents the execution of finally block from k8s infra.

sorry, not following - startMachines is supposed to be called from internalStart where watchers closed / deleted in finally. Could you describe potential bug that could prevent closing watchers currently ?

Sorry, my bad. We don't need finally here.

garagatyi · 2018-04-17T12:04:06Z

@ibuziuk please, do not push to the branch with force. Each force push make reviewers review whole PR from scratch

garagatyi · 2018-04-17T12:12:59Z

BTW looks like you need to address @skabashnyuk 's comment before merging since we can't add new dependency without a CQ

ibuziuk · 2018-04-17T13:05:30Z

@garagatyi fixed dep CQ problem - changed to com.google.common.base.Strings

ibuziuk · 2018-04-17T13:45:38Z

@l0rd have added PR for docs eclipse-che/che-docs#395

garagatyi · 2018-04-17T14:29:08Z

@ibuziuk is there any other cases that you are going to cover in unit tests?

ibuziuk · 2018-04-17T14:38:41Z

@garagatyi yeah, working on adding more tests

l0rd · 2018-04-17T15:29:51Z

assembly/assembly-wsmaster-war/src/main/webapp/WEB-INF/classes/che/che.properties

@@ -347,6 +347,10 @@ che.infra.kubernetes.workspace_start_timeout_min=8
 # Defines the timeout in minutes that limits the period for which Kubernetes Ingress become ready
 che.infra.kubernetes.ingress_start_timeout_min=5

+# If during workspace startup an unrecoverable event defined in the property occurs,
+# terminate workspace immediately instead of waiting until timeout
+che.infra.kubernetes.workspace_unrecoverable_events=Failed Mount,Failed Scheduling,Failed


Failed to pull image has become Failed

it was actually done deliberate, but I can rework

Yes please revert that. Having Failed in a list of kubernetes events is confusing.

@lord ok, just wanted to clarify that reason of the event for "Failed to pull image" is actually just Failed, so I assume that Failed is quite common reason for k8s / openshift events. Failed to pull image is part of the event message

ibuziuk · 2018-04-18T10:17:07Z

@l0rd @garagatyi changed property value, added tests. Please, review
PR for updating docs to make it consistent with property - eclipse-che/che-docs#397

garagatyi · 2018-04-18T11:05:14Z

...c/main/java/org/eclipse/che/workspace/infrastructure/openshift/OpenShiftInternalRuntime.java

@@ -92,7 +97,11 @@ protected void startMachines() throws InfrastructureException {
    }
    // TODO https://github.com/eclipse/che/issues/7653
    // project.pods().watch(new AbnormalStopHandler());
-    // project.pods().watchContainers(new MachineLogsPublisher());
+
+    project.pods().watchContainers(new MachineLogsPublisher());


Sorry, my bad. We don't need finally here.

garagatyi · 2018-04-18T11:19:29Z

.../java/org/eclipse/che/workspace/infrastructure/kubernetes/KubernetesInternalRuntimeTest.java

@@ -352,8 +412,9 @@ public void throwsInfrastructureExceptionWhenKubernetesNamespaceCleanupFailed()
  @Test
  public void testRepublishContainerOutputAsMachineLogEvents() throws Exception {
    final MachineLogsPublisher logsPublisher = internalRuntime.new MachineLogsPublisher();
-    final ContainerEvent out1 = mockContainerEvent("pulling image", "07/07/2007 19:01:22");
-    final ContainerEvent out2 = mockContainerEvent("image pulled", "07/07/2007 19:08:53");
+    final ContainerEvent out1 =


You have added logs publishing in the runtime start process and this is very cool. Can you also add a unit test for this case?

isn't it already handled by https://github.com/eclipse/che/pull/9426/files#diff-08f3ea6fea2790aa766edf8d57d37897R275 e.g. handler for publishing events is added during runtime start?

Well, it just checks that some subscribers are watching it. We have a separate test for logs publisher, so maybe we can also check that one of the watchers is logs publisher and another one is unrecoverable events listener? But it is not critical, so feel free to leave it as is if you want.

garagatyi · 2018-04-18T11:25:16Z

.../java/org/eclipse/che/workspace/infrastructure/kubernetes/KubernetesInternalRuntimeTest.java

@@ -366,10 +427,34 @@ public void testRepublishContainerOutputAsMachineLogEvents() throws Exception {
    assertTrue(captor.getAllValues().containsAll(machineLogs));
  }

+  @Test
+  public void testHandleUnrecoverableEvent() throws Exception {
+    final UnrecoverableEventHanler unrecoverableEventHanler =


Please, test unrecoverable events matching/non-matching in the whole flow instead of with a separated component. We don't have any guarantees that handler is called as it is supposed in this test case.
Please cover cases:
non-matching event
matching by the reason field
matching by the message field

It would be nice to verify that stop of the runtime is really called after appearing of the matching event.

@garagatyi was not able to figure out how exactly to test unrecoverable events in starting flow, but have added separate tests for matching by the reason / matching by the message. Also, not sure how exactly verifying that stop of the runtime e.g. internalStop should be tested - currently done by verifying that namespace cleanup was called

Are you ok to merge it in the current state, since I do not really want to face merge conflicts, and we could discuss more tests cases which I would add later in separate PR ?

l0rd

LGTM. Thanks @ibuziuk!

garagatyi

Looks good! I believe that tests that we were discussing are not mandatory for the PR.

ibuziuk · 2018-04-19T09:58:49Z

@garagatyi ok, thanks I will try to add more tests for those event handlers during the sprint

…/ openshift runtime if unrecoverable event occurs during workspace startup Signed-off-by: Ilya Buziuk <ibuziuk@redhat.com>

ibuziuk requested review from garagatyi, riuvshin and skabashnyuk as code owners April 13, 2018 17:21

benoitf added the status/code-review This issue has a pull request posted for it and is awaiting code review completion by the community. label Apr 13, 2018

ibuziuk force-pushed the rh-che-557 branch from 7e5dfd6 to 4b4d999 Compare April 15, 2018 09:27

skabashnyuk requested review from mshaposhnik and sleshchenko April 16, 2018 10:14

garagatyi requested a review from akorneta April 16, 2018 10:42

sleshchenko reviewed Apr 16, 2018

View reviewed changes

garagatyi reviewed Apr 16, 2018

View reviewed changes

ibuziuk force-pushed the rh-che-557 branch from 4b4d999 to 1155bd9 Compare April 16, 2018 14:03

akorneta approved these changes Apr 16, 2018

View reviewed changes

ibuziuk force-pushed the rh-che-557 branch from 1155bd9 to f063f9a Compare April 16, 2018 15:19

ibuziuk force-pushed the rh-che-557 branch from f063f9a to 6482315 Compare April 16, 2018 17:58

sleshchenko approved these changes Apr 17, 2018

View reviewed changes

ibuziuk force-pushed the rh-che-557 branch from 6482315 to a7b251f Compare April 17, 2018 08:31

skabashnyuk reviewed Apr 17, 2018

View reviewed changes

l0rd approved these changes Apr 17, 2018

View reviewed changes

ibuziuk force-pushed the rh-che-557 branch from a7b251f to d49012c Compare April 17, 2018 11:02

garagatyi reviewed Apr 17, 2018

View reviewed changes

ibuziuk mentioned this pull request Apr 17, 2018

rh-che-557: Adding docs for configuring K8s / OpenShift unrecoverable… eclipse-che/che-docs#395

Merged

l0rd requested changes Apr 17, 2018

View reviewed changes

skabashnyuk approved these changes Apr 18, 2018

View reviewed changes

ibuziuk mentioned this pull request Apr 18, 2018

rh-che-557: Making doc consistent with unrecoverable event property eclipse-che/che-docs#397

Merged

garagatyi reviewed Apr 18, 2018

View reviewed changes

l0rd approved these changes Apr 18, 2018

View reviewed changes

garagatyi approved these changes Apr 19, 2018

View reviewed changes

rh-che eclipse-che#557: Adding property and handler for stopping k8s …

402703d

…/ openshift runtime if unrecoverable event occurs during workspace startup Signed-off-by: Ilya Buziuk <ibuziuk@redhat.com>

ibuziuk force-pushed the rh-che-557 branch from b1cd07c to 402703d Compare April 19, 2018 11:15

ibuziuk merged commit ac3b401 into eclipse-che:master Apr 19, 2018

ibuziuk mentioned this pull request Apr 19, 2018

Provide a better error message when PV failed to be attached redhat-developer/rh-che#557

Closed

benoitf added this to the 6.5.0 milestone Apr 19, 2018

benoitf removed the status/code-review This issue has a pull request posted for it and is awaiting code review completion by the community. label Apr 19, 2018

sleshchenko mentioned this pull request Apr 26, 2018

Workspaces can not be started after change image in the "Machine" section #9542

Closed

rh-che #557: Adding property and handler for stopping k8s / openshift workspace if unrecoverable event occurs during workspace startup #9426

rh-che #557: Adding property and handler for stopping k8s / openshift workspace if unrecoverable event occurs during workspace startup #9426

Conversation

ibuziuk commented Apr 13, 2018 • edited Loading

What does this PR do?

What issues does this PR fix or reference?

Release Notes

Docs PR

codenvy-ci commented Apr 13, 2018

codenvy-ci commented Apr 13, 2018

codenvy-ci commented Apr 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ibuziuk Apr 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sleshchenko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ibuziuk Apr 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

garagatyi commented Apr 17, 2018

garagatyi commented Apr 17, 2018

ibuziuk commented Apr 17, 2018

ibuziuk commented Apr 17, 2018

garagatyi commented Apr 17, 2018

ibuziuk commented Apr 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ibuziuk Apr 18, 2018 • edited Loading

Choose a reason for hiding this comment

ibuziuk commented Apr 18, 2018 • edited Loading

Choose a reason for hiding this comment

ibuziuk commented Apr 13, 2018 •

edited

Loading

ibuziuk Apr 16, 2018 •

edited

Loading

ibuziuk Apr 17, 2018 •

edited

Loading

ibuziuk Apr 18, 2018 •

edited

Loading

ibuziuk commented Apr 18, 2018 •

edited

Loading