KEP-22: Add initial design on diagnostics bundles #1310

mpereira · 2020-01-24T16:35:25Z

mpereira · 2020-01-24T16:38:40Z

The operator developer experience section needs to be fleshed out. It is planned that @gerred will take a look at this whenever he gets a chance.

mpereira · 2020-01-24T16:39:49Z

(Pinging @marccampbell as well)

gerred · 2020-02-11T13:29:27Z

Added a lot of bundle-level information here.

keps/0022-diagnostics-bundle.md

ANeumann82 · 2020-02-11T14:19:58Z

keps/0022-diagnostics-bundle.md

+
+Also, may specs may include an `objectRef`. It ALWAYS has the following keys:
+
+- **kind**: The Kubernetes Kind referenced. For example, this may be a


Can this be a CR as well? I would assume so, but in that case do we need a GVK? Or is a GK enough?

Hmmm.. Not sure it would make sense to allow CRs here. But in that case we need to specify what kinds we exactly allow here. Probably something that boils down to something runnable?

I wonder if there's a convention we can follow. You could potentially have a CR that is representative of a set of pods - see the metacontroller DaemonJob example. Additionally, could you just eventually reference a KUDO Instance that is created as part of an operator (see: Flink) and get ALL PODS inside of Flink?

There is some convention here around selectors and labels inside of Deployment, Statefulset, and Pod. Though, service has that too. Could you select ALL PODS that are selected by that Service? It seems weird at first, but it could be very useful (give me the traffic information for all pods that are currently being load balanced by this service).

What's a CR?

CR stands here for "Custom Resource"

ANeumann82 · 2020-02-11T14:27:46Z

keps/0022-diagnostics-bundle.md

+- **taskRef**: Name of the task to run. **NOTE**: We MAY need a Pause and Resume
+  task to be able to copy files and run commands during the running of a task.
+  Otherwise, we may want to make this an arbitrary job.


Could we maybe utilize the PipeTask for this?

Potentially. There were a few semantics I was worried about and I think that'll just come up in implementation though. Mostly around - what if we're piping around VERY large files? Debugging training models, etc. Maybe that's then a "this app was architected poorly" thing and we should have file limits.

We should probably have file limits in any case ;) But they can probably be bigger than the 1MB of the PipeTask, yes. Also it might be useful to copy more than one file out of a task. I think the bundle.resources.Task type seems to be a feature that might be lower on our prio list for the implementation

agreed. what I was looking to solve here was preventing mega-pods filled with unrelated things. for example, seems silly to have to have nodetool sitting inside of your cassandra pods, but maybe this is false and nobody actually cares. But you may want to debug with dtrace, while not having to ship dtrace along, as a perhaps better example.

reminds me, we will need an optional container field for multi-container pods and exec.

Or maybe it's as simple as resources get created by this are deleted at the end of a diagnostics run, enabling you to have the subsequent copy and command bundle tasks that you need 🤔

yeah, this is probably low priority for now compared to Copy and Command.

Having resources deleted at the end of diagnostics and then using Copy&Command sounds actually not too bad, that would be easy to implement.

keps/0022-diagnostics-bundle.md

ANeumann82 · 2020-02-11T14:34:14Z

keps/0022-diagnostics-bundle.md

+- **name**: The human readable name of the filter.
+- **regex** (optional): Regular expression, not encased in slashes, to use.
+  Regex flags are not supported, and are global and case sensitive.
+- **objectRef** (optional): Required if `jsonPath` is present.


In this context it would probably make sense to be able to reference CRs, i assume? So we probably need to think about what an objectRef exactly should be

Agreed. objectRef, fwiw, is a standard convention. Here we have a bit of a partial one since you're not specifying the full GVK, so I guess we're not technically following convention here 🤔 but we will need to for CRs.

gerred · 2020-02-11T15:32:57Z

odd these tests are failing, assuming they're flakes.

runyontr · 2020-02-12T01:04:37Z

Overall looks like a great KEP that will provide some good value for debugging. The only thing I would follow up with, is that given a diagnostic bundle: Now what?

We should think about how these get submitted to the Bug team (Operator or KUDO), and maybe make the attachment to a Github issue part of the CLI tooling
Once the bug team receives the bundle, we should be building some sort of tool that can parse/explore it more easily and maybe search for common/reocurring issues/solutions

ANeumann82 · 2020-02-12T13:45:26Z

keps/0022-diagnostics-bundle.md

+- **objectRef** (optional): Required if `jsonPath` is present.
+- **jsonPath** (optional): JSONPath referencing a non-object key in the
+  referenced object. All instances of the value of this key will be removed and
+  replaced with `**FILTERED**`. Required if `objectRef` is present.


Might be a good idea to have an optional replacement value that defaults to "FILTERED".
In this case we could have a filter for IP-Addresses that replaces IPs with "XXX.XXX.XXX.XXX" so we can distinguish between different filtered values?

zen-dog

Nice work! I left a bunch of questions/proposals.

zen-dog · 2020-02-12T12:46:55Z

keps/0022-diagnostics-bundle.md

+
+    Checks:
+
+    - Does the Kubernetes cluster have enough resources to install the operator


Is such a preflight check really feasible? How does one determine the amount of memory/disc/cpus/gpus in the whole cluster? And even if they're available right now, how about in 1s? What about things like PVC and PVs? How does one determine everything is needed for a specific plan to run successfully? Do you dry-run execute it and render all resources in the CLI?
Anyway, I believe preflight checks should be a separate issue/KEP/proposal.

Good idea, we'll split out preflight check. @mpereira's original work included it, but the scope of just the support diagnostics is so large it makes sense to take on it's own.

keps/0022-diagnostics-bundle.md

zen-dog · 2020-02-12T13:38:46Z

keps/0022-diagnostics-bundle.md

+          command: # Can be string or array
+            - nslookup
+            - google.com
+          objectRef:


What if you wanted to run the command on a specific set of pods (but not necessarily whole StatefulSet). Label selectors might be useful here too (see my comment above ^^).

ya I like label selectors a lot as an option.

(still, think reference-able objects is an even better long term solution, graph graph graph 📈 😉 )

Added label selector as @zen-dog suggested, skipped matchExpression as its usage doesn't seem so clear at the moment
Removed templates and name field, because

we agree on disliking templates in operator.yaml

I can be wrong here, but if objects can be named according to a certain template (pattern), they can as well be labelled according to the pattern

in the worst case we'll just add them

Could you also remove the objectRef?

zen-dog · 2020-02-12T13:45:31Z

keps/0022-diagnostics-bundle.md

+        key: "zookeeper-configuration"
+        kind: Copy
+        spec:
+          path: /opt/zookeeper/server.properties


We should allow an array of files here. Otherwise, we'll have to specify N tasks for N files.

👍 good call

keps/0022-diagnostics-bundle.md

zen-dog · 2020-02-12T14:08:03Z

keps/0022-diagnostics-bundle.md

+complexity of selecting resources to run commands and files on.
+
+Steps in a bundle run serially. To prevent the KUDO controller manager from
+crashing, the collector process runs in another pod as a job. **TODO**:


Ok, I've taken another close look at what data we want to collect and all of it is available from the CLI. We have a mix of k get/describe/logs/events and k exec/cp for executing commands and saving stdout/copy out files from the pods. Frankly, all of that could be a rather simple kudo-cli command.

If our past taught us anything, then we know that logs from the big clusters can easily exceed gigabytes (even gzipped). Handling them, either in the Manager or in a separate Pod, will require arbitrary size PVCs?

A user of an operator collecting these diagnostics might not have RBAC to run all of these locally, ESPECIALLY exec/copy - which can present security issues.

That said, I'm open to this approach too. It simplifies a lot of things. :)

Could things like resumable/retried diagnostic-fetching (wget -C) be feasibly implemented as a set of operations under a single CLI command?

Handling them, either in the Manager or in a separate Pod, will require arbitrary size PVCs?

That's a good point. It should at a minimum be called out in documentation if that's the path taken, imo.

A user of an operator collecting these diagnostics might not have RBAC to run all of these locally, ESPECIALLY exec/copy - which can present security issues.

Fair point. Though I expect most users have access to "their stuff" or having some kind of admin on the line with the right access. It seems like a small price to pay for a massive implementation simplification 🤷‍♀

Could things like resumable/retried diagnostic-fetching (wget -C) be feasibly implemented as a set of operations under a single CLI command?

That's the easiest way to implement them IMHO. Everything else adds layers of complexity: extra image to maintain, extra Pod/Job to deploy, extra things that can failover to think about...

A user of an operator collecting these diagnostics might not have RBAC to run all of these locally, ESPECIALLY exec/copy - which can present security issues.

Oooh.. this really opens a can of worms.

This seems to imply that we treat the controller as a privileged entity, and therefore treat OperationVersions as trusted resources (because their diagnostics spec can prescribe whatever), and Instances as untrusted ones. However I do not think we mention this anywhere in the operator developer docs - for example that the developer should treat parameter values as untrusted input.

I think for now we should do something like:

say in the KEP that it's unspecified where the diagnostics collection runs,

actually run it from the CLI for now, and require RBAC to be opened up as needed

And in parallel we can start thinking about KUDO security model more broadly.

kensipe · 2020-03-10T14:19:30Z

@gerred ping... what are the next steps here? are you still owner of this issue.

kensipe · 2020-03-10T14:20:59Z

needs rebase with dco signoff

kensipe · 2020-03-10T18:42:37Z

Thanks @vemelin-epm for taking this over. What is expected next? Is this WIP? or ready for review?

porridge

This is great work, but I think it's getting bogged down with too much detail. I'd say let's finish the problem statement and rough design overview and merge it. And then do incremental improvements, because this is already taking a long time.

keps/0022-diagnostics-bundle.md

porridge · 2020-03-11T11:00:52Z

keps/0022-diagnostics-bundle.md

+
+## Prior art, inspiration, resources
+
+### Diagnostics


Please also take a look at sonobuoy. I've never used it myself, but heard that it's pretty extensible, so please investigate that what we're designing could not be done simply with a sonobuoy plugin. Also a sentence or two of explanation why we are not reusing/extending one of these "prior arts" (these related to k8s of course) would be good.

I like sonobuoy a lot and am open to using it. I've only used it for running conformance tests in the past. If the implementation makes sense more as a series of Sonobuoy plugins that are then just part of someone's spec, that's great. We should just balance this with adding it as an external dependency as another service we need to manage.

keps/0022-diagnostics-bundle.md

porridge · 2020-03-11T11:20:30Z

keps/0022-diagnostics-bundle.md

+### bundle.resources.HTTP
+
+- **serviceRef**: Object containing references to a Kubernetes service. This is
+  scoped to KUDO-only services.


Please clarify what a "KUDO-only service" is. Personally I'm puzzled by this statement.

I am puzzled as well. @gerred could you please shed some light here?

We're just assuming that the only Kubernetes services we can reached were made by KUDO.

porridge · 2020-03-11T11:21:45Z

keps/0022-diagnostics-bundle.md

+Filters are a list of filters. They contain the following keys:
+
+- **name**: The human readable name of the filter.
+- **regex** (optional): Regular expression, not encased in slashes, to use.


OK, but what is the semantics of this regex? What is it applied against? Does a match mean "include in the output" or "omit" or "redact"?

semantics: based on the example above I assume it's meant to be applied on the files of certain structure (e.g. the host regex above aims at yaml), but it's not obvious how it's application scope should be restricted. @gerred could you please say some words here?

what match means: I think the current text implies quite consistently that filtering is understood as redacting (say, with asterisks) but not removing the sensitive content and therefore whatever matches should be redacted. So, unless my understanding is wrong, I'd personally prefer not to add more explanations.

Yeah, the original intent here was that after all collection occurred, everything passed through the filters / redaction. @vemelin-epm is correct on the second point.

porridge · 2020-03-11T11:23:00Z

keps/0022-diagnostics-bundle.md

+          path: /opt/zookeeper/server.properties
+          objectRef:
+            kind: StatefulSet # Runs on ALL pods in the statefulset
+            name: "{{ .InstanceName }}-zookeeper"


I agree with @zen-dog here. Let's not do this. Let's use label sets to select objects as the convention requires.

porridge · 2020-03-11T11:32:26Z

keps/0022-diagnostics-bundle.md

+complexity of selecting resources to run commands and files on.
+
+Steps in a bundle run serially. To prevent the KUDO controller manager from
+crashing, the collector process runs in another pod as a job. **TODO**:


A user of an operator collecting these diagnostics might not have RBAC to run all of these locally, ESPECIALLY exec/copy - which can present security issues.

Oooh.. this really opens a can of worms.

This seems to imply that we treat the controller as a privileged entity, and therefore treat OperationVersions as trusted resources (because their diagnostics spec can prescribe whatever), and Instances as untrusted ones. However I do not think we mention this anywhere in the operator developer docs - for example that the developer should treat parameter values as untrusted input.

I think for now we should do something like:

say in the KEP that it's unspecified where the diagnostics collection runs,

actually run it from the CLI for now, and require RBAC to be opened up as needed

And in parallel we can start thinking about KUDO security model more broadly.

keps/0022-diagnostics-bundle.md

porridge

I think we're getting there. A bunch of nitpicks inline, and I really think we should change the word filter since it implies omitting information rather than replacing it with something else. I'd suggest redact but maybe someone has a better alternative.

porridge · 2020-03-26T07:46:31Z

keps/0022-diagnostics-bundle.md

+- At least not initially: collection of metrics from monitoring services (e.g.,
+  Prometheus, Statsd, etc.).
+- Automatic fixing of faults
+- Preflight checks


It's not entirely clear to me what this point means... 🤔

keps/0022-diagnostics-bundle.md

porridge · 2020-03-26T08:01:25Z

keps/0022-diagnostics-bundle.md

+- **Command**: Run a command on a running pod and copy the stdout/err. Higher level
+  resources can also be used, which will run the command on all pods selected by
+  that resource.
+- **Task**: Run a KUDO task and copy the stdout and other arbitrary files.


I'm not sure I understand this... a "KUDO task" in general does not have notions of "stdout" or "files". Some specific task types might have something close to those.
However both this and the bundle.resources.Task mention below are so hand-wavy that I'd suggest removing them for now. The paragraph above does say "subject to change" so we can always introduce something like this later, once we have a better idea what's needed.

keps/0022-diagnostics-bundle.md

porridge · 2020-03-26T08:11:27Z

keps/0022-diagnostics-bundle.md

+use magical strings with templates. Future iterations of this will reduce the
+complexity of selecting resources to run commands and files on.
+
+Filtering is an important part of diagnostics collection. It enables diagnostics


I'd suggest renaming "filtering" with "redaction" everywhere, since this is what it is about. "Filtering" makes me thing of grep while "redaction" is sed 's///' in my mind.

porridge · 2020-03-26T08:15:36Z

keps/0022-diagnostics-bundle.md

+By default, KUDO filters all resources (and custom resources) of values
+contained within the KUDO Instance's secrets. This is configurable with the


Hm, this seems like a promise that is hard to deliver on... for example the secret may be a JSON file which contains a password ({"user":"admin", "password": "foo"}). How do we make KUDO recognize that it should elide that password when it encounters it in a log file in a different context (... logging into 1.2.3.4 with password foo.)?

True. But we probably cannot avoid doing this anyway, otherwise the bundle cannot be safely shared. For files with known structure (e.g. yaml) it will be pretty doable. Logs will present a problem indeed, however we can just blindly black out whatever matches any secret as the initial approach, or there may be configurable phrase-based replacement. I may be wrong here, but I believe that placing sensitive information in the logs is a dubious practice after all

porridge · 2020-03-26T08:17:19Z

keps/0022-diagnostics-bundle.md

+  resources can also be used, which will run the command on all pods selected by
+  that resource.
+- **Task**: Run a KUDO task and copy the stdout and other arbitrary files.
+- **HTTP**: Make an HTTP request from the KUDO controller manager to a named


I'm not sure we want to promise that it will be the controller manager who does the fetching... Perhaps this should just say it should be fetched by a process running within the cluster?

I think, the reason for executing it from the manager is mentioned in the L258. We want to support air-gapped clusters (so no fetching of arbitrary images) and don't want operator developers to maintain curl-containers (though busybox has wget so that might be not an issue).

In general, I think all these "diagnostics forms" were meant to be run from the KUDO manager as some sort of tasks: not necessarily as plan-tasks but something similar (@gerred correct me if I'm wrong). Now, at least for the general diagnostics, we switched to the cli-based approach. We might do the same for extended diagnostics too. E.g. a HTTP request might be a kubectl exec busybox-pod wget ...

keps/0022-diagnostics-bundle.md

zen-dog

Another pass on it. I think this is in good-enough shape that we can merge it as provisional and update later. The biggest question for me would be whether the extended diagnostics collection happens on the client-side, triggered by the CLI in the cluster, or triggered by the KUDO manager as some sort of tasks. We can mention it and leave it open for now.

@vemelin-epm PTAL and address all the nits and let's merge this.

keps/0022-diagnostics-bundle.md

zen-dog · 2020-03-30T16:36:02Z

keps/0022-diagnostics-bundle.md

+  resources can also be used, which will run the command on all pods selected by
+  that resource.
+- **Task**: Run a KUDO task and copy the stdout and other arbitrary files.
+- **HTTP**: Make an HTTP request from the KUDO controller manager to a named


I think, the reason for executing it from the manager is mentioned in the L258. We want to support air-gapped clusters (so no fetching of arbitrary images) and don't want operator developers to maintain curl-containers (though busybox has wget so that might be not an issue).

In general, I think all these "diagnostics forms" were meant to be run from the KUDO manager as some sort of tasks: not necessarily as plan-tasks but something similar (@gerred correct me if I'm wrong). Now, at least for the general diagnostics, we switched to the cli-based approach. We might do the same for extended diagnostics too. E.g. a HTTP request might be a kubectl exec busybox-pod wget ...

zen-dog · 2020-03-30T17:03:24Z

keps/0022-diagnostics-bundle.md

+An individual bundle resource is represented as an element in the list inside of the
+`diagnostics.bundle.resources` key. Resources ALWAYS have the following keys:
+
+- **description**: The human-readable name of the file.


Probably optional since the name will be enough most of the time.

zen-dog · 2020-03-31T07:53:40Z

keps/0022-diagnostics-bundle.md

+          command: # Can be string or array
+            - nslookup
+            - google.com
+          objectRef:


Could you also remove the objectRef?

zen-dog · 2020-03-31T08:13:23Z

keps/0022-diagnostics-bundle.md

+- **spec**: The attributes of a particular kind. This is different for every
+  kind.
+
+Also, specs may include an `objectRef`. It ALWAYS has the following keys:


This and subsequent mentionings of the objectRef are outdated since we're going with the selector?

keps/0022-diagnostics-bundle.md

porridge

LGTM, but please:

do mention sonobuoy in the prior art section, since you did investigate it.
replace the TODO at the bottom with some table (see other KEPs for an example)

Co-Authored-By: Aleksey Dukhovniy <adukhovniy@mesosphere.io> Co-Authored-By: Marcin Owsiany <mowsiany@D2iQ.com> Signed-off-by: Gerred Dillon <hello@gerred.org> Signed-off-by: Andreas Neumann <aneumann@mesosphere.com> Signed-off-by: Aleksey Dukhovniy <alex.dukhovniy@googlemail.com> Signed-off-by: Vasilii Emelin <vasilii_emelin@epam.com>

Co-Authored-By: Gerred Dillon <hello@gerred.org> Co-Authored-By: Aleksey Dukhovniy <alex.dukhovniy@googlemail.com> Co-Authored-By: Andreas Neumann <aneumann@mesosphere.com> Co-Authored-By: Marcin Owsiany <mowsiany@D2iQ.com> Signed-off-by: Vasilii Emelin <vasilii_emelin@epam.com>

mpereira · 2020-04-17T15:06:45Z

🎉

mpereira requested review from alenkacz, gerred, kensipe, nfnt and zen-dog as code owners January 24, 2020 16:35

mpereira mentioned this pull request Jan 24, 2020

[TRACKING] Support Bundles / Supportability / Metrics #1100

Closed

1 task

mpereira assigned gerred Jan 24, 2020

mpereira added the kind/kep Kudo Enhancement label Jan 24, 2020

gerred added this to the 0.11.0 milestone Feb 11, 2020

ANeumann82 reviewed Feb 11, 2020

View reviewed changes

keps/0022-diagnostics-bundle.md Show resolved Hide resolved

ANeumann82 reviewed Feb 11, 2020

View reviewed changes

keps/0022-diagnostics-bundle.md Outdated Show resolved Hide resolved

ANeumann82 reviewed Feb 11, 2020

View reviewed changes

keps/0022-diagnostics-bundle.md Outdated Show resolved Hide resolved

ANeumann82 reviewed Feb 11, 2020

View reviewed changes

keps/0022-diagnostics-bundle.md Outdated Show resolved Hide resolved

ANeumann82 reviewed Feb 11, 2020

View reviewed changes

ANeumann82 reviewed Feb 12, 2020

View reviewed changes

zen-dog reviewed Feb 12, 2020

View reviewed changes

zen-dog force-pushed the kep-22-diagnostics-bundle branch from 5b3d92c to 6a84fa8 Compare March 6, 2020 14:25

kensipe modified the milestones: 0.11.0, 0.12.0 Mar 10, 2020

kensipe self-assigned this Mar 10, 2020

zen-dog changed the title ~~Add initial KEP-22: Diagnostics Bundle.~~ KEP-22: Add initial design on diagnostics bundles Mar 10, 2020

zen-dog force-pushed the kep-22-diagnostics-bundle branch 2 times, most recently from cc60ca2 to 51bc070 Compare March 10, 2020 14:54

vemelin-epm self-assigned this Mar 10, 2020

porridge requested changes Mar 11, 2020

View reviewed changes

porridge reviewed Mar 12, 2020

View reviewed changes

keps/0022-diagnostics-bundle.md Outdated Show resolved Hide resolved

vemelin-epm force-pushed the kep-22-diagnostics-bundle branch from cf966fe to 0cbaf7b Compare March 25, 2020 13:34

vemelin-epm requested a review from porridge March 25, 2020 13:35

porridge requested changes Mar 26, 2020

View reviewed changes

porridge added the kep/22 Diagnostics bundles label Mar 26, 2020

zen-dog reviewed Mar 31, 2020

View reviewed changes

kensipe modified the milestones: 0.12.0, 0.13.0 Apr 9, 2020

vemelin-epm force-pushed the kep-22-diagnostics-bundle branch 2 times, most recently from 8685305 to d592a88 Compare April 14, 2020 08:40

vemelin-epm requested review from zen-dog and porridge April 14, 2020 08:42

porridge approved these changes Apr 14, 2020

View reviewed changes

vemelin-epm force-pushed the kep-22-diagnostics-bundle branch from c7e00bc to 55741cc Compare April 15, 2020 13:06

vemelin-epm force-pushed the kep-22-diagnostics-bundle branch from 55741cc to 447eccb Compare April 15, 2020 13:08

vemelin-epm requested a review from ANeumann82 April 15, 2020 13:30

gerred approved these changes Apr 17, 2020

View reviewed changes

vemelin-epm merged commit 5ca644e into master Apr 17, 2020

vemelin-epm added this to Done in KUDO Global via automation Apr 17, 2020

vemelin-epm deleted the kep-22-diagnostics-bundle branch April 17, 2020 14:10


		Also, may specs may include an `objectRef`. It ALWAYS has the following keys:

		- kind: The Kubernetes Kind referenced. For example, this may be a


		Checks:

		- Does the Kubernetes cluster have enough resources to install the operator

		By default, KUDO filters all resources (and custom resources) of values
		contained within the KUDO Instance's secrets. This is configurable with the

KEP-22: Add initial design on diagnostics bundles #1310

KEP-22: Add initial design on diagnostics bundles #1310

Conversation

mpereira commented Jan 24, 2020

mpereira commented Jan 24, 2020

mpereira commented Jan 24, 2020 • edited Loading

gerred commented Feb 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gerred commented Feb 11, 2020

runyontr commented Feb 12, 2020

Choose a reason for hiding this comment

zen-dog left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kensipe commented Mar 10, 2020

kensipe commented Mar 10, 2020

kensipe commented Mar 10, 2020

porridge left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vemelin-epm Mar 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

porridge left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zen-dog left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

porridge left a comment

Choose a reason for hiding this comment

mpereira commented Apr 17, 2020

mpereira commented Jan 24, 2020 •

edited

Loading

vemelin-epm Mar 18, 2020 •

edited

Loading