The histogram aggregator plugin was added. #2387

vlamug · 2017-02-09T16:55:38Z

Required for all PRs:

CHANGELOG.md updated (we recommend not updating this until the PR has been approved by a maintainer)
Sign CLA (if not already signed)
README.md updated (if adding a new plugin)

The ability of collecting metrics for histogram was added.

sparrc · 2017-02-10T13:45:34Z

thank you for the contribution @vlamug

We also already have a PR for adding basic aggregator stats to telegraf: #2167. I will be working on adding quantiles either to this one, or to a separate plugin.

one major problem with merging your plugin is that it re-implements measurement filtering, which telegraf already does, and plugin authors should not be re-implementing this.

For these reasons I'm going to close this PR.

d-ulyanov · 2017-02-10T14:12:03Z

Hi, @sparrc!

it re-implements measurement filtering

Is there another possibility to do not apply aggregations for all metrics?
If yes - please, tell how to do it.
If not - then aggregator will be applied to all measurements. And your tool for monitoring will eat more CPU than more significant applications on the same server.

I will be working on adding quantiles

quantiles calculated on client is not the same as histograms: it precalculated on client - so you're not able to calculate cross-server quantiles at all.

sparrc · 2017-02-10T14:35:25Z

If yes - please, tell how to do it.

https://github.com/influxdata/telegraf/blob/master/docs/CONFIGURATION.md#measurement-filtering

quantiles calculated on client is not the same as histograms: it precalculated on client - so you're not able to calculate cross-server quantiles at all.

ah, sorry I thought that the "buckets" were quantiles. What is a "histogram" stat? I think you might be using a definition from a specific product/workflow, can you link to some documentation on what this is exactly?

d-ulyanov · 2017-02-10T14:58:32Z

@sparrc thanks for suggestion!

About quantiles. There are two ways how you can calculate it.

Calculate quantiles inside telegraf. You can aggregate metrics during some interval and write it to output as:
cpu usage_user=30 usage_user_quantile_99th=35
In such implementation you can't calculate quantile on your cluster. You can just calculate avg(cpu_usage_user) - but such value is bad from math point of view.
Calculate measurement distribution inside telegraf. In this case you define list of buckets (or ranges) and count hits on each bucket. For example, you can define following buckets for cpu usage:
0..10, 10..20, ..., 90..100. Such solution allows you to understand measurement distribution and it allows you to calculate any quantile including quantile for whole cluster, for example, using Prometheus.
This approach implemented in pull request.
https://prometheus.io/docs/concepts/metric_types/#histogram

Which modifications should be done to move forward with this pull request (except for filtering)?

sparrc · 2017-02-10T15:07:23Z

ah, I see.

Apologies for the misunderstanding, I'll reopen, please ping me when you've implemented it to use builtin filtering and I'll review.

sparrc · 2017-02-10T18:03:04Z

etc/telegraf.conf

+#   ## The example of config to aggregate histogram for all fields of specified metric.
+#   [[aggregators.histogram.config]]
+#     ## The set of buckets.
+#     buckets = [0.0, 15.6, 34.5, 49.1, 71.5, 80.5, 94.5, 100.0]


each bucket is an "upper inclusive bound", correct? you should document that

you're right. Sure

Done. Added to documentation.

sparrc · 2017-02-10T18:03:41Z

plugins/aggregators/histogram/README.md

+
+### Tags:
+
+All measurements have tag `le`. This tag has the border value of bucket.


what does le mean exactly? Also document that it's the "upper inclusive bound"

It is right border of bucket, which means, that value of metric is less than the value of this bucket.

For example, we have the value "9" of metric "usage_user" and we have the following buckets:
[0.0, 10.0, 30.0, 50.0, 80.0, 100.0]

Then the tag "le" will have the value "10" and this tag will be added for the field "usage_user".

So, we will have the following result in output like this:

... cpu,host=localhost,cpu=cpu1,le=10.0 usage_user_bucket=1 ...

It is right border of bucket, which means, that value of metric is less than the value of this bucket.

I believe it should actually be "less than or equal"? do you have a unit test to verify this?

I believe it should actually be "less than or equal"?

Yes, you are right, sorry for my wrong.

do you have a unit test to verify this?

Yes, unit test checks this case.

sparrc · 2017-02-10T18:06:29Z

plugins/aggregators/histogram/README.md

+
+```
+cpu_usage_system_bucket{cpu="cpu-total",host="local",le="0"} 0
+cpu_usage_system_bucket{cpu="cpu-total",host="local",le="10"} 24


please provide output in influx-format (you can get this easily using the --test flag)

also, does the prometheus output properly put the correct types on these?

prometheus works well - we already tested full integration: from metrics collection to visualisation in Grafana

this is not negotiable, every other plugin provides it in influx, so you need to do it the same

Ok, we see, we will provide the example in influx-format.

sparrc · 2017-02-10T18:14:46Z

plugins/aggregators/histogram/histogram.go

+
+		if value, ok := convert(value); ok {
+			index := sort.SearchFloat64s(buckets, value)
+			if index < len(agr.histogramCollection[field]) {


why is this check necessary?

This algorithm has been taken from golang prometheus clint. And it is needed to prevent the increase the bucket "+Inf" in two times. But we changed the algorithm a bit and this condition will be removed after our changes.

So it is not necessary here.

Additional checks removed. Done

sparrc · 2017-02-10T18:19:48Z

plugins/aggregators/histogram/histogram.go

+
+// Push returns histogram values for metrics
+func (h *HistogramAggregator) Push(acc telegraf.Accumulator) {
+	var isResetNeeded = false


what is this exactly? can you explain why this is necessary?

As you noticed, the method Reset() is empty. We did it, in order to be able to accumulate hits in buckets. For example:

If we reset cache in the method Reset() and the value "period" is short(30 second for example), then hits will not accumulate enough, i.e. the hits in buckets will be so small that we will get not quite correct distribution. Therefore the method Reset() is empty.

However, after some time, along with the empty method Reset() the value of hist will be more than the value uint64. So we decided to add the variable isResetNeeded for reseting hits cache.

In line 153 you can find the condition, which difenes the time of the resetting the cache. We use the value math.MaxInt64. It is quite suitable for a sufficient accumulation of hits in order to have correct distribution.

MaxInt64 seems like overkill to get a "correct distribution"

If you want this merged into the main repo you're going to have to make it reset when Reset() is called.

I'm not 100% sure I agree that it should be managing the int64 rollovers itself either, but I'll consider it if you really don't have a way of predicting when an int64 rollover might occur.

Thank for your observation.

We decided, that MaxInt64 is really overkill. Furthermore, our buckets have uint64 type:

type counts []uint64

Therefore, we can remove the reset cache and just accumulate hist in buckets.

So, we offer to remove the using the variable 'isResetNeeded' and the method 'resetCache()'. The method 'Reset()' will be empty.

What do you think about it?

yes, that works, although you should change the counts to be int64 instead of uint64, the reason being that influxdb won't accept numbers over int64 (I believe prometheus may accept numbers over this, but you will begin to get odd rounding behavior)

Ok, we will change from uint64 to int64.

sparrc · 2017-02-13T13:14:17Z

EDIT see below

You should also add a "Histogram" type to the telegraf metric definition here: https://github.com/influxdata/telegraf/blob/master/metric.go, and then add an "AddHistogram" function here: https://github.com/influxdata/telegraf/blob/master/agent/accumulator.go

this is all so that the prometheus_client exporter will expose these metrics as proper "histogram" metrics rather than untyped.

You'll also need to update the prometheus exporter here: https://github.com/influxdata/telegraf/blob/master/plugins/outputs/prometheus_client/prometheus_client.go#L131

sparrc · 2017-02-13T15:55:25Z

EDIT on the above comment, this might quite a bit more complicated than I realized, but I'd also like to hear from you guys who are presumably using these histograms with prometheus.

Do you run into problems with these metrics being "Untyped" rather than "Histogram" types? how do you deal with that and how does it change the workflow?

vlamug · 2017-02-14T08:38:24Z

You should also add a "Histogram" type to the telegraf metric definition here: https://github.com/influxdata/telegraf/blob/master/metric.go, and then add an "AddHistogram" function here: https://github.com/influxdata/telegraf/blob/master/agent/accumulator.go

this is all so that the prometheus_client exporter will expose these metrics as proper "histogram" metrics rather than untyped.

You'll also need to update the prometheus exporter here: https://github.com/influxdata/telegraf/blob/master/plugins/outputs/prometheus_client/prometheus_client.go#L131

@sparrc it doesn't make sense to add histogram type in telegraf - finally - its just counter with le field.
In Prometheus case it doesn’t make sense at all. Untyped and histogram - it’s just suggestions to users who want to see metrics by eyes. All of them are printed as comments as you can see in output.

Also, we think, that the sending of histogram type will confuse others, because they will think that histogram makes sense in outputs.

In addition, we've tried to implement aggregator without changes in telegraf core.

I'd also like to hear from you guys who are presumably using these histograms with prometheus.

As for us - we prefer histograms instead of precalculated quantiles in most cases.

Do you run into problems with these metrics being "Untyped" rather than "Histogram" types?

We have no problems with type “Untyped”. Prometheus doesn’t use this comments at all as I already mentioned.

how do you deal with that and how does it change the workflow?

We already use our fork with histograms - it works well. We’ve build histograms in Grafana based on collected metrics. For example, we made graphs for CPU usage on cluster. There are no changes in workflow.

vlamug · 2017-02-15T14:56:45Z

@sparrc we added all changes by code-review. Please, check it, when you have free time. Thanks.

d-ulyanov · 2017-02-17T09:26:16Z

Hi, @sparrc!
Thanks for including pull request to v.1.3.0. When are you planning the release?
Are we ready to describe changes in CHANGELOG.md?

vlamug · 2017-02-21T12:29:35Z

Hello, @sparrc , we updated the CHANGELOG.md file.

danielnelson

Looks great just a few things to make it fit style wise. There is also a bunch of grammar stuff in the README, but we can fix that up in a second pass. Also, I'd like to get a review from @goller to make sure the schema plays nice with Chronograf.

danielnelson · 2017-04-20T22:52:10Z

plugins/aggregators/histogram/README.md

+
+#### Explanation
+
+The field `metric_fields` is the list of metric parameters. For example, the metric `cpu` has the following parameters:


To avoid confusion we should use Telegraf terminology ("field" instead of "parameter"). There are several other places this occurs.

danielnelson · 2017-04-21T00:56:00Z

plugins/aggregators/histogram/histogram.go

+// sampleConfig is config sample of histogram aggregation plugin
+var sampleConfig = `
+  # # Configuration for aggregate histogram metrics
+  # [[aggregators.histogram]]


Don't comment sampleConfig, also remove this section header and comment above it, it gets included outside of the sample config. If you generate the config you will see what I mean.

danielnelson · 2017-04-21T01:03:10Z

plugins/aggregators/histogram/histogram.go

+// Add adds new hit to the buckets
+func (h *HistogramAggregator) Add(in telegraf.Metric) {
+	id := in.HashID()
+	agr, ok := h.cache[id]


h.cache will have an item for every series. To avoid this becoming large users should be directed to exclude any metrics not aggregated in the docs. I think we should check the measurement name first and return as well based on the configuration, that way it should remain fast even if filters are not setup correctly.

danielnelson · 2017-04-21T01:07:24Z

plugins/aggregators/histogram/histogram.go

+
+			for index, bucket := range buckets {
+				count += counts[index]
+				addFields(acc, aggregate, field, strconv.FormatFloat(bucket, 'f', 1, 64), count)


If the buckets are less than 0.1 apart we won't be able to emit them accurately. Perhaps we need a precision option?

It seems to me, that the having such option will be difficult to understand. Maybe we just increase the value of precision, for example to 3? Then we write this limitation in the docs.

I agree it would be confusing. What do you think about specifying the buckets as a list of strings, a little more annoying to type in but then we could report exactly as inputted.

But in any case, we will need to convert strings to float64. What if we will determine the precision dynamically? I mean, we will get the bucket and determine the number of digits after the comma. It will look something like this:

strconv.FormatFloat(bucket, 'f', determinePrecision(bucket), 64)

If we do the list of strings idea, you will need to keep both the string and float64 versions of the array around, and it would complicate the sorting of buckets. So, definitely some drawbacks there.

The determinePrecision idea is fine from a usage point of view, so long as it works.

Ok, I will try to implement the determinePrecision function. I will add changes soon.

danielnelson · 2017-04-21T01:38:45Z

plugins/aggregators/histogram/histogram.go

+}
+
+// checkAndGetBuckets checks the order of buckets and returns them.
+func (h *HistogramAggregator) getBuckets(metric string, field string) []float64 {


Rename function or update the comment.

danielnelson · 2017-04-24T22:33:24Z

plugins/aggregators/histogram/histogram.go

+	for key, val := range agr.tags {
+		tags[key] = val
+	}
+	tags[bucketTag] = bucketTagVal


Why do we need to make a copy here?

I cant remember what the problem was. But now it works correctly without any copying. I will remove this piece of code.

I just remembered! When we pass tags without copying, it is passed by reference. As result, all metrics have tag "le" with value "+inf". It was discovered by tests.
The debug output of tags map without copying:
map[tag_name:tag_value le:+Inf] map[tag_name:tag_value le:+Inf] map[tag_name:tag_value le:+Inf] map[le:+Inf tag_name:tag_value] map[tag_name:tag_value le:+Inf] map[tag_name:tag_value le:+Inf]

With coping:
map[tag_name:tag_value le:0.000] map[tag_name:tag_value le:10.000] map[tag_name:tag_value le:20.000] map[tag_name:tag_value le:30.000] map[tag_name:tag_value le:40.000] map[tag_name:tag_value le:+Inf]

danielnelson · 2017-04-24T23:01:05Z

plugins/aggregators/histogram/histogram.go

+			count := int64(0)
+
+			for index, bucket := range buckets {
+				count += counts[index]


This creates a cumulative histogram. We should mention this somewhere in the docs.

danielnelson · 2017-04-24T23:12:20Z

plugins/aggregators/histogram/histogram.go

+// Push returns histogram values for metrics
+func (h *HistogramAggregator) Push(acc telegraf.Accumulator) {
+	for _, aggregate := range h.cache {
+		for field, counts := range aggregate.histogramCollection {


We should push one metric per series.

So instead of this:

cpu,cpu=cpu1,host=localhost,le=0.0 usage_idle_bucket=0i 1486998330000000000 cpu,cpu=cpu1,host=localhost,le=0.0 usage_system_bucket=0i 1486998330000000000

There should just be one metric:

cpu,cpu=cpu1,host=localhost,le=0.0 usage_idle_bucket=0i,usage_system_bucket=0i 1486998330000000000

But the number of hits to buckets will be different for this fields "usage_system" and "usage_idle".

Not sure I understand, the two lines are equivalent and each field could have different values:

cpu,cpu=cpu1,host=localhost,le=0.0 usage_idle_bucket=5i,usage_system_bucket=42i 1486998330000000000

When the tagset is different then it should be separate:

cpu,cpu=cpu1,host=localhost,le=0.0 usage_idle_bucket=5i,usage_system_bucket=42i 1486998330000000000 cpu,cpu=cpu1,host=localhost,le=5.0 usage_idle_bucket=3i,usage_system_bucket=5i 1486998330000000000

But for this we need to perform an additional action: the grouping fields by value. I think it is overhead. Why do not you want to leave it at that?

The reasons for it are that it would reduce the size of the output and also it would match the rest of the metric creation in Telegraf so it will work as expected in regards to the metric buffers.

Ok, thanks. I will do this changes soon.

danielnelson · 2017-04-25T00:08:49Z

plugins/aggregators/histogram/README.md

+to the value of this tag. For example, let assume that we have the metric value 10 and the following buckets:
+[5, 10, 30, 70, 100]. Then the tag `le` will have the value 10, because the metrics value is passed into bucket with
+right border value `10`.
+


We should add some InfluxQL sample queries.

You want so that we add some InfluxQL queries, right? Do you suggest doing this for a better understating? If so, could you provide us any plugin or aggregator, where we can take an example of queries?

Actually, don't worry I'll add them in later on.

danielnelson · 2017-04-26T00:50:48Z

plugins/aggregators/histogram/README.md

+
+#### Description
+
+The histogram aggregator plugin aggregates values of specified metric\`s parameters. The metric is emitted every


Use metric's. Also could you wrap this file at 78 chars?

Not required

sebito91 · 2017-05-11T15:16:16Z

This would be amazing, thanks for putting in the effort to get it started!

vlamug · 2017-05-11T17:05:10Z

@danielnelson I added necessary changes. Please, check it. Also, we need to solve the problem about bucket decision. I am waiting you response.

Also, regarding failed checks: as I understand the problem is in environment of running checks. The error:
zookeeper: unrecognized service

Can anybody restart checks again after a while?

@sebito91 thank you)

danielnelson · 2017-05-11T19:08:23Z

Rebasing will fix the zookeeper issue.

…ts was deleted.

The refactoring of the "Add" method. The "metric" attribute of structure "metricHistogramCollection" was renamed to "name". The value of var "sampleConfig" was changed. Some changes were added to readme file.

Refactoring of the method getBuckets(). Using sortBuckets method to sort buckets, if it is needed.

Now the precision is fully used.

danielnelson · 2017-07-25T00:56:08Z

@vlamug Sorry, I had forgotten how it worked.

I found an issue in the Telegraf startup that could cause metrics collected on startup to not be sent to the aggregator. I made this PR to address it: #3050.

With that change applied, I still see some one thing I'm not expecting. With this config and output, it appears to me that some metrics are not grouping how we wanted:

[[aggregators.histogram]]
  namepass = ["test"]
  period = "20s"
  drop_original = false

  [[aggregators.histogram.config]]
    buckets = [0.0, 1.0, 1.5, 2.0]
    metric_name = "test"
    metric_fields = ["foo", "bar"]

test bar=2,foo=1 1500939196441962181
test bar=1,foo=1 1500939197383665400
test,le=0 bar_bucket=0i,foo_bucket=0i 1500939220000000000
test,le=1 foo_bucket=2i 1500939220000000000
test,le=1.5 foo_bucket=2i 1500939220000000000
test,le=2 bar_bucket=2i,foo_bucket=2i 1500939220000000000
test,le=+Inf bar_bucket=2i,foo_bucket=2i 1500939220000000000
test,le=1 bar_bucket=1i 1500939220000000000
test,le=1.5 bar_bucket=1i 1500939220000000000

Here is what I expected the output to be:

test bar=2,foo=1 1500939196441962181
test bar=1,foo=1 1500939197383665400
test,le=0 foo_bucket=0i,bar_bucket=0i 1500939220000000000
test,le=1 foo_bucket=2i,bar_bucket=1i 1500939220000000000
test,le=1.5 foo_bucket=2i,bar_bucket=1i  1500939220000000000
test,le=2 foo_bucket=2i,bar_bucket=2i 1500939220000000000
test,le=+Inf foo_bucket=2i,bar_bucket=2i 1500939220000000000

…ersByBuckets Histogram aggregator now groups metric fields by bucket

2nick · 2017-07-26T12:33:08Z

@danielnelson I've implemented grouping which works as described - could you please, check it?

danielnelson · 2017-07-28T00:33:21Z

Looks good, I think it's ready! Could you just remove the edits to CHANGELOG.md. Lately I've just been updating this file when I merge since it has so much change contention.

vlamug · 2017-07-28T08:41:14Z

@danielnelson thanks for your code-review. We removed the edits from CHANGELOG.md.

sparrc closed this Feb 10, 2017

sparrc reopened this Feb 10, 2017

121watts added the in progress label Feb 10, 2017

sparrc previously requested changes Feb 10, 2017

View reviewed changes

sparrc added this to the 1.3.0 milestone Feb 16, 2017

This was referenced Mar 29, 2017

Added pprof tool #2512

Merged

Added cpu_usage_total measurement in CPU input plugin #2594

Closed

danielnelson suggested changes Apr 26, 2017

View reviewed changes

danielnelson requested a review from goller April 26, 2017 00:53

danielnelson modified the milestones: 1.3.0, 1.4.0 Apr 27, 2017

nhaugo removed the in progress label Jun 13, 2017

vlamug force-pushed the aggregator-histogram branch from b198963 to 5a5fd6a Compare July 19, 2017 15:58

vlamug and others added 17 commits July 24, 2017 15:21

The histogram aggregator plugin was added.

5fcd278

The gofmt tool was applied.

9f9327f

The problem with test was fixed.

d24d011

Changed the config format of histogram.

53b63fa

The using uint64 instead int64. The checking of overflow of bucket hi…

232c432

…ts was deleted.

The go fmt tool was applied.

17ea04e

The tests were fixed.

79ea10f

Added the additional information about buckets to README.md.

30aa7f9

Added item to Features section in the file CHANGELOG.md.

2d94051

Changes by code-review.

af29572

The refactoring of the "Add" method. The "metric" attribute of structure "metricHistogramCollection" was renamed to "name". The value of var "sampleConfig" was changed. Some changes were added to readme file.

Changes by code-review. Part 2.

023f3f6

Refactoring of the method getBuckets(). Using sortBuckets method to sort buckets, if it is needed.

Using copying of tags in the method: addFields

ca40497

Some small changes in tests.

d69f245

The converting of bucket to string was fixed.

d902a12

Now the precision is fully used.

The grouping metric fields by value was added.

2873efc

The unit test were fixed.

f4e3867

The changelog file was changed.

36f356c

vlamug force-pushed the aggregator-histogram branch 2 times, most recently from 70250bf to 36f356c Compare July 24, 2017 13:13

Alexander Tunik and others added 2 commits July 26, 2017 15:12

Histogram aggregator now groups metric fields by bucket

e8f4476

Merge pull request #1 from vlamug/aggregator-history-groupFieldsCount…

abc2324

…ersByBuckets Histogram aggregator now groups metric fields by bucket

danielnelson approved these changes Jul 28, 2017

View reviewed changes

The info about histogram plugin was deleted from CHANGELOG file.

700c52b

danielnelson merged commit 9c0aadf into influxdata:master Jul 31, 2017

danielnelson mentioned this pull request Jul 31, 2017

"Histogram" statistics aggregator plugin #1662

Closed

maxunt pushed a commit that referenced this pull request Jun 26, 2018

Add histogram aggregator plugin (#2387)

265a558


		### Tags:

		All measurements have tag `le`. This tag has the border value of bucket.


		#### Explanation

		The field `metric_fields` is the list of metric parameters. For example, the metric `cpu` has the following parameters:


		#### Description

		The histogram aggregator plugin aggregates values of specified metric\`s parameters. The metric is emitted every

The histogram aggregator plugin was added. #2387

The histogram aggregator plugin was added. #2387

Conversation

vlamug commented Feb 9, 2017 • edited Loading

Required for all PRs:

sparrc commented Feb 10, 2017

d-ulyanov commented Feb 10, 2017 • edited Loading

sparrc commented Feb 10, 2017

d-ulyanov commented Feb 10, 2017

sparrc commented Feb 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d-ulyanov Feb 16, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sparrc commented Feb 13, 2017 • edited Loading

sparrc commented Feb 13, 2017

vlamug commented Feb 14, 2017

vlamug commented Feb 15, 2017

d-ulyanov commented Feb 17, 2017 • edited Loading

vlamug commented Feb 21, 2017

danielnelson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vlamug May 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sebito91 commented May 11, 2017

vlamug commented May 11, 2017 • edited Loading

danielnelson commented May 11, 2017

danielnelson commented Jul 25, 2017

2nick commented Jul 26, 2017

danielnelson commented Jul 28, 2017

vlamug commented Jul 28, 2017

vlamug commented Feb 9, 2017 •

edited

Loading

d-ulyanov commented Feb 10, 2017 •

edited

Loading

d-ulyanov Feb 16, 2017 •

edited

Loading

sparrc commented Feb 13, 2017 •

edited

Loading

d-ulyanov commented Feb 17, 2017 •

edited

Loading

vlamug May 11, 2017 •

edited

Loading

vlamug commented May 11, 2017 •

edited

Loading