Allow storing tags as object fields in ES - better kibana support #1018

pavolloffay · 2018-08-23T13:30:29Z

Resolves #906
Supersedes #980, #982

Store configured or all tags as object fields. For example: {"tag": {"http@method": "GET", "span@kind": "server"}}.
Binary tags are always stored as nested objects - there is no way how to reproduce the type - it would be always a string.
The default behaviour does not use tag fields. It's a subject to a change. By default we could store only standard tags as object field - no breaking change
with default index.mapping.total_fields.limit" it is possible to store ~940 unique span tag keys. If the limit is reached no new unique tag can be stored - ES throws an exception. It basically means span will not be stored in the current index
tag with float value 1.0 will be always returned as int 123 Stored number as interface is always a float olivere/elastic#891

Introduced 3 flags:

      --es.tag-as-fields                                      Store all span and process tags as object fields. If true .tag-file-path is ignored.
      --es.tag-file-path string                               Optional path to a file containing tag keys which will be stored as object fields. Each key should be on a separate line.
      --es.tag-key-de-dot-char string                         The character used to replace dots (".") in tag keys stored as object fields. (default "@")

Other changes

copied json model to plugin/storage/es/spanwriter/dbmodel

Todos after this is merged:

move Service struct from writer to dbmodel package
Use json/dbmodel for ES dependencies #1017

Adding screenshot from kibana. The search pattern is tag.span@kind:server AND process.tag.ip:10.33.144.223

codecov · 2018-08-23T13:39:06Z

Codecov Report

Merging #1018 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master   #1018    +/-   ##
=======================================
  Coverage     100%    100%            
=======================================
  Files         139     141     +2     
  Lines        6419    6604   +185     
=======================================
+ Hits         6419    6604   +185

Impacted Files	Coverage Δ
model/converter/thrift/jaeger/to_domain.go	`100% <ø> (ø)`	⬆️
plugin/storage/es/spanstore/dbmodel/from_domain.go	`100% <100%> (ø)`
plugin/storage/es/spanstore/dbmodel/to_domain.go	`100% <100%> (ø)`
.../storage/es/spanstore/dbmodel/process_hashtable.go	`100% <100%> (ø)`
plugin/storage/es/factory.go	`100% <100%> (ø)`	⬆️
plugin/storage/es/spanstore/writer.go	`100% <100%> (ø)`	⬆️
plugin/storage/es/options.go	`100% <100%> (ø)`	⬆️
plugin/storage/es/spanstore/reader.go	`100% <100%> (ø)`	⬆️
plugin/storage/es/spanstore/service_operation.go	`100% <100%> (ø)`	⬆️
cmd/ingester/app/consumer/offset/manager.go	`100% <0%> (ø)`	⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a67504...8a713a0. Read the comment docs.

pavolloffay · 2018-08-29T10:06:14Z

model/json/model.go

-	Tags        []KeyValue `json:"tags"`
+	ServiceName string                 `json:"serviceName"`
+	Tags        []KeyValue             `json:"tags"`
+	TagsMap     map[string]interface{} `json:"tagsMap,omitempty"`


Please comment if you know a better name.

Ideally the name used in the kibana query would be tags, but I guess then that may cause backward compatibility issues?
Another alternative would be to use tag instead of tagsMap - so when referenced it would be tag.orderId (for example).

Yes we cannot use tags as it is used for legacy tags (as nested objects).

+1 on tag.

pavolloffay · 2018-08-29T10:10:09Z

plugin/storage/es/spanstore/dbmodel/to_domain.go

+		dKey := strings.Replace(k, ":", ".", -1)
+		tag := model.KeyValue{Key: dKey}
+		// TODO v.(int/64/32) didn't work it is always a float
+		if pInt, err := strconv.ParseInt(fmt.Sprintf("%v", v), 10, 64); err == nil {


ES always returns float64 as a number - even if it was stored as int. This works but it's not that pretty.

pavolloffay · 2018-08-29T10:11:53Z

plugin/storage/es/spanstore/reader.go

 	}
+
+	// TODO we could query tagsMap only if the tag is configured
+	// but configuration can change over time


I mean what if previous deployment was storing tags as object fields. If the new deployment is not using tags as fields query would not find old tags.

pavolloffay · 2018-08-29T15:56:11Z

This is ready for review

objectiser · 2018-08-30T08:24:43Z

plugin/storage/es/options.go

@@ -37,6 +37,9 @@ const (
 	suffixBulkActions       = ".bulk.actions"
 	suffixBulkFlushInterval = ".bulk.flush-interval"
 	suffixIndexPrefix       = ".index-prefix"
+	suffixTagsFile          = ".tag-file-path"
+	suffixAllTagsAsObject   = ".tag-as-fields"
+	suffixTagDeDotChar      = ".tag-key-de-dot-char"


Not sure '-de-' part is required in the name? I think tag-key-dot-charis good enough.

This is usually called de dot https://www.elastic.co/blog/introducing-the-de_dot-filter

objectiser · 2018-08-30T08:33:28Z

plugin/storage/es/options.go

@@ -75,6 +78,7 @@ func NewOptions(primaryNamespace string, otherNamespaces ...string) *Options {
 				BulkWorkers:       1,
 				BulkActions:       1000,
 				BulkFlushInterval: time.Millisecond * 200,
+				TagDeDotChar:      "@",


What was the issue with using ':'?

From my ES past, I vaguely remember there being able to mark a field as not_analyzed so it was indexed but using the actual value without being processed. I see that this has been replaced by the concept of keyword in 5.x. Doesn't this avoid the issue of using '.' in the tag name?

What was the issue with using ':'?

It would have to be escaped. It's used in kibana query tag.key:foo. To avoid dots in field name we would have to set enabled:false on the field.

pavolloffay · 2018-08-30T13:06:23Z

I have published docker images in https://hub.docker.com/u/pavolloffay/ with tag PR1018. For example pavolloffay:jaeger-collector:PR1018. Or use docker run --net=host -e SPAN_STORAGE_TYPE=elasticsearch -e ES_TAG_AS_FIELDS=true pavolloffay/all-in-one:PR1018 to simply run it (it expects ES to be running on the host).

jpkrohling

Haven't finished yet, but wanted to share my review sooner.

jpkrohling · 2018-08-31T08:16:40Z

plugin/storage/es/spanstore/dbmodel/from_domain.go

+	if kvs == nil {
+		kvs = make([]KeyValue, 0)
+	}
+	return kvs, tagsMap


If keyValues is empty (Len=0), you'll end up with a kvs with Len=0 and a nil tagsMap. Wouldn't it make more sense to initialize the tagsMap on line 89?

Good question, it make sense if tagMap is used by default. By default it's not used and not present in the index.

jpkrohling · 2018-08-31T08:18:02Z

plugin/storage/es/spanstore/dbmodel/from_domain.go

@@ -0,0 +1,137 @@
+// Copyright (c) 2017 Uber Technologies, Inc.


Copyright :)

(as well as other files in this PR)

what is wrong with it? It's automatically handled by make fmt

s/2017 Uber Technologies, Inc/2018 The Jaeger Authors/

The updadeLicense script seems to be correct. Actually these files were copied (with changes) from domain package. Do you want to change the license?

If they are just copies, then no, keep the old copyright.

if with changes you can add 2018 The Jaeger Authors line above

jpkrohling · 2018-08-31T08:22:41Z

plugin/storage/es/spanstore/dbmodel/model.go

+// References and ProcessID, respectively.
+// When converting to ES model, ProcessID and Warnings should be omitted. Even if
+// included, ES with dynamic settings off will automatically ignore unneeded fields.
+type Span struct {


Is this struct specific to ES?

yes it's in es/dbmodel package. I think the comment is not accurate. I will fix it

jpkrohling · 2018-09-03T10:16:12Z

plugin/storage/es/spanstore/dbmodel/from_domain_test.go

+		var expectedSpan Span
+		require.NoError(t, json.Unmarshal(jsonStr, &expectedSpan))
+
+		testJSONEncoding(t, i, jsonStr, embeddedSpan)


Could you wrap each test in a t.Run() with an appropriate name? Like, "TestFromDomainEmbedProcess#fixture%d"

jpkrohling · 2018-09-03T10:21:04Z

plugin/storage/es/spanstore/dbmodel/to_domain.go

 }

-type toDomain struct{}
+// ToDomain is used to convert Span to model.Span
+type ToDomain interface {


Same question as before: is this specific to ES? Couldn't this be made at a higher level, to be used by other storage backends? Otherwise, what's the point of changing from a struct to an interface?

It is ES specific. Each storage define it's own model e.g. see C* package.

The biggest advantage is keeping the struct private and exposing only the interface with builder function. It makes it clear how it is constructed.

jpkrohling · 2018-09-03T10:22:11Z

plugin/storage/es/spanstore/dbmodel/to_domain.go

+	for k, v := range tagsMap {
+		dKey := strings.Replace(k, deDotChar, ".", -1)
+		tag := model.KeyValue{Key: dKey}
+		// TODO v.(int/64/32) didn't work it is always a float


What's the TODO here? Sounds more like a regular comment to me :)

jpkrohling · 2018-09-03T10:25:30Z

plugin/storage/es/spanstore/dbmodel/to_domain.go

+		if pInt, err := strconv.ParseInt(fmt.Sprintf("%v", v), 10, 64); err == nil {
+			tag.VType = model.Int64Type
+			tag.VInt64 = pInt
+		} else if val, ok := v.(float64); ok {


Isn't it more idiomatic in Go to use a switch/case with the type?

switch v.(type) { case float64: tag.VType = model.Float64Type tag.VFloat64 = val ... }

jpkrohling · 2018-09-03T10:27:11Z

plugin/storage/es/spanstore/dbmodel/to_domain.go

+		} else if val, ok := v.([]byte); ok {
+			tag.VType = model.BinaryType
+			tag.VBinary = val
+		}


In theory, it can also be something else. How about adding a log statement for that case?

That should not really happen. In this case it's better to return an error.

plugin/storage/es/spanstore/reader_test.go

plugin/storage/es/spanstore/writer_test.go

jpkrohling · 2018-09-03T10:40:52Z

plugin/storage/es/spanstore/writer_test.go

+	s := &model.Span{Tags: []model.KeyValue{{Key: "foo", VStr: "bar"}},
+		Process: &model.Process{Tags: []model.KeyValue{{Key: "bar", VStr: "baz"}}}}
+	for _, test := range testCases {
+		mSpan := test.writer.spanConverter.FromDomainEmbedProcess(s)


Wrap every iteration with its own t.Run() with an appropriate name. In case of a failure, it should be easy to identify what's the combination that failed.

jpkrohling · 2018-09-03T10:41:12Z

plugin/storage/integration/domain_trace_compare_test.go

@@ -34,11 +34,18 @@ func CompareSliceOfTraces(t *testing.T, expected []*model.Trace, actual []*model
 	}
 	if !assert.EqualValues(t, expected, actual) {
 		for _, err := range pretty.Diff(expected, actual) {
+			//fmt.Println("Actual")


Will this be removed before merging?

plugin/storage/integration/elasticsearch_test.go

pavolloffay · 2018-09-04T16:29:12Z

@Monnoroch @golonzovsky could you please review/run it?

yurishkuro · 2018-09-05T02:54:19Z

@pavolloffay as I understand it, this is backwards compatible?

Regarding the CLI flags, proposal for minor adjustment:

      --es.tags-as-fields.always                                      Store all span and process tags as object fields. If true .tag-file-path is ignored.
      --es.tags-as-fields.config-file string                          Optional path to a file containing tag keys which will be stored as object fields. Each key should be on a separate line.
      --es.tags-as-fields.dot-replacement string                      The character used to replace dots (".") in tag keys stored as object fields. (default "@")

This way if people are using config files instead of CLI then all the new options are in the same section.

@black-adder can you review this?

pavolloffay · 2018-09-05T07:28:13Z

It is backward compatible. The proposed flags look fine except I would rename .always to all.

yurishkuro · 2018-09-05T12:35:15Z

plugin/storage/es/spanstore/dbmodel/to_domain.go

+	// v.(int/64/32) does not work. It is always a float
+	if pInt, err := strconv.ParseInt(fmt.Sprintf("%v", v), 10, 64); err == nil {
+		tag.VType = model.Int64Type
+		tag.VInt64 = pInt


should use constructor functions from model instead of direct struct manipulation, e.g.

return model.Int64(k, pint)

yurishkuro · 2018-09-05T12:35:50Z

plugin/storage/es/spanstore/dbmodel/to_domain.go

+	return kvs, nil
+}
+
+func convertTagField(v interface{}, k string, deDotChar string) (*model.KeyValue, error) {


return by value, no need to cause heap allocations: (model.KeyValue, error)

yurishkuro · 2018-09-05T12:38:48Z

plugin/storage/es/spanstore/dbmodel/to_domain.go

-func SpanToDomain(span *json.Span) (*model.Span, error) {
-	return toDomain{}.spanToDomain(span)
+// NewToDomain creates ToDomain
+func NewToDomain(deDotChar string) ToDomain {


why do we need this struct public? Would it be enough to have a public function SpanToDomain()?

yurishkuro · 2018-09-05T12:41:19Z

plugin/storage/es/spanstore/reader.go

+	MetricsFactory          metrics.Factory
+	serviceOperationStorage *ServiceOperationStorage
+	IndexPrefix             string
+	TagDeDotChar            string


can we name this consistent with the flag, i.e. TagDotReplacement

plugin/storage/es/spanstore/reader.go

yurishkuro · 2018-09-05T12:51:22Z

plugin/storage/es/spanstore/dbmodel/model.go

+	OperationName string      `json:"operationName"`
+	References    []Reference `json:"references"`
+	StartTime     uint64      `json:"startTime"` // microseconds since Unix epoch
+	// Span adds a StartTimeMillis field to the standard JSON span.


rm this line

yurishkuro · 2018-09-05T12:53:31Z

plugin/storage/es/spanstore/dbmodel/model.go

+	Tags            []KeyValue             `json:"tags"`
+	Tag             map[string]interface{} `json:"tag,omitempty"`
+	Logs            []Log                  `json:"logs"`
+	Process         *Process               `json:"process,omitempty"`


any reason why this has to be a pointer and 'omitempty'? it's not optional here.

good catch, we can make it mandatory it was just copied from json package.

yurishkuro · 2018-09-05T12:56:02Z

plugin/storage/es/spanstore/dbmodel/model.go

+	StartTimeMillis uint64                 `json:"startTimeMillis"`
+	Duration        uint64                 `json:"duration"` // microseconds
+	Tags            []KeyValue             `json:"tags"`
+	Tag             map[string]interface{} `json:"tag,omitempty"`


nit: I recommend moving StartTimeMillis and Tag fields to the bottom of the struct and adding comments to Tag.

This affects how the json is stored to ES. I would like to keep tags and tag close to each other.

The order the fields are defined in the Go struct are indicative of how they are stored in ES? That's good to know! Would probably be nice to have a code comment with this remark.

yurishkuro · 2018-09-05T12:56:34Z

plugin/storage/es/spanstore/dbmodel/model.go

+type Process struct {
+	ServiceName string                 `json:"serviceName"`
+	Tags        []KeyValue             `json:"tags"`
+	Tag         map[string]interface{} `json:"tag,omitempty"`


yurishkuro · 2018-09-05T12:59:29Z

plugin/storage/es/spanstore/dbmodel/process_hashtable.go

+	"github.com/jaegertracing/jaeger/model"
+)
+
+type processHashtable struct {


when is this used? if we're embedding the Process in the Span we don't need to do deduping. Are we deduping at query time? Do we need to? I think domain-to-ui converter will dedupe anyway.

codecov-io · 2018-09-05T16:06:06Z

Codecov Report

Merging #1018 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master   #1018    +/-   ##
=======================================
  Coverage     100%    100%            
=======================================
  Files         139     140     +1     
  Lines        6451    6620   +169     
=======================================
+ Hits         6451    6620   +169

Impacted Files	Coverage Δ
model/converter/thrift/jaeger/to_domain.go	`100% <ø> (ø)`	⬆️
plugin/storage/es/spanstore/dbmodel/from_domain.go	`100% <100%> (ø)`
plugin/storage/es/spanstore/dbmodel/to_domain.go	`100% <100%> (ø)`
plugin/storage/es/factory.go	`100% <100%> (ø)`	⬆️
plugin/storage/es/spanstore/writer.go	`100% <100%> (ø)`	⬆️
plugin/storage/es/options.go	`100% <100%> (ø)`	⬆️
plugin/storage/es/spanstore/reader.go	`100% <100%> (ø)`	⬆️
plugin/storage/es/spanstore/service_operation.go	`100% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 774c01b...0ca2187. Read the comment docs.

pavolloffay · 2018-09-06T14:04:25Z

can somebody re-review/approve?

kacper-jackiewicz · 2018-09-07T14:35:13Z

@pavolloffay
I am just testing Jaeger with your new images. Everything seems to work fine on Kibana side(searches/visualizations). Just one remark on Jaeger-Query side. Searching for tags works only if we use new format so for ex. http@method="GET", but the tags are shown with old dotted format, which might be misleading for users.

pavolloffay · 2018-09-07T14:43:36Z

@kacper-jackiewicz thanks for running it. It's a bug query side should use the original key format.

Signed-off-by: Pavol Loffay <ploffay@redhat.com>

pavolloffay · 2018-09-07T16:09:56Z

Can somebody re-review so it can be merged?

cc) @jpkrohling @yurishkuro @black-adder

jpkrohling

LGTM and I think it's good to go for a RC.

jpkrohling · 2018-09-11T09:43:31Z

plugin/storage/es/factory.go

+			return nil, err
+		}
+	}
+	p := esSpanStore.SpanWriterParams{Client: f.primaryClient,


No need to change IMO, but I'm wondering why did you assign it to a var instead of passing it directly to the NewSpanWriter like you did with the other calls.

jpkrohling · 2018-09-11T09:51:35Z

plugin/storage/es/spanstore/dbmodel/model.go

+	StartTimeMillis uint64                 `json:"startTimeMillis"`
+	Duration        uint64                 `json:"duration"` // microseconds
+	Tags            []KeyValue             `json:"tags"`
+	Tag             map[string]interface{} `json:"tag,omitempty"`


The order the fields are defined in the Go struct are indicative of how they are stored in ES? That's good to know! Would probably be nice to have a code comment with this remark.

yurishkuro · 2018-09-11T14:44:55Z

I have published docker images in https://hub.docker.com/u/pavolloffay/ with tag PR1018.

@pavolloffay can you ask someone to test with those images? Ideally existing users, as well as those who asked for the new functionality

masteinhauser · 2018-09-12T02:06:58Z

I have published docker images in https://hub.docker.com/u/pavolloffay/ with tag PR1018.

@pavolloffay can you ask someone to test with those images? Ideally existing users, as well as those who asked for the new functionality

Hi, how can I help? We have been using the new Elasticsearch backend for ~100 days now with a fairly significant amount of span volume and lots of additional tags being added. This feature is incredibly relevant for us.

Signed-off-by: Pavol Loffay <ploffay@redhat.com>

pavolloffay · 2018-09-12T12:49:47Z

@masteinhauser or anybody else could you please test this PR?

Images with this functionality are available in pavolloffay/ with tag PR1018 https://hub.docker.com/u/pavolloffay/. Here are binaries https://drive.google.com/open?id=15jujV5Y-lEXaXAgsQgWh2Y-urxrgenqFhttps://drive.google.com/open?id=15jujV5Y-lEXaXAgsQgWh2Y-urxrgenqF I could not upload binaries here due to the size limit.

Use --es.tags-as-fields.all flag to enable this feature.

Signed-off-by: Pavol Loffay <ploffay@redhat.com>

masteinhauser · 2018-09-15T15:23:46Z

I finally had some time to roll this out and test it this morning. Everything appears to be working as expected, once I actually set the correct environment variables for the collector. No other config changes appeared to be needed, although I deployed agent, collector, and query based on the images made available here https://hub.docker.com/u/pavolloffay/

This of course only works for newly collected spans, which is perfectly acceptable for us. We'll likely be maintaining this image and deployment in production until the next Jaeger release is made with this available.

Incredibly useful feature for our developers. Thank you for the work on getting this done!

pavolloffay · 2018-09-18T09:45:20Z

@masteinhauser thanks for trying it out! I guess we can merge it and release it.

yurishkuro · 2018-10-08T19:19:04Z

plugin/storage/es/factory.go

+		Client:            f.primaryClient,
+		Logger:            f.logger,
+		MetricsFactory:    f.metricsFactory,
+		MaxLookback:       cfg.GetMaxSpanAge(),


@pavolloffay why did we rename it? It's confusing that config params do not match the storage params.

I don't think we renamed anything in this PR

The flag name is --es.max-span-age and configuration cfg.GetMaxSpanAge(). MaxLookback is only used in reader maybe also writer.

I think we should change everything to maxSpanAge

I can change it, there are also other refactoring opportunities

pavolloffay requested review from black-adder, jpkrohling, vprithvi and yurishkuro as code owners August 23, 2018 13:30

ghost assigned pavolloffay Aug 23, 2018

ghost added the review label Aug 23, 2018

pavolloffay changed the title ~~WIP Es schema object disable raw~~ WIP: Change ES tag schema to support tag search in Kibana - final Aug 23, 2018

pavolloffay force-pushed the es-schema-object-disable-raw branch 4 times, most recently from 6bb7f85 to 524b0a6 Compare August 29, 2018 09:20

pavolloffay added the storage/elasticsearch label Aug 29, 2018

pavolloffay changed the title ~~WIP: Change ES tag schema to support tag search in Kibana - final~~ Allow storing tags as object fields in ES - better kibana support Aug 29, 2018

pavolloffay commented Aug 29, 2018

View reviewed changes

pavolloffay mentioned this pull request Aug 29, 2018

Span format is not well suited for ES #906

Closed

objectiser reviewed Aug 30, 2018

View reviewed changes

jpkrohling reviewed Aug 31, 2018

View reviewed changes

jpkrohling reviewed Sep 3, 2018

View reviewed changes

pavolloffay force-pushed the es-schema-object-disable-raw branch from a2be925 to ab5fdc2 Compare September 4, 2018 09:09

pavolloffay mentioned this pull request Sep 4, 2018

Stored number as interface is always a float olivere/elastic#891

Closed

yurishkuro reviewed Sep 5, 2018

View reviewed changes

pavolloffay force-pushed the es-schema-object-disable-raw branch 2 times, most recently from 2777b56 to a422014 Compare September 6, 2018 07:40

pavolloffay added 3 commits September 7, 2018 18:04

Allow stoting tags as object fields for better kibana support

97fae92

Signed-off-by: Pavol Loffay <ploffay@redhat.com>

Replace dot in query

3925950

Signed-off-by: Pavol Loffay <ploffay@redhat.com>

Fix CI

d777e97

Signed-off-by: Pavol Loffay <ploffay@redhat.com>

pavolloffay force-pushed the es-schema-object-disable-raw branch from 5f34e00 to d777e97 Compare September 7, 2018 16:09

jpkrohling approved these changes Sep 11, 2018

View reviewed changes

Return immediately

69a554a

Signed-off-by: Pavol Loffay <ploffay@redhat.com>

Add comment

0ca2187

Signed-off-by: Pavol Loffay <ploffay@redhat.com>

pavolloffay merged commit 3fd0677 into jaegertracing:master Sep 18, 2018

ghost removed the review label Sep 18, 2018

pavolloffay mentioned this pull request Sep 19, 2018

Parse ES tag field jaegertracing/spark-dependencies#42

Closed

yurishkuro reviewed Oct 8, 2018

View reviewed changes

pavolloffay mentioned this pull request Oct 11, 2018

Refactor es storage #1111

Merged

ScottKaye mentioned this pull request Dec 4, 2018

When reading numeric strings from an Elasticsearch store with --es.tags-as-fields.all, the string is parsed as a number #1236

Closed

mehta-ankit mentioned this pull request Apr 25, 2022

ES_TAGS_AS_FIELDS_ALL gives an Array instead of Dictionary. jaegertracing/helm-charts#360

Closed

		@@ -0,0 +1,137 @@
		// Copyright (c) 2017 Uber Technologies, Inc.

Allow storing tags as object fields in ES - better kibana support #1018

Allow storing tags as object fields in ES - better kibana support #1018

Conversation

pavolloffay commented Aug 23, 2018 • edited Loading

codecov bot commented Aug 23, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavolloffay commented Aug 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavolloffay commented Aug 30, 2018

jpkrohling left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavolloffay commented Sep 4, 2018

yurishkuro commented Sep 5, 2018

pavolloffay commented Sep 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Sep 5, 2018 • edited by codecov bot Loading

Codecov Report

pavolloffay commented Sep 6, 2018

kacper-jackiewicz commented Sep 7, 2018

pavolloffay commented Sep 7, 2018

pavolloffay commented Sep 7, 2018

jpkrohling left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yurishkuro commented Sep 11, 2018

masteinhauser commented Sep 12, 2018

pavolloffay commented Sep 12, 2018 • edited Loading

masteinhauser commented Sep 15, 2018

pavolloffay commented Sep 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavolloffay commented Aug 23, 2018 •

edited

Loading

codecov bot commented Aug 23, 2018 •

edited

Loading

pavolloffay commented Sep 5, 2018 •

edited

Loading

codecov-io commented Sep 5, 2018 •

edited by codecov bot

Loading

pavolloffay commented Sep 12, 2018 •

edited

Loading