HTTPS Sender #602

iori-yja · 2019-03-07T14:51:30Z

Which problem is this PR solving?

Allow to send to collector via https, even when the collector's https sidecar is self-signed, key-continuity-based backend: HTTPS sender #595
Related: Allow secure communication between components jaeger#458

Short description of the changes

Added those below:

Optional Certificate Pinning
Optional Certificate verification disabling (used only with pinning)
Integration test over HTTPS

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

…with Pinning Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

jpkrohling · 2019-03-07T15:02:29Z

Looks like it failed the checkstyle:

[ant:checkstyle] [ERROR] /home/travis/build/jaegertracing/jaeger-client-java/jaeger-crossdock/src/main/java/io/jaegertracing/crossdock/JerseyServer.java:147: Line is longer than 120 characters (found 140). [LineLength]

https://travis-ci.org/jaegertracing/jaeger-client-java/jobs/503139012#L2534

Signed-off-by: Iori YONEJI <fivo.11235813@gmail.com>

iori-yja · 2019-03-07T17:51:40Z

Oh, that was too stupid. I believe it's fixed now.
Also, crossdock check is still broken. It turns out that the cause is Bad Gateway on call to jaeger-collector from its https-proxy, but it seems weird...

yurishkuro · 2019-03-08T05:49:44Z

Also, crossdock check is still broken. It turns out that the cause is Bad Gateway on call to jaeger-collector from its https-proxy, but it seems weird...

if you found this in the build logs, could you copy them here? Many PRs have been seeing crossdock failures recently, I suspect a common cause.

iori-yja · 2019-03-10T15:57:58Z

I've added an nginx proxy in the docker-compose.yml. This service's log shows no route to host(pasted below).
For debug purpose, I added host-port mapping (14443:14443) as in the current diff and saw Bad Gateway response from localhost:14443 with my browser (chromium, curl) on the host machine.

% docker-compose -f jaeger-crossdock/docker-compose.yml -f jaeger-crossdock/jaeger-docker-compose.yml logs jaeger-collector-https-proxy
Attaching to jaeger-crossdock_jaeger-collector-https-proxy_1
jaeger-collector-https-proxy_1  | 2019/03/10 11:55:40 [error] 6#6: *1 connect() failed (113: No route to host) while connecting to upstream, client: 192.168.64.8, server: jaeger-collector-https-proxy, request: "POST /api/traces?format=jaeger.thrift HTTP/1.1", upstream: "http://192.168.64.9:14268/api/traces?format=jaeger.thrift", host: "jaeger-collector-https-proxy:14443"
jaeger-collector-https-proxy_1  | 192.168.64.8 - - [10/Mar/2019:11:55:40 +0000] "POST /api/traces?format=jaeger.thrift HTTP/1.1" 502 157 "-" "okhttp/3.9.0" "-"

I don't think the network problem is caused by reporter misconfiguration (as I changed it) because it succeeded to establish a session to the proxy, and also I don't think collector's issue because plain http version (java-http) is able to report to it.

This failure happens very likely but merely it succeeds, if I remember correctly (I couldn't reproduce now).

iori-yja · 2019-03-10T16:17:37Z

I tried once again which was successful this time (no change).

% make crossdock-clean && make
(snip)
2019/03/10 15:57:22 HTTP: HEAD http://test_driver:8080/
HTTP: Service test_driver:8080 is up

All services are up after 15.10839437s!

Executing Matrix...

...................................................................................................................................

131/131 passed (0/131 skipped)

Tests passed!

% git diff
% docker-compose -f jaeger-crossdock/docker-compose.yml -f jaeger-crossdock/jaeger-docker-compose.yml logs jaeger-collector-https-proxy
Attaching to jaeger-crossdock_jaeger-collector-https-proxy_1
jaeger-collector-https-proxy_1  | 192.168.80.8 - - [10/Mar/2019:15:57:25 +0000] "POST /api/traces?format=jaeger.thrift HTTP/1.1" 202 0 "-" "okhttp/3.9.0" "-"

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

codecov · 2019-03-11T08:02:49Z

Codecov Report

Merging #602 into master will increase coverage by 0.12%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master     #602      +/-   ##
============================================
+ Coverage     89.53%   89.66%   +0.12%     
- Complexity      542      563      +21     
============================================
  Files            68       69       +1     
  Lines          1949     2070     +121     
  Branches        251      263      +12     
============================================
+ Hits           1745     1856     +111     
- Misses          129      133       +4     
- Partials         75       81       +6

Impacted Files	Coverage Δ	Complexity Δ
.../src/main/java/io/jaegertracing/Configuration.java	`93.33% <100.00%> (-2.18%)`	`44.00 <0.00> (ø)`
...egertracing/internal/reporters/RemoteReporter.java	`85.05% <100.00%> (+2.49%)`	`7.00 <0.00> (ø)`
...rtracing/internal/reporters/CompositeReporter.java	`71.42% <0.00%> (-28.58%)`	`6.00% <0.00%> (-1.00%)`
...gertracing/internal/reporters/LoggingReporter.java	`81.81% <0.00%> (-9.10%)`	`4.00% <0.00%> (-1.00%)`
...a/io/jaegertracing/internal/utils/RateLimiter.java	`94.73% <0.00%> (-5.27%)`	`5.00% <0.00%> (ø%)`
...ernal/baggage/RemoteBaggageRestrictionManager.java	`94.91% <0.00%> (-5.09%)`	`11.00% <0.00%> (+1.00%)`	⬇️
...n/java/io/jaegertracing/internal/JaegerTracer.java	`88.96% <0.00%> (-0.05%)`	`26.00% <0.00%> (ø%)`
...aegertracing/internal/propagation/BinaryCodec.java	`100.00% <0.00%> (ø)`	`17.00% <0.00%> (?%)`
...ain/java/io/jaegertracing/internal/JaegerSpan.java	`93.51% <0.00%> (+0.06%)`	`48.00% <0.00%> (+1.00%)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb0fa21...4e0a704. Read the comment docs.

* Make grpc reporter default and add retry Signed-off-by: Pavol Loffay <ploffay@redhat.com> * Polish Signed-off-by: Pavol Loffay <ploffay@redhat.com> * Fix port Signed-off-by: Pavol Loffay <ploffay@redhat.com> * Polish Signed-off-by: Pavol Loffay <ploffay@redhat.com> * Use higher retry Signed-off-by: Pavol Loffay <ploffay@redhat.com> * Increase retry to 100 Signed-off-by: Pavol Loffay <ploffay@redhat.com>

yurishkuro · 2019-03-11T09:43:43Z

jaeger-core/src/main/java/io/jaegertracing/Configuration.java

+   *  TLS certificates (in comma-separated BASE64 SHA256 Hash) for certificates pinning,
+   *  used in case of HTTPS communication to the endpoint.
+   */
+  public static final String JAEGER_TLS_CERTIFICATE_PINNING = JAEGER_PREFIX + "TLS_CERTIFICATE_PINNING";


Not thrilled with the name. Is it typical to pass certs via env vars? We've recently added many TLS options to Jaeger backend, and the certs are always passed via files.

I had three reasons to pass this via environment variable over file.

It should be good to be consistent with other reporter configurations most of them can be done via fromEnv() otherwise manipulated with Internal*.

This must be SHA256 Hash of Certificate or Public Key (https://square.github.io/okhttp/3.x/okhttp/okhttp3/CertificatePinner.html) so this is kept short.

In Lambda, one of the motivations for this, managing environment variables is easier than doing so for file. Environment variables are shown and settable plainly in the console, while files are to be uploaded directly or indirectly (S3) to the console as zipped format.

…TTPSender Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

… any trust anchor Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

Signed-off-by: Iori YONEJI <fivo.11235813@gmail.com>

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

yurishkuro · 2019-03-14T16:45:26Z

I feel like this is adding way too much code to the client.

yurishkuro · 2019-03-12T18:22:17Z

jaeger-crossdock/docker-compose.yml

+            - jaeger-collector
+
+    jaeger-collector-https-proxy:
+        image: nginx:alpine


it would be better to support TLS in the collector directly, rather than use nginx, since we already added similar functionality to gRPC:

--collector.grpc.tls Enable TLS --collector.grpc.tls.cert string Path to TLS certificate file --collector.grpc.tls.key string Path to TLS key file --collector.http-port int The HTTP port for the collector service (default 14268)

This is nice to have for jaeger, and I may add the option to jaeger soon, but this proxy is for client test for now. I will replace this after --collector.http.tls options are merged.

yurishkuro · 2019-03-14T16:38:51Z

jaeger-crossdock/https-proxy/Dockerfile

+ADD build/proxy.key /etc/nginx/proxy.key
+
+EXPOSE 8080 14443
+CMD ["sh", "-c", "while ! { wget --spider -S http://jaeger-collector:14269; }; do echo waiting...; sleep 1; done; nginx -g 'daemon off;'"]


comment would be useful, what is being queried on port 14269?

yurishkuro · 2019-03-14T16:39:26Z

jaeger-crossdock/rules.mk

+crossdock: gradle-compile crossdock-proxy-secret-gen crossdock-download-jaeger
+	docker-compose -f $(XDOCK_YAML) -f $(XDOCK_JAEGER_YAML) kill java-udp java-http java-https
+	docker-compose -f $(XDOCK_YAML) -f $(XDOCK_JAEGER_YAML) rm -f java-udp java-http java-https
+	docker-compose -f $(XDOCK_YAML) -f $(XDOCK_JAEGER_YAML) build java-udp java-http java-https


should this also include jaeger-collector-https-proxy ?

I would move these into a make var, to DRY

yurishkuro · 2019-03-14T16:42:15Z

jaeger-crossdock/https-proxy/authority.key

+MHcCAQEEIAA24MB8sxrLG1an0nG1DCH6J32iqrtborxFOjdqWNCmoAoGCCqGSM49
+AwEHoUQDQgAEAhYnw9zYU0G3VZ48nNlT5jAs096pX0zeHM/yxiJe+DS5Yj0EJXM/
+0A1Of2zqxbyJpaEIFcqTmTTyTXCE7I6B6Q==
+-----END EC PRIVATE KEY-----


are these being generated by jaeger-crossdock/https-proxy/gen.sh every time?

No, this is root certificate which is used to generate proxy's certificate by gen.sh. Hard-coding is ugly by the way, but sometimes test project includes.

The Public key pin is coded here:
https://github.com/jaegertracing/jaeger-client-java/pull/602/files#diff-9d280e9b01ffb0fc60647fe5d2a8286cR74

Could you please document somewhere how you generated these files?

OK, I will.

yurishkuro · 2019-03-14T16:43:14Z

jaeger-thrift/src/main/java/io/jaegertracing/thrift/internal/senders/HttpSender.java

@@ -66,6 +77,7 @@ public void send(Process process, List<Span> spans) throws SenderException {
    try {
      response = httpClient.newCall(request).execute();
    } catch (IOException e) {
+      e.printStackTrace();


don't we have that elsewhere?

That was my mistake not to remove debug code. I will delete.

yurishkuro · 2019-03-14T16:43:42Z

jaeger-thrift/src/main/java/io/jaegertracing/thrift/internal/senders/HttpSender.java

+    }
+
+    public Builder acceptSelfSigned() {
+      // This dangerous operation will only take effect if pinning is used.


method comment?

Not intended but should be.

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

jpkrohling · 2019-03-18T14:55:15Z

jaeger-crossdock/https-proxy/authority.crt

@@ -0,0 +1,10 @@
+-----BEGIN CERTIFICATE-----


What's the expiration date for this one?

Noted in gen.sh

(#602 (comment))

jpkrohling · 2019-03-18T14:56:01Z

jaeger-crossdock/https-proxy/authority.key

+MHcCAQEEIAA24MB8sxrLG1an0nG1DCH6J32iqrtborxFOjdqWNCmoAoGCCqGSM49
+AwEHoUQDQgAEAhYnw9zYU0G3VZ48nNlT5jAs096pX0zeHM/yxiJe+DS5Yj0EJXM/
+0A1Of2zqxbyJpaEIFcqTmTTyTXCE7I6B6Q==
+-----END EC PRIVATE KEY-----


Could you please document somewhere how you generated these files?

jpkrohling · 2019-03-18T14:56:46Z

jaeger-crossdock/https-proxy/proxy.template.conf

+    server_name ${SERVER};
+    location / {
+        root /usr/share/nginx/html/;
+				index index.html;


The indentation here seems a bit off

jaeger-crossdock/https-proxy/proxy.template.conf

jpkrohling · 2019-03-18T15:00:14Z

jaeger-crossdock/https-proxy/proxy.template.conf

+    listen 14443 ssl;
+    server_name ${SERVER};
+
+    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;


I know this is only for testing, but could we have a production-grade configuration here? Like, a secure set of ciphers and options like ssl_prefer_server_ciphers on. This way, we'd be testing with production settings, making sure that all involved components are up to modern standards.

For reference, this is what I used to have on a few servers of mine, which was "current" a couple of years ago:

server { listen [::]:443 ssl http2; listen 443 ssl http2; server_name acme.example.com; ssl_certificate /etc/pki/tls/certs/acme.example.com.bundle.cert; ssl_certificate_key /etc/pki/tls/private/acme.example.com.key; ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # probably needs an update to include TLSv1.3? ssl_prefer_server_ciphers on; ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH !EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"; add_header Strict-Transport-Security max-age=31536000; root /usr/share/nginx/html/acme.example.com; }

jpkrohling · 2019-03-18T15:05:06Z

jaeger-thrift/src/main/java/io/jaegertracing/thrift/internal/senders/HttpSender.java

+           * Instead, check the sha256 hash value by custom verifier. */
+          acceptSelfSigned(clientBuilder, hostname, pins);
+        }
+      } catch (Exception e) {


This deserves a log message, even if at debug level (but possibly at info or even warn, as this is called only once)

I afraid it isn't, because it is very similar to https://github.com/jaegertracing/jaeger-client-java/pull/602/files/a0607f48cd7370959b8f937a208fdc517a03b82f#diff-3330f8a6420c943a90a1b22b0abd0815R56. But it might be OK to fail earlier than build itself.

Not sure I understood what you mean. The code you linked throw an exception, the code here is swallowing one. Instead of swallowing, we should log it at some level (info in my opinion).

The try above the catch is needed because of URI to be constructed by the endpoint which is supplied by the configuration. If the endpoint is invalid, building HttpSender at few lines later would fail. But I agree with that it shouldn't swallow all kind of Exception, so I changed it to catch only URISyntaxException and throw it earlier.

jaeger-thrift/src/main/java/io/jaegertracing/thrift/internal/senders/HttpSender.java

jpkrohling · 2019-03-18T15:14:06Z

jaeger-thrift/src/main/java/io/jaegertracing/thrift/internal/senders/HttpSender.java

+                }
+              }
+              // TODO: For TOFU (trust on first use) usecase, print the chain
+              throw new CertificateException();


If there's anything wrong, it would be very hard to figure out what is wrong. I see the following possible error conditions:

Empty chain (bad configuration from the user's part)

No matches between the hostname and the subject CNs from the chain

The check failed

It would be good to have an exception thrown with the appropriate message, even if it's a bit more code.

CertificateException is thrown when the certificate couldn't be verified by the TrustManager. If another (like default) manager might succeed the verification even after this manager failed, and if it eventually turned to be successful, the connection must be sane.

To follow this (strange?) interface, I throw CertificateException whenever this manager fails to verify the connection.

But probably other exception could be thrown to fail in case of "The check failed" but CN matches including server's chain was empty.

But probably other exception could be thrown to fail in case of "The check failed" but CN matches including server's chain was empty.

This is good for trust-on-first-use use-case to print actual pins of the server, because calculating the pin by openssl command is a little bothering.

+1 for having different exceptions for the check failed case

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

pavolloffay · 2019-04-26T07:58:04Z

@iori-yja is this ready from your side? @jpkrohling could you please check if your review has been fixed?

jpkrohling

This PR is missing some tests, but the actual code looks sane to me.

Tests I'd like to see:

By default, HTTPS URL should not accept self-signed certs
HTTPS URL succeeds when the server sends a self-signed cert and acceptSelfSigned() is used
HTTP + acceptSelfSigned() is noop (or has no bad consequences)

jaeger-thrift/src/main/java/io/jaegertracing/thrift/internal/senders/HttpSender.java

iori-yja · 2019-06-15T22:47:25Z

Sorry, forgot to check; I will catch up this week

Edit: will add some tests

jaegertracing#602 (review)

jaegertracing#602 (review) Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

iori-yja · 2019-06-24T10:37:23Z

HTTPS URL succeeds when the server sends a self-signed cert and acceptSelfSigned() is used

java-https does this in crossdock tests.

By default, HTTPS URL should not accept self-signed certs

This is the difficult part. How can I set "must-fail" test in crossdock?

HTTP + acceptSelfSigned() is noop (or has no bad consequences)

setTLSCertificatePinning() test does this check, but the name may be misleading. I've changed it to sendPlainHttpWithCertificatePinning().

…hCertificatePinning Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

iori-yja added 7 commits March 4, 2019 17:20

add certificate pinner method in HttpSender

9ce0e90

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

add senderConfiguration option to specify the endpoint certificate

1d492c8

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

add unsafeNoopVerifier

c99e96f

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

change server certificate pinning interfaces

7a02c67

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

xdock test

59b1b0f

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

allow disabling certificate verification for integration test

9240b22

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

add test-purpose CA key and enforce "verification-disabling" feature …

81b429a

…with Pinning Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

iori-yja mentioned this pull request Mar 7, 2019

HTTPS sender #595

Closed

iori-yja added 2 commits March 8, 2019 01:15

fix crossdock checkstyle failure

264db82

Signed-off-by: Iori YONEJI <fivo.11235813@gmail.com>

re-allow to disable certificate verification

d5127ff

Signed-off-by: Iori YONEJI <fivo.11235813@gmail.com>

iori-yja force-pushed the tls-http-thrift branch from aad57ee to d5127ff Compare March 7, 2019 17:42

fix two simple typoes involving with CertPinning configuration

aec7e1e

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

yurishkuro reviewed Mar 11, 2019

View reviewed changes

iori-yja added 3 commits March 11, 2019 19:20

restrict and rename relaxation option of HTTPS cert verification in H…

5e9e6e1

…TTPSender Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

remove debug code with HTTPS from HTTPSender

cdd4c79

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

add hand-check pinning verifier for self-signed certification without…

9892e8d

… any trust anchor Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

yurishkuro mentioned this pull request Mar 11, 2019

Increase grpc retry to 1000 in xdock jaegertracing/jaeger#1419

Merged

crossdock fix (wait for nginx proxy that waits for collector)

9479bb3

Signed-off-by: Iori YONEJI <fivo.11235813@gmail.com>

iori-yja force-pushed the tls-http-thrift branch 5 times, most recently from ed82e0d to 175fa1d Compare March 13, 2019 03:56

iori-yja force-pushed the tls-http-thrift branch from 78aba28 to 9f0a0b2 Compare March 13, 2019 07:31

fix crossdock test avoiding ambiguous volume behavior

59342f1

Signed-off-by: Iori YONEJI <fivo.11235813@gmail.com>

iori-yja force-pushed the tls-http-thrift branch from 9f0a0b2 to 59342f1 Compare March 13, 2019 07:37

iori-yja added 2 commits March 13, 2019 16:56

Merge remote-tracking branch 'origin/master' into tls-http-thrift

74f8ffb

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

add test for HTTPSender's new feature (pinning)

da96565

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

iori-yja force-pushed the tls-http-thrift branch from 133d8db to da96565 Compare March 13, 2019 08:43

increase crossdock timeout

386b537

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

yurishkuro reviewed Mar 14, 2019

View reviewed changes

refine comments and remove debug print as review in jaegertracing#602

a0607f4

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

jpkrohling reviewed Mar 18, 2019

View reviewed changes

iori-yja added 4 commits March 20, 2019 23:24

add more careful error handling in HttpSender

bc856fd

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

print pins if pinning failed for self-signed server

8530278

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

add note for pre-generated, for-crossdock certificate and fix its conf

17b1e7b

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

Log SenderException in RemoteReporter, including Trust verification

b10ab02

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

iori-yja force-pushed the tls-http-thrift branch 2 times, most recently from a2daf6d to 1c3cb04 Compare March 21, 2019 11:10

add a test "testReporterWithSenderException" for RemoteReporter

f45ed78

Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

iori-yja force-pushed the tls-http-thrift branch from 1c3cb04 to f45ed78 Compare March 21, 2019 11:38

jpkrohling reviewed Apr 30, 2019

View reviewed changes

iori-yja added a commit to iori-yja/jaeger-client-java that referenced this pull request Jun 21, 2019

fix as review comments in pullrequestreview-232113179

5d01046

jaegertracing#602 (review)

fix as review comments in pullrequestreview-232113179

55ac341

jaegertracing#602 (review) Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

iori-yja force-pushed the tls-http-thrift branch from 5d01046 to 55ac341 Compare June 21, 2019 11:47

change a test case name setTLSCeritificatePinning -> sendPlainHttpWit…

4e0a704

…hCertificatePinning Signed-off-by: Iori Yoneji <yoneji_i@worksap.co.jp>

fmhwong mentioned this pull request Sep 25, 2019

Secure tracing data to Jaeger backend OpenLiberty/open-liberty#8897

Closed

Asisranjan mentioned this pull request Mar 12, 2021

Support of https sender - Jaeger client to collector #771

Closed

yurishkuro closed this in 8ba71a6 Jan 10, 2022

HTTPS Sender #602

HTTPS Sender #602

Conversation

iori-yja commented Mar 7, 2019

Which problem is this PR solving?

Short description of the changes

jpkrohling commented Mar 7, 2019 • edited

iori-yja commented Mar 7, 2019

yurishkuro commented Mar 8, 2019

iori-yja commented Mar 10, 2019

iori-yja commented Mar 10, 2019

codecov bot commented Mar 11, 2019 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yurishkuro commented Mar 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iori-yja Mar 20, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavolloffay commented Apr 26, 2019

jpkrohling left a comment • edited

Choose a reason for hiding this comment

iori-yja commented Jun 15, 2019 • edited

iori-yja commented Jun 24, 2019

jpkrohling commented Mar 7, 2019 •

edited

codecov bot commented Mar 11, 2019 •

edited

iori-yja Mar 20, 2019 •

edited

jpkrohling left a comment •

edited

iori-yja commented Jun 15, 2019 •

edited