Migrates AWS Elasticsearch Storage to armeria/jackson #135

codefromthecrypt · 2019-07-20T01:09:07Z

Obviates #134 which I've lost write access to :P

codefromthecrypt · 2019-07-20T01:26:53Z

self tracing is a little bit messed up, but I don't think that's related to this. I think main thing was sorting out the contents of the jar.

While this works, we shouldn't merge until we are on a final version. There are a few dents to knock out upstream such as logging the health check. Plus we shouldn't release zipkin 2.16 until the next armeria release I think, as that could cleanup some code here also.

codefromthecrypt · 2019-07-20T01:27:34Z

by self-tracing messed up I mean that there are no child spans of the "accept-spans" call. There should be a bulk http POST request

codefromthecrypt · 2019-07-20T01:28:52Z

autoconfigure/pom.xml

@@ -72,8 +72,15 @@
        <configuration>
          <!-- https://github.com/spring-projects/spring-boot/issues/3426 transitive exclude doesn't work -->
          <excludeGroupIds>
-            io.zipkin.zipkin2,io.zipkin.reporter2,org.springframework.boot,org.springframework,com.fasterxml.jackson.core,com.google.auto.value,com.google.guava,com.squareup.moshi,com.squareup.okio,com.squareup.okhttp
+            com.github.openzipkin.zipkin,io.zipkin.reporter2,org.springframework.boot,org.springframework,com.fasterxml.jackson.core,com.google.auto.value,com.google.guava,org.reactivestreams,com.google.code.findbugs,javax.annotation,org.slf4j,io.netty,io.micrometer,org.hdrhistogram,org.latencyutils


this first entry needs to change back to io.zipkin.zipkin2 before merge. basically a global replace back needs ot happen

...rc/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/ElasticsearchDomainEndpoint.java

...h-aws/src/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/AWSSignatureVersion4.java

anuraaga · 2019-07-20T07:09:37Z

...h-aws/src/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/AWSSignatureVersion4.java

+
+  @Override public HttpResponse execute(ClientRequestContext ctx, HttpRequest req) {
+    return HttpResponse.from(
+        req.aggregateWithPooledObjects(ctx.eventLoop(), ctx.alloc())


I think if there are issues with self-tracing, it's because we need to do our contextAwareEventLoop trick here.

haven't handled this one yet

...h-aws/src/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/AWSSignatureVersion4.java

codefromthecrypt · 2019-07-29T04:20:41Z

@anuraaga I noticed that we made some utilities for moshi which aren't relevant when using jackson. This helped cleanup the code a good bit I think. Thanks for the other feedback as well. PTAL

codefromthecrypt · 2019-07-29T04:21:09Z

...csearch-aws/src/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/AWSCredentials.java

@@ -13,18 +13,16 @@
 */
 package zipkin.autoconfigure.storage.elasticsearch.aws;

-import zipkin2.internal.Nullable;


I got rid of all internal type usage in this PR

codefromthecrypt · 2019-07-29T04:21:48Z

...rc/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/ElasticsearchDomainEndpoint.java

-            .url(baseUrl.newBuilder("2015-01-01/es/domain").addPathSegment(domain).build())
-            .tag("get-es-domain")
-            .build();
+        AggregatedHttpRequest.of(HttpMethod.GET, "/2015-01-01/es/domain/" + domain);


TODO: name this "get-es-domain" like we did upstream

Maybe not worth it - IIUC, this is always going to be client-initiated and not sampled.

codefromthecrypt · 2019-07-29T04:22:59Z

...rc/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/ElasticsearchDomainEndpoint.java

+    try {
+      AggregatedHttpResponse res = client.execute(describeElasticsearchDomain).aggregate().join();
+      status = res.status();
+      // As the domain endpoint is read only once per startup. We don't worry about pooling etc.


@anuraaga let me know if this is true or not

codefromthecrypt · 2019-07-29T04:27:22Z

ps you have karma so feel free to update the branch. I have to look at other things a bit

...h-aws/src/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/AWSSignatureVersion4.java

anuraaga · 2019-07-29T04:30:22Z

...rc/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/ElasticsearchDomainEndpoint.java

-            .url(baseUrl.newBuilder("2015-01-01/es/domain").addPathSegment(domain).build())
-            .tag("get-es-domain")
-            .build();
+        AggregatedHttpRequest.of(HttpMethod.GET, "/2015-01-01/es/domain/" + domain);


Maybe not worth it - IIUC, this is always going to be client-initiated and not sampled.

anuraaga · 2019-07-29T04:32:00Z

...rc/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/ElasticsearchDomainEndpoint.java

+    try {
+      AggregatedHttpResponse res = client.execute(describeElasticsearchDomain).aggregate().join();
+      status = res.status();
+      // As the domain endpoint is read only once per startup. We don't worry about pooling etc.


...rc/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/ElasticsearchDomainEndpoint.java

anuraaga · 2019-07-29T04:35:44Z

...s/src/test/java/zipkin/autoconfigure/storage/elasticsearch/aws/AWSSignatureVersion4Test.java

+    Endpoint endpoint = Endpoint.of(
+        "search-zipkin-2rlyh66ibw43ftlk4342ceeewu.ap-southeast-1.es.amazonaws.com",
+        server.httpPort())
+        .withIpAddr("127.0.0.1");


codefromthecrypt · 2019-07-29T05:12:00Z

Maybe not worth it - IIUC, this is always going to be client-initiated and not sampled.

actually I think on first request it would be server initiated.. we can check.

codefromthecrypt · 2019-07-29T07:14:47Z

oops..

2019-07-29 15:12:42.278  WARN 78315 --- [-worker-nio-4-2] .a.c.b.RequestContextCurrentTraceContext : Attempted to propagate trace context, but no request context available. Did you forget to use RequestContext.contextAwareExecutor() or RequestContext.makeContextAware()?

com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext$LogRequestContextWarningOnce$NoRequestContextException: null
	at com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext$LogRequestContextWarningOnce$ClassLoaderHack.<clinit>(RequestContextCurrentTraceContext.java:266) ~[armeria-brave-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext$LogRequestContextWarningOnce.get(RequestContextCurrentTraceContext.java:254) ~[armeria-brave-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext$LogRequestContextWarningOnce.get(RequestContextCurrentTraceContext.java:247) ~[armeria-brave-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.RequestContext.mapCurrent(RequestContext.java:108) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext.getRequestContextOrWarnOnce(RequestContextCurrentTraceContext.java:232) ~[armeria-brave-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.brave.RequestContextCurrentTraceContext.get(RequestContextCurrentTraceContext.java:143) ~[armeria-brave-0.89.0.jar!/:?]
	at brave.Tracer.nextSpan(Tracer.java:472) ~[brave-5.6.9.jar!/:?]
	at brave.http.HttpClientHandler.nextSpan(HttpClientHandler.java:127) ~[brave-instrumentation-http-5.6.9.jar!/:?]
	at brave.http.HttpClientHandler.handleSend(HttpClientHandler.java:89) ~[brave-instrumentation-http-5.6.9.jar!/:?]
	at com.linecorp.armeria.client.brave.BraveClient.execute(BraveClient.java:121) ~[armeria-brave-0.89.0.jar!/:?]
	at com.linecorp.armeria.client.brave.BraveClient.execute(BraveClient.java:56) ~[armeria-brave-0.89.0.jar!/:?]
	at zipkin.autoconfigure.storage.elasticsearch.aws.AWSSignatureVersion4.lambda$execute$2(AWSSignatureVersion4.java:98) ~[elasticsearch-aws.jar!/:?]
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
	at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:98) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:363) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.notifySubscriberOfCloseEvent(FixedStreamMessage.java:126) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.EmptyFixedStreamMessage.doRequest(EmptyFixedStreamMessage.java:32) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.request(FixedStreamMessage.java:88) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$SubscriptionImpl.request(AbstractStreamMessage.java:309) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.HttpMessageAggregator.onSubscribe(HttpMessageAggregator.java:55) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.lambda$subscribe$1(FixedStreamMessage.java:117) ~[armeria-0.89.0.jar!/:?]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:515) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]

codefromthecrypt · 2019-07-30T15:31:15Z

stil didn't work but I can't even read the screen anymore. good time to stop. thanks for chasing up

codefromthecrypt · 2019-07-30T15:34:09Z

actually it did work, just it got an exception..
com.linecorp.armeria.client.endpoint.EndpointGroupException: com.linecorp.armeria.client.endpoint.dns.DnsAddressEndpointGroup@686c4360 is empty

I need to move the other fix here about the slacker resolution :P

codefromthecrypt · 2019-07-30T15:38:37Z

finally!

ok this works, so we can now do cleanups etc. as long as we super-verify that this continues to work I'm also ok with re-enabling the health check stuff. thx again for the epic support @anuraaga

During openzipkin/zipkin-aws#135 we found that Armeria will add a second `host` header even if one is already set in the request headers before sending them. This can break behavior and I suppose must break some HTTP spec too. I wasn't sure whether to respect or ignore the user's setting, but with this change, if a user themselves set an authority and host to different values, we'll leave the host as is. While this is a bad user, I guess we should trust they did it on purpose. I had to write the unit test in Jetty since during the conversion from HTTP/1 headers to Armeria headers, the duplicate hosts got normalized.

codefromthecrypt · 2019-08-02T10:40:59Z

./mvnw clean install -DskipTests
mv ./autoconfigure/storage-elasticsearch-aws/target/zipkin-autoconfigure-storage-elasticsearch-aws-0.17.3-SNAPSHOT-module.jar elasticsearch-aws.jar
TAG=master-SNAPSHOT
curl -sSL https://jitpack.io/com/github/openzipkin/zipkin/zipkin-server/${TAG}/zipkin-server-${TAG}-exec.jar > zipkin.jar
SELF_TRACING_ENABLED=true ES_AWS_DOMAIN=zipkin-ha ES_HTTP_LOGGING=BODY STORAGE_TYPE=elasticsearch    java     -Dloader.path='elasticsearch-aws.jar,elasticsearch-aws.jar!/lib'     -Dspring.profiles.active=elasticsearch-aws     -cp zipkin.jar     org.springframework.boot.loader.PropertiesLauncher

codefromthecrypt · 2019-08-09T05:29:24Z

running wrk and trying to do health check I get 503..

 2019-08-09 13:28:43 ⌚  MacBook-Pro-7 in ~/oss/zipkin-aws
± |elasticsearch-armeria {1} ?:4 ✗| → curl -s localhost:9411/actuator/prometheus|grep .0
503 Service Unavailable

codefromthecrypt · 2019-08-09T05:30:20Z

sorry this is health check and also prometheus. I'll try to figure out what's up..

codefromthecrypt · 2019-08-09T05:33:47Z

with logging on, you can see the health request go back and forth, but the health check itself is impatient I guess?

request/response for domain endpoint

2019-08-09 13:32:35.579  INFO 9762 --- [-worker-nio-2-2] c.l.a.c.l.LoggingClient                  : Request: {startTime=2019-08-09T05:32:35.578Z(1565328755578079), length=0B, duration=754µs(754310ns), scheme=none+h1, headers=[:method=GET, :path=/2015-01-01/es/domain/zipkin, :authority=es.ap-southeast-1.amazonaws.com, :scheme=https, x-amz-date=20190809T053235Z, authorization=AWS4-HMAC-SHA256 Credential=AKIA4IPHLCEKIQATBOUN/20190809/ap-southeast-1/es/aws4_request, SignedHeaders=host;x-amz-date, Signature=1558887a5267a800b3bdfbb9790842d4b77c51ac8624e369eb7cae49063ac446, b3=6d9dda115a2c437b-61db88a71f38f649-0-6d9dda115a2c437b, accept-encoding=gzip,deflate, user-agent=armeria/unknown]}
2019-08-09 13:32:35.741  INFO 9762 --- [-worker-nio-2-2] c.l.a.c.l.LoggingClient                  : Response: {startTime=2019-08-09T05:32:35.740Z(1565328755740153), length=1540B, duration=917µs(917564ns), headers=[:status=200, x-amzn-requestid=0d8ea2ae-ba67-11e9-9296-479f22947f13, content-type=application/json, content-length=1540, date=Fri, 09 Aug 2019 05:32:18 GMT], content={"DomainStatus":{"ARN":"arn:aws:es:ap-southeast-1:842835628308:domain/zipkin","AccessPolicies":"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"es:*\",\"Resource\":\"arn:aws:es:ap-southeast-1:842835628308:domain/zipkin/*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":\"36.2.165.225\"}}}]}","AdvancedOptions":{"rest.action.multi.allow_explicit_index":"true"},"CognitoOptions":{"Enabled":false,"IdentityPoolId":null,"RoleArn":null,"UserPoolId":null},"Created":true,"Deleted":false,"DomainId":"842835628308/zipkin","DomainName":"zipkin","EBSOptions":{"EBSEnabled":true,"Iops":null,"VolumeSize":10,"VolumeType":"gp2"},"ElasticsearchClusterConfig":{"DedicatedMasterCount":null,"DedicatedMasterEnabled":false,"DedicatedMasterType":null,"InstanceCount":1,"InstanceType":"t2.small.elasticsearch","ZoneAwarenessConfig":null,"ZoneAwarenessEnabled":false},"ElasticsearchVersion":"6.7","EncryptionAtRestOptions":{"Enabled":false,"KmsKeyId":null},"Endpoint":"search-zipkin-2rlyh66ibw43ftlk4342ceeewu.ap-southeast-1.es.amazonaws.com","Endpoints":null,"LogPublishingOptions":null,"NodeToNodeEncryptionOptions":{"Enabled":false},"Processing":false,"ServiceSoftwareOptions":{"AutomatedUpdateDate":1.56709052E9,"Cancellable":false,"CurrentVersion":"R20190724","Description":"There is no software update available for this domain.","NewVersion":"","UpdateAvailable":false,"UpdateStatus":"COMPLETED"},"SnapshotOptions":{"AutomatedSnapshotStartHour":0},"UpgradeProcessing":false,"VPCOptions":null}}}

health check relating to the above call

curl -s localhost:9411/health
{"status":{"status":"DOWN"},"zipkin":"Timed out computing health status. This often means your storage backend is unreachable."}

codefromthecrypt · 2019-08-09T05:39:14Z

hmm.. trying a different domain.. at last the prometheus endpoint can show the endpoints, which I can hit, but of course these are signed urls..

$ curl -s localhost:9411/actuator/prometheus|grep health
# HELP armeria_client_endpointGroup_healthy  
# TYPE armeria_client_endpointGroup_healthy gauge
armeria_client_endpointGroup_healthy{authority="search-zipkin-2rlyh66ibw43ftlk4342ceeewu.ap-southeast-1.es.amazonaws.com:443",ip="52.77.100.49",name="elasticsearch",} 0.0
armeria_client_endpointGroup_healthy{authority="search-zipkin-2rlyh66ibw43ftlk4342ceeewu.ap-southeast-1.es.amazonaws.com:443",ip="54.251.169.105",name="elasticsearch",} 0.0
armeria_client_endpointGroup_count{name="elasticsearch",state="unhealthy",} 2.0
armeria_client_endpointGroup_count{name="elasticsearch",state="healthy",} 0.0
$ curl -s https://search-zipkin-2rlyh66ibw43ftlk4342ceeewu.ap-southeast-1.es.amazonaws.com/_cluster/health
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}

codefromthecrypt · 2019-08-09T05:41:31Z

if I disable client-side health-check, it is back..

run

ES_AWS_DOMAIN=zipkin ES_HTTP_LOGGING=BODY  STORAGE_TYPE=elasticsearch java     -Dloader.path='elasticsearch-aws.jar,elasticsearch-aws.jar!/lib'     -Dspring.profiles.active=elasticsearch-aws -cp zipkin.jar     org.springframework.boot.loader.PropertiesLauncher --zipkin.storage.elasticsearch.health-check.enabled=false

check

$ curl -s localhost:9411/health
{"status":"UP","zipkin":{"status":"UP","details":{"ElasticsearchStorage{initialEndpoints=aws://ap-southeast-1/zipkin, index=zipkin}":{"status":"UP"}}}}

…aws into elasticsearch-armeria-adrian

…y input).

...h-aws/src/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/AWSSignatureVersion4.java

codefromthecrypt · 2019-08-09T09:53:12Z

works, though on a bad network a ton of timeout messages and an interesting one about http/2 reset. I think we should consider adding com.linecorp.armeria.client.ResponseTimeoutException to the list of exceptions that indicate StorageComponent.isOverCapacity though it is debateable..

com.linecorp.armeria.client.ResponseTimeoutException: null

2019-08-09 17:48:19.459  WARN 24215 --- [worker-nio-2-13] c.l.a.c.HttpResponseDecoder              : Unexpected exception:

com.linecorp.armeria.client.ResponseTimeoutException: null

2019-08-09 17:48:19.466  WARN 24215 --- [-worker-nio-2-2] c.l.a.c.HttpResponseDecoder              : Unexpected exception:

com.linecorp.armeria.client.ResponseTimeoutException: null

2019-08-09 17:48:19.467  WARN 24215 --- [-worker-nio-2-4] c.l.a.c.HttpResponseDecoder              : Unexpected exception:

com.linecorp.armeria.client.ResponseTimeoutException: null

2019-08-09 17:48:19.469  WARN 24215 --- [-worker-nio-2-4] c.l.a.c.HttpRequestSubscriber            : [id: 0xfa0a1a64, L:/192.168.1.108:64213 - R:search-zipkin-2rlyh66ibw43ftlk4342ceeewu.ap-southeast-1.es.amazonaws.com/54.251.169.105:443][h2] Unexpected exception:

io.netty.handler.codec.http2.Http2Exception$StreamException: Maximum active streams violated for this endpoint.
	at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:147) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.checkNewStreamAllowed(DefaultHttp2Connection.java:896) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:748) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:668) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:176) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:151) ~[netty-codec-http2-4.1.38.Final.jar!/:4.1.38.Final]
	at com.linecorp.armeria.internal.Http2ObjectEncoder.doWriteHeaders(Http2ObjectEncoder.java:73) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.internal.HttpObjectEncoder.writeHeaders(HttpObjectEncoder.java:56) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.HttpRequestSubscriber.write0(HttpRequestSubscriber.java:279) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.HttpRequestSubscriber.writeFirstHeader(HttpRequestSubscriber.java:173) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.HttpRequestSubscriber.onSubscribe(HttpRequestSubscriber.java:151) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.subscribe(FixedStreamMessage.java:115) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage.subscribe(AbstractStreamMessage.java:93) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage.subscribe(AbstractStreamMessage.java:84) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.HeaderOverridingHttpRequest.subscribe(HeaderOverridingHttpRequest.java:113) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.HttpSessionHandler.invoke(HttpSessionHandler.java:157) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.HttpClientDelegate.doExecute(HttpClientDelegate.java:234) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.HttpClientDelegate.acquireConnectionAndExecute(HttpClientDelegate.java:136) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.HttpClientDelegate.execute(HttpClientDelegate.java:83) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.HttpClientDelegate.execute(HttpClientDelegate.java:42) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.encoding.HttpDecodingClient.execute(HttpDecodingClient.java:92) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.encoding.HttpDecodingClient.execute(HttpDecodingClient.java:39) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at zipkin2.server.internal.elasticsearch.ZipkinElasticsearchStorageConfiguration.lambda$esTracing$1(ZipkinElasticsearchStorageConfiguration.java:144) ~[classes!/:?]
	at com.linecorp.armeria.client.FunctionalDecoratingClient.execute(FunctionalDecoratingClient.java:47) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.brave.BraveClient.execute(BraveClient.java:131) ~[armeria-brave-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.client.brave.BraveClient.execute(BraveClient.java:56) ~[armeria-brave-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at zipkin.autoconfigure.storage.elasticsearch.aws.AWSSignatureVersion4.lambda$execute$2(AWSSignatureVersion4.java:93) ~[elasticsearch-aws.jar!/:?]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
	at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:98) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:363) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.notifySubscriberOfCloseEvent(FixedStreamMessage.java:126) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.stream.OneElementFixedStreamMessage.doNotify(OneElementFixedStreamMessage.java:70) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.stream.OneElementFixedStreamMessage.doRequest(OneElementFixedStreamMessage.java:55) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.request(FixedStreamMessage.java:88) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$SubscriptionImpl.request(AbstractStreamMessage.java:309) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.HttpMessageAggregator.onSubscribe(HttpMessageAggregator.java:55) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.lambda$subscribe$1(FixedStreamMessage.java:117) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at com.linecorp.armeria.common.AbstractRequestContext.lambda$makeContextAware$1(AbstractRequestContext.java:69) ~[armeria-no_long_polling_on_stop-armeria-0.4.0.Final-g4147712-1847.jar!/:?]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:515) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]

codefromthecrypt · 2019-08-10T14:37:37Z

retested against released binaries. still works

codefromthecrypt · 2019-08-10T15:19:14Z

another epic thanks @anuraaga

During openzipkin/zipkin-aws#135 we found that Armeria will add a second `host` header even if one is already set in the request headers before sending them. This can break behavior and I suppose must break some HTTP spec too. I wasn't sure whether to respect or ignore the user's setting, but with this change, if a user themselves set an authority and host to different values, we'll leave the host as is. While this is a bad user, I guess we should trust they did it on purpose. I had to write the unit test in Jetty since during the conversion from HTTP/1 headers to Armeria headers, the duplicate hosts got normalized.

anuraaga and others added 5 commits July 20, 2019 10:04

Migrate storage-elasticsearch-aws to use new armeria version.

81b2090

make things use snapshot version for now

4448a5c

Slight progress but not working

157842f

progress health endpoint works, but reads hang

bd4dea1

switches to jackson

c67559e

codefromthecrypt mentioned this pull request Jul 20, 2019

Migrate storage-elasticsearch-aws to use new armeria version. #134

Closed

Fixes dependency scope

4824dbe

codefromthecrypt requested a review from anuraaga July 20, 2019 01:28

codefromthecrypt commented Jul 20, 2019

View reviewed changes

...rc/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/ElasticsearchDomainEndpoint.java Outdated Show resolved Hide resolved

anuraaga reviewed Jul 20, 2019

View reviewed changes

switches to jackson instead of internals and feeback fixes

e4b942f

codefromthecrypt commented Jul 29, 2019

View reviewed changes

anuraaga reviewed Jul 29, 2019

View reviewed changes

anuraaga and others added 2 commits July 29, 2019 13:54

Release and context aware.

54a1e0d

review feedback

d852367

anuraaga approved these changes Jul 29, 2019

View reviewed changes

Adrian Cole added 2 commits July 29, 2019 13:13

fuzz

d4c139a

reverts get back to path for null safety

210433f

codefromthecrypt mentioned this pull request Jul 29, 2019

Implement ByteBufHttpData.toInputStream line/armeria#1919

Merged

Adrian Cole added 2 commits July 29, 2019 15:23

context aware

0d8f679

polish

3dec418

host dodge

2f57293

implement slacker dns resolution, but it makes test output terrible

38dea08

This was referenced Jul 30, 2019

Don't add a second translated header when it already exists. line/armeria#1942

Merged

Use unique attribute key. openzipkin/zipkin#2724

Closed

anuraaga and others added 3 commits August 2, 2019 20:59

Dance with me

bf29107

Clarify

80bc6c8

More newer

9d61933

codefromthecrypt mentioned this pull request Aug 9, 2019

Updates to latest armeria openzipkin/zipkin#2743

Merged

anuraaga added 2 commits August 9, 2019 15:02

Merge branch 'elasticsearch-armeria' of github.com:openzipkin/zipkin-…

68875f7

…aws into elasticsearch-armeria-adrian

Don't parse JSON if response is empty (readTree returns null for empt…

ad41787

…y input).

codefromthecrypt commented Aug 9, 2019

View reviewed changes

...h-aws/src/main/java/zipkin/autoconfigure/storage/elasticsearch/aws/AWSSignatureVersion4.java Outdated Show resolved Hide resolved

Adrian Cole added 2 commits August 9, 2019 17:13

trustin and no response handling

63abf0b

now it works

0c21756

official

00cd022

codefromthecrypt merged commit 3005c60 into master Aug 10, 2019

codefromthecrypt deleted the elasticsearch-armeria branch August 10, 2019 15:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrates AWS Elasticsearch Storage to armeria/jackson #135

Migrates AWS Elasticsearch Storage to armeria/jackson #135

codefromthecrypt commented Jul 20, 2019

codefromthecrypt commented Jul 20, 2019

codefromthecrypt commented Jul 20, 2019

codefromthecrypt Jul 20, 2019

anuraaga Jul 20, 2019

codefromthecrypt Jul 29, 2019

codefromthecrypt commented Jul 29, 2019

codefromthecrypt Jul 29, 2019

codefromthecrypt Jul 29, 2019

anuraaga Jul 29, 2019

codefromthecrypt Jul 29, 2019

anuraaga Jul 29, 2019

codefromthecrypt commented Jul 29, 2019

anuraaga Jul 29, 2019

anuraaga Jul 29, 2019

anuraaga Jul 29, 2019

codefromthecrypt commented Jul 29, 2019

codefromthecrypt commented Jul 29, 2019

codefromthecrypt commented Jul 30, 2019

codefromthecrypt commented Jul 30, 2019

codefromthecrypt commented Jul 30, 2019

codefromthecrypt commented Aug 2, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 10, 2019

codefromthecrypt commented Aug 10, 2019

Migrates AWS Elasticsearch Storage to armeria/jackson #135

Migrates AWS Elasticsearch Storage to armeria/jackson #135

Conversation

codefromthecrypt commented Jul 20, 2019

codefromthecrypt commented Jul 20, 2019

codefromthecrypt commented Jul 20, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codefromthecrypt commented Jul 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codefromthecrypt commented Jul 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codefromthecrypt commented Jul 29, 2019

codefromthecrypt commented Jul 29, 2019

codefromthecrypt commented Jul 30, 2019

codefromthecrypt commented Jul 30, 2019

codefromthecrypt commented Jul 30, 2019

codefromthecrypt commented Aug 2, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 9, 2019

codefromthecrypt commented Aug 10, 2019

codefromthecrypt commented Aug 10, 2019