Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker example fails with "bucketName is empty" and "DEADLINE_EXCEEDED" #96

Closed
dradetsky opened this issue Nov 7, 2020 · 6 comments
Closed
Labels
bug Something isn't working

Comments

@dradetsky
Copy link
Contributor

dradetsky commented Nov 7, 2020

I'm trying to get otel working. I was following the instructions here:

https://aws-otel.github.io/docs/setup/docker-images

After running into a number of weird issues trying to instrument my own app & generate traces, I tried to simplify the problem by just running the example emitter so I could work backwards. Unfortunately, the instructions do not work for me.

I filled out the following env vars in the docker compose:

      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
      - AWS_SESSION_TOKEN
      - AWS_REGION

the first 3 with output from the awsapps sso page, and the last as us-east-2. When running I had two interesting issues. First, it attempts to connect to localhost:4567 for some reason that's not explained

aws-ot-collector_1   | 2020-11-07T00:05:09.996Z INFO    service/service.go:252  Everything is ready. Begin running and processing data.
ot-metric-emitter_1  |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4567: Connection refused
ot-metric-emitter_1  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
ot-metric-emitter_1  |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4567: Connection refused
ot-metric-emitter_1  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
ot-metric-emitter_1  |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4567: Connection refused
ot-metric-emitter_1  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
ot-metric-emitter_1  |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4567: Connection refused
ot-metric-emitter_1  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
ot-metric-emitter_1  |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4567: Connection refused
ot-metric-emitter_1  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
ot-metric-emitter_1  |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4567: Connection refused
ot-metric-emitter_1  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
ot-metric-emitter_1  |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 4567: Connection refused
ot-metric-emitter_1  | [opentelemetry.auto.trace 2020-11-07 00:05:20:224 +0000] [main] INFO io.opentelemetry.javaagent.tooling.TracerInstaller - Installed span exporter: io.opentelemetry.exporters.otlp.OtlpGrpcSpanExporter

Don't know what that's about.

Anyway, it then has two following issues repeatedly

ot-metric-emitter_1  | [opentelemetry.auto.trace 2020-11-07 00:05:22:278 +0000] [grpc-default-executor-0] WARN io.opentelemetry.exporters.otlp.OtlpGrpcMetricExporter - Failed to export metrics
ot-metric-emitter_1  | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 974524962ns. [buffered_nanos=976275332, waiting_for_connection]
ot-metric-emitter_1  |  at io.grpc.Status.asRuntimeException(Status.java:533)
ot-metric-emitter_1  |  at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:515)
ot-metric-emitter_1  |  at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
ot-metric-emitter_1  |  at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
ot-metric-emitter_1  |  at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
ot-metric-emitter_1  |  at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
ot-metric-emitter_1  |  at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
ot-metric-emitter_1  |  at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
ot-metric-emitter_1  |  at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
ot-metric-emitter_1  |  at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
ot-metric-emitter_1  |  at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510)
ot-metric-emitter_1  |  at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
ot-metric-emitter_1  |  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630)
ot-metric-emitter_1  |  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518)
ot-metric-emitter_1  |  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692)
ot-metric-emitter_1  |  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681)
ot-metric-emitter_1  |  at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
ot-metric-emitter_1  |  at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
ot-metric-emitter_1  |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
ot-metric-emitter_1  |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
ot-metric-emitter_1  |  at java.lang.Thread.run(Thread.java:748)

and also

ot-metric-emitter_1  |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
ot-metric-emitter_1  |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Executing request GET http://localhost:4567/span1 HTTP/1.1
ot-metric-emitter_1  | Executing request GET http://localhost:4567/span2 HTTP/1.1
ot-metric-emitter_1  | emit metric with return time 1,/span2,200
ot-metric-emitter_1  | emit metric with http request size 926 byte, /span2
ot-metric-emitter_1  | ----------------------------------------
ot-metric-emitter_1  | 9472e6246b032793
ot-metric-emitter_1  | emit metric with return time 35,/span1,200
ot-metric-emitter_1  | emit metric with http request size 198 byte, /span1
ot-metric-emitter_1  | ----------------------------------------
ot-metric-emitter_1  | d01381d7026198e5,9472e6246b032793
ot-metric-emitter_1  | java.lang.RuntimeException: bucketName is empty
ot-metric-emitter_1  |  at com.amazon.aocagent.S3Service.uploadTraceData(S3Service.java:36)
ot-metric-emitter_1  |  at com.amazon.aocagent.App.lambda$main$0(App.java:81)
ot-metric-emitter_1  |  at spark.ResponseTransformerRouteImpl$1.handle(ResponseTransformerRouteImpl.java:47)
ot-metric-emitter_1  |  at spark.http.matching.Routes.execute(Routes.java:61)
ot-metric-emitter_1  |  at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:134)
ot-metric-emitter_1  |  at spark.embeddedserver.jetty.JettyHandler.doHandle(JettyHandler.java:50)
ot-metric-emitter_1  |  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1584)
ot-metric-emitter_1  |  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
ot-metric-emitter_1  |  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
ot-metric-emitter_1  |  at org.eclipse.jetty.server.Server.handle(Server.java:501)
ot-metric-emitter_1  |  at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
ot-metric-emitter_1  |  at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
ot-metric-emitter_1  |  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
ot-metric-emitter_1  |  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:272)
ot-metric-emitter_1  |  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
ot-metric-emitter_1  |  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
ot-metric-emitter_1  |  at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
ot-metric-emitter_1  |  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
ot-metric-emitter_1  |  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
ot-metric-emitter_1  |  at java.lang.Thread.run(Thread.java:748)

Was there supposed to be a third image included in the docker-compose or something? I'm very confused. I'd have looked at the sample app source code, but I can't find it. Is it public?

@dradetsky
Copy link
Contributor Author

So I think I figured out where that bucketName is empty issue was coming from:

https://github.com/aws-observability/aws-otel-test-framework/blob/deprecating/java-sdk-emitter/src/main/java/com/amazon/aocagent/S3Service.java#L31

Although defining the env variables mentioned here just led to new errors. I note the branch name "deprecating" and have a feeling that asking for support for getting this working is not the right move.

Are there any plans to make newer/better/working sample emitters & thus a good example for devs to use so they can work backwards?

@mxiamxia
Copy link
Member

mxiamxia commented Nov 9, 2020

Hi @dradetsky, thank you for reaching out us. The Java sample app (data emitter) has been moved to a branch and we do lack the good document for it. We'll update our Getting Started docs on it later. Thanks.

To your problems,

  1. bucketName is empty error is a side error which should not impact the functionality of the emitter. Can you please ignore it for now and we're fixing it.
  2. When sample app (Java WebApp) is up, there is a script to make http requests on localhost:4567 to mimic the regular business traffic that will generate Trace data to AWS X-Ray backend via AWS OTel. There is gap when WebApp is fully started before the calls are made. So we see few localhost port 4567: Connection refused at the starting time. We'll fix it.

The two problems you have should not impact Metrics and Traces data to be sent to AWS consoles with the right AWS Credentials. Have your found the data in your AWS account console with Docker Compose example at the end?

Thanks again for the feedbacks and please let us know any issue you have and we'd like to discuss and improve.

@dradetsky
Copy link
Contributor Author

@mxiamxia fwiw, I managed to get my local environment working by replacing the sample app there with this thing from the js code: https://github.com/aws-observability/aws-otel-js/tree/master/integ-test/js-sdk-emitter.

IMO, that thing is basically as complicated as the sample app needs to be. Even slightly more. As a developer, the very first thing I'm trying to do is figure out whether I have my local environment set up correctly, whether my containers can connect to aws, whether I'm getting my aws credentials loaded properly (see #100 btw), and that sort of thing. I don't yet care about mimicing regular business traffic, I just want a 'hello world xray' kind of thing to let me know my setup works. After that, I might want a sample app that in some way mimics real traffic, but maybe not. In any way, it's nowhere near as important as the sanity-check value provided by a stupid-simple thing like that integration test container.

@mxiamxia
Copy link
Member

I totally agree with you the setup should be simple. Our goal is to have our examples simple enough and developers can also take it as a reference sample code for their applications. Thanks for your contribution (#100) : )

To address some of your feedbacks, we made some improvements on our Java sample app.(if you still need it). It removes all the unexpected errors you've seen and generate Traces and Metrics to AWS. (Documentations are WIP)
https://github.com/aws-observability/aws-otel-collector/pull/101/files

I also sent a PR for creating Metrics with JS SDK to our aws-otel-js aws-observability/aws-otel-js#31. Hope it can provide some help to you. We'll update our Getting Started docs as well soon.

Thanks!

@dradetsky
Copy link
Contributor Author

@mxiamxia I haven't tried the metrics thing, but it looks like exactly the sort of thing I wanted. Or at least, that I wanted yesterday before I figured it out the hard way. But still, nice!

JasonXZLiu pushed a commit to open-o11y/aws-otel-collector that referenced this issue Nov 19, 2020
@mxiamxia mxiamxia added the bug Something isn't working label Nov 20, 2020
@mxiamxia
Copy link
Member

Thanks. Closed the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants