Single X-ray Daemon for Multiple Containers

[sandygvs]

Hi Experts,

Currently we have 50+ containers running on ECS cluster, each container has it's own side-car xray containers. In that case for 50 application containers we end up running 100~ containers(app + xray). Is there any way that we can have single xray daemon per ECS host will help us in trace all the logs/request from all the running application containers from the particular ECS host. so that we can avoid using side-car container for each app. containers. Please let me know if you more information on this

Appreciate your help on this!!!

[yogiraj07]

Hi @sandygvs ,
Can you take a look at ECS documentation for the X-Ray daemon. You can run X-Ray daemon in the separate container and make sure all the X-Ray instrumented apps containers has X-Ray daemon address set to the X-Ray container. This address will be picked by X-Ray SDK and the traces will be send to the X-Ray daemon. Please consider security best practices while exposing the ports of the daemon container.

Let me know if you have any questions.

Thank you,
Yogi

[sandygvs]

Hi @yogiraj07,

Thanks for the update, we are using both awsvpc and bridge networking mode. As per the document mentioned above, AWS_XRAY_DAEMON_ADDRESS is pointing to localhost. I would like to know that is there any config changes we need or we need to have separate ECS service for x-ray in this case. Here i have attached the task definition for both networking mode.
xray.zip

[yogiraj07]

Hi @sandygvs ,
By default, X-Ray daemon runs on localhost:2000 address, however, this is configurable using command line args or cfg.yaml. The daemon docker image has an example of how to configure daemon address.

Once the X-Ray container is configured, you need to correctly point your application containers to the X-Ray ECS container daemon address using AWS_XRay_DAEMON_ADDRESS field. X-Ray SDK will read this value and send data generated by your application to the configured daemon address.

I think you need to re-check the port mapping section for the X-Ray daemon task definition (protocol should be udp) as well as set AWS_XRay_DAEMON_ADDRESS to <xray_container>:2000 instead of localhost:2000 for bridge mode.

Thanks,
Yogi

[sandygvs]

Hi @yogiraj07 ,

As you mentioned above, { "name" : "AWS_XRAY_DAEMON_ADDRESS", "value" : "xray-daemon:2000" } to be included as part of application task definition so that this address will be picked by X-Ray SDK and the traces will be send to the X-Ray daemon??
In this case, x-ray daemon to be run as separate ECS service for both bridge and awsvpc networking mode??

[yogiraj07]

Hi @sandygvs ,

As you mentioned above, { "name" : "AWS_XRAY_DAEMON_ADDRESS", "value" : "xray-daemon:2000" } to be included as part of application task definition so that this address will be picked by X-Ray SDK and the traces will be send to the X-Ray daemon??

Yes this is correct. The linking should be provided to bridge networking mode only, as per documentation.

In this case, x-ray daemon to be run as separate ECS service for both bridge and awsvpc networking mode??

Yes, however, please make sure, the X-Ray SDK - X-Ray daemon link works. This can be verified by looking at X-Ray SDK logs and X-Ray daemon logs. The UDP emitter used in the SDK sends data to configured X-Ray Daemon address. If the address is correctly configured, the daemon logs will have entry 1 segments successfully sent. This implies that the segment is sent by X-Ray SDK (application container) to the daemon (daemon container) and the daemon sends to X-Ray service.

Please let me know if you need any help.

Thanks,
Yogi

[sandygvs]

Thanks for the update, let me try the above mentioned approach and keep you posted on progress..

[sandygvs]

Hi @yogiraj07,

I would like to understand how traces are being captured in x-ray if i have multiple containers running across multiple zones. I could see Avg. Response time from the trace result, is that response time is consolidated time of all the containers which are running across different zones. Is there anyway how i can segregate the traces on basis of AZ's. It's is very critical piece of info we needed for our application as far as performance is considered.

Much Appreciate your help on this!!!

[atreyd]

@sandygvs Have you got the solution for this requirement ?

[willarmiros]

Hi @atreyd,
Are you looking for a solution to the AZ separation or to running one daemon container for multiple application containers?

[atreyd]

@willarmiros I am looking for the X-ray daemon container for entire cluster

[willarmiros]

@atreyd please see the conversation on #53, specifically the more recent posts on the ECS-recommended way to run daemons on a cluster. The gist is that you can designate the X-Ray daemon as a daemon service and ECS will automatically deploy it to every container instance in your cluster and manage/scale it on your behalf.

The alternative, to actually have one daemon per cluster, involves setting up your daemon as one task, exposing its port 2000, and then modifying the task definition of all your other X-Ray instrumented tasks to link their AWS_XRAY_DAEMON_ADDRESS environment variable to your daemon task as described here: https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ecs.html.

[atreyd]

@willarmiros Thanks for the clarifications.

I was able to run the X-ray as a DAEMON service in the ECS cluster and was able to to view that the X-ray container is running on each host in the cluster on 0.0.0.0:2020.

However I have below questions:

1. Can I configure AWS_XRAY_DAEMON_ADDRESS as 0.0.0.0:2020 or it should be localhost:2020 only for the other task definitions ?
2. Can I use Service Discovery Record instead of using localhost/0.0.0.0 for xray daemon ?
3. Is there a way we can send traces to the x-ray daemon without instrumenting the logs. i.e. from a file or location on the host machine ?

[atreyd]

@willarmiros

Any pointers on the questions ?

[srprash]

Hi @atreyd
I may be able to answer some of your questions.

1. Yes, you can configure 0.0.0.0:2000 as the AWS_XRAY_DAEMON_ADDRESS if you have your daemon running on that address.
2. You may be able to use Service Discovery if you are able to expose the daemon address to the task definitions. Please see here for considerations.
3. The SDKs typically send trace data to daemon by opening a socket to the daemon address. If you want to send traces to the X-Ray service without using an SDK, you might want to look into using the APIs directly.

[phamquy]

@atreyd please see the conversation on #53, specifically the more recent posts on the ECS-recommended way to run daemons on a cluster. The gist is that you can designate the X-Ray daemon as a daemon service and ECS will automatically deploy it to every container instance in your cluster and manage/scale it on your behalf.

The alternative, to actually have one daemon per cluster, involves setting up your daemon as one task, exposing its port 2000, and then modifying the task definition of all your other X-Ray instrumented tasks to link their AWS_XRAY_DAEMON_ADDRESS environment variable to your daemon task as described here: https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ecs.html.

The example in the doc is for the use case where the xray-daemon setup as a sidecar container. how would it look like if the xray-daemon is setup as a separate task?

[Nitinkumar2206]

I am running the daemon/node by using service type “Daemon”. However, there is an issue pointing application container to it. Localhost:2000 or 0.0.0.0:2000 end up in connection refused error. As for me in bridge networking I can only address containers via IP and in awsvpc each task is getting its own eni.

I am running XRay daemon in its own service.
Has any one able to achieve this so far? Especially without service discover setup.

[willarmiros]

Hi @Nitinkumar2206,

Can you please send your task definitions for your daemon and replica services? This might help us resolve this issue with the ECS team so we can support the use cases mentioned in this issue.

[Nitinkumar2206]

Thanks @willarmiros . Below are the task definitions:

I am using your sample app in a container : https://github.com/aws-samples/aws-xray-sdk-node-sample

Application Task Definition :

containerDefinitions:
  portMappings:
  - hostPort: 3000
    protocol: tcp
    containerPort: 3000
  cpu: 1024
  environment:
  - name: AWS_XRAY_DAEMON_ADDRESS
    value: localhost:2000      ## Tried 0.0.0.0:2000 , direct IP address reference works. IP:2000
  memory: 256
  image: ecr
  essential: true
  name: sampleapp
memory: '512'
taskRoleArn: xxxx
compatibilities:
- EC2
taskDefinitionArn: xxxx
family: smapleapp
requiresCompatibilities:
- EC2
networkMode: bridge
cpu: '1024'
revision: 54
status: ACTIVE

X-Ray Daemon :

containerDefinitions:
  portMappings:
  - hostPort: 2000
    protocol: udp
    containerPort: 2000
  command:
  - "--log-level=dev"
  linuxParameters: 
  cpu: 128
  environment:
  - name: AWS_DEFAULT_REGION
    value: ap-southeast-2
  memoryReservation: 300
  image: amazon/aws-xray-daemon:latest
  name: xray-daemon
memory: '512'
taskRoleArn: arn:aws:iam::xxxxxx:role/xxxxx
compatibilities:
- EC2
taskDefinitionArn: xxxx
family: xray-daemon
requiresCompatibilities:
- EC2
networkMode: bridge
cpu: '1024'
revision: 2
status: ACTIVE

[Nitinkumar2206]

I used AWS_XRAY_DAEMON_ADDRESS:169.254.172.1:2000 after referring to answers at aws/containers-roadmap#165. Looks like its working both for containers using awsvpc and bridge containers.

[willarmiros]

Going to close this out to avoid clutter, and since #53 will now track the request to add docs around running daemon as a service.