Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail to run cloud debug enabled task with x-ray enabled #1464

Open
zxkane opened this issue Dec 24, 2019 · 3 comments
Open

Fail to run cloud debug enabled task with x-ray enabled #1464

zxkane opened this issue Dec 24, 2019 · 3 comments
Labels

Comments

@zxkane
Copy link

@zxkane zxkane commented Dec 24, 2019

Describe the bug

After enabling cloud debug for a task of ECS with x-ray enabled, the new cloud debug task failed to run due to the x-ray side exits with code 1.

To reproduce

  1. create a ECS cluster with task x-ray enabled
  2. enable cloud debug above task in IntelliJ
  3. observe the status of cloud debug enabled new task in ecs cluster, the task always restarted due to the x-ray sidecar container exits with error code

Expected behavior

The x-ray sidecar in task still works like without enabling cloud debug. The cloud debug enabled task fails making the cloud debug not working.

Screenshots

Your Environment

  • OS: macOSX 10.14
  • JetBrains' Product: IntelliJ community
  • JetBrains' Product Version: 2019.3.1
  • Toolkit Version: 1.9-193 / cloud-debug cli 1.0.233
  • SAM CLI Version:
  • JVM/Python Version: java 11.0.5 2019-10-15 LTS

Additional context

@zxkane zxkane added the bug label Dec 24, 2019
@raducataws

This comment has been minimized.

Copy link

@raducataws raducataws commented Jan 8, 2020

Hi @zxkane,

Unable to reproduce it with the following steps:

  1. Onboard an existing ECS service with X-Ray using the official docker image;
  2. Enabled Cloud Debugging for the ECS service;
  3. Successfully start a debug session.

To move forward, please provide more details on the reason why the x-ray sidecar container fails to start. Also, the task definition for your ECS service may help us identify the issue.

@zxkane

This comment has been minimized.

Copy link
Author

@zxkane zxkane commented Jan 9, 2020

I'm using office X-Ray docker image(amazon/aws-xray-daemon) from docker hub. But I added a healthy command to check if the X-Ray daemon is running or not.

Actually I found the injected X-Ray with cloud debug exited with code non-zero. See below screenshot from ECS console.

image

Below is the task definition for your reference,

{
  "ipcMode": null,
  "executionRoleArn": "arn:aws:iam::845861764576:role/eCommenceClusterStack-ExecutionRolefrontendservice-7CTWZ0YY8QU2",
  "containerDefinitions": [
    {
      "dnsSearchDomains": [],
      "logConfiguration": null,
      "entryPoint": [
        "/aws/cloud-debug/common/agent",
        "--selector",
        "x-ray-for-frontendservice-mainline",
        "/usr/bin/xray",
        "-t",
        "0.0.0.0:2000",
        "-b",
        "0.0.0.0:2000"
      ],
      "portMappings": [],
      "command": [],
      "linuxParameters": null,
      "cpu": 32,
      "environment": [],
      "resourceRequirements": null,
      "ulimits": [],
      "dnsServers": [],
      "mountPoints": [
        {
          "readOnly": null,
          "containerPath": "/aws/cloud-debug",
          "sourceVolume": "cloud-debug-volume"
        }
      ],
      "workingDirectory": null,
      "secrets": [],
      "dockerSecurityOptions": [],
      "memory": null,
      "memoryReservation": 256,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "amazon/aws-xray-daemon",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": [
        {
          "containerName": "cloud-debug-sidecar-container",
          "condition": "HEALTHY"
        }
      ],
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": {
        "retries": 3,
        "command": [
          "CMD-SHELL",
          "true"
        ],
        "timeout": 5,
        "interval": 30,
        "startPeriod": null
      },
      "essential": false,
      "links": [],
      "hostname": null,
      "extraHosts": [],
      "pseudoTerminal": null,
      "user": "1337",
      "readonlyRootFilesystem": null,
      "dockerLabels": {},
      "systemControls": [],
      "privileged": null,
      "name": "x-ray-for-frontendservice-mainline"
    },
    {
      "dnsSearchDomains": [],
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": [],
        "options": {
          "awslogs-group": "eCommenceClusterStack-eCommenceLogGroupB3C03A6A-85DJ9G716HKU",
          "awslogs-datetime-format": "%Y-%m-%d %H:%M:%S",
          "awslogs-region": "ap-northeast-1",
          "awslogs-stream-prefix": "frontendservice"
        }
      },
      "entryPoint": [
        "/aws/cloud-debug/common/agent",
        "--selector",
        "container-frontendservice-mainline",
        "sh",
        "-c"
      ],
      "portMappings": [
        {
          "hostPort": 8080,
          "protocol": "tcp",
          "containerPort": 8080
        }
      ],
      "command": [
        "java -jar /app/frontend-1.0.0-SNAPSHOT.jar"
      ],
      "linuxParameters": null,
      "cpu": 0,
      "environment": [
        {
          "name": "dubbo.registry.address",
          "value": "nacos://NacosServiceCloudmapService-ya25WqvvBRzp.eCommenceCloudMapNamesapce:8848"
        },
        {
          "name": "service.product.url",
          "value": "http://productservice.eCommenceCloudMapNamesapce:8082"
        },
        {
          "name": "VERSION",
          "value": "1.0.0.SNAPSHOT_20191128075702"
        }
      ],
      "resourceRequirements": null,
      "ulimits": [],
      "dnsServers": [],
      "mountPoints": [
        {
          "readOnly": null,
          "containerPath": "/aws/cloud-debug",
          "sourceVolume": "cloud-debug-volume"
        }
      ],
      "workingDirectory": null,
      "secrets": [],
      "dockerSecurityOptions": [],
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "845861764576.dkr.ecr.ap-northeast-1.amazonaws.com/ecommence/frontendservice:1.0.0.SNAPSHOT_20191128075702",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": [
        {
          "containerName": "x-ray-for-frontendservice-mainline",
          "condition": "HEALTHY"
        },
        {
          "containerName": "cloud-debug-sidecar-container",
          "condition": "HEALTHY"
        }
      ],
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": {
        "retries": 3,
        "command": [
          "CMD-SHELL",
          "true"
        ],
        "timeout": 5,
        "interval": 30,
        "startPeriod": null
      },
      "essential": false,
      "links": [],
      "hostname": null,
      "extraHosts": [],
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": {},
      "systemControls": [],
      "privileged": null,
      "name": "container-frontendservice-mainline"
    },
    {
      "dnsSearchDomains": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "cloud-debug-frontendservice-mainline-group",
          "awslogs-region": "ap-northeast-1",
          "awslogs-stream-prefix": "cloud-debug-frontendservice-mainline-stream"
        }
      },
      "entryPoint": null,
      "portMappings": [],
      "command": null,
      "linuxParameters": null,
      "cpu": 0,
      "environment": [],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [
        {
          "readOnly": null,
          "containerPath": "/aws/cloud-debug",
          "sourceVolume": "cloud-debug-volume"
        }
      ],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": 300,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "831759287394.dkr.ecr.us-west-2.amazonaws.com/amberwing-sidecar:1.0",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": {
        "retries": 3,
        "command": [
          "CMD-SHELL",
          "true"
        ],
        "timeout": 5,
        "interval": 30,
        "startPeriod": null
      },
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "cloud-debug-sidecar-container"
    }
  ],
  "placementConstraints": [],
  "memory": "2048",
  "taskRoleArn": "arn:aws:iam::845861764576:role/eCommenceClusterStack-TaskRolefrontendservice1B395-1X4SQPE10IIMU",
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "taskDefinitionArn": "arn:aws:ecs:ap-northeast-1:845861764576:task-definition/cloud-debug-frontendservice-mainline:4",
  "family": "cloud-debug-frontendservice-mainline",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.ecr-auth"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.docker-plugin.local"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.container-health-check"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-ecr-pull"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.24"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.30"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.container-ordering"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "1024",
  "revision": 4,
  "status": "ACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": [
    {
      "name": "cloud-debug-volume",
      "host": null,
      "dockerVolumeConfiguration": {
        "autoprovision": null,
        "labels": null,
        "scope": "task",
        "driver": "local",
        "driverOpts": null
      }
    }
  ]
}

I also managed above ECS cluster via AWS CDK. You can play with it if you want to use this stack to reproduce the issue(freely to replace the project container to yours).

@ovidiupw

This comment has been minimized.

Copy link

@ovidiupw ovidiupw commented Jan 10, 2020

@zxkane

Thanks for providing us with more information. We'll try to reproduce this on our side and get back to you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.