Troubleshooting

Problems

1. Error running agent container
2. Enable sending stacktrace to clients

1. Error running agent container

Symptoms

• Docker hosts do not have memory and cpu statistics displayed.
• Admiral agent container is in state ERROR
• Admiral agent container logs contains:

Admiral Agent started successfully!
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x401310]

goroutine 1 [running]:
panic(0x7f4920, 0xc82000a0b0)
    /usr/local/go/src/runtime/panic.go:464 +0x3e6
main.startDockerEventListener()
    /go/src/containerproxymapper/containerproxymapper.go:33 +0x1d0
main.main()
    /go/src/containerproxymapper/containerproxymapper.go:21 +0x108

Description

Admiral agent uses unix socket file for communication with docker daemon. The path to it should be /var/run/docker.sock

Solution

Add (or edit) docker starting arguments to contain following option

-H unix:///var/run/docker.sock'

Resources

• Configuring and running Docker

2. Enable sending stacktrace to clients

Description

By default error stacktrace is not sent with the error response, so the client receives error like:

{
    statusCode: 500,
    documentKind: "com:vmware:xenon:common:ServiceErrorResponse",
    errorCode: 0
}

Solution

This can be configured to make it easier when troubleshooting an issue. Parameter XENON_ENABLE_STACKTRACE should be set to true either when starting the container or afterwards:

• when running the container add the following argument to the docker command

-e XENON_ENABLE_STACKTRACE=true

• when container is already running (let's say its name is admiral-container), execute following command and then restart the container:

docker exec -u root admiral-container ex -sc '%s/disableStackTraceCollection=true/disableStackTraceCollection=false/g|x' /entrypoint.sh
docker restart admiral-container

Now stacktrace is propagated to the client, e.g.:

{
    stackTrace: [
        "com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1340)",
        "com.vmware.xenon.common.Operation.complete(Operation.java:1239)",
        "com.vmware.xenon.common.FactoryService.handleGet(FactoryService.java:556)",
        "com.vmware.xenon.common.FactoryService.handleRequest(FactoryService.java:304)",
        "com.vmware.xenon.common.FactoryService.handleRequest(FactoryService.java:273)",
        "com.vmware.xenon.common.ServiceHost.lambda$queueOrScheduleRequestInternal$45(ServiceHost.java:4301)",
        "java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)",
        "java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)",
        "java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)",
        "java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)",
        "java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)",
    ],
    statusCode: 500,
    documentKind: "com:vmware:xenon:common:ServiceErrorResponse",
    errorCode: 0
}

To restore the default behavior execute:

docker exec -u root admiral-container ex -sc '%s/disableStackTraceCollection=false/disableStackTraceCollection=true/g|x' /entrypoint.sh
docker restart admiral-container