New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running dotnet in Dockerfile fails with "lttng-ust-comm.c:1446: lttng_ust_init: Assertion `!ret' failed." #1582

Closed
mmalolepszy opened this Issue Feb 25, 2016 · 15 comments

Comments

Projects
None yet
8 participants
@mmalolepszy

Problem

When trying to create new docker image using dotnet command in Dockerfile, it fails with following error:

Step 9 : RUN dotnet restore
 ---> Running in b28d56b2ace2
dotnet: lttng-ust-comm.c:1446: lttng_ust_init: Assertion `!ret' failed.
Aborted
The command '/bin/sh -c dotnet new && dotnet restore' returned a non-zero code: 134

Steps to reproduce

Example dockerfile:

FROM ubuntu:trusty

RUN apt-get -y update && apt-get -y install wget libcurl3 libicu52 liblldb-3.6 liblttng-ust0 libunwind8 libssl-dev clang-3.5

RUN wget https://dotnetcli.blob.core.windows.net/dotnet/beta/Installers/Latest/dotnet-ubuntu-x64.latest.deb
RUN dpkg -i dotnet-ubuntu-x64.latest.deb

RUN mkdir /app
WORKDIR /app

RUN dotnet new && dotnet restore
ENTRYPOINT ["dotnet", "run"]

My environment

  • host: Win 10 Enterprise x64
  • docker: 1.10.2, build c3959b1
  • dotnet: latest (1.0.0.001556, but observed also yesterday on 1.0.0.001517)

Additional observations

  • If you strip last two lines from the above Dockerfile, the image builds successfully. When you run this image, first execution of dotnet fails, but the next one runs ok:
$ docker run -it dotnetbug
root@95ec2aeac8d3:/app# dotnet new && dotnet restore
dotnet: lttng-ust-comm.c:1446: lttng_ust_init: Assertion `!ret' failed.
Aborted
root@95ec2aeac8d3:/app# dotnet new && dotnet restore
Created new C# project in /app.
log  : Restoring packages for /app/project.json...
(...)
  • Same behavior can be observed on microsoft/dotnet:latest image:
$ docker run -it microsoft/dotnet:latest
root@9a569dd5251a:/# dotnet
dotnet: lttng-ust-comm.c:1446: lttng_ust_init: Assertion `!ret' failed.
Aborted
root@9a569dd5251a:/# dotnet
.NET Command Line Interface
Usage: dotnet [common-options] [command] [arguments]
(...)

Notes

  • I found a reference to similar (if not exactly the same) problem here: RendleLabs/docknet-docker#1
  • I don't have access to another docker host, so I can't check it in other environments.

/cc @blackdwarf

@ppanyukov

This comment has been minimized.

Show comment
Hide comment
@ppanyukov

ppanyukov Feb 25, 2016

It fails the first time you run it, everything is ok after that. As a workaround I have this in my scripts:

# First time we run this it crashes so run here for the first time
dotnet -v 1>/dev/null 2>/dev/null

A bit annoying but does the job (for now).

It fails the first time you run it, everything is ok after that. As a workaround I have this in my scripts:

# First time we run this it crashes so run here for the first time
dotnet -v 1>/dev/null 2>/dev/null

A bit annoying but does the job (for now).

@blackdwarf

This comment has been minimized.

Show comment
Hide comment
@blackdwarf

blackdwarf Feb 25, 2016

Contributor

@dleeapho have you guys encountered this?

Contributor

blackdwarf commented Feb 25, 2016

@dleeapho have you guys encountered this?

@blackdwarf blackdwarf added the bug label Feb 25, 2016

@blackdwarf blackdwarf added this to the RTM-March milestone Feb 25, 2016

@TheRealPiotrP

This comment has been minimized.

Show comment
Hide comment
@TheRealPiotrP

TheRealPiotrP Feb 25, 2016

Collaborator

@MichaelSimons can you take a first stab at triaging & understanding this?

Collaborator

TheRealPiotrP commented Feb 25, 2016

@MichaelSimons can you take a first stab at triaging & understanding this?

@CarlosLanderas

This comment has been minimized.

Show comment
Hide comment
@CarlosLanderas

CarlosLanderas Feb 26, 2016

Same problem with microsoft/dotnet:latest.

Step 4 : RUN dotnet new
---> Running in e1fe235968f7
dotnet: lttng-ust-comm.c:1446: lttng_ust_init: Assertion `!ret' failed.
Aborted
The command '/bin/sh -c dotnet new' returned a non-zero code: 134.

The workaround @ppanyukov suggested is not working for me:

Step 4 : RUN dotnet -v 1>/dev/null 2>/dev/null
---> Running in 2d0d912d74a0
The command '/bin/sh -c dotnet -v 1>/dev/null 2>/dev/null' returned a non-zero code: 134

Same problem with microsoft/dotnet:latest.

Step 4 : RUN dotnet new
---> Running in e1fe235968f7
dotnet: lttng-ust-comm.c:1446: lttng_ust_init: Assertion `!ret' failed.
Aborted
The command '/bin/sh -c dotnet new' returned a non-zero code: 134.

The workaround @ppanyukov suggested is not working for me:

Step 4 : RUN dotnet -v 1>/dev/null 2>/dev/null
---> Running in 2d0d912d74a0
The command '/bin/sh -c dotnet -v 1>/dev/null 2>/dev/null' returned a non-zero code: 134

@mmalolepszy

This comment has been minimized.

Show comment
Hide comment
@mmalolepszy

mmalolepszy Feb 26, 2016

@CarlosLanderas Workaround suggested by @ppanyukov works. For my case I made a dotnet.sh script:

#!/bin/bash

dotnet 1>/dev/null 2>/dev/null
dotnet $@

and I use it from my Dockerfile like this:

COPY dotnet.sh .
RUN bash dotnet.sh restore

@CarlosLanderas Workaround suggested by @ppanyukov works. For my case I made a dotnet.sh script:

#!/bin/bash

dotnet 1>/dev/null 2>/dev/null
dotnet $@

and I use it from my Dockerfile like this:

COPY dotnet.sh .
RUN bash dotnet.sh restore
@CarlosLanderas

This comment has been minimized.

Show comment
Hide comment
@CarlosLanderas

CarlosLanderas Feb 26, 2016

@mmalolepszy Thank you very much, It is working now using the script approach.

@mmalolepszy Thank you very much, It is working now using the script approach.

@MichaelSimons

This comment has been minimized.

Show comment
Hide comment
@MichaelSimons

MichaelSimons Mar 1, 2016

Contributor

I do not believe this issue is specific to dotnet. You can reproduce the same behavior with the following Dockerfile

FROM buildpack-deps:trusty-scm

RUN apt-get update
RUN apt-get install -y liblttng-ust0

RUN LD_PRELOAD=liblttng-ust-cyg-profile.so.0 ls

This appears to be a regression in docker between 1.9.1 and 1.10.2. It also appears Boot2Docker related as I cannot reproduce it in an Ubuntu environment.

A better workaround other than RUN dotnet -v 1>/dev/null 2>/dev/null would be to set the LTTNG_UST_REGISTER_TIMEOUT environment variable to 0 (no wait) or -1 (infinite wait) as the default value (3000) as well as any positive number appears to be the source of the failing assert.

I am following up with both Docker and LTTNG to get them to address this issue.

Contributor

MichaelSimons commented Mar 1, 2016

I do not believe this issue is specific to dotnet. You can reproduce the same behavior with the following Dockerfile

FROM buildpack-deps:trusty-scm

RUN apt-get update
RUN apt-get install -y liblttng-ust0

RUN LD_PRELOAD=liblttng-ust-cyg-profile.so.0 ls

This appears to be a regression in docker between 1.9.1 and 1.10.2. It also appears Boot2Docker related as I cannot reproduce it in an Ubuntu environment.

A better workaround other than RUN dotnet -v 1>/dev/null 2>/dev/null would be to set the LTTNG_UST_REGISTER_TIMEOUT environment variable to 0 (no wait) or -1 (infinite wait) as the default value (3000) as well as any positive number appears to be the source of the failing assert.

I am following up with both Docker and LTTNG to get them to address this issue.

@nmschulte

This comment has been minimized.

Show comment
Hide comment
@nmschulte

nmschulte Mar 1, 2016

Just posting to confirm both workarounds; I reported rendlelabs/docknet-docker#1, let me know if there's more I can help with.

$ docker run --rm --entrypoint="/bin/bash" -it rendlelabs/docknet:latest
root@48f0396dacb3:/# dotnet -v
dotnet: lttng-ust-comm.c:1446: lttng_ust_init: Assertion `!ret' failed.
Aborted (core dumped)
root@48f0396dacb3:/# dotnet -v
.NET Command Line Tools (1.0.0-dev-001606)
Usage: dotnet [common-options] [command] [arguments]

Arguments:
  [command]     The command to execute
  [arguments]   Arguments to pass to the command

Common Options (passed before the command):
  -v|--verbose  Enable verbose output
  --version     Display .NET CLI Version Info

Common Commands:
  new           Initialize a basic .NET project
  restore       Restore dependencies specified in the .NET project
  build         Builds a .NET project
  publish       Publishes a .NET project for deployment (including the runtime)
  run           Compiles and immediately executes a .NET project
  repl          Launch an interactive session (read, eval, print, loop)
  pack          Creates a NuGet package
$ docker run --rm --entrypoint="/bin/bash" -it rendlelabs/docknet:latest
root@7a2028cda8fe:/# LTTNG_UST_REGISTER_TIMEOUT=0 dotnet -v
.NET Command Line Tools (1.0.0-dev-001606)
Usage: dotnet [common-options] [command] [arguments]

Arguments:
  [command]     The command to execute
  [arguments]   Arguments to pass to the command

Common Options (passed before the command):
  -v|--verbose  Enable verbose output
  --version     Display .NET CLI Version Info

Common Commands:
  new           Initialize a basic .NET project
  restore       Restore dependencies specified in the .NET project
  build         Builds a .NET project
  publish       Publishes a .NET project for deployment (including the runtime)
  run           Compiles and immediately executes a .NET project
  repl          Launch an interactive session (read, eval, print, loop)
  pack          Creates a NuGet package

Just posting to confirm both workarounds; I reported rendlelabs/docknet-docker#1, let me know if there's more I can help with.

$ docker run --rm --entrypoint="/bin/bash" -it rendlelabs/docknet:latest
root@48f0396dacb3:/# dotnet -v
dotnet: lttng-ust-comm.c:1446: lttng_ust_init: Assertion `!ret' failed.
Aborted (core dumped)
root@48f0396dacb3:/# dotnet -v
.NET Command Line Tools (1.0.0-dev-001606)
Usage: dotnet [common-options] [command] [arguments]

Arguments:
  [command]     The command to execute
  [arguments]   Arguments to pass to the command

Common Options (passed before the command):
  -v|--verbose  Enable verbose output
  --version     Display .NET CLI Version Info

Common Commands:
  new           Initialize a basic .NET project
  restore       Restore dependencies specified in the .NET project
  build         Builds a .NET project
  publish       Publishes a .NET project for deployment (including the runtime)
  run           Compiles and immediately executes a .NET project
  repl          Launch an interactive session (read, eval, print, loop)
  pack          Creates a NuGet package
$ docker run --rm --entrypoint="/bin/bash" -it rendlelabs/docknet:latest
root@7a2028cda8fe:/# LTTNG_UST_REGISTER_TIMEOUT=0 dotnet -v
.NET Command Line Tools (1.0.0-dev-001606)
Usage: dotnet [common-options] [command] [arguments]

Arguments:
  [command]     The command to execute
  [arguments]   Arguments to pass to the command

Common Options (passed before the command):
  -v|--verbose  Enable verbose output
  --version     Display .NET CLI Version Info

Common Commands:
  new           Initialize a basic .NET project
  restore       Restore dependencies specified in the .NET project
  build         Builds a .NET project
  publish       Publishes a .NET project for deployment (including the runtime)
  run           Compiles and immediately executes a .NET project
  repl          Launch an interactive session (read, eval, print, loop)
  pack          Creates a NuGet package
@MichaelSimons

This comment has been minimized.

Show comment
Hide comment
@MichaelSimons

MichaelSimons Mar 1, 2016

Contributor

This change in behavior is a result of the new seccomp feature that was added to Docker in 1.10.

The default profile is restricting syscalls that LTTng is making which is causing the assert. LTTng acknowledged the assert behavior is wrong and will try to get a more specific error message. They are also going to identify which syscalls they make so that Docker users can create the appropriate profile to use with the library.

The seccomp feature is not enabled/supported on all platforms by default which is why some Docker users are not affected by this.

Until the syscalls made by LTTng are identified, another workaround would be to use the unconfined profile which I believe is equivalent to the pre 1.10 behavior. This obviously is just a workaround for now.

docker run --rm -it --security-opt seccomp:unconfined microsoft/dotnet

Contributor

MichaelSimons commented Mar 1, 2016

This change in behavior is a result of the new seccomp feature that was added to Docker in 1.10.

The default profile is restricting syscalls that LTTng is making which is causing the assert. LTTng acknowledged the assert behavior is wrong and will try to get a more specific error message. They are also going to identify which syscalls they make so that Docker users can create the appropriate profile to use with the library.

The seccomp feature is not enabled/supported on all platforms by default which is why some Docker users are not affected by this.

Until the syscalls made by LTTng are identified, another workaround would be to use the unconfined profile which I believe is equivalent to the pre 1.10 behavior. This obviously is just a workaround for now.

docker run --rm -it --security-opt seccomp:unconfined microsoft/dotnet

@MichaelSimons

This comment has been minimized.

Show comment
Hide comment
@MichaelSimons

MichaelSimons Mar 2, 2016

Contributor

The syscall LTTng makes was identified - restart_syscall. (Ref http://lists.lttng.org/pipermail/lttng-dev/2016-March/025573.html) You can define a custom profile based on the default (https://github.com/docker/docker/blob/master/profiles/seccomp/default.json) and add this syscall instead of using unconfined.

Waiting to see if Docker and LTTng can come up with a solution that works out of the box.

Contributor

MichaelSimons commented Mar 2, 2016

The syscall LTTng makes was identified - restart_syscall. (Ref http://lists.lttng.org/pipermail/lttng-dev/2016-March/025573.html) You can define a custom profile based on the default (https://github.com/docker/docker/blob/master/profiles/seccomp/default.json) and add this syscall instead of using unconfined.

Waiting to see if Docker and LTTng can come up with a solution that works out of the box.

@MichaelSimons MichaelSimons referenced this issue in dotnet/dotnet-docker-nightly Mar 16, 2016

Merged

Temporarily work around LTTng assert #2

@MichaelSimons

This comment has been minimized.

Show comment
Hide comment
@MichaelSimons

MichaelSimons Mar 17, 2016

Contributor

Docker made a change (moby/moby#21117) to allow the restart_syscall by default. With the next release of Docker (1.11.0), this issue will be addressed and you will no longer encounter it with the default experience. Additionally a change (lttng/lttng-ust@8aadb54) was made to the LTTng library to provide a better error experience when this does occur so that users will be provided with more information on what happened rather than a failed assert.

Both the microsoft/dotnet and microsoft/dotnet-preview images were updated with a workaround so that users do not encounter this issue with the out of box experience. This allows customers to kick the tires and have a better initial experience with the dotnet images.

It is recommended that if you encounter this issue within your own Dockerfiles that you use one of the workarounds mentioned earlier in this thread (change the seccomp profile you are running with or set the LTTNG_UST_REGISTER_TIMEOUT variable accordingly) or update your Docker to 1.11.0 once it is released.

Contributor

MichaelSimons commented Mar 17, 2016

Docker made a change (moby/moby#21117) to allow the restart_syscall by default. With the next release of Docker (1.11.0), this issue will be addressed and you will no longer encounter it with the default experience. Additionally a change (lttng/lttng-ust@8aadb54) was made to the LTTng library to provide a better error experience when this does occur so that users will be provided with more information on what happened rather than a failed assert.

Both the microsoft/dotnet and microsoft/dotnet-preview images were updated with a workaround so that users do not encounter this issue with the out of box experience. This allows customers to kick the tires and have a better initial experience with the dotnet images.

It is recommended that if you encounter this issue within your own Dockerfiles that you use one of the workarounds mentioned earlier in this thread (change the seccomp profile you are running with or set the LTTNG_UST_REGISTER_TIMEOUT variable accordingly) or update your Docker to 1.11.0 once it is released.

tonysneed added a commit to tonysneed/docknet-docker that referenced this issue Apr 30, 2016

@roederja2

This comment has been minimized.

Show comment
Hide comment
@roederja2

roederja2 Apr 27, 2017

I still get this issue with the latest MSFT provided docker -sdk image (1.1.1). The LTTNG_UST_REGISTER_TIMEOUT=0 workaround works.

I still get this issue with the latest MSFT provided docker -sdk image (1.1.1). The LTTNG_UST_REGISTER_TIMEOUT=0 workaround works.

@MichaelSimons

This comment has been minimized.

Show comment
Hide comment
@MichaelSimons

MichaelSimons Apr 27, 2017

Contributor

@roederja2, what version of docker are you using?

Contributor

MichaelSimons commented Apr 27, 2017

@roederja2, what version of docker are you using?

@roederja2

This comment has been minimized.

Show comment
Hide comment
@roederja2

roederja2 Apr 27, 2017

Docker version 1.10.3, build 79ebcd8-unsupported

So this still has the bug, but I thought you said you also did something on the dotnetcore side.

Docker version 1.10.3, build 79ebcd8-unsupported

So this still has the bug, but I thought you said you also did something on the dotnetcore side.

@MichaelSimons

This comment has been minimized.

Show comment
Hide comment
@MichaelSimons

MichaelSimons Apr 27, 2017

Contributor

@roederja2, we did add a workaround but have since removed it in out latest images because the versions of Docker that had this issue are no longer supported by Docker.

Contributor

MichaelSimons commented Apr 27, 2017

@roederja2, we did add a workaround but have since removed it in out latest images because the versions of Docker that had this issue are no longer supported by Docker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment