Alpine Linux "Bus error" #762

Closed
makovich opened this Issue Dec 29, 2016 · 9 comments

Comments

Projects
None yet
2 participants
@makovich
Contributor

makovich commented Dec 29, 2016

While playing around nghttp2 proxy server in Docker after a pause of ~1..2 months I've found that it stops to work and fails with "Bus error" message. It was working.

/ # nghttpx
Bus error

Since I've updated Docker app itself and rebuilt my images, I decided to make some checks before writing here.

  1. I use this package nearly two months ago and it has no updates in the mentioned period of time. Same 1.70 version. I tried to run it with Alpine 3.2, 3.4 and fresh one 3.5. Same fail.

  2. Downgrade of Docker on mac, Vagrant's maier/alpine-3.3.1-x86_64 and same steps on Windows machine haven't succeeded.

  3. Build from master branch (one of 1.80 RCs) shows the same error.

  4. I tried to track down any dependencies changes within Alpine's nghttp2 package for that past two months and found only libgcc and libstdc++ version updates (5.3.0 -> 6.3.0). When I installed this previous 5.3.0 versions the error message have changed but still too cryptic to me:

/ # nghttpx
Error relocating /usr/bin/nghttpx: _ZTINSt6thread6_StateE: symbol not found
Error relocating /usr/bin/nghttpx: _ZTINSt6thread6_StateE: symbol not found
Error relocating /usr/bin/nghttpx: _ZNSt6thread6_StateD2Ev: symbol not found
Error relocating /usr/bin/nghttpx: _ZNSt6thread15_M_start_threadESt10unique_ptrINS_6_StateESt14default_deleteIS1_EEPFvvE: symbol not found

Right now I stuck with this slightly unappealing message and don't know what my next steps could look like.

Here is Dockerfile that can speed up things:

FROM alpine:3.5
RUN apk add --no-cache nghttp2

# or for previous Alpine version

FROM alpine:3.4
RUN apk add --no-cache nghttp2 --repository http://nl.alpinelinux.org/alpine/edge/main --allow-untrusted

# this helps to stick libgcc/libstdc++ versions (must be run before apk add nghttp2)
RUN apk add --no-cache libgcc=5.3.0-r0 --repository http://dl-cdn.alpinelinux.org/alpine/v3.4/main --allow-untrusted
RUN apk add --no-cache libstdc++=5.3.0-r0 --repository http://dl-cdn.alpinelinux.org/alpine/v3.4/main --allow-untrusted
@tatsuhiro-t

This comment has been minimized.

Show comment
Hide comment
@tatsuhiro-t

tatsuhiro-t Dec 30, 2016

Collaborator

I confirmed that nghttpx crashes with bus error on alpine 3.5.
https://bugs.alpinelinux.org/issues/5389 may be related to this issue.

Collaborator

tatsuhiro-t commented Dec 30, 2016

I confirmed that nghttpx crashes with bus error on alpine 3.5.
https://bugs.alpinelinux.org/issues/5389 may be related to this issue.

@tatsuhiro-t tatsuhiro-t added the nghttpx label Dec 30, 2016

@tatsuhiro-t

This comment has been minimized.

Show comment
Hide comment
@tatsuhiro-t

tatsuhiro-t Dec 30, 2016

Collaborator

I rebuilt nghttp2 on alpine 3.5, and it works even with jemalloc. I found that enabling neverbleed (--with-neverbleed) causes bus error, but it came from different execution path.

Collaborator

tatsuhiro-t commented Dec 30, 2016

I rebuilt nghttp2 on alpine 3.5, and it works even with jemalloc. I found that enabling neverbleed (--with-neverbleed) causes bus error, but it came from different execution path.

@makovich

This comment has been minimized.

Show comment
Hide comment
@makovich

makovich Dec 30, 2016

Contributor

I was able to build and run nghttp2 without jemalloc. As ./configure --help shows, neverbleed feature is turned off by default, so maybe it's a falsy path. How did you build it successfully with jemalloc?

Here how my steps are looking:

FROM alpine:3.5

RUN apk add --no-cache \
        alpine-sdk \
        libtool \
        autoconf \
        automake \
        libev-dev \
        zlib-dev \
        c-ares-dev \
        libressl-dev \
        jemalloc-dev

RUN git clone --depth 1 --single-branch --branch 'v1.18.0' https://github.com/nghttp2/nghttp2.git

WORKDIR /nghttp2

RUN autoreconf -i \
 && automake \
 && autoconf \
 && ./configure \
        --disable-static \
        --without-libxml2 \
        --without-spdylay \
        --without-jemalloc \
        --disable-python-bindings \
 && make

CMD ["./src/nghttpx", "--version"]
Contributor

makovich commented Dec 30, 2016

I was able to build and run nghttp2 without jemalloc. As ./configure --help shows, neverbleed feature is turned off by default, so maybe it's a falsy path. How did you build it successfully with jemalloc?

Here how my steps are looking:

FROM alpine:3.5

RUN apk add --no-cache \
        alpine-sdk \
        libtool \
        autoconf \
        automake \
        libev-dev \
        zlib-dev \
        c-ares-dev \
        libressl-dev \
        jemalloc-dev

RUN git clone --depth 1 --single-branch --branch 'v1.18.0' https://github.com/nghttp2/nghttp2.git

WORKDIR /nghttp2

RUN autoreconf -i \
 && automake \
 && autoconf \
 && ./configure \
        --disable-static \
        --without-libxml2 \
        --without-spdylay \
        --without-jemalloc \
        --disable-python-bindings \
 && make

CMD ["./src/nghttpx", "--version"]
@tatsuhiro-t

This comment has been minimized.

Show comment
Hide comment
@tatsuhiro-t

tatsuhiro-t Dec 30, 2016

Collaborator

The only difference is that I used openssl-dev instead of libressl-dev.

Collaborator

tatsuhiro-t commented Dec 30, 2016

The only difference is that I used openssl-dev instead of libressl-dev.

@tatsuhiro-t

This comment has been minimized.

Show comment
Hide comment
@tatsuhiro-t

tatsuhiro-t Dec 30, 2016

Collaborator

Now I can run nghttpx with both jemalloc and neverbleed turned on.

Collaborator

tatsuhiro-t commented Dec 30, 2016

Now I can run nghttpx with both jemalloc and neverbleed turned on.

@makovich

This comment has been minimized.

Show comment
Hide comment
@makovich

makovich Dec 30, 2016

Contributor

Well, I'm not lucky enough as you are. :) With jemalloc included I see only SegFault/BusError dance. Are you sure that it is working on your machine? To me all looks like you definitely nailed down the root cause — musl does not allow to replace malloc implementation.

I think the issue could be closed. Maybe it would be good to add a warn note to readme's requirements section right after jemalloc point. I can make a PR if you will.

Also I'm going to ping nghttp2 Alpine package maintainer. He need to make update there.

Contributor

makovich commented Dec 30, 2016

Well, I'm not lucky enough as you are. :) With jemalloc included I see only SegFault/BusError dance. Are you sure that it is working on your machine? To me all looks like you definitely nailed down the root cause — musl does not allow to replace malloc implementation.

I think the issue could be closed. Maybe it would be good to add a warn note to readme's requirements section right after jemalloc point. I can make a PR if you will.

Also I'm going to ping nghttp2 Alpine package maintainer. He need to make update there.

@tatsuhiro-t

This comment has been minimized.

Show comment
Hide comment
@tatsuhiro-t

tatsuhiro-t Dec 31, 2016

Collaborator

It looks like neverbleed randomly crashes nghttpx, but it depends on binary, I mean that sometimes I could build working nghttpx, but sometimes not.
Since my build works with jemalloc, now I'm not sure bus error is really caused by that.
I'm using Linux host to run alpine container. I accept patches to add documentation about the possible failure with jemalloc on alpine.

Collaborator

tatsuhiro-t commented Dec 31, 2016

It looks like neverbleed randomly crashes nghttpx, but it depends on binary, I mean that sometimes I could build working nghttpx, but sometimes not.
Since my build works with jemalloc, now I'm not sure bus error is really caused by that.
I'm using Linux host to run alpine container. I accept patches to add documentation about the possible failure with jemalloc on alpine.

@tatsuhiro-t

This comment has been minimized.

Show comment
Hide comment
@tatsuhiro-t

tatsuhiro-t Dec 31, 2016

Collaborator

Right, I manged to find that the cause of bus error with neverbleed is actually due to the incompatibility between musl and jemalloc we discussed earlier. neverbleed_init uses strdup, and it looks like it uses its own malloc instead of jemalloc implementation. OTOH, free(3) uses jemalloc version, and we get bus error.
So you are right, the combination of musl and jemalloc is the culprit.

Collaborator

tatsuhiro-t commented Dec 31, 2016

Right, I manged to find that the cause of bus error with neverbleed is actually due to the incompatibility between musl and jemalloc we discussed earlier. neverbleed_init uses strdup, and it looks like it uses its own malloc instead of jemalloc implementation. OTOH, free(3) uses jemalloc version, and we get bus error.
So you are right, the combination of musl and jemalloc is the culprit.

@makovich

This comment has been minimized.

Show comment
Hide comment
@makovich

makovich Jan 3, 2017

Contributor

I've left a message to Alpine package maintainer. Also you can pick the #768 PR up if needed. Thank you for your grandiosus work!

Contributor

makovich commented Jan 3, 2017

I've left a message to Alpine package maintainer. Also you can pick the #768 PR up if needed. Thank you for your grandiosus work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment