glibc: corrupted double-linked list

[phillipp]

In one of your deployments we saw that all SSL websites stopped working. Attempts to connect failed:

curl -v https://www.[foobar].net/
* About to connect() to www.[foobar] port 443 (#0)
*   Trying 91.216.248.**... connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to www.[foobar].net:443 
* Closing connection #0
curl: (35) Unknown SSL protocol error in connection to www.[foobar].net:443 

The bud log showed the following output:

(wrn) [9845] client 0x4c9acd0 on frontend SNI from json failed: "Failed to load or parse JSON: <SNI Response>"
*** glibc detected *** bud: corrupted double-linked list: 0x00000000116daa00 ***

Kernel ring buffer/dmesg:

[345812.084141] traps: bud[3256] general protection ip:5cad48 sp:7fffdcf1ca30 error:0 in bud[400000+2ae000]

A rebstart of bud solved the problem. What do you need to dig further into the problem?

[indutny]

@phillipp thanks for reporting this!

Do you have a core dump of this crash?

[phillipp]

Unfortunately not. What would be the best way to setup this up for the next time?

[odeke-em]

@phillipp if it is possible, would you mind giving the config or a redacted config. I just would like to be able to reproduce this.

[indutny]

@phillipp I would do something like http://askubuntu.com/questions/53956/how-can-i-enable-core-dump . Please be aware that it will create large files per each crashing process. If you don't have much of them - it should not be a big deal, though.

[phillipp]

The config is:

{
  "workers": 4,

  "restart_timeout": 250,

  "log": {
    "level": "info",
    "facility": "user",
    "stdio": true,
    "syslog": true
  },

  "availability": {
    "max_retries": 5,
    "retry_interval": 250,
    "death_timeout": 1000,
    "revive_interval": 2500
  },

  "frontend": {
    "interfaces": [
      { "port": 443, "host": "::" }
    ],
    "keepalive": 3600,
    "server_preference": true,
    "cert": "default.crt",
    "key": "default.key"
  },

  "user": "bud",
  "group": "bud",

  "backend": [{
    "port": 10010,
    "host": "127.0.0.1",
    "keepalive": 3600,
    "x-forward": true
  }],

  "sni": {
    "enabled": true,
    "port": 9000,
    "host": "127.0.0.1",
    "url": "/bud/sni/%s"
  },

  "stapling": {
    "enabled": false,
    "port": 9000,
    "host": "127.0.0.1",

    "url": "/bud/stapling/%s"
  },

  "contexts": []
}

[phillipp]

We saw this error message again and despite bud is running with the ulimit here was no core dump taken. Will there really be taken a core dump on this error? bud is not crashing, just showing the message and behaving incorrectly after.

[indutny]

@phillipp is it always preceded by that JSON warning? Do you have multiple logs for this?

[phillipp]

Today we had another one. bud continues to run, as before.

(dbg) [24208] client 0x82dabe0 on backend ssl_cert_cb {2}
(dbg) [26505] client 0x11ff1f70 on frontend SSL_read() => -1
(dbg) [26505] client 0x11ff1f70 on frontend uv_write(1802) iovcnt: 1
(dbg) [26505] client 0x11ff1f70 on frontend immediate write
(dbg) [26505] client 0x11ff1f70 on frontend write_cb => 1802
(dbg) [26505] client 0x11ff1f70 on backend read_start
(dbg) [26505] client 0x11ff1f70 on frontend recycle
(dbg) [26505] client 0x11ff1f70 on frontend SSL_read() => -1
(dbg) [24208] client 0x82dabe0 on frontend SSL_read() => -1
(dbg) [24208] client 0x82dabe0 on frontend uv_write(5838) iovcnt: 1
(dbg) [24208] client 0x82dabe0 on frontend immediate write
(dbg) [24208] client 0x82dabe0 on frontend write_cb => 5838
(dbg) [24208] client 0x82dabe0 on backend read_start
(dbg) [24208] client 0x82dabe0 on frontend recycle
(dbg) [24208] client 0x82dabe0 on frontend SSL_read() => -1
*** glibc detected *** bud: corrupted double-linked list: 0x00000000084fe560 ***
(dbg) [26506] client 0x12dd6010 on frontend SSL_read() => -1
(dbg) [26506] client 0x12dd6010 on frontend uv_write(5275) iovcnt: 1
(dbg) [26506] client 0x12dd6010 on frontend immediate write
(dbg) [26506] client 0x12dd6010 on frontend write_cb => 5275
(dbg) [26506] client 0x12dd6010 on backend read_start

Core dumps are enabled and apport is running, but didn't save a report. So I'm thinking this is just a glibc warning/error. The core may not be dumped because the program is not crashing.

SO answers suggest that there is a race condition where multiple callers free an object at the same time.

[indutny]

@phillipp the problem is that I don't use threads in bud at all. I have one idea that might be worth exploring.

Let's try building bud with Address Sanitizer. I have just pushed 1c45275 to the master branch. Please give a try to it:

./gyp_bud asan
make -C out/ -j8
./out/Release/bud # <- binary

[indutny]

Ideally I expect it to give us more information on where this error is happening.

[indutny]

Thanks!

[odeke-em]

Ooh this is so cool, I am acquainted with one or two of the Address Sanitizer authors so can't wait to see the results of using it.

[indutny]

@phillipp I believe we may have fixed this in the recent release.

[phillipp]

                                                                                  I think I'll again grep the logs for this just to be sure. You think this could be related to the use-after-free?

[indutny]

@phillipp it is very very likely. Please let me know if we haven't fixed it!

[phillipp]

Yes, now I see that this looks like the same problem. LGTM!

[indutny]

Fantastic! Still no crashes, right?

[phillipp]

                                                                                  None so far. On none of the servers. Looks great, I think it's fixed! Thanks for the effort!

[indutny]

Hooray! Two issues in a row 👍