bud leaks file descriptiors (heavily) #78
Comments
|
I have added some more debug output from my previous "investigation" and found that all connections that lead to the log lines in the last example (the very show connections that do nothing and abort) are from the networks of a single entity. We rejected all traffic from their nets and the file descriptors are no longer increasing. So there seems to be an easily exploitable DoS attack vector, regardless if the client was aware and actively using it or not. Any ideas what might cause this? |
|
@phillipp have you tried lowering |
|
@phillipp I think I know what's happening. Will fix it in a bit, not sure how I missed it. |
|
@phillipp may I ask you to give a try to this patch? diff --git a/src/client.c b/src/client.c
index 9ff97e5..c097ade 100644
--- a/src/client.c
+++ b/src/client.c
@@ -968,8 +968,7 @@ void bud_client_shutdown_cb(uv_shutdown_t* req, int status) {
/* If either closing, or shutdown both sides - kill both sockets! */
} else if (side->close == kBudProgressRunning ||
client->frontend.shutdown == client->backend.shutdown ||
- (side == &client->frontend &&
- !client->config->frontend.allow_half_open)) {
+ !client->config->frontend.allow_half_open) {
bud_client_close(client, bud_client_ok(side));
}
} |
|
Hm... actually, this seems to be too aggressive. |
|
@phillipp how about this? diff --git a/src/client.c b/src/client.c
index 9ff97e5..34fe283 100644
--- a/src/client.c
+++ b/src/client.c
@@ -971,6 +971,16 @@ void bud_client_shutdown_cb(uv_shutdown_t* req, int status) {
(side == &client->frontend &&
!client->config->frontend.allow_half_open)) {
bud_client_close(client, bud_client_ok(side));
+ } else if (side == &client->backend &&
+ client->backend.reading == kBudProgressNone) {
+ bud_client_error_t cerr;
+
+ DBG_LN(&client->backend, "read_start after shutdown");
+ cerr = bud_client_read_start(client, &client->backend);
+ if (bud_is_ok(cerr.err))
+ client->backend.reading = kBudProgressRunning;
+ else
+ bud_client_close(client, cerr);
}
}
diff --git a/test/basic-test.js b/test/basic-test.js
index 217da81..25506a9 100644
--- a/test/basic-test.js
+++ b/test/basic-test.js
@@ -1,5 +1,6 @@
var assert = require('assert');
var https = require('https');
+var net = require('net');
var fixtures = require('./fixtures');
var request = fixtures.request;
var caRequest = fixtures.caRequest;
@@ -125,4 +126,16 @@ describe('Bud TLS Terminator/Basic', function() {
});
});
});
+
+ describe('EOF on frontend', function() {
+ var sh = fixtures.getServers();
+
+ it('should support basic termination', function(cb) {
+ var socket = net.connect(sh.frontend.port);
+ socket.on('close', function() {
+ cb();
+ });
+ socket.end();
+ });
+ });
}); |
|
Great, we've deployed the new version and will see if the problem pops up again. |
|
Thanks! |
So a few hours ago we got the first symptoms of a new problem where clients can't connect to bud anymore. The debug output from cul looks like this:
Bud debug output says:
Other workers seem to work normally. Over time more and more of the workers start to do the same.
So I was thinking that some problem occurred in the requests before and left the worker in a bad state, and found that bud opens too many files (output from the same worker before):
So as this problem started only after #76 was fixed, maybe the fix introduced a new bug that leads to HTTP requests no longer closed and thus leaking file descriptors. Could that be possible, that the fix introduced that problem? Right now the problem seems to emerge very fast as we see a worker hit the ulimit (1024) in only a couple of minutes.
It is possible though that there is another problem as I see strange debug output from the workers:
So something is going on there that may be the cause of the leak. Any ideas?
The text was updated successfully, but these errors were encountered: