Problems while trying to get SG into production in Ubuntu

[dariocravero]

Hi @stakach,

How're things?

Found a weird bug when trying to get SG in production you can probably help with: whenever a route is missing (say, a file wasn't included or something) it will just fail with the following error message:

connection error: data received after completed connection: close message (CLOSED_CONNECTION)

The server will hang up failing with the same error for any upcoming connections.

This might be an issue that goes beyond SG though. I've tried using libuv in GH's master, the latest version (0.11.21) and the compiled version that comes with the gem. I always get the same issue.
The production environment is (uname -a) Linux uxtemple 3.8.0-29-generic #42~precise1-Ubuntu SMP Wed Aug 14 16:19:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux.

I get the same error on my Mac (although it doesn't hang the server, it just shows it when you're quitting it) when using the latest libuv. When I use the compiled version that comes with the libuv gem it doesn't show it.

Would you have any idea of what might be going on here?

Thanks,
Darío

[stakach]

That probably is a spider gazelle issue.
The error comes out of the http-parser when a connection is re-used after a Connection: Close header has been sent to the client. So an issue with persistent connections.

I'm running in production on the 0.1.5 release without issue. Does that version work for you? Just so we can isolate the problem.

[stakach]

The error could be here: https://github.com/cotag/spider-gazelle/blob/master/lib/spider-gazelle/gazelle.rb#L110

We should replace @parser_cache << connection.state with

state = connection.state
state.reset!
@parser_cache << state

Do you want to see if that fixes the issue?

[dariocravero]

Amazing! That fixes it right away :) Thanks a lot!

[dariocravero]

There's still something weird going on when you close the application, perhaps connections need to be closed too or something? This is what I'm getting after shutting down SG:

ruby: src/unix/loop.c:93: uv_loop_delete: Assertion `uv_loop_close(loop) == 0' failed.

It's the same message I got while trying to run the tests on libuv, see this. They sorted it out by closing the loop before deleting it.

Thoughts?

[stakach]

Awesome! Was the only real place I could think of that may be causing the issue.

Hmm. How are you shutting down SG? Close is definitely called before delete, although there are still handles on the loop that would be open.

Possibly related to this? https://github.com/cotag/spider-gazelle/blob/master/lib/spider-gazelle/gazelle.rb#L150

Basically once we signal shutdown we need to wait for all the connections / requests to complete and then stop the loop. We'll also use the same process for live migrations.