laminard crashing #146
Comments
|
Hi! Commented out memory checking in kj/memory.h, rebuild with -g -O0 and got (heavy log output, following in browser): Guess it's the missing disposer? But I fail to gain overview over the overall code flow. |
|
Thanks for the detailed report. Unfortunately I can't reproduce it locally, but I have a potential fix for your last backtrace. Since defer-initial-log-notification-until-fulfiller-instantiated.patch.txt |
|
Hi! With this fix in place, laminard doesn't crash any longer. The frontend feels a bit "weird", though. It seems that, while such a massively sending run is active, the chunked transfer towards the web client isn't served? In real life, that's probably not a problem, because even for my current jobs (builds for gcc/binutils/glibc), there are times of high-volume logging ( |
|
That last patch turned out to be wrong, it causes the whole log to be sent out to all clients when a new client connects. It looks like in general the multiple simultaneous log clients was broken anyway, hopefully is fixed in this patch: logwatcher-ensure-fulfiller-instantiated-before-returning-to-eventloop.patch.txt Please let me know if this one works for you too.
I can't think of a good reason for that, I could speculate that laminar is too busy reading (it is highly asynchronous, but single threaded) the output in order to write it in time. Perhaps tuning buffer sizes could improve this; I agree it's probably not a real issue. |
|
While your first patch worked well for me, the second (in #146 (comment)) does not. With this applied, watching a binutils' log live on the web frontend, it gets intermixed with the (newly introduced) "admin" job (generating links and queueing all jobs.) |
|
Oh and it seems I had one crash with the second patch compared to no crash with the first one. |
|
Weird, I guess I need to think about this patch some more. Any chance you have a backtrace of the crash? |
|
Unfortunately, no backtrace. I gave it a try (actually: my base is the Debian source package, so I applied my recursive-artifacts patch and your second no-crash-on-web-watching) and had this one crash. Thereafter, I changed it back to your first patch. That test box is off right now, but I'll "revert" to your second patch asap and attach gdb. |
|
Now running again with your second patch applied, under What I quite recognize is unresponsiveness in the web frontend. However, opening a good number of live logs in several tabs results in "loading nothing" or "just loading the page itself without the log". Once a tab has picked up following the log, it seems to properly follow all the time. Is |
|
So no crash so far with the second patch? From the strace output and your description it sounds like only the pipes to the job output are being serviced, and the connections from the frontend are neglected. I can't see any reason for this...
Nope. At least not intentionally. It might be interesting to try a compile and run against the latest capnproto - debian is still shipping 0.7.0 even in sid |
|
No crash again up to now. ...and for watching at a goot number of parallel build logs in the web browser, I think that there's simple explanation: Web browsers limit the number of connections (overall and towards one host), so I guess that limit just applied. |
|
I'll consider the crash fixed in b16991b, please reopen if you can reproduce. |
|
Just FTR, I had another unexpected crash/exit with this message in laminar.log:
What is a bit unfortunate is that right now, it doesn't pick up on queued runs. Aren't these just in the database? So recovering could be done by pulling outstanding runs from the database? |
|
Got hit by that again. So now running with tcpdump on port 80 and with GDB attached, breaking on abort, exit and _exit. Lets see what we can catch. (So my current setup has about 770 jobs defined, and I think I didn't ever make it through all of them by just queueing them all in and waiting for completion. Working around that by checking the database for the least run jobs and starting them. But in the end, I'll of course want to get that up'n'running properly.) So recovery after crash (re-starting queued jobs) would be high on my wish list right now. |
|
Thanks for your help debugging this, it will be nice to get to the bottom of it. Regarding restarting queued jobs, I don't particularly like the sound of this. Laminar could be down for quite a while (e.g. power cut) and trying to restart its queue could just make a big mess if the resources the job scripts expect have gone stale in the meantime. Anyway it isn't quite as simple as you suggest, because the database does not store enough information to reschedule the queued run. I would prefer to just fix the bug and leave the existing behaviour with queued jobs. Crashes like the one you describe I think are unacceptable. |
|
I yet have to understand it, but I guess it's a bug in libcapnp. It breaks in src/kj/async-io-unix.c++:301, but I still have to understand how to trigger it. |
|
My guess is that this is happening from kj's http code, but I don't know (cf. capnproto issue 1219) who should fix it. |
|
I must admit that I horribly fail to get the kj / libcapn code. Next gdb round trying to catch this. I really hope to get to trace down these kj-generated exceptions on closed network FDs. Right now, running on the small box, it'll only loose the two running jobs, it'll soon be more like 32 ... |
|
Are you sure the crash is due to that message? I suspect what you are seeing is this, which is not actually fatal despite "fatal" being in the error message... Anyway a backtrace would be needed to investigate this further |
|
This time not a crash, but rather a
Unfortunately, no complete stack trace since GDB had issues with the DWARFs... But maybe you can make something out of it (ie. a non-catched exception?) |
|
That was, by the way, along with a
Notice the stray following output of "laminard version 1.0 started"? Odd, but haven't looked at the sources. |
|
I pushed 0281030 to deal with the exit() - I didn't think that a dropped http client would bubble up to that exception handler, but otherwise all attached resources should be freed because they are attached to the promise. So I just made the exception handler non-fatal.
Laminard Anyway that would be the obvious explanation why you would see "laminard version 1.0 started" - because |
|
So first day/night of operation. Good news: laminard didn't crash/exit. Bad news: It closed it's port 8080 listening socket. No debugger attached, this is the log: |
|
Got bitten by it again: It doesn't |
|
Note to self: need to write a test case to reproduce this |
|
I have a sliiight hint maybe: I think this happens when I have the main page opened in my mobile phone (chrome browser) and just leave the browser and put the mobile phone to sleep (short press on the on/off button.) It seems to not happen when I haven't touched the laminar server with my mobile phone after having it restarted. |
|
Yeah, I'm quite certain by now: This exception (formerly leading to an exit, now leading to the 8080 fd being closed) gets triggered when I keep the Laminar dashboard open and (common in Germany) my IPv4 router gets a new external IP address assigned. The former address will be either not handed out within a short timeframe (would result in a timeout or filling buffer during send), or should lead to a RST (new IPv4 address holder doesn't know about the former connection.) |
|
Tested this two more nights: It seems to be as I expected. When there's an open connection (event-stream) and my external IP address changes, the server closes its listening port 8080 after some time. So any client can probably lead to a denial-of-service situation with simply going offline without shutting down the TCP connection first. |
|
I tried to reproduce this by reducing |
|
My brain really bends looking at the C++'ized approach to the POSIX API. I think my next try will be to find out where the listening socket can be closed and break on all those places (guessing it's only one or two actually.) But yeah, it's hard to trigger but quite fatal once it happened. Just to document it: To not loose the currently running run(s), I'm setting the default workers to zero once I'm locked out through the API and wait for the runs to settle. Then restart and re-queue what was lost. |
Hi!
I'm a novice laminar user and noticed that for me, laminard quite easily crashes. This is with Debian unstable (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983645), which is quite up-to-date with upstream. However, as this seems to be not Debian specific, I came here to discuss it.
Basically, I see that starting a job with lots of output:
(with
/tmp/linesexisting) is enough to do. With such a job running, direct your browser to follow the log (ie. http://localhost:8080/jobs/lines/4). It'll take a second or two to crash for me.On a systemd-driven system, laminard will probably be restarted, but the run is lost.
As for the bug, I wasn't able to track it down yet. My guess is that it somehow overflows its read buffer (
readDescription()) crushing therunpointer before it callshttp->notifyLog()inlaminar.cpp, but I may be wrong on that. I am, however, quite confident that it can easily be triggered with a job emitting lots of output and viewing the log online.Thanks a lot,
Jan-Benedict Glaw
The text was updated successfully, but these errors were encountered: