Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
[dev.icinga.com #9322] sending multiple Livestatus commands rejects all except the first #3038
This issue has been migrated from Redmine: https://dev.icinga.com/issues/9322
Created by sni on 2015-05-27 08:33:08 +00:00
Livestatus normally accepts multiple commands at once and executes them ond after another. However, it seems like Icinga2 only reads the first command from the socket and discards all other commands. It would be nice to support multiple commands with a single connection.
2015-10-16 13:12:05 +00:00 by (unknown) a9f14f1
2015-10-16 13:16:48 +00:00 by (unknown) 6b9013c
Updated by iosif69 on 2015-05-28 07:44:03 +00:00
the same behavior, Icinga2 (v2.3.4), thruk 1.88-4
Updated by sni on 2015-05-28 09:01:19 +00:00
So it's a string of commands, separated by two newlines. Something like this should work:
Updated by mittma on 2015-10-06 07:45:00 +00:00
Same problem here!
PS: My programming experience in C** is completely rusted, but it seems to me that LivestatusListener::ClientHandler in livestatuslistener.cpp is the guilty one. There in the second for-Loop (reading each line) you have the following code:
So, if the line has no content, reading stops. Stupid question: is it possible to use "continue" instead of "break"? It may work, since there is an additional check for Eof I think:
Please don't be too harsh with me if that's complete nonsense, but I clutch at every straw, because I want to use Icinga 2 AND Thruk together!
Updated by tomwork on 2015-10-07 00:48:16 +00:00
Well, I came with the same conclusion yesterday, like you I read the same code and I have the same problem about C** skills. If I find the time I will try to hack things around to try to make it work. Trial and error ;-)
Desperate, I also tried to tell thruk to only send one \n instead of two (so no empty lines) but that does not work either. Thruk being coded in Perl it was easier to try. Recompiling icinga2 is a different exercise, hopefully, there is a makefile to just rebuild the livestatus library instead of the whole shebang.
Updated by krausm on 2015-10-07 06:32:52 +00:00
We are currently working on integrating Icinga2 within "OMD Labs Edition" (see https://labs.consol.de/de/omd/ for details), using Thruk as its main user interface.
Thereby sni found this issue, and we already discussed it with dnsmichi.
According to him, the culprit is here: https://github.com/Icinga/icinga2/blob/master/lib/livestatus/livestatuslistener.cpp#L195
He outlined as a possible solution, to change the break and store the request into a map (request1, lines, request2, lines), that can afterwwards be processed within a loop.
Limiting factors currently are either C** skills or time ...
Updated by mittma on 2015-10-07 07:57:13 +00:00
thanks a lot for the info! The OMD Labs Edition surely looks promising and I will have a closer look on it. I didn't know that mod_gearman is possible with Icinga2...
Yes, both can be found rather sparsely...
Updated by tomwork on 2015-10-09 01:19:37 +00:00
Below is the livestatus LQL language description. As we all already know, what Thruk does is correct. However icinga2 stops processing once it find an empty line which actually goes against the LQL "protocol". We can also imagine a quick and non performing hack on the Thruk side where each command is sent separately instead of a series of commands but that's an ugly workaround.
"Just as with GET, a query is terminated either by closing the connection or by sending a newline. COMMAND automatically implies keep alive and behave like GET when KeepAlive is set to on. That way you can mix GET and COMMAND quries in one connection. "
Updated by krausm on 2015-10-09 13:28:26 +00:00
TL;DR: my current progress is "It would seem to work, BUT..."
Thruk now seems to behave as expected in short tests, i.e. multiple commands work.
BUT, there are exceptions occurring in the debug log:
The livestatus query seems to terminate the stream, while someone still tries to read or write to/from it.
Does anyone have an idea, how to solve the stream handling? My knowledge about c** and boost is way too little, to be helpful here...
Updated by tomwork on 2015-10-16 07:07:30 +00:00
I tried your logic and yes it works but there is this 'bad file descriptor' error. I believe that's because it hits EOF and Stream::Readline still tries to read from an empty socket or something like that I guess. I believe that without this exception it may not exit the for loop anyway. The 'else break' is the for loop exit door. You hijack'd it, it works but I cannot understand why.
I tried different ways of my own, sadly, I couldn't make it work nicer. Also having these magical object does not help because there is a fair bit of logic in the stream readline as well so that could be where the problem comes from. Another way avoiding C** code changes would be to build a unixsock proxy to intercept thruk's '_bulk_send' commands and split them in different command so icinga2 sees them as separate COMMAND.
Also, I wanted to come back about something I said earlier, the LQL says "a query is terminated either by closing the connection or by sending a newline." That's what icinga2 does. So the bulk_send done by Thruk is incorrect or is meant to be managed via different connections on the receiving end then but I am no expert, just a livestatus newbie trying to have it a go.
To debug livestatus I came with the following to see what's going on.
I think I will stop there for now, I burnt a day trying different (random;) things. I may try another day or build a 'proxy'.
Updated by gbeutner on 2015-10-16 13:14:35 +00:00
Wow, you're going to hate the patch for this one - considering how much effort you've already put into this.
Icinga uses ReadLine() to read data from the socket. However, because we don't know exactly where the line ends we just read a 'few' bytes - and put whatever data we can't process right now (i.e. after the first line) into a data structure called StreamReadContext. The problem in this case is that the StreamReadContext variable is declared inside the for loop and destroyed after the first query, i.e. we're losing whatever data we've read from the stream. Oops.
The fix is to move the StreamReadContext variable outside of the loop.
Updated by cmh on 2015-12-11 12:35:09 +00:00
I'm still seeing issues with this, unsure if it should be re-opened. If I select many services and apply a comment, most of them will get the comment, but there is usually one, sometimes more - at the end which doesn't get the comment. I've started copying the comment, selecting all that I want, and applying the comment, then seeing which ones didn't get it, selecting those, applying the comment again. Sometimes if I choose enough (>20?) at once I'll have to do this process 3-4 times.