Introduce tooldaq.js -- a new API for the web server

[jini-zh]

Introduce tooldaq.js which is intended to replace functions.js eventually. It should be able to do everything that functions.js does now, but with error handling and hopefully more concise interface. See docs/tooldaq.mkd for the documentation.

I may have overengineered the makeTable function. Let me know what you think about it.

New files and changes to old ones:

• cgi-bin/command.cgi is used to send commands. It does the same thing as sendcommand.cgi, but emits no HTML code, just the command reply as plain text, and also returns code 500 with an error message if something goes wrong.
• cgi-bin/db-query.cgi is used to talk to the database. It can return data in JSON or in CSV format, both of which can be handled in tooldaq.js. CSV format should use less bandwidth, but sometimes JSON is more convenient. Hopefully it will replace sqlquery.cgi, sqlqueryjson.cgi and sqlquerystring.cgi.
• There is now services.txt in html-Detector and html-Standalone* which is a symlink to /tmp/table_file. That way httpd doesn't need to execute a CGI script to distribute the table. This should make tablecontent5.cgi obsolete.
• cgi-bin/backgroundSD2 was changed to not emit Content-type header and HTML code. They are ignored anyway in functions.js's GetSDTable, so it doesn't break old code. Now the table file (services.txt) can be served as plain text.

[marc1uk]

in command.cpp i think it would be nice to be able to turn debugging on/off when there are issues at runtime without needing to re-compile. Perhaps it could even be a flag passed to the command, as that would let you ensure you get debug logs for the command you're interested in, rather than just whatever happened to be the last command executed. The latter is fine when any command will do, but if it's a specific query, it's difficult to catch (in fact that's why my bash versions always append to a common log rather than replacing it).

[marc1uk]

• Sorry this one's probably on me, but remove the ZMQ_IMMEDIATE setting for the socket - I believe the correct option is in fact ZMQ_ROUTER_MANDATORY, so perhaps you could instead swap to that, although it isn't yet in WCTE and should probably be tested there first.

• I realise we have boost available, but since we're considering dropping it, is it possible/preferable to use strftime instead of boost::posix_time::to_iso_extended_string

• is it worth checking the return code of the psql call as well? That might be more robust than a regex on the output string, especially if there are errors not specifically from postgres.

• i wonder if we should modularise things a little, once you start to get 1,000 lines of code in one file it's hard to keep track... this doesn't include the ROOT plotting functions, so those will need to go in as well...

[jini-zh]

in command.cpp i think it would be nice to be able to turn debugging on/off when there are issues at runtime without needing to re-compile.

Done.

Sorry this one's probably on me, but remove the ZMQ_IMMEDIATE setting for the socket - I believe the correct option is in fact ZMQ_ROUTER_MANDATORY, so perhaps you could instead swap to that, although it isn't yet in WCTE and should probably be tested there first.

I have removed ZMQ_IMMEDIATE, but setting ZMQ_ROUTER_MANDATORY makes zmq fail with "Invalid argument" error message. This option appears to be for pub sockets only while this is a req socket.

I realise we have boost available, but since we're considering dropping it, is it possible/preferable to use strftime instead of boost::posix_time::to_iso_extended_string

Done. I used std::chrono functions for that.

is it worth checking the return code of the psql call as well?

I considered that. The problem is that we can check the return code only after psql has returned and closed its output stream, while we need to set the status code at the beginning of our output. That means that we need to capture psql output in a variable first, and it can be very large. Current implementation checks the first line of the stream and pipes the rest to the client. If the client decides that they don't want all of the data, they can close the stream and the query will stop midway. I think the way it works now is well enough. If you want it more robust, then in my opinion it's better to implement it in C++ rather than bash. I can do that when we have a problem with it.

i wonder if we should modularise things a little

Good point. I am thinking of abstracting plot functions to a class that would govern a plot on a div. It could provide methods for updating and modifying the plot. A similar class should probably be written for services. They can be moved out to separate modules. However, I must work now on other projects, so this will have to wait for a while.

[brichards64]

delete standalone 2