Add a scoreboard /server-status function

[robbat2]

Please add a /server-status scoreboard functionality like Apache's scoreboard, so a sysadmin can have insight into the request processing (and why all threads might be stuck).

This would be used in Ceph's RadosGW per upstream issue http://tracker.ceph.com/issues/12102

[bel2125]

As to my knowledge, scoreboard is a Perl library, used to generate "server status" pages for the admin, like http://apache.org/server-status.

Are you looking for a similar interface to get some diagnostic data while the server is running?
In a scripting language? Lua? CGI? Or a C API?

This diagnosis is only working if the server is working and still ready to process a request to the server status page. There would be also ways for diagnosis if the server is not responding anymore, but of course this can not be a status page generated by the server.

Is there a specific problem you need to investigate? What diagnostic information would be most valuable for you?

I have never used Ceph, so I don't know the Ceph diagnosis infrastructure. Of course, a new civetweb diagnosis infrastructure should be suitable for as many applications as possible. In particular, it should be also suitable for embedded systems and I would prefer if I also get some data if the server can not handle any request (anymore).
How is the diagnosis for Ceph working? Is it also some server status page?

[robbat2]

Ceph RadosGW has no scoreboard or status-page of it's own presently; The Ceph OSDs do have a means of inspecting operations in flight.

Apache scoreboard is a mod_perl module that generates & parses /server-status like-data, even when /server-status itself is disabled. It has a server-side portion (plus a small C-only APXS module) that serves up at /scoreboard, plus a consumer portion that reads said data.

CivetWeb could implement either the /scoreboard functionality [1] or the /server-status functionality; either would be useful to me.

Here's the small C code:
http://cpansearch.perl.org/src/MJH/Apache-Scoreboard-2.10/apxs/send.c

/server-status is probably more useful to other people, as you can quickly view it as a sysadmin, whereas /scoreboard is a binary blob that needs special parsing.

[bel2125]

I think a scoreboard / server-status is important.
I plan to do a release in the next days, and take care about this topic after the release.

[jharbott]

Oops, didn't want to spam the issue here. What I did (after giving up on trying to cherry-pick only these particular patches) is take the master branch up to the Step 20 patch and rebase it onto the fork that Ceph is currently using, which only gave me a moderate number of conflicts. Now I can at least create a server stats output on shutdown, will need some more integration work in order to end up with a status page that can be read during normal operations. Sample data looks like this currently:

{
"memory" : {
"blocks" : 104,
"used" : 146472,
"maxUsed" : 146472
},
"connections" : {
"active" : 0,
"maxActive" : 1,
"total" : 3
},
"requests" : {
"total" : 6
}
}

See https://github.com/x-ion-de/ceph/tree/wip-listen4-server-status for what I have done so far.

@bel2125 Any idea when 1.10 will be released? Seems easier to have Ceph folk adapt this from a stable version.

[bel2125]

@bel2125 Any idea when 1.10 will be released? Seems easier to have Ceph folk adapt this from a stable version.

My intention was to do this before my summer vacation, but this won't be possible. So, somewhere in August.

I had a look at this issue ~2 days ago, and was still not sure if it can be considered "completed".
What do you think? Is some important information missing?
Yet more important: Do you think the output format is the right choice - did it integrate well for your status page? Or do you just show the plain text output? Do you have a demo of a real page running somewhere?
One idea was also to use a Lua websocket state with a timer, to collect historical data (to plot connections over time).

[jharbott]

Using json as output format is fine for me. The intention in my case is not to integrate the stats into a visible page directly, but have the data polled by some monitoring system like prometheus in order to see graphs of e.g. the number of requests over time. Currently we have to parse the log files for this.

Regarding the selected information, maybe adding uptime and bytes sent/received would be good.

[bel2125]

This topic was postponed for more than a year, but now I think now we reached a point where all information required for a site like http://apache.org/server-status is available.
Probably there are another thousand things one could add to the server diagnostics data, and maybe I will add some, but I don't think it makes sense to track all of them with this GitHub issue (#243).

[bel2125]

This is the currently available data:

{
    "system":{
        "version":"1.10",
        "os":"Linux #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 (4.4.0-64-generic) - x86_64",
        "features":4095,
        "feature_list":"Server: Files HTTPS CGI IPv6 WebSockets Lua JavaScript Cache",
        "lua_version":"502 (Lua 5.2.4)",
        "javascript":"Duktape 1.5.2",
        "build":"Aug 6 2017",
        "compiler":"gcc: 5.4.0",
        "data_model":"int:2/4/8/8, float:4/8/16, char:1/4, ptr:8, size:8, time:8"
    },
    "summary":{
        "memory":{
            "blocks":1466,
            "used":2895494,
            "maxUsed":2895494
        },
        "connections":{
            "active":2,
            "maxActive":2,
            "total":8
        },
        "requests":{
            "total":6
        },
        "data":{
            "read":0,
            "written":14651
        },
        "time":{
            "uptime":32,
            "start":"Sun, 06 Aug 2017 20:35:12 GMT",
            "now":"Sun, 06 Aug 2017 20:35:44 GMT"
        }
    },
    "common":{
        "memory":{
            "blocks":84,
            "used":821969,
            "maxUsed":847564
        }
    }

Created by the Lua test page: http://localhost:8080/page_status.lua (if built WITH_LUA=1), and there is more data if MG_EXPERIMENTAL_INTERFACES is activated.

I'm closing this issue now. If someone is missing something important, feel free to either post a comment here (within the next month) or open a new issue (any time).