No Monitor status when ceph cluster only have one monitor.

[mkkie]

Hello, this is my ceph cluster.

$ ceph -s
    cluster 7530a176-e733-4b76-bc66-76c516157960
     health HEALTH_OK
     monmap e1: 1 mons at {B08-1-MS1=172.28.0.30:6789/0}
            election epoch 2, quorum 0 B08-1-MS1
     osdmap e26: 3 osds: 3 up, 3 in
      pgmap v383: 64 pgs, 1 pools, 49108 MB data, 12277 objects
            145 GB used, 10997 GB / 11142 GB avail
                  64 active+clean

$ ceph -v
ceph version 0.94.7 (d56bdf93ced6b80b07397d57e3fa68fe68304432)

When I run ./ceph-dash.py, I could not found any monitor status on Ceph Dashboard.

Here is my screenshot.

And I try to add another monitor, finally I can see monitor status.

Thanks!

[mkkie]

I think the problem is caused by restapi.
When only one monitor is running,
the json format is

"timechecks":{
"epoch":1,
"round":0,
"round_status":"finished"
}

Only after we add another monitor, it can get the value from mons.

"timechecks":{
"epoch":2,
"mons":[],
"round":2,
"round_status":"finished"
}

Maybe fix this lines would be help.

timechekMons = data['health']['timechecks']['mons'];

https://github.com/Crapworks/ceph-dash/blob/master/app/static/js/ceph.dash.js#L327

$.each(timechekMons, function(index, mon_health) {
                    if (mon['name'] == mon_health['name']) {
                        health = mon_health['health'];
                    }
                });

https://github.com/Crapworks/ceph-dash/blob/master/app/static/js/ceph.dash.js#L428

Thanks.

[Crapworks]

Hi @mkkie ,

can you send me the full json output with one monitor? I don't currently have a possibility to look at it myself. If I see the difference in the json I can maybe fix this. The point is, I need to see if there actually is one monitor, and I don't know where to find the information elsewhere without the full json.

Thanks,
Christian

[mkkie]

Hello @Crapworks ,

I'm using ceph version 10.2.2 now, but the problem still there.

Here are the json files:
1. Only one monitor

{
    "election_epoch": 3,
    "fsid": "eab2be8a-90c8-4919-9384-c6e2742c3956",
    "fsmap": {
        "by_rank": [],
        "epoch": 1
    },
    "health": {
        "detail": [],
        "health": {
            "health_services": [{
                "mons": [{
                    "avail_percent": 94,
                    "health": "HEALTH_OK",
                    "kb_avail": 1818661196,
                    "kb_total": 1920339980,
                    "kb_used": 4108120,
                    "last_updated": "2016-10-14 01:56:39.715564",
                    "name": "node-c76b2c",
                    "store_stats": {
                        "bytes_log": 3726788,
                        "bytes_misc": 1178,
                        "bytes_sst": 0,
                        "bytes_total": 3727966,
                        "last_updated": "0.000000"
                    }
                }]
            }]
        },
        "overall_status": "HEALTH_OK",
        "summary": [],
        "timechecks": {
            "epoch": 3,
            "round": 0,
            "round_status": "finished"
        }
    },
    "monmap": {
        "created": "2016-10-14 01:53:08.712856",
        "epoch": 1,
        "fsid": "eab2be8a-90c8-4919-9384-c6e2742c3956",
        "modified": "2016-10-14 01:53:08.712856",
        "mons": [{
            "addr": "192.168.33.31:6789/0",
            "name": "node-c76b2c",
            "rank": 0
        }]
    },
    "osdmap": {
        "osdmap": {
            "epoch": 28,
            "full": false,
            "nearfull": false,
            "num_in_osds": 5,
            "num_osds": 5,
            "num_remapped_pgs": 0,
            "num_up_osds": 5
        }
    },
    "pgmap": {
        "bytes_avail": 19988592058368,
        "bytes_total": 19988796190720,
        "bytes_used": 204132352,
        "data_bytes": 0,
        "num_pgs": 256,
        "pgs_by_state": [{
            "count": 256,
            "state_name": "active+clean"
        }],
        "version": 47
    },
    "quorum": [0],
    "quorum_names": ["node-c76b2c"]
}

2. Two monitors

{
    "election_epoch": 4,
    "fsid": "eab2be8a-90c8-4919-9384-c6e2742c3956",
    "fsmap": {
        "by_rank": [],
        "epoch": 1
    },
    "health": {
        "detail": [],
        "health": {
            "health_services": [{
                "mons": []
            }]
        },
        "overall_status": "HEALTH_OK",
        "summary": [],
        "timechecks": {
            "epoch": 4,
            "mons": [{
                "health": "HEALTH_OK",
                "latency": 0.0,
                "name": "node-c76b2c",
                "skew": 0.0
            }, {
                "health": "HEALTH_OK",
                "latency": 0.525225,
                "name": "node-70dcb4",
                "skew": 0.0
            }],
            "round": 2,
            "round_status": "finished"
        }
    },
    "monmap": {
        "created": "2016-10-14 01:53:08.712856",
        "epoch": 2,
        "fsid": "eab2be8a-90c8-4919-9384-c6e2742c3956",
        "modified": "2016-10-14 01:57:24.478937",
        "mons": [{
            "addr": "192.168.33.31:6789/0",
            "name": "node-c76b2c",
            "rank": 0
        }, {
            "addr": "192.168.33.32:6789/0",
            "name": "node-70dcb4",
            "rank": 1
        }]
    },
    "osdmap": {
        "osdmap": {
            "epoch": 28,
            "full": false,
            "nearfull": false,
            "num_in_osds": 5,
            "num_osds": 5,
            "num_remapped_pgs": 0,
            "num_up_osds": 5
        }
    },
    "pgmap": {
        "bytes_avail": 19988592549888,
        "bytes_total": 19988796190720,
        "bytes_used": 203640832,
        "data_bytes": 0,
        "num_pgs": 256,
        "pgs_by_state": [{
            "count": 256,
            "state_name": "active+clean"
        }],
        "version": 49
    },
    "quorum": [0, 1],
    "quorum_names": ["node-c76b2c", "node-70dcb4"]
}

Thanks.

[Crapworks]

So basically with one monitor the health informations are in

["health"]["health"]["health_services"]["mons"]

and with two or more monitors the health information are here?

["health"]["timechecks"]["mons"]

Wow, thats... intetesting. I will see what I can do with that.

[Crapworks]

Hi @mkkie

can you please checkout the branch "one_monitor_fix" and test if this works with one and more monitors?

[mkkie]

Hi @Crapworks

I have tried the branch "one_monitor_fix", and the problem still there.

My Chrome tell me that some error in ceph.dash.js.

I put the following code before line 432 in ceph.dash.js.
console.log(timechekMons);

If I run two monitors, console shows timechekMons is an object,
but I when i remove another monitor, console shows timechekMons is undefined.

[Crapworks]

Hi @mkkie

I've overlookes the fact that health_services is an array, not a dict (which is even more ugly).
Can pull the latest code and try again?

[mkkie]

Thank you @Crapworks

I pull the latest code and it works fine!