Rolling Update : create ceph mgr keyring => "Error EINVAL: bad entity name"

[mistur]

Hello,

I'm doing rolling update tests from jewel to luminous and the playbook fails on "create ceph mgr keyring"

Ceph mgr keyring are created early during the update and, at this time, MONs are still running jewel version (even packages may have been upgraded, services have not been restarted yet) thus I guess, MONs don't know anything about MGR and the command failed

As soon as I restart the MONs in luminous, ceph mgr keyring are created without any issue.

ceph-ansible log :
rolling_update-mgr-error.log

ceph-ansible config for jewel deployement (ceph-ansible stable-2.2)
cluster-migration-jewel.zip

ceph-ansible config for luminous deployement (ceph-ansible stable-3.0)
cluster-migration-luminous.zip

Yoann

[mistur]

Hello,

Anyone available to give a look on this issue ? Thanks

Best regards,

Yoann

[leseb]

Weird, this shouldn't fail since we have https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-mon/tasks/ceph_keys.yml#L75

[leseb]

Since we point to the last item of the list, the previous mon should have been updated already. Thus the request should just work on any mon.

[mistur]

does " - inventory_hostname == groups[mon_group_name]|last" means mons daemon have been restarted ?

[mistur]

in this extract from my ansible-playbook log, after line 11, the playbook update ceph package to luminous, at line 358 [ceph-mon : start the monitor service] just checks if mons is started but does not restarted daemon, then at line 422, [ceph-mon : create ceph mgr keyring(s) when mon is not containerized] try to create mgr key. In this case, I restarted mons manually before, so there were no error.

mgr_error.log

[mistur]

You can find here the full log with -vvv with the bug :
rolling_update-mgr-error2.log.txt

[mistur]

something like that seems to fixe my issue :

mistur@bc9ae94

but I guess it may have side effect later, I don't know how to say to run the flush_handlers only in the case of rolling update

[mistur]

Hello,

I have migrated my cluster with --step option , I have restarted MONs after package upgrade and the playbook has ended without any issue.

Best regards,

Yoann