ceph-common: purge ceph.conf file

[leseb]

Since ##461 we have been having the ability to override ceph default
options. Previously we had to add a new line in the template and then
another variable as well. Doing a PR for one option was such a pain. As
a result, we now have tons of options that we need to maintain across
all the ceph version, yet another painful thing to do.
This commit removes all the ceph options so they are handled by ceph
directly. If you want to add a new option, feel free to to use the
ceph_conf_overrides variable of your group_vars/all.

Risks, for those who have been managing their ceph using ceph-ansible
this is not a trivial change as it will trigger a change in your
ceph.conf and then restart all your ceph services. Moreover if you did
some specific tweaks as well, prior to run ansible you should update the
ceph_conf_overrides variable to reflect your previous changes.

To avoid service restart, you need to know a bit of ansible for this,
but generally the idea would be to run ansible on a dummy host to
generate the ceph.conf, then scp this file to all your ceph hosts and
you should be good.

Closes: #693

Signed-off-by: Sébastien Han seb@redhat.com

[leseb]

@andrewschoen looks like we are seeing the same error again on the Ubuntu VM :(

[leseb]

@jdurgin, @andrewschoen, @alfredodeza feel free to chime :)

[andrewschoen]

test this please

[andrewschoen]

@leseb this looks ok to me, it certainly cleans things up. @jdurgin had mentioned to me that some of these defaults currently in ceph.conf are good for older versions of ceph, do we perhaps want to support version-specific configuration files?

[andrewschoen]

@leseb the trusty failure was because the CI was picking up a node without an extra device attached. I think I've got that cleared up, but we will probably need to run the tests one more time.

[vasukulkarni]

I think the rbd_client_log/dir should also go away, with selinux enabled this can create issues if they are not in default directories and policy exists only for default directories(for hammer/jewel), for other distro's it isn't an issue.

[andrewschoen]

test this please

[jdurgin]

looks good in terms of getting rid of settings that are suboptimal for jewel.

For other versions, I do think having a per-version config might make sense - e.g. the osd recovery settings commonly had to be adjust in hammer, which is probably where a lot of these came from. Maybe those could just go in a sample playbook as overrides or something? I'm not too familiar with the structure of things in ansible.

[leseb]

Ok I'll try to add an example section for those who are looking at deploying Hammer then.

[andrewschoen]

Test this please

[leseb]

test this please

[andrewschoen]

@leseb what else needs to be done for this one? Can I help?

[leseb]

@andrewschoen I just need to rebase and then add some doc to address the Hammer use case I guess.

[andrewschoen]

I think we want to keep rbd_client_admin_socket_path, but lose the rest of the rbd_* options here.

[leseb]

test this please

[leseb]

test this please

[mattt416]

Looks like the defaults for rbd_cache will also need to be removed.

[mattt416]

Hi @leseb,

The restarts on the back of config changes is concerning obviously, but I think what is potentially more concerning is the significant impact to behavior that is going to result from this change. I've tried to look at what will be removed and made a note where ceph-ansible and upstream ceph values differ:

Setting	ceph-ansible	ceph
cephx require signatures	true	false
cephx cluster require signatures	true	false
osd pool default pg num	128	8
osd pool default pgp num	128	8
rbd concurrent management ops	20	10
rbd default map options	rw	''
rbd default format	2	1
mon osd down out interval	600	300
mon osd min down reporters	7	1
mon clock drift allowed	0.15	0.5
mon clock drift warn backoff	30	5
mon osd report timeout	900	300
mon pg warn max per osd	0	300
mon osd allow primary affinity	true	false
filestore merge threshold	40	10
filestore split multiple	8	2
osd op threads	8	2
filestore op threads	8	2
osd recovery max active	5	15
osd max backfills	2	10
osd recovery op priority	2	63
osd recovery max chunk	1048576	8 << 20
osd scrub sleep	0.1	0
osd disk thread ioprio class	idle	''
osd disk thread ioprio priority	0	-1
osd deep scrub stride	1048576	524288
osd scrub chunk max	5	25

These are defaults I'm still working on tracking down:

Setting	ceph-ansible	ceph
osd mkfs type	xfs
osd crush update on start	true

Obviously, we can override all these locally but this is going to have a big impact on anyone who is not watching what is happening in this repo. I think some further discussions need to be had here before this goes in.

With all that said, I do think this is the right long-term approach. :)

--Matt

[leseb]

Hi @mattt416,

You raising a really good point and thanks for the in-depth analysis. However I'm not really sure how to provide a compatible and non-breaking change.
What could be done for the next release of ceph-ansible, is to highlight this as a huge change.
So users getting the next ceph-ansible version will hopefully read the release note :).
The other option would be to keep previous default in override_config but I'm not such a big fan of this.

Any other idea?

[alfredodeza]

test this please

[leseb]

test this please

[mattt416]

Hi @leseb,

TBH I didn't even realise you were using tags! Having those in place helps a lot because it means you can be mindful of breaking changes when the major version is bumped.

We will discuss internally what to do for rpc-openstack -- whether we override locally to bring back a similar configuration or try to run with a more vanilla setup and only override things when we have a specific need.

Thanks!

--Matt

[leseb]

test this please

[leseb]

test this please

[leseb]

test this please

[leseb]

@mattt416 how did the discussion go?
I probably need to keep some examples for Hammer and before in the meantime.

[leseb]

@jdurgin I'm currently documenting some of the variables that we would recommend to keep if we deploy a cluster before Jewel. Could you tag some of the variables we need highlight? (just by looking at @mattt416's table maybe).

Thanks!

[jdurgin]

@leseb I think the main ones are the recovery settings highlighted already in this pr. Others are pretty hardware-dependent, or not too important I think. You could point at the old config before this pr so folks could compare perhaps.

[leseb]

@jdurgin good point, will do thanks!