ncm-ceph: support for luminous, bluestore, ceph-volume and per host osd deployment #1230

kwaegema · 2018-01-24T13:11:56Z

Recreate component for luminous support, with bluestore, ceph-volume instead of ceph-deploy for osd creation.
This component will only work with Luminous > 12.2.2 and beyond. For Jewel and earlier there is still the v1 of component

Replaces #1224 (which was getting far too large).

Fixes #740, fixes #741, fixes #742, fixes #743, fixes #776.

jrha · 2018-01-24T13:18:30Z

Why the new PR?

kwaegema · 2018-01-24T13:26:28Z

because some browsers were having difficulties loading the page :)

stdweird · 2018-01-24T13:31:04Z

ncm-ceph/src/main/pan/components/ceph/v1/schema-osd.pan

@@ -30,6 +30,8 @@ type ceph_osd = {
    'journal_path' ? string
    'crush_weight' : double(0..) = 1.0
    'labels' ? string[1..]
+    'storetype'? string with match(SELF, '^(blue|file)store$')


use choice? but this is probably not correct (new entry in old schema)? also panlint fixes and remove next empty line?

stdweird · 2018-01-24T13:32:30Z

ncm-ceph/src/main/pan/components/ceph/v2/schema.pan

+    'filestore_xattr_use_omap' ? boolean
+    'fsid' : type_uuid
+    'mon_cluster_log_to_syslog' : boolean = true
+    'mon_initial_members' : type_network_name [1..]


no space before [

stdweird · 2018-01-24T13:33:03Z

ncm-ceph/src/main/pan/components/ceph/v2/schema.pan

+
+@documentation{ overarching ceph cluster type, with osds, mons and msds }
+type ceph_cluster = {
+    'monitors' : ceph_monitor {3..} # with match


no space before{

stdweird · 2018-01-24T13:33:34Z

ncm-ceph/src/main/pan/components/ceph/v2/schema.pan

+    'max_add_osd_failures' : long(0..) = 0
+};
+
+type ceph_supported_version = string with match(SELF, '[0-9]+\.[0-9]+(\.[0-9]+)?'); # TODO  minimum 12.2.2


\d instead of [0-9]

support globs?

stdweird · 2018-01-24T13:33:50Z

ncm-ceph/src/main/pan/components/ceph/v2/schema.pan

+};
+
+type ceph_supported_version = string with match(SELF, '[0-9]+\.[0-9]+(\.[0-9]+)?'); # TODO  minimum 12.2.2
+type ceph_deploy_supported_version = string with match(SELF, '[0-9]+\.[0-9]+\.[0-9]+'); # TODO minimum 1.5.39


same as above

stdweird · 2018-01-24T13:37:59Z

ncm-ceph/src/main/perl/Ceph/Cluster.pm

+            return;
+        }
+    }
+    $self->debug(1, "Node ready to receive ceph-commands");


what does this mean?

stdweird · 2018-01-24T13:39:36Z

ncm-ceph/src/main/perl/Ceph/Cluster.pm

+    return $success || $self->write_init_cfg();
+
+}
+sub deploy


insert newline above

stdweird · 2018-01-24T13:40:02Z

ncm-ceph/src/main/perl/Ceph/Cluster.pm

+
+    $self->debug(5, "deploy hash:", Dumper($map));
+    $self->verbose("Running ceph-deploy commands. This can take some time when adding new daemons.");
+    foreach my $hostname (sort keys(%{$map})) {


no {} required

stdweird · 2018-01-24T13:40:24Z

ncm-ceph/src/main/perl/Ceph/Cluster.pm

+    my ($self, $map) = @_;
+
+    $self->debug(5, "deploy hash:", Dumper($map));
+    $self->verbose("Running ceph-deploy commands. This can take some time when adding new daemons.");


report names of hosts

stdweird · 2018-01-24T13:41:46Z

ncm-ceph/src/main/perl/Ceph/ClusterMap.pm

+    my ($ec, $jstr) = $self->{Cluster}->run_ceph_command([qw(mon dump)], 'get mon map', nostderr => 1) or return;
+    my $monsh = decode_json($jstr);
+    foreach my $mon (@{$monsh->{mons}}){
+        $self->add_existing('mon', $mon->{name}, { addr => $mon->{addr}});


remove space before addr

stdweird · 2018-01-24T14:37:25Z

ncm-ceph/README.md

+* Copy bootstrap-osd key for ncm-download 
+
+Do for each OSDServer (If deployhosts is osd server, start with this one)
+* remove osds from a server


is there a way to check ceph state that removing osds does not cause real issues? no dataloss should occur before zapping the disks

I added to wait before rebalance is ok (after ceph osd out)

stdweird · 2018-01-24T14:38:23Z

ncm-ceph/src/main/perl/Ceph/ClusterMap.pm

+    my $mdshs = decode_json($jstr);
+    my $fsmap = $mdshs->{fsmap};
+    foreach my $fs (@{$fsmap->{filesystems}}){
+        foreach my $mds (values %{$fs->{mdsmap}->{info}}) {


loop over sorted keys

why keys? I dont need them

you can loop over sorted values if you want, as long as it reproducable. if you don;t need the keys, use a list?

this is json output of ceph, keys are pids or someting. I'll sort the values then

stdweird · 2018-01-24T14:38:57Z

ncm-ceph/src/main/perl/Ceph/ClusterMap.pm

+    my ($self, $hostname, $mon) = @_;
+    $self->debug(3, "Comparing mon $hostname");
+    my $ceph_mon = $self->{ceph}->{$hostname}->{mon};
+    if ($ceph_mon->{addr} =~ /^0\.0\.0\.0:0/) {


there's no othe rway to check this?

This is a corner case seen with an earlier version. running the ceph-deploy create initial command sometimes went wrong and the ceph-mon is in a weird existing not configured state. Don't know if this can still happen or how to reproduce this

please add this in a comment

stdweird · 2018-01-24T14:40:20Z

ncm-ceph/src/main/perl/Ceph/ClusterMap.pm

+    $self->{quattor}->{$name}->{daemons}->{$type} = $daemon;
+    $self->{quattor}->{$name}->{fqdn} = $daemon->{fqdn};
+    return 1;
+


remove empty line

stdweird · 2018-01-24T14:41:17Z

ncm-ceph/src/main/perl/Ceph/ClusterMap.pm

+{
+    my ($self) = @_;
+    my @actions = (\&mon_hash, \&mgr_hash, \&mds_hash);
+    foreach my $type (@actions){


space before {

stdweird · 2018-01-24T14:41:38Z

ncm-ceph/src/main/perl/Ceph/ClusterMap.pm

+    my ($self) = @_;
+    my $quattor = $self->{Cluster}->{cluster};
+    $self->debug(2, "Building information from quattor");
+    while (my ($hostname, $mon) = each(%{$quattor->{monitors}})) {


no each, loop over sorted keys

stdweird · 2018-01-24T14:41:55Z

ncm-ceph/src/main/perl/Ceph/ClusterMap.pm

+        $self->add_quattor('mon', $hostname, $mon);
+        $self->add_quattor('mgr', $hostname, $mon); # add a mgr for each mon
+    }
+    while (my ($hostname, $mds) = each(%{$quattor->{mdss}})) {


no each, loop over sorted keys

stdweird · 2018-01-24T14:47:30Z

ncm-ceph/src/main/perl/Ceph/Luminous.pm

+=item 6. Run the component again to start the configuration of the new cluster
+
+=item 7. When the component now runs on OSD servers, it will deploy the local OSDs
+=back


add newline

stdweird · 2018-01-24T14:49:00Z

ncm-ceph/src/main/pan/components/ceph/v2/schema.pan

+    'max_add_osd_failures' : long(0..) = 0
+};
+
+type ceph_supported_version = string with match(SELF, '[0-9]+\.[0-9]+(\.[0-9]+)?'); # TODO  minimum 12.2.2


support globs?

stdweird · 2018-01-24T14:53:12Z

ncm-ceph/src/main/perl/Ceph/OSDserver.pm

+    }
+    # ceph-volume lvm create --bluestore --data /dev/sdk
+    my $devpath = "/dev/" . unescape($name);
+    my $succes = $self->run_command([qw(ceph-volume lvm create), "--$attrs->{storetype}", "--data", $devpath],


succes -> success

stdweird · 2018-01-24T14:53:32Z

ncm-ceph/src/main/perl/Ceph/OSDserver.pm

+
+
+}
+sub deploy


insert newline

stdweird · 2018-01-24T14:53:40Z

ncm-ceph/src/main/perl/Ceph/OSDserver.pm

+    }
+    $self->debug(1, "Deployed osd $name");
+    return 1;
+


remove newlines

stdweird · 2018-02-07T17:23:15Z

ncm-ceph/src/main/pan/components/ceph/config.pan

+variable CEPH_SCHEMA_VERSION ?= 'v1';
+
+include format('components/${project.artifactId}/%s/schema', CEPH_SCHEMA_VERSION);
+


can you add site overrid like https://github.com/quattor/maven-tools/pull/170/files#diff-fb9fe79c1696b94dcb0e714774c01f61R71

jrha · 2018-03-29T16:01:08Z

@stdweird what's the likelihood of this being ready for merge in 18.3?

stdweird · 2018-03-29T18:28:34Z

@jrha need a new release of the buildtools first, and i probably need to rereview this. but we use this in production for new and upgraded ceph clusters

Also add Commands helper module

Also ClusterMap helper Module

kwaegema · 2018-04-25T14:48:46Z

@jrha needs maven-tools 1.58 for building 😄

jrha · 2018-06-08T12:21:26Z

@stdweird and @kwaegema maven-tools 1.58 is now available.

jrha · 2018-06-08T14:57:55Z

@hpcugentbot retest this please

jrha · 2018-07-02T12:07:23Z

Merged as is, we can open more issues if they crop up.

kwaegema force-pushed the lumin branch from 4bc330a to 6a510c2 Compare January 24, 2018 13:15

jrha added this to the 18.3 milestone Jan 24, 2018

stdweird reviewed Jan 24, 2018

View reviewed changes

stdweird reviewed Feb 7, 2018

View reviewed changes

kwaegema force-pushed the lumin branch 3 times, most recently from 5bcddc0 to 3d0ae26 Compare February 12, 2018 15:39

kwaegema force-pushed the lumin branch from c920a42 to adbc55e Compare March 12, 2018 10:15

jrha modified the milestones: 18.3, 18.6 Apr 3, 2018

kwaegema added 7 commits April 25, 2018 16:00

ncm-ceph: init second version of component, new schema and osds per host

d912a72

ncm-ceph: add v2 and init new component

907d627

ncm-ceph: init writing cfgfile and tests

78d926f

ncm-ceph: OSDserver prepare and deploy methods, parse pv and make tests

9b5d2ef

Also add Commands helper module

ncm-ceph: more health and cluster start

a10fa17

ncm-ceph: make some schema changes and refactor

8e1218e

ncm-ceph: start Cluster module: cluster comparison, building

b649e55

Also ClusterMap helper Module

kwaegema added 22 commits April 25, 2018 16:00

ncm-ceph: fix remarks part1

0857e59

ncm-ceph: use CAF::Path

c4400d3

ncm-ceph: cleanup pan code

07b6c22

ncm-ceph: remarks processed part 2

738bb0c

ncm-ceph: add Component Redirect and tqu test

40f27f9

ncm-ceph: cleanup for new tests

3434e8d

ncm-ceph: use choice instead of matching

eb10be3

ncm-ceph: wrapper around file_exists and timeout option of CAF

400ea7a

ncm-ceph: re-add $EC

7189b03

ncm-ceph: add timeout to cluster states call

6911332

ncm-ceph: set default timeout to 0

f999aff

ncm-ceph: update documentation

b09f929

ncm-ceph: fix bugs and remarks

9232b5c

ncm-ceph: add class support for osds

1b66dc1

ncm-ceph: ceph osd crush class support and tests

83b9a39

ncm-ceph: fix config $project.artifact

31fc852

ncm-ceph: add extra global config options

69f4a98

ncm-ceph: readd options for mds

256d65b

ncm-ceph: add dmcrypt support

dfa8efd

ncm-ceph: ceph as owner for ceph-deploy ceph.conf

fc9c5eb

ncm-ceph: add authentication name to crush class commands

509ffc9

ncm-ceph: luminous renamed options

9d8fdc7

kwaegema force-pushed the lumin branch from adbc55e to 9d8fdc7 Compare April 25, 2018 14:20

jrha approved these changes Jul 2, 2018

View reviewed changes

jrha merged commit 45f5903 into quattor:master Jul 2, 2018

kwaegema deleted the lumin branch December 3, 2018 10:39

		variable CEPH_SCHEMA_VERSION ?= 'v1';

		include format('components/${project.artifactId}/%s/schema', CEPH_SCHEMA_VERSION);



		}
		sub deploy

ncm-ceph: support for luminous, bluestore, ceph-volume and per host osd deployment #1230

ncm-ceph: support for luminous, bluestore, ceph-volume and per host osd deployment #1230

Conversation

kwaegema commented Jan 24, 2018 • edited by jrha Loading

jrha commented Jan 24, 2018

kwaegema commented Jan 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrha commented Mar 29, 2018 • edited Loading

stdweird commented Mar 29, 2018

kwaegema commented Apr 25, 2018

jrha commented Jun 8, 2018

jrha commented Jun 8, 2018

jrha commented Jul 2, 2018

kwaegema commented Jan 24, 2018 •

edited by jrha

Loading

jrha commented Mar 29, 2018 •

edited

Loading