owner for osds and journal device partition are rnot set correctly with infernalis

[mistur]

OS : Ubuntu 14.04
ceph version : infernalis

Journals are on ssd raw partition :

group_vars/osds :

crush_location: false
osd_crush_location: "'root={{ ceph_crush_root }} rack={{ ceph_crush_rack }} host={{ ansible_hostname }}'"
devices: - /dev/sdb - /dev/sdc
raw_multi_journal: true
raw_journal_devices:
- /dev/sdl1
- /dev/sdl2

I must create SSD partitons before to run the playbook,

brw-rw---- 1 root disk 8, 177 jan 27 22:00 /dev/sdl1
brw-rw---- 1 root disk 8, 178 jan 27 22:00 /dev/sdl2

since infernalis, OSD runs with the user ceph so both osd disks and journal partitions seem to be owned by the ceph user.

to set the owner of the disk I run that's script before

for disk in sdb sdc ; do
if ! grep -q ${disk} /etc/udev/rules.d/89-ceph-journal.rules ; then
echo "KERNEL==\"${disk}?\" SUBSYSTEM==\"block\" OWNER=\"ceph\" GROUP=\"disk\" MODE=\"0660\"" >> /etc/udev/rules.d/89-ceph-journal.rules
fi
done

so when the ceph-disk create OSDs partitions, udev set the good owner.

maybe ceph-ansible should fill /etc/udev/rules.d/89-ceph-journal.rules ?

and for the journal partition, it's not possible to change the owner of the partition before because the user ceph is not create yet, and the playbook does not change the owner of journal partition. so OSDs fails to start, we must change the user of journal partitions and run the playbook again to starts OSDSs

[leseb]

The udev rule is a good solution, however an easier fix would be to run chown 64045:64045 on the proper devices prior to deploy. 64045 is the uuid of the ceph user that will be created during the package installation.

Can you test this?

[leseb]

Just tried on Ubuntu 14.04 and I can't reproduce your issue. I can create an OSD with a dedicated journal and I don't hahve any problem.

Can you share the error trace?

[mistur]

the playbook seem to finish well but no osd is started, if I try to start manualy on the OSDs server, I have this message 👍

2016-01-28 09:00:47.041809 7f5dd1406940 -1 filestore(/var/lib/ceph/tmp/mnt.BxlMix) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.BxlMix/journal: (13) Permission denied 2016-01-28 09:00:47.041860 7f5dd1406940 -1 OSD::mkfs: ObjectStore::mkfs failed with error -13 2016-01-28 09:00:47.041909 7f5dd1406940 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.BxlMix: (13) Permission denied ERROR:ceph-disk:Failed to activate ceph-disk: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '27', '--monmap', '/var/lib/ceph/tmp/mnt.BxlMix/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.BxlMix', '--osd-journal', '/var/lib/ceph/tmp/mnt.BxlMix/journal', '--osd-uuid', '732b9c6c-738d-44e5-a1d3-c836c545aeb2', '--keyring', '/var/lib/ceph/tmp/mnt.BxlMix/keyring', '--setuser', 'ceph', '--setgroup', 'ceph']' returned non-zero exit status 1 got monmap epoch 1 2016-01-28 09:00:47.500454 7f5bc33a7940 -1 filestore(/var/lib/ceph/tmp/mnt.O_lF_u) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.O_lF_u/journal: (13) Permission denied 2016-01-28 09:00:47.500511 7f5bc33a7940 -1 OSD::mkfs: ObjectStore::mkfs failed with error -13 2016-01-28 09:00:47.500564 7f5bc33a7940 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.O_lF_u: (13) Permission denied ERROR:ceph-disk:Failed to activate ceph-disk: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '25', '--monmap', '/var/lib/ceph/tmp/mnt.O_lF_u/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.O_lF_u', '--osd-journal', '/var/lib/ceph/tmp/mnt.O_lF_u/journal', '--osd-uuid', '1e95ba97-218e-4e48-bedb-fd83a1a6066f', '--keyring', '/var/lib/ceph/tmp/mnt.O_lF_u/keyring', '--setuser', 'ceph', '--setgroup', 'ceph']' returned non-zero exit status 1

And if I change the owner of the device partition, OSDs start correctly

[leseb]

Did you try to simply set:

raw_journal_devices:
- /dev/sdl
- /dev/sdl

And let ceph-disk configure the device.
One a test machine I have one OSD and one dedicated journal and I have the following permissions:

root@ceph-ansible:~# stat /dev/vdc
  File: ‘/dev/vdc’
  Size: 0               Blocks: 0          IO Block: 4096   block special file
Device: 5h/5d   Inode: 1244        Links: 1     Device type: fd,20
Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    6/    disk)
Access: 2016-01-27 23:47:59.685809423 +0000
Modify: 2016-01-27 23:47:37.597748007 +0000
Change: 2016-01-27 23:47:37.597748007 +0000
 Birth: -
root@ceph-ansible:~# stat /dev/vdc1
  File: ‘/dev/vdc1’
  Size: 0               Blocks: 0          IO Block: 4096   block special file
Device: 5h/5d   Inode: 111559      Links: 1     Device type: fd,21
Access: (0660/brw-rw----)  Uid: (64045/    ceph)   Gid: (64045/    ceph)
Access: 2016-01-27 23:48:01.073813177 +0000
Modify: 2016-01-27 23:48:10.377838026 +0000
Change: 2016-01-27 23:48:10.377838026 +0000
 Birth: -
root@ceph-ansible:~# stat /dev/vdb
  File: ‘/dev/vdb’
  Size: 0               Blocks: 0          IO Block: 4096   block special file
Device: 5h/5d   Inode: 1243        Links: 1     Device type: fd,10
Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    6/    disk)
Access: 2016-01-27 23:48:00.649812031 +0000
Modify: 2016-01-27 23:48:00.649812031 +0000
Change: 2016-01-27 23:48:00.649812031 +0000
 Birth: -
root@ceph-ansible:~# stat /dev/vdb1
  File: ‘/dev/vdb1’
  Size: 0               Blocks: 0          IO Block: 4096   block special file
Device: 5h/5d   Inode: 112750      Links: 1     Device type: fd,11
Access: (0660/brw-rw----)  Uid: (64045/    ceph)   Gid: (64045/    ceph)
Access: 2016-01-27 23:47:59.845809856 +0000
Modify: 2016-01-27 23:47:58.037804951 +0000
Change: 2016-01-27 23:47:58.037804951 +0000
 Birth: -

Here vdb is the OSD and vdc is the journal.

[leseb]

The OSD don't start because the journal device was partitioned outside of the play and thus had wrong permissions. Closing this.