Use arbitrary ceph user

[egroeper]

I'm currently testing the pro version (3.0.6) of seafile.
The manual says, that I should copy the ceph admin keyring to the seafile node. But I created a separate keyring/user with lowered privileges and want to use that.

It seems like you are calling rados_create(&cluster, NULL), which means you don't set id/username. That way I would have to use the default admin user.
Could you add a configuration option in ccnet.conf to set username?
Is there any other possibility to set the username/id?

I tried to add 'export CEPH_ARGS="--id "' to seafile.sh, but that didn't work out.

[egroeper]

Binary patching /data/haiwen/seafile-server-latest/seafile/lib/librados.so.2 (of course your username has to be (max) 8 characters long) is our current workaround. But of course not a real solution.

[lins05]

@killing

On Tue, Aug 19, 2014 at 9:56 PM, egroeper notifications@github.com wrote:

Binary patching
/data/haiwen/seafile-server-latest/seafile/lib/librados.so.2 (of course
your username has to be (max) 8 characters long) is our current workaround.
But of course not a real solution.

—
Reply to this email directly or view it on GitHub
#800 (comment).

[killing]

Hi @egroeper

I've tried to 'export CEPH_ARGS="--id foo --keyring=/etc/ceph/ceph.client.foo.keyring"'. That works find in our test environment. Did you miss the --keyring argument?

[egroeper]

@killing:
Sorry. I had to get the test setup working again...
Yes. I did not use keyring argument, but keyring file for the id is defined in ceph.conf.
Of course I now tested with additional keyring parameter as well, but that didn't work either.
However ceph --id seafile -s works fine.
Where do you specify "export CEPH_ARGS..."? I appended it to the other export lines in seafile.sh:26.

Thanks for your help!

[killing]

I just export CEPH_ARGS before running seafile.sh.
The ceph rados API says rados_create() will see the environment variables. So using "--id " should work.
What error do you get in seafile.log? What do you mean by the configuration doesn't work?

[egroeper]

export | fgrep CEPH
declare -x CEPH_ARGS="--id seafile --keyring=/etc/ceph/client.seafile"
./seafile.sh start

results in:

[09/17/14 16:09:29] ../common/obj-backend-ceph.c(391): [Block backend] Cannot connect to cluster
[09/17/14 16:09:29] [Obj backend] Failed to init ceph: pool name is seafile-fs.
[09/17/14 16:09:29] ../common/block-backend-ceph.c(630): [Block backend] Cannot connect to cluster
[09/17/14 16:09:29] [Block backend] Failed to init ceph: pool name is seafile-blocks.
[09/17/14 16:09:29] ../common/obj-backend-ceph.c(391): [Block backend] Cannot connect to cluster
[09/17/14 16:09:29] [Obj backend] Failed to init ceph: pool name is seafile-commits.
[09/17/14 16:09:29] ../common/mq-mgr.c(60): [mq client] mq cilent is started
[09/17/14 16:09:29] ../common/mq-mgr.c(106): [mq mgr] publish to hearbeat mq: seaf_server.heartbeat
[09/17/14 16:09:29] listen-mgr.c(115): listen on port 12001 for block tranfer

Whereas when I binary patch the library everything works. The username then has to be "seafi" as we are limited to 5 characters.
It seems like the fileserver doesn't get the env variable.

By the way:

ceph -s
    cluster 9d1886c0-ce93-4729-a790-6f38b8a1114e
     health HEALTH_OK
...

Of course I checked capabilities. seafi and seafile have the same ceph capabilities (permissions).

[killing]

Hi,

After reading the source code of librados, I find that only the CEPH_KEYRING env variable is read in rados_create(). So it's necessary to add an option to seafile.conf for specifying the id.

[egroeper]

Good to hear we now know what is needed to fix this (and why). Since this part of the code is closed source, I'm awaiting your fix.

[killing]

Hi @egroeper
The new beta version has been added to seacloud.cc group. And the doc for Ceph setup has been updated too. Please give it a try.

[egroeper]

This works. Thanks!

[egroeper]

Please apply the following patch to make it work with seafdav, too:

diff -ur seahub/thirdpart/seafobj/backends/ceph.py /home/user/seahub/thirdpart/seafobj/backends/ceph.py
--- seahub/thirdpart/seafobj/backends/ceph.py   2014-06-13 04:06:21.000000000 +0000
+++ /home/user/seahub/thirdpart/seafobj/backends/ceph.py    2015-01-16 13:44:10.000000000 +0000
@@ -9,9 +9,10 @@
 from seafobj.utils.ceph_utils import ioctx_set_namespace

 class CephConf(object):
-    def __init__(self, ceph_conf_file, pool_name):
+    def __init__(self, ceph_conf_file, pool_name, ceph_client_id):
         self.pool_name = pool_name
         self.ceph_conf_file = ceph_conf_file
+        self.ceph_client_id = ceph_client_id

 class IoCtxPool(object):
     '''since we need to set the namespace before read the object, we need to
@@ -21,7 +22,7 @@
     def __init__(self, conf, pool_size=5):
         self.conf = conf
         self.pool = Queue.Queue(pool_size)
-        self.cluster = rados.Rados(conffile=conf.ceph_conf_file)
+        self.cluster = rados.Rados(conffile=conf.ceph_conf_file, rados_id=conf.ceph_client_id)
         self.lock = threading.Lock()

     def get_ioctx(self, repo_id):
diff -ur seahub/thirdpart/seafobj/objstore_factory.py /home/user/seahub/thirdpart/seafobj/objstore_factory.py
--- seahub/thirdpart/seafobj/objstore_factory.py    2014-11-13 20:56:30.000000000 +0000
+++ /home/user/seahub/thirdpart/seafobj/objstore_factory.py 2015-01-16 13:26:54.000000000 +0000
@@ -7,10 +7,11 @@
 def get_ceph_conf(cfg, section):
     config_file = cfg.get(section, 'ceph_config')
     pool_name = cfg.get(section, 'pool')
+    ceph_client_id = cfg.get(section, 'ceph_client_id')

     from seafobj.backends.ceph import CephConf

-    return CephConf(config_file, pool_name)
+    return CephConf(config_file, pool_name, ceph_client_id)

 def get_s3_conf(cfg, section):
     key_id = cfg.get(section, 'key_id')

[killing]

Thanks! Will be included in 4.0.4 version.