CentOS 7 Issues

[adambmedent]

Hey all, very cool project and hoping to make good use of it. Would like to get benji working on C7, based on the install steps it was pretty straight forward. However I can't seem to get around this specific issue.

benji) [root@ceph-backups backy2]# benji backup --snapshot-name 2019-03-20T10:37:1 --rbd-hints /tmp/vm-101-disk-0.diff rbd://Test/vm-101-disk-0 vm-101-disk-0
INFO: $ /usr/local/benji/bin/benji backup --snapshot-name 2019-03-20T10:37:1 --rbd-hints /tmp/vm-101-disk-0.diff rbd://Test/vm-101-disk-0 vm-101-disk-0
ERROR: Module file benji.io.rbd not found or related import error.

Am I missing something major?

[adambmedent]

I setup a fresh CentOS7 VM and ran through the steps and hit the same issue.

Pretty simple config.

configurationVersion: '1'
databaseEngine: sqlite:////tmp/benji.sqlite
defaultStorage: Ceph-Backup

storages:

• name: Ceph-Backup
  storageId: 1
  module: rbd
  configuration:
  path: /Ceph-Backups2

ios:

• name: file
  module: file

• name: rbd
  module: rbd

I get the same error when initializing the database, unless I change my storage module to file.

[root@ceph-backups ~]# benji database-init
INFO: $ /bin/benji database-init
ERROR: An exception of type ConfigurationError occurred: Module file benji.io.rbd not found or related import error.

[olifre]

To access RBD, you also need to compile the Ceph RBD python bindings (ideally from the Ceph version you have in use). I think these are sadly not packaged for CentOS 7 in a Python3 version. What we use, is:

CEPH_VERSION=13.2.5
pip install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rados
pip install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd

[adambmedent]

Fantastic information and I appreciate it! Is there any distro that is better suited for benji?

[olifre]

Is there any distro that is better suited for benji?

I'll leave that for others to answer, since we are also using CentOS 7 as it is so well supported by Ceph 😉 .
But I think that with CentOS 8 sometime later this year (hopefully) which comes with Python 3, things will improve also for us.

[adambmedent]

Running into a issue missing the RBD development headers.

[root@ceph-backups ~]# pip3.6 install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd
Collecting git+https://github.com/ceph/ceph@v13.2.5#subdirectory=src/pybind/rbd
Cloning https://github.com/ceph/ceph (to v13.2.5) to /tmp/pip-xm500h70-build
Complete output from command python setup.py egg_info:
/tmp/pip-xm500h70-build/src/pybind/rbd/tmp0bqxxywk/rbd_dummy.c:3:24: fatal error: rbd/librbd.h: No such file or directory
#include <rbd/librbd.h>
^
compilation terminated.

Compile Error: RBD development headers not found

[adambmedent]

Running into a issue missing the RBD development headers.

[root@ceph-backups ~]# pip3.6 install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd
Collecting git+https://github.com/ceph/ceph@v13.2.5#subdirectory=src/pybind/rbd
Cloning https://github.com/ceph/ceph (to v13.2.5) to /tmp/pip-xm500h70-build
Complete output from command python setup.py egg_info:
/tmp/pip-xm500h70-build/src/pybind/rbd/tmp0bqxxywk/rbd_dummy.c:3:24: fatal error: rbd/librbd.h: No such file or directory
#include <rbd/librbd.h>
^
compilation terminated.

Compile Error: RBD development headers not found

Got around this by installing the following.

yum install librados-devel librbd-devel

Now I am seeing this.

[root@ceph-backups ~]# pip3.6 install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd
Collecting git+https://github.com/ceph/ceph@v12.2.11#subdirectory=src/pybind/rbd
Cloning https://github.com/ceph/ceph (to v12.2.11) to /tmp/pip-l35mn9rd-build
Complete output from command python setup.py egg_info:
WARNING: Cython is not installed.
ERROR: Cannot find Cythonized file rbd.c

Cython is indeed installed as well.

[root@ceph-backups tmp]# rpm -qa | grep Cython
Cython-0.19-5.el7.x86_64

[olifre]

That version of cython is not a python 3.6 version. It seems there is no Cython for Python 3.6 in EPEL.

We are using a python 3.6 virtualenv in which we install cython, the rbd/rados libraries and Benji itself.

[adambmedent]

Great input, got me a bit further. Decided to try out the python virtualenv but getting hung up still after getting the Ceph RBD python bindings in place.

(benji) [root@ceph-backups ~]# pip install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd
Collecting git+https://github.com/ceph/ceph@v12.2.11#subdirectory=src/pybind/rbd
Cloning https://github.com/ceph/ceph (to revision v12.2.11) to /tmp/pip-req-build-3o4cqquh
Requirement already satisfied (use --upgrade to upgrade): rbd==2.0.0 from git+https://github.com/ceph/ceph@v12.2.11#subdirectory=src/pybind/rbd in /usr/local/benji/lib/python3.6/site-packages

Still results in the same error.

(benji) [root@ceph-backups ~]# /usr/local/benji/bin/benji backup --snapshot-name 2019-03-18T14:38:53 --rbd-hints /tmp/vm-100-disk-0.diff rbd://Test/vm-100-disk-0@2019-03-18T14:38:53 vm-100-disk-0
INFO: $ /usr/local/benji/bin/benji backup --snapshot-name 2019-03-18T14:38:53 --rbd-hints /tmp/vm-100-disk-0.diff rbd://Test/vm-100-disk-0@2019-03-18T14:38:53 vm-100-disk-0
ERROR: Module file benji.io.rbd not found or related import error.

No luck finding benji.io.rbd.

(benji) [root@ceph-backups ~]# find / | grep benji.io.rbd
/usr/local/benji/lib/python3.6/site-packages/benji/io/rbd.py
/usr/local/benji/lib/python3.6/site-packages/benji/schemas/v1/benji.io.rbd.yaml

[olifre]

Mhm - the path:

/usr/local/benji/lib/python3.6/site-packages/benji/io/rbd.py

shoult contain the class and load fine.

Here's the complete set of things we do (since we do it via puppet, I had to convert things a bit):

1. Add IUS repos for Python 3.6 ( https://ius.io/ )
2. Install gcc and git.
3. Install librados-devel and librbd-devel from Ceph repos.
4. Install python36u-devel, python36u-pip, python36u-libs, python36u-setuptools and create a virtualenv with Python 3.6.
5. Install aes-keywrap, pycryptodome, Cython and rados / rbd (as described above) via pip3 in the VirtualEnv.
6. Finally, install Benji via pip3 with features compression,readcache.

That being said - do you get any error if, inside the VirtualEnv, you run python and then type:

import benji.io.rbd

and hit "return"?

[adambmedent]

Mhm - the path:

/usr/local/benji/lib/python3.6/site-packages/benji/io/rbd.py

shoult contain the class and load fine.

Here's the complete set of things we do (since we do it via puppet, I had to convert things a bit):

1. Add IUS repos for Python 3.6 ( https://ius.io/ )
2. Install gcc and git.
3. Install librados-devel and librbd-devel from Ceph repos.
4. Install python36u-devel, python36u-pip, python36u-libs, python36u-setuptools and create a virtualenv with Python 3.6.
5. Install aes-keywrap, pycryptodome, Cython and rados / rbd (as described above) via pip3 in the VirtualEnv.
6. Finally, install Benji via pip3 with features compression,readcache.

That being said - do you get any error if, inside the VirtualEnv, you run python and then type:

import benji.io.rbd

and hit "return"?

Here is that output.

(benji) [root@ceph-backups ~]# import benji.io.rbd
import: unable to open X server `' @ error/import.c/ImportImageCommand/369.

One thing I want to mention. In order to get rados/rdb I had to install python-devel because it was looking for python-config. Shouldn't it be using python3.6-config which was installed.

[elemental-lf]

@adambmedent, CentOS 7 is fine, but, as @olifre already mentioned, is missing the Ceph Python modules for Python 3. The Docker images are also CentOS 7 based and compile the necessary modules themselves. Some workarounds are required as you've already noticed. Please have a look at the command sequence in https://github.com/elemental-lf/benji/blob/8446afbad81f36432a9c0dabf0be59f1ed42b238/images/benji/Dockerfile, lines 11 to 31.
If you install Docker (or another compatible container runtime) you could also use the pre-compiled Docker image that we provide. With the right options to docker run put into a script wrapper you could make it behave like a native CLI program. You wouldn't have to deal with the dependencies then, but you'd have to live with the Ceph version used in the image.

[adambmedent]

@adambmedent, CentOS 7 is fine, but, as @olifre already mentioned, is missing the Ceph Python modules for Python 3. The Docker images are also CentOS 7 based and compile the necessary modules themselves. Some workarounds are required as you've already noticed. Please have a look at the command sequence in https://github.com/elemental-lf/benji/blob/8446afbad81f36432a9c0dabf0be59f1ed42b238/images/benji/Dockerfile, lines 11 to 31.
If you install Docker (or another compatible container runtime) you could also use the pre-compiled Docker image that we provide. With the right options to docker run put into a script wrapper you could make it behave like a native CLI program. You wouldn't have to deal with the dependencies then, but you'd have to live with the Ceph version used in the image.

I think this might have answered my questions.

RUN python3.6 -m venv $VENV_DIR &&
ln -s /usr/bin/python3.6-config $VENV_DIR/bin/python-config &&
ln -s /usr/bin/python3.6m-x86_64-config $VENV_DIR/bin/python3.6m-x86_64-config &&
. $VENV_DIR/bin/activate &&
pip install --upgrade 'pip==10.0.*' &&
sed -i "s|self.run_command(['clone', '-q', url, dest])|self.run_command(['clone', '--depth=1', '--branch', rev if rev else 'master', '-q', url, dest])|" $(python -c 'import site; print(site.getsitepackages()[0])')/pip/_internal/vcs/git.py

So I shouldn't install python-devel to get python-config. I should create a symbolic link. Thats gotta be my issue. I am going to start fresh right now and see what I end up with.

I am not overly familiar with docker so I am hoping to just run our standard C7 load because we are C7 shop.

[adambmedent]

The symbolic links got me further and I am no longer getting the same error. I now have benji.io.rbd.

(benji) [root@linux151 benji]# find / | grep benji.io.rbd
/usr/lib/python3.6/site-packages/benji/io/benji.io.rbd

(benji) [root@linux151 bin]# benji database-init
INFO: $ /usr/local/benji/bin/benji database-init
ERROR: An exception of type ConfigurationError occurred: Module file benji.storage.rbd not found or related import error.

[elemental-lf]

@adambmedent I just noticed: You can skip the sed hack, it should work without. This hack is just so that the Ceph repository gets cloned with --depth=1 because it's quite large and we're only interested in the most recent revision. This reduces build time.

[elemental-lf]

Sorry, I overlooked this one:

(benji) [root@linux151 bin]# benji database-init
INFO: $ /usr/local/benji/bin/benji database-init
ERROR: An exception of type ConfigurationError occurred: Module file benji.storage.rbd not found or related import error.

Could you please run /usr/local/benji/bin/benji --log-level DEBUG database-init and report the output back here? And while you're at it could you try to import Ceph's rbd manually? Something like python3.6 -c 'import rbd' and see what is says... No output would mean that it could be imported correctly.
I suspect that the problem is that you either need to (re)create the virtual environment with --system-site-packages or you need to install Ceph's rados and rbd modules into the virtual environment instead of globally. I would suggest the latter.

[adambmedent]

Ok I started with a brand new CentOS7 VM and did the following steps to a T.

• yum install -y git gcc
• Install ceph repo package
• yum install -y deltarpm tzdata epel-release https://centos7.iuscommunity.org/ius-release.rpm
• yum install -y python36u-devel python36u-pip python36u-libs python36u-setuptools librados-devel librbd-devel
• python3.6 -m venv /usr/local/benji
• VENV_DIR=/usr/local/benji
• ln -s /usr/bin/python3.6-config $VENV_DIR/bin/python-config
• ln -s /usr/bin/python3.6m-x86_64-config $VENV_DIR/bin/python3.6m-x86_64-config
• . $VENV_DIR/bin/activate
• pip install --upgrade 'pip==10.0.*'
• pip install cython
• CEPH_VERSION=12.2.11
• pip install "git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rados"
• pip install "git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd"
• pip install benji[compression,readcache,s3,b2]

There were no errors at all and it looked to be successful, but I am still hitting the same exact issue.

(benji) [root@linux151 ~]# benji database-init
INFO: $ /usr/local/benji/bin/benji database-init
ERROR: An exception of type ConfigurationError occurred: Module file benji.storage.rbd not found or related import error.

Here is the database-init debug output.

(benji) [root@linux151 ~]# /usr/local/benji/bin/benji --log-level DEBUG database-init
INFO: $ /usr/local/benji/bin/benji --log-level DEBUG database-init
DEBUG: commands.database_init(**{})
DEBUG: Using block hash BLAKE2b with kwargs {'digest_bits': 256}.
DEBUG: Resolved schema for benji.io.base-v1: {'configuration': {'type': 'dict', 'nullable': True, 'required': True, 'schema': {'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}}}}.
DEBUG: Resolved schema for benji.io.file-v1: {'configuration': {'type': 'dict', 'nullable': True, 'required': True, 'schema': {'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}}}}.
DEBUG: Resolved schema for benji.io.base-v1: {'configuration': {'type': 'dict', 'nullable': True, 'required': True, 'schema': {'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}}}}.
DEBUG: Resolved schema for benji.io.rbd-v1: {'configuration': {'type': 'dict', 'nullable': True, 'required': True, 'schema': {'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'cephConfigFile': {'type': 'string', 'empty': False, 'default': '/etc/ceph/ceph.conf'}, 'clientIdentifier': {'type': 'string', 'empty': False, 'default': 'admin'}, 'newImageFeatures': {'type': 'list', 'empty': False, 'default': ['RBD_FEATURE_LAYERING'], 'schema': {'type': 'string', 'regex': '^RBD_FEATURE_.*'}}}, 'empty': False}}.
DEBUG: An exception of type ConfigurationError occurred: Module file benji.storage.rbd not found or related import error.
Traceback (most recent call last):
File "/usr/local/benji/lib64/python3.6/site-packages/benji/factory.py", line 50, in _import_modules
module = importlib.import_module('{}.{}.{}'.format(package, cls._MODULE, module))
File "/usr/lib64/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'benji.storage.rbd'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/benji/lib64/python3.6/site-packages/benji/scripts/benji.py", line 338, in main
func(**func_args)
File "/usr/local/benji/lib64/python3.6/site-packages/benji/commands.py", line 492, in database_init
benji_obj = Benji(self.config, init_database=True)
File "/usr/local/benji/lib64/python3.6/site-packages/benji/benji.py", line 50, in init
StorageFactory.initialize(self.config)
File "/usr/local/benji/lib64/python3.6/site-packages/benji/factory.py", line 74, in initialize
cls._import_modules(config, storages)
File "/usr/local/benji/lib64/python3.6/site-packages/benji/factory.py", line 53, in _import_modules
package, cls._MODULE, module))
benji.exception.ConfigurationError: Module file benji.storage.rbd not found or related import error.
ERROR: An exception of type ConfigurationError occurred: Module file benji.storage.rbd not found or related import error.

(benji) [root@linux151 ~]# python3.6 -c 'import rbd'
(benji) [root@linux151 ~]#

I was in the python virtual enviroment when I installed the rados and rbd modules.

[adambmedent]

My ceph cluster is 12.2.11, so I figured I would stick with this version on my dedicated benji VM. I am starting to wonder if that is my issue as well? Maybe I need to move to the newer version for the benji VM?

[elemental-lf]

My ceph cluster is 12.2.11, so I figured I would stick with this version on my dedicated benji VM. I am starting to wonder if that is my issue as well? Maybe I need to move to the newer version for the benji VM?

Hmm, I've think I've also tested with Luminous last year and Benji only uses very basic calls to the RBD API which have existed for years.

[adambmedent]

My ceph cluster is 12.2.11, so I figured I would stick with this version on my dedicated benji VM. I am starting to wonder if that is my issue as well? Maybe I need to move to the newer version for the benji VM?

Hmm, I've think I've also tested with Luminous last year and Benji only uses very basic calls to the RBD API which have existed for years.
You write:

I was in the python virtual enviroment when I installed the rados and rbd modules.
But you also have written this:

Not quite sure what you mean here?

[elemental-lf]

Sorry, I've deleted this as I was on the wrong track, but I pressed the comment button by accident, please ignore it.
But I just found this in your output:

ModuleNotFoundError: No module named 'benji.storage.rbd'

And this matches the fact the you can load Ceph's rbd module fine. So why is Python not able to find benji.storage.rbd?

[adambmedent]

Sorry, I've deleted this as I was on the wrong track, but I pressed the comment button by accident, please ignore it.
But I just found this in your output:

ModuleNotFoundError: No module named 'benji.storage.rbd'

And this matches the fact the you can load Ceph's rbd module fine. So why is Python not able to find benji.storage.rbd?

It doesn't seem to exist any were. Do you happen to know what provides this module?

(benji) [root@linux151 ~]# find / | grep benji.storage.rbd
(benji) [root@linux151 ~]#

[elemental-lf]

First of all your grep is wrong as the dots translate to directories in Python. But I found the issue. I didn't see the forest for the trees. You're actually trying to load a module that doesn't exist. In your configuration:

name: Ceph-Backup
 storageId: 1
 module: rbd # Change this to file
 configuration:
 path: /Ceph-Backups2

Storage modules are file, s3 or b2. IO modules are file and rbd.

[elemental-lf]

First of all your grep is wrong as the dots translate to directories in Python.

Actually I was wrong, the dots are okay as they match any character.

[adambmedent]

Yep, I am in action now.

When I was hitting the issue of missing "benji.io.rbd" I started messing around with that conf file and totally forgot. Once I changed it back, benji is now working. I appreciate the assistance and I will let you guys know how things go!

[elemental-lf]

That's great to hear! Please report back any issues you encounter. And if you have any feature ideas, I'd be interested to hear them, too.