Skip to content
Ceph object store designed for Samsung Key-Value SSD
C++ C Terra HTML Python Assembly Other
Branch: master
Clone or download
Latest commit 64875c8 Aug 16, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
admin KVCEPH v0.8 Apr 19, 2019
alpine KVCEPH v0.8 Apr 19, 2019
bin KVCEPH v0.8 Apr 19, 2019
build add helper scripts and a sample configuration file Apr 22, 2019
ceph-erasure-code-corpus KVCEPH v0.8 Apr 19, 2019
ceph-object-corpus KVCEPH v0.8 Apr 19, 2019
cmake/modules KVCEPH v0.8 Apr 19, 2019
debian KVCEPH v0.8 Apr 19, 2019
doc KVCEPH v0.8 Apr 19, 2019
etc KVCEPH v0.8 Apr 19, 2019
examples KVCEPH v0.8 Apr 19, 2019
fio KVCEPH v0.8 Apr 19, 2019
fusetrace KVCEPH v0.8 Apr 19, 2019
keys KVCEPH v0.8 Apr 19, 2019
man KVCEPH v0.8 Apr 19, 2019
mirroring KVCEPH v0.8 Apr 19, 2019
qa KVCEPH v0.8 Apr 19, 2019
selinux KVCEPH v0.8 Apr 19, 2019
share KVCEPH v0.8 Apr 19, 2019
src get_freespace commit Aug 16, 2019
systemd KVCEPH v0.8 Apr 19, 2019
udev KVCEPH v0.8 Apr 19, 2019
.gitignore KVCEPH v0.8 Apr 19, 2019
.gitmodule_mirrors KVCEPH v0.8 Apr 19, 2019
.gitmodules KVCEPH v0.8 Apr 19, 2019
.gitreview KVCEPH v0.8 Apr 19, 2019
.mailmap KVCEPH v0.8 Apr 19, 2019
.organizationmap KVCEPH v0.8 Apr 19, 2019
.peoplemap KVCEPH v0.8 Apr 19, 2019
1 KVCEPH v0.8 Apr 19, 2019
AUTHORS KVCEPH v0.8 Apr 19, 2019
CMakeLists.txt KVCEPH v0.8 Apr 19, 2019
CONTRIBUTING.rst KVCEPH v0.8 Apr 19, 2019
COPYING KVCEPH v0.8 Apr 19, 2019
COPYING-GPL2 KVCEPH v0.8 Apr 19, 2019
COPYING-LGPL2.1 KVCEPH v0.8 Apr 19, 2019
CURRENT KVCEPH v0.8 Apr 19, 2019
ChangeLog KVCEPH v0.8 Apr 19, 2019
CodingStyle KVCEPH v0.8 Apr 19, 2019
Doxyfile KVCEPH v0.8 Apr 19, 2019
INSTALL KVCEPH v0.8 Apr 19, 2019
KVCEPH v0.8 User guide.pdf Modified Documentation for KV-SSD support May 1, 2019
NEWS KVCEPH v0.8 Apr 19, 2019
PendingReleaseNotes KVCEPH v0.8 Apr 19, 2019
README KVCEPH v0.8 Apr 19, 2019
README.FreeBSD KVCEPH v0.8 Apr 19, 2019
README.aix KVCEPH v0.8 Apr 19, 2019
README.alpine.md KVCEPH v0.8 Apr 19, 2019
README.ceph KVCEPH v0.8 Apr 19, 2019
README.git-subtree KVCEPH v0.8 Apr 19, 2019
README.md KVCEPH v0.8 Apr 19, 2019
README.solaris KVCEPH v0.8 Apr 19, 2019
README.xio KVCEPH v0.8 Apr 19, 2019
SubmittingPatches.rst KVCEPH v0.8 Apr 19, 2019
ceph.spec.in KVCEPH v0.8 Apr 19, 2019
do_cmake.sh KVCEPH v0.8 Apr 19, 2019
do_freebsd.sh KVCEPH v0.8 Apr 19, 2019
doc_deps.deb.txt KVCEPH v0.8 Apr 19, 2019
install-deps.sh KVCEPH v0.8 Apr 19, 2019
make-apk.sh KVCEPH v0.8 Apr 19, 2019
make-debs.sh KVCEPH v0.8 Apr 19, 2019
make-dist KVCEPH v0.8 Apr 19, 2019
make-srpm.sh KVCEPH v0.8 Apr 19, 2019
pom.xml KVCEPH v0.8 Apr 19, 2019
run-make-check.sh KVCEPH v0.8 Apr 19, 2019

README.md

KVCeph - CEPH for Samsung Key-Value SSDs

KVCeph introduces a new CEPH object store, KvsStore, that is designed to support Samsung KV SSDs. The following three components are added to Ceph's Luminous release:

  • KvsStore - an object store for KV SSDs
  • Onode Prefetcher - a utility class that can prefetch Onodes (object metadata)
  • EventDriven Request Scheduler - a replacement for the existing sharded_opwq scheduler.

Checking out the source

You can clone from github with

git clone git@github.com/OpenMPDK/KVSSD.git

or, if you are not a github user,

git clone https://github.com/OpenMPDK/KVSSD.git

KVCeph is located at /KVSSD/application/KVCeph/.

Prerequisites

1. Install dependent packages

./install-deps.sh

2. Install a KV kernel driver

cd /KVSSD/PDK/driver/PCIe/kernel_driver/kernel_vX.X.X
make 
make install 
./re_insmod.sh

3. Format Samsung KV NVMe SSDs

sudo nvme format (device path) -s1 -n1

Building KVCeph

No extra steps are necessary to build KVCeph. The original Ceph build instructions can be used.

For example:

cd /KVSSD/application/KVCeph/
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=./ceph-runtime ..
make
make install  (This will install the binaries under the ceph-runtime directory)

For a list of Ceph cmake options: https://github.com/ceph/ceph/

Configuring KVCeph

Prior to running KVCeph, the following two KvsStore configuration parameters need to be added to ceph.conf.

  • osd objectstore = kvsstore - sets the type of object store to use

  • kvsstore_dev_path = /dev/path - sets the path to a Samsung KV SSD

where osd objectstore needs to be defined in the [global] section in ceph.conf, and kvsstore_dev_path should be set in each OSD section. Please also see a sample configuration file, /KVCeph/build/sample.conf, which demonstrates the use of these parameters.

Running a test cluster

For convenience, the Ceph utility vstart.sh is modified to support KvsStore. To run a test cluster with a vstart.sh:

1. Set the device path in vstart.sh

By default, kvsstore_dev_path is set to /dev/nvme0n1 in vstart.sh. 
Please modify the line that defines kvsstore_dev_path in vstart.sh if needed.

2. Run a test cluster with KvsStore

cd build
make vstart        # builds just enough to run vstart
sudo CEPH_DEV=1 MON=1 MDS=0 OSD=1 ../src/vstart.sh -n -x --kvsstore 
./bin/ceph -s

Running a cluster in deploy mode

The instruction for setting up and deploying CEPH clusters can be found at Original Ceph Deploy Manual Page .

Additionally, KVCeph requires the parameter, kvsstore_dev_path, to be set per OSD in ceph.conf, and the KV SSD kernel driver needs to be loaded in each node. The ceph.conf for KVCeph would look like the following.

	[global]
		...
		osd_objectstore = kvsstore
		...
	[osd.0]
		# settings affect osd.0 only
		kvsstore_dev_path = (nvme device path 1)
		cluster addr = (cluster network IP of Node 0)
		public addr =  (public network IP of Node 0)
		host = (hostname)
		...
	[osd.1]
	    ... 
		kvsstore_dev_path = (nvme device path 2)
		...
	[osd.2]
		...
		kvsstore_dev_path = (nvme device path 3)
		...

Unit tests

The following command will run kvsstore unittests.

cd build
./unittest_f.sh
You can’t perform that action at this time.