Storage Driver: Ceph Object Storage (RADOS) #443

gierschv · 2015-04-27T15:28:36Z

This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.

Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:

All the objects data are stored with opaque UUID names prefixed
(e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
All the hierarchy information are stored in rados omaps, where the
omap object identifier is the virtual directory name, the keys in
a specific are the relative filenames and the values the blob
object identifier (or empty value for a sub directory).

e.g. For the following hierarchy:

/directory1
/directory1/object1
/directory1/object2
/directory1/directory2/object3

The omap "/directory1" will contains the following key / values:
- "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
- "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
- "directory2" ""
The omap "/directory1/directory2" will contains:
- "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"
The MOVE is implemented by changing the reference to a specific
blob in its parent virtual directory omap.

Todo for this WIP driver:

Stripes rados objects to a fixed size (e.g. 4M). The idea is
to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) will be stored as a
Xattr in the first chunk object.
Driver tests
List all the keys of the omaps without a fixed limit of keys (used
to list virtual directories).

Related: #40
Dependencies: dotnwat/go-ceph#15, dotnwat/go-ceph#16 and dotnwat/go-ceph#19 (merged)

Signed-off-by: Vincent Giersch vincent.giersch@ovh.net

gierschv · 2015-04-27T15:39:49Z

@simon3z @noahdesu you might be interested :)

stevvooe · 2015-04-30T01:54:10Z

@gierschv This is looking promising. Can't wait to see the testsuites passing.

Will you be able to maintain this in the future?
How can we test this with circleci (or other ci)?

gierschv · 2015-04-30T16:22:19Z

@stevvooe Yes, I can maintain it in the future. For the CI, I installed a ceph micro cluster in Circle CI, as done in go-ceph for travis and it works pretty fine:

ok      github.com/docker/distribution/registry/storage/driver/rados    58.569s coverage: 78.3% of statements

stevvooe · 2015-05-01T01:49:31Z

registry/storage/driver/rados/rados.go

+// Remove a file from the files hierarchy
+func (d *driver) DeleteOid(path string) error {
+	// Remove object reference
+	directory := filepath.Dir(path)


Considering that we don't want the path layout to change when running on unix or windows, this should use the path package rather than filepath.

Ok, I replaced filepath by path.

gierschv · 2015-05-04T18:54:13Z

Added dotnwat/go-ceph#19 (which is now merged) for the third todo ("List all the keys of the omaps without a fixed limit of keys"). I'll work on it tomorrow.

stevvooe · 2015-05-04T19:29:12Z

registry/storage/driver/rados/rados.go

+}
+
+// Reference a object and its hierarchy
+func (d *driver) PutOid(objectPath string, oid string) error {


Do these methods need to be exported?

stevvooe · 2015-05-05T00:21:29Z

@gierschv Another thought here: Will a user require to the librados libraries to be available to build the registry? If so, should we hide rados support behind a compilation flag. We'll build the official images with rados built-in but developers shouldn't be required to build with it.

gierschv · 2015-05-05T13:55:11Z

@stevvooe Indeed, for now it requires the librados to build it.
I thought about adding a build tag using -tags. Do you guys prefer an environment variable passed when building, or a check if rados/librados.h is available? If so, should I add a separate shell script as done in the docker engine (i.e. hack/make.sh), or add it here directly in the Makefile (e.g. by adding something like DOCKER_BUILDTAGS=$(shell echo '\#include <rados/librados.h>' | gcc -E - &> /dev/null || echo 'exclude_rados'))

stevvooe · 2015-05-05T18:37:38Z

@gierschv I think explicit build tags are the right approach. No need for a separate build script, as that is outside the scope of this PR. Just make sure the build tag is easy to use and let's add some documentation to building.md.

dotnwat · 2015-05-05T20:56:22Z

@gierschv is it possible to reference a specific tag or branch for go-ceph from dependencies such as this pull request? I would like to avoid cutting a stable release since the API is still be hashed out, but would be happy to cut a branch or tag since it looks like we have more than a couple users now.

gierschv · 2015-05-20T01:41:26Z

@noahdesu @dmp42 @stevvooe Finally fixed setup script for circle. Thanks Olivier for the debug access :-)

This driver implements the storagedriver.StorageDriver interface and uses Ceph Object Storage as storage backend. Since RADOS is an object storage and no hierarchy notion, the following convention is used to keep the filesystem notions stored in this backend: * All the objects data are stored with opaque UUID names prefixed (e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164). * All the hierarchy information are stored in rados omaps, where the omap object identifier is the virtual directory name, the keys in a specific are the relative filenames and the values the blob object identifier (or empty value for a sub directory). e.g. For the following hierarchy: /directory1 /directory1/object1 /directory1/object2 /directory1/directory2/object3 The omap "/directory1" will contains the following key / values: - "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164" - "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0" - "directory2" "" The omap "/directory1/directory2" will contains: - "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9" * The MOVE is implemented by changing the reference to a specific blob in its parent virtual directory omap. This driver stripes rados objects to a fixed size (e.g. 4M). The idea is to keep small objects (as done by RBD on the top of RADOS) that will be easily synchronized accross OSDs. The information of the original object (i.e total size of the chunks) is stored as a Xattr in the first chunk object. Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>

stevvooe · 2015-05-30T00:08:45Z

LGTM!

Do we want this to ship with the official registry? Do we need to make adjustments to the dockerfile to ensure this works?

gierschv · 2015-05-30T00:12:49Z

@stevvooe I can change the Dockerfile if we ship the drive with the registry :) Please let me know, I'll make the changes.

stevvooe · 2015-05-30T01:07:55Z

@gierschv Want to do that in a separate PR?

gierschv · 2015-05-30T01:09:27Z

@stevvooe ok, I'll do that.

Storage Driver: Ceph Object Storage (RADOS)

RyanDeng · 2015-06-08T09:28:52Z

@gierschv I still got error while make

../github.com/noahdesu/go-ceph/rados/conn.go:5:29: fatal error: rados/librados.h: No such file or directory

Is that you missed add // +build include_rados in /registry/storage/driver/rados/rados.go and rados_test.go? Since the error didn't appear after I added this line

gierschv · 2015-06-08T10:34:36Z

@RyanDeng This tag is present in the package main, so the rados package should only be imported if the tag is present. I'll check for the test.

RyanDeng · 2015-06-08T10:44:19Z

@gierschv Thanks for looking into this. I saw the tag in distribution/cmd/registry/rados.go so I'm also curious why here the error appeared. Could be some missing in my GOPATH

stevvooe · 2015-06-08T20:04:24Z

@RyanDeng @gierschv I think the issue is that the tests are still built when using the makefile. Perhaps, the build flag needs to be added to the test files. You can safely ignore this error when running a build.

RyanDeng · 2015-06-09T02:08:22Z

@stevvooe that makes sense. Thanks for your advice.

stevvooe · 2015-06-09T02:16:36Z

@gierschv @RyanDeng Would it be possible to get this worked out in a PR?

RyanDeng · 2015-06-09T03:11:41Z

@stevvooe @gierschv Please review it. Thanks

farcaller · 2015-08-01T09:53:09Z

Would be nice to have some basic documentation along with this, in terms of what's needed to run the backend properly.

So far I've tried adding /etc/ceph/ceph.conf from my cluster and making a one-user keychain at /etc/ceph/keychain, passing in pool and user names, and getting stuck at api error -1.

yadutaf · 2015-08-02T20:34:32Z

Have you looked at these?

stevvooe · 2015-08-03T17:33:40Z

@farcaller @yadutaf Please don't comment on closed or merged issues.

Storage Driver: Ceph Object Storage (RADOS)

GordonTheTurtle added the status/0-needs-triage label Apr 27, 2015

stevvooe added the feature label Apr 28, 2015

gierschv force-pushed the driver-rados branch 11 times, most recently from a10d23a to 287ebfc Compare April 30, 2015 16:09

stevvooe reviewed May 1, 2015
View reviewed changes

gierschv force-pushed the driver-rados branch from 287ebfc to 33a1540 Compare May 4, 2015 13:17

stevvooe reviewed May 4, 2015
View reviewed changes

gierschv force-pushed the driver-rados branch from 33a1540 to 64aa994 Compare May 5, 2015 13:44

gierschv changed the title ~~[WIP] Storage Driver: Ceph Object Storage (RADOS)~~ Storage Driver: Ceph Object Storage (RADOS) May 5, 2015

gierschv changed the title ~~Storage Driver: Ceph Object Storage (RADOS)~~ [WIP] Storage Driver: Ceph Object Storage (RADOS) May 5, 2015

stevvooe mentioned this pull request May 5, 2015

Storage support openstack swift #493

Merged

gierschv force-pushed the driver-rados branch 4 times, most recently from 2abc046 to 3bc9587 Compare May 20, 2015 01:25

gierschv force-pushed the driver-rados branch from 3bc9587 to 394eea0 Compare May 20, 2015 01:44

stevvooe added a commit that referenced this pull request May 30, 2015

Merge pull request #443 from gierschv/driver-rados

318af0b

Storage Driver: Ceph Object Storage (RADOS)

stevvooe merged commit 318af0b into distribution:master May 30, 2015

dmp42 mentioned this pull request May 31, 2015

Docs for storage drivers versioned in external repositories #577

Closed

gierschv deleted the driver-rados branch June 1, 2015 09:40

gierschv mentioned this pull request Jun 1, 2015

Dockerfile: build with the RADOS driver #579

Merged

RyanDeng mentioned this pull request Jun 9, 2015

add build include_rados tag to prevent MAKE failing #611

Closed

stevvooe mentioned this pull request Jul 1, 2015

Implement Ali OSS storage driver #669

Closed

thaJeztah pushed a commit to thaJeztah/distribution that referenced this pull request Apr 22, 2021

Merge pull request distribution#443 from gierschv/driver-rados

1428853

Storage Driver: Ceph Object Storage (RADOS)

thaJeztah pushed a commit to thaJeztah/distribution that referenced this pull request Jan 19, 2022

Merge pull request distribution#443 from gierschv/driver-rados

af987fd

Storage Driver: Ceph Object Storage (RADOS)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage Driver: Ceph Object Storage (RADOS) #443

Storage Driver: Ceph Object Storage (RADOS) #443

gierschv commented Apr 27, 2015

gierschv commented Apr 27, 2015

stevvooe commented Apr 30, 2015

gierschv commented Apr 30, 2015

stevvooe May 1, 2015

gierschv May 4, 2015

gierschv commented May 4, 2015

stevvooe May 4, 2015

stevvooe commented May 5, 2015

gierschv commented May 5, 2015

stevvooe commented May 5, 2015

dotnwat commented May 5, 2015

gierschv commented May 20, 2015

stevvooe commented May 30, 2015

gierschv commented May 30, 2015

stevvooe commented May 30, 2015

gierschv commented May 30, 2015

RyanDeng commented Jun 8, 2015

gierschv commented Jun 8, 2015

RyanDeng commented Jun 8, 2015

stevvooe commented Jun 8, 2015

RyanDeng commented Jun 9, 2015

stevvooe commented Jun 9, 2015

RyanDeng commented Jun 9, 2015

farcaller commented Aug 1, 2015

yadutaf commented Aug 2, 2015

stevvooe commented Aug 3, 2015

Storage Driver: Ceph Object Storage (RADOS) #443

Storage Driver: Ceph Object Storage (RADOS) #443

Conversation

gierschv commented Apr 27, 2015

gierschv commented Apr 27, 2015

stevvooe commented Apr 30, 2015

gierschv commented Apr 30, 2015

stevvooe May 1, 2015

Choose a reason for hiding this comment

gierschv May 4, 2015

Choose a reason for hiding this comment

gierschv commented May 4, 2015

stevvooe May 4, 2015

Choose a reason for hiding this comment

stevvooe commented May 5, 2015

gierschv commented May 5, 2015

stevvooe commented May 5, 2015

dotnwat commented May 5, 2015

gierschv commented May 20, 2015

stevvooe commented May 30, 2015

gierschv commented May 30, 2015

stevvooe commented May 30, 2015

gierschv commented May 30, 2015

RyanDeng commented Jun 8, 2015

gierschv commented Jun 8, 2015

RyanDeng commented Jun 8, 2015

stevvooe commented Jun 8, 2015

RyanDeng commented Jun 9, 2015

stevvooe commented Jun 9, 2015

RyanDeng commented Jun 9, 2015

farcaller commented Aug 1, 2015

yadutaf commented Aug 2, 2015

stevvooe commented Aug 3, 2015