Add ceph rados plugin #5744

bjzhang · 2018-11-06T12:34:05Z

In ipfs/go-datastore#101, @magik6k agree that we could contribute a new ceph rados datastore for go-ipfs datastore(public repo: https://github.com/ipbit/go-ds-rados).
As you know, RADOS(Reliable Autonomic Distributed Object Store) provides an extremely scalable storage service for variably sized objects. Basically, it is a distributed key-value store which is suitable for storing files.

These series patches implement the datastore plugin for ceph rados. Such plugin is based on our public repo: https://github.com/ipbit/go-ds-rados . Test passed in our environment. In our work, We read carefully and follow the guideline in https://github.com/ipfs/community/blob/master/CONTRIBUTING_GO.md .

Any feedback is welcome. Thanks in advance.

magik6k

Thanks!

We require the whole dependency tree to be under gx, so you'll need to gx import go-ceph in the datastore implementation and release the package once again. I published go-ceph to gx under QmbdFveSZQFqLNB9VXwaj4WFEX424h1tU2gqcTCZxN5oDT, so you can just do:

[in go-ds-rados]
$ gx import QmbdFveSZQFqLNB9VXwaj4WFEX424h1tU2gqcTCZxN5oDT
$ gx release patch
[pin/copy hash]
[cd to go-ipfs]
$ gx update [new go-ds-rados hash]
[commit/push]

Datastore implementation looks good, I still need to test it, but I'm definitely interested in running this for some of my nodes

plugin/loader/preload.go

magik6k · 2018-11-06T12:55:17Z

plugin/loader/preload_list

@@ -8,3 +8,4 @@ ipldgit github.com/ipfs/go-ipfs/plugin/plugins/git 0
 badgerds github.com/ipfs/go-ipfs/plugin/plugins/badgerds 0
 flatfs github.com/ipfs/go-ipfs/plugin/plugins/flatfs 0
 levelds github.com/ipfs/go-ipfs/plugin/plugins/levelds 0
+rados github.com/ipfs/go-ipfs/plugin/plugins/rados 0


Should be commented out for now

bjzhang · 2018-11-06T14:15:41Z

Hi, @magik6k thanks for importing go-ceph for me. Meanwhile go-ceph/rados require librados-dev in ubuntu. I saw that ubuntu 14.04 is used in circleci of go-ipfs, while I use ubuntu 18.04 in my environment. I am not very familiar with circleci. It seems that I should a "- run: sudo apt install librados-dev" in "go test" jobs?

It is a little bit late for me(UTC+8), I will try to make circleci work tomorrow.

bjzhang · 2018-11-06T14:23:53Z

Another off-topic question: I encounter 'ERROR: publishing: ' when I try to 'gx publish' rados of go-ceph and its library and headers. The code is here. I know @magik6k already import the whole go-ceph. But I could not understand why I am failed. Could you please give me some hint? Sorry I am not familiar with gx.

magik6k · 2018-11-06T14:59:12Z

Another off-topic question: I encounter 'ERROR: publishing: ' when I try to 'gx publish'

Currently gx can't deal with subpackages, so you should be importing whole repo - https://github.com/ipbit/go-ceph/blob/gx-rados-publish/rados/package.json#L7

bjzhang · 2018-11-07T12:17:08Z

Hi, here is the v2 of my pr. Changes:

gx import go-ceph
Skip the go-ds-rados test if we could not connect to ceph cluster through CEPH_CONF;
make building of go-ipfs successful by installing librados-dev in travis.

I am a little bit confused by the plugin of datastore. I do not know why make test_go_expensive of go-ipfs will fail.

# gx/ipfs/QmbdFveSZQFqLNB9VXwaj4WFEX424h1tU2gqcTCZxN5oDT/go-ceph/rados
../../../gx/ipfs/QmbdFveSZQFqLNB9VXwaj4WFEX424h1tU2gqcTCZxN5oDT/go-ceph/rados/ioctx.go:249:17: could not determine kind of name for C.rados_nobjects_list_close
../../../gx/ipfs/QmbdFveSZQFqLNB9VXwaj4WFEX424h1tU2gqcTCZxN5oDT/go-ceph/rados/ioctx.go:608:19: could not determine kind of name for C.rados_nobjects_list_get_pg_hash_position
../../../gx/ipfs/QmbdFveSZQFqLNB9VXwaj4WFEX424h1tU2gqcTCZxN5oDT/go-ceph/rados/ioctx.go:253:10: could not determine kind of name for C.rados_nobjects_list_next
../../../gx/ipfs/QmbdFveSZQFqLNB9VXwaj4WFEX424h1tU2gqcTCZxN5oDT/go-ceph/rados/ioctx.go:245:9: could not determine kind of name for C.rados_nobjects_list_open
../../../gx/ipfs/QmbdFveSZQFqLNB9VXwaj4WFEX424h1tU2gqcTCZxN5oDT/go-ceph/rados/ioctx.go:612:2: could not determine kind of name for C.rados_nobjects_list_seek

In my understanding, it is responsibility of datastore plugin(such as go-ds-rados) to make sure all the test passed. It should be decoupled with go-ipfs.

Thanks.

magik6k · 2018-11-07T18:41:46Z

The errors you get are cgo errors, but I have no idea why it even tries to compile the plugin for tests here. cc @Kubuxu for help.

Optimally we wouldn't have to build this plugin on CI as the cgo dependency is rather painful, I'm not sure what are the better options here

magik6k · 2018-11-07T18:26:39Z

.travis.yml

@@ -16,6 +16,9 @@ env:
  - TEST_NO_FUSE=1 TEST_VERBOSE=1 TEST_SUITE=test_go_expensive
  - TEST_NO_FUSE=1 TEST_VERBOSE=1 TEST_SUITE=test_sharness_expensive

+before_install:
+  - sudo apt-get install -y librados-dev


Won't work on osx - https://travis-ci.org/ipfs/go-ipfs/jobs/451847996

fix it in today's pr

License: MIT Signed-off-by: Shaoxiong Li <frank.li@ipfsbit.com>

So we can get go-ds-rados v0.0.3 License: MIT Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

License: MIT Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

So we can get go-ds-rados v0.0.4 License: MIT Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

License: MIT Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

bjzhang · 2018-11-08T13:51:08Z

The errors you get are cgo errors, but I have no idea why it even tries to compile the plugin for tests here. cc @Kubuxu for help.

After check the environment of travis ci, I found that there are some changes of librados-dev after ubuntu 1404 in which travis build the package. I could build successful in ubuntu 1804.

Optimally we wouldn't have to build this plugin on CI as the cgo dependency is rather painful, I'm not sure what are the better options here

Agree. How about create new xxx-plugin project, such radosds-plugin. And build them in supported environment. Then load them(plugin.Open(mod)) dynamically when ipfs daemon start.

whyrusleeping · 2018-11-20T04:03:38Z

This should probably just exist in its own repo, with nice instructions for users to build and install the plugin. The whole point of these plugins is so that they can live in places other than the main go-ipfs repo.

We can even link to it from the readme here, but adding it to the codebase feels a bit too heavy

whyrusleeping · 2018-11-20T04:04:19Z

That said though, this is super cool. I love seeing people extend ipfs in different ways like this :)

bjzhang · 2018-11-21T07:10:12Z

Hi, Thanks. When you are talking about "too heavy", do you mean cgo or the entire go ceph rados datastore?

We use swift to access ceph in our openstack cloud. From my view, there is no need to add one more layer above ceph when we use the ipfs.
And on the other hand, we create nentropy which build on the top of badger. Nentropy is a pure go project. We already finish the functional test and basic performance test. We are planning to do stress test. Are you interested in such datastore, if it could be used in production?

BTW, I upload the rados profile in our go-ipfs-config/profile.go yesterday. I will write a README.

bjzhang · 2018-11-23T08:20:46Z

Hi, @whyrusleeping

I notice that go-ipfs could load the plugins in ~/.ipfs/plugins. I could provide rados-plugin as separate repo and provide document about how to use it.

Is it make sense to you?

whyrusleeping · 2018-11-27T19:15:46Z

@bjzhang Yeah, i’m talking about the whole thing. Most users of go-ipfs won’t need this, and so to the average user, this is just ‘extra bloaty features’. But that’s exactly why we made the plugins load able through the plugins directory. Making this a separate package with nice build instructions is the right way to move things forward. Plus, once we do that, we can feature your plugin in the readme (and probably announce it in our newsletter so other interested people can hear about it).

bjzhang · 2018-11-30T13:43:16Z

Making this a separate package with nice build instructions is the right way to move things forward. Plus, once we do that, we can feature your plugin in the readme (and probably announce it in our newsletter so other interested people can hear about it).

Cool. What could I do for this part? Is there some detailed requirement of it?

Stebalien · 2018-12-26T23:48:46Z

@bjzhang,

Sorry for the delay, I've created an example plugin here: https://github.com/ipfs/go-ipfs-example-plugin. That should be a good starting point for your datastore plugin.

Stebalien · 2018-12-27T00:50:56Z

New documentation: https://github.com/ipfs/go-ipfs/blob/f706bc69ba5f27b845ba594b2ca28e63dd17529f/docs/plugins.md

bjzhang · 2018-12-27T02:53:22Z

@bjzhang,

Sorry for the delay, I've created an example plugin here: https://github.com/ipfs/go-ipfs-example-plugin. That should be a good starting point for your datastore plugin.

Thanks @Stebalien . I will follow your document and create a pr to available-plugins

Stebalien · 2018-12-27T03:20:23Z

Thanks! I'm going to close this PR in the mean time.

bjzhang requested a review from Kubuxu as a code owner November 6, 2018 12:34

magik6k reviewed Nov 6, 2018

View reviewed changes

bjzhang force-pushed the feat/radosds-plugin branch 4 times, most recently from d1edc2e to 8c64688 Compare November 7, 2018 12:01

magik6k reviewed Nov 7, 2018

View reviewed changes

passerby888 and others added 5 commits November 8, 2018 21:38

plugin/plugins: add rados implementation for datastore backend

26f5c91

License: MIT Signed-off-by: Shaoxiong Li <frank.li@ipfsbit.com>

Update go-ds-rados

22dc8a6

So we can get go-ds-rados v0.0.3 License: MIT Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

travis: add librados-dev

290c27f

License: MIT Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

Update go-ds-rados

4cac906

So we can get go-ds-rados v0.0.4 License: MIT Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

Only build radosds on Linux

1ab91e2

License: MIT Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

bjzhang force-pushed the feat/radosds-plugin branch from 8c64688 to 1ab91e2 Compare November 8, 2018 13:39

Stebalien closed this Dec 27, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ceph rados plugin #5744

Add ceph rados plugin #5744

bjzhang commented Nov 6, 2018

magik6k left a comment

magik6k Nov 6, 2018

bjzhang Nov 6, 2018

bjzhang Nov 7, 2018

bjzhang commented Nov 6, 2018

bjzhang commented Nov 6, 2018

magik6k commented Nov 6, 2018

bjzhang commented Nov 7, 2018 •

edited

magik6k commented Nov 7, 2018

magik6k Nov 7, 2018

bjzhang Nov 8, 2018

bjzhang commented Nov 8, 2018

whyrusleeping commented Nov 20, 2018

whyrusleeping commented Nov 20, 2018

bjzhang commented Nov 21, 2018

bjzhang commented Nov 23, 2018 •

edited

whyrusleeping commented Nov 27, 2018

bjzhang commented Nov 30, 2018

Stebalien commented Dec 26, 2018

Stebalien commented Dec 27, 2018

bjzhang commented Dec 27, 2018

Stebalien commented Dec 27, 2018

Add ceph rados plugin #5744

Add ceph rados plugin #5744

Conversation

bjzhang commented Nov 6, 2018

magik6k left a comment

Choose a reason for hiding this comment

magik6k Nov 6, 2018

Choose a reason for hiding this comment

bjzhang Nov 6, 2018

Choose a reason for hiding this comment

bjzhang Nov 7, 2018

Choose a reason for hiding this comment

bjzhang commented Nov 6, 2018

bjzhang commented Nov 6, 2018

magik6k commented Nov 6, 2018

bjzhang commented Nov 7, 2018 • edited

magik6k commented Nov 7, 2018

magik6k Nov 7, 2018

Choose a reason for hiding this comment

bjzhang Nov 8, 2018

Choose a reason for hiding this comment

bjzhang commented Nov 8, 2018

whyrusleeping commented Nov 20, 2018

whyrusleeping commented Nov 20, 2018

bjzhang commented Nov 21, 2018

bjzhang commented Nov 23, 2018 • edited

whyrusleeping commented Nov 27, 2018

bjzhang commented Nov 30, 2018

Stebalien commented Dec 26, 2018

Stebalien commented Dec 27, 2018

bjzhang commented Dec 27, 2018

Stebalien commented Dec 27, 2018

bjzhang commented Nov 7, 2018 •

edited

bjzhang commented Nov 23, 2018 •

edited