Question: mimic release support

[nuqz]

Hi everyone. First of all I want to thank you for this useful package.

When I try to use it with mimic release, I get this error: Operation not supported. I want to know following things:

• I use librados-dev 10.x.y from debian stretch repositories. As I can understand it is not suitable, right? How I should install librados for mimic release?
• Does this package support mimic release? If not - which ceph release is the latest suitable for this package?

[dotnwat]

Hi @nuqz. When you see operation not supported, is it only when running the tests? I've seen this too running tests on Mimic, but it was a minor issue that shouldn't affect applications using go-ceph. I have a large pending update that works with Mimic (and fixes those errors), and I'll be merging that soon.

As for compatibility, I think you should be fine with the 10.x.y library. The current approach I'm taking is to fix all the outstanding issues with the current code base, which should work with Jewel to Mimic, but has some deprecation warnings with newer libraries. Once that is solid, we'll tag it or create a branch for fixes.

Then move forward with breaking changes that would be nice to have.

[nuqz]

@noahdesu No, not only when running tests. I have a program, which works fine with jewel release. I have deployed mimic release using Docker (host network). Then I have updated go-ceph package and built application. When I run the program, it throws the error: operation not supported.

The program is very simple:

1. Create connection instance with rados.NewConn(), read default config file conn.ReadDefaultConfigFile(), connect conn.Connect();
2. Write/append key(s) to existing pool: create new IO context conn.OpenIOContext(poolName), ioctx.WriteFull(k, v)/ioctx.Append(k, v), destroy context ioctx.Destroy();
3. Destroy connection conn.Shutdown().

Is there anything I should change?

Upd: I have added some log messages to the program and now I know where the error occours - conn.Connect() throws it. I went deeper and compared my old ceph.conf with the new one. The difference was in global section:

[global]
...
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
...

I added these lines to the new ceph.conf and now I see: rados: No such file or directory (after conn.Connect()). What could I miss?

Upd. 2: I have changed auth ... required options to none and it started working. I have ceph.client.admin.keyring in /etc/ceph on the client machine (it is also Ceph host machine). As I can see there is no more differencies between old /etc/ceph and the new one, except the old one has an empty rbdmap file. Can you help me to figure out why auth is not working?

Upd. 3: Fixed /etc/ceph directory and its contents permissions (Docker owned it by 167/167 uid/gid, changed group to my user and added g+r permission) and auth started working. @noahdesu you helped me avoid a lot of unnecessary actions, thank you very much.

[dotnwat]

Hi @nuqz thanks for this detailed update. just a quick clarification: are you running jewel / mimic mixed (e.g. jewel client mimic server?). it was a little unclear the relation between how you were using jewel and mimic at the beginning of the post.

[nuqz]

@noahdesu, sorry for a huge delay. I'm not sure what is the client in this case.

My first cluster was powered by Jewel version inside the virtual machine (VirtualBox). I used librados-dev (10.2.5-7.2) from Debian Stretch repository and the latest version (downloaded it near March-April this year) of go-ceph.

After that, when I had new hardware, I deployed a cluster powered by the Mimic release, imported my pool data and updated the go-ceph package. The librados-dev library has not been updated.

The program code remained the same.

p.s. I also have a question about performance. There are several goroutines that write information to the pool. What would be better:

1. create 1 connection per application and IO contexts per goroutine's write operation;
2. create pool of connections and IO contexts per goroutine's write operation?

[dotnwat]

Hi @nuqz . The go-ceph project now has CI setup for mimic and luminous so we should see any issue now that arise between the two. If you are still running into any mimic issues I'd love to get unit tests created for those issues.

Regarding performance, that is a good question. I haven't invested this at all. An io context is thread safe, and in general can be shared by many threads running in the same application assuming they are interacting with the same pool. However, I haven't thought much about how goroutines will effect correctness since they aren't real threads. This is something we need to look at!

[phlogistonjohn]

This issue is quite old and appears to be mostly resolved. Thus I am closing the issue to cut down on our backlog of items. Please feel free to reopen the issue or, preferably, open a new issue to discuss the performance topic more.