rbd: support rbd on FreeBSD (without KRBD) #12798

wjwithagen · 2017-01-05T11:33:43Z

Cmake changes to exclude KRBD and keyutils from building because FreeBSD does not support that.
Code changes to:

Stub krbd kernel calls
small incompatabilities, missing includes.
Fix Clang casts errors

Signed-off-by: Willem Jan Withagen wjw@digiware.nl

wjwithagen · 2017-01-05T15:46:46Z

Hi I'm getting this casting error in Clang that doesn't want to go away, with any of the casting version, other than static_cast.
Depressing is that GNU really doesn't like that, so either we fix the code, or I if-else-endif it.

I hope you have some suggestions.

/usr/srcs/Ceph/work/ceph/src/test/librbd/test_mock_fixture.h:28:27: error: reinterpret_cast from 'nullptr_t' to 'ContextWQ *' is not allowed
  ContextWQ *context_wq = reinterpret_cast<ContextWQ *>(wq);
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/srcs/Ceph/work/ceph/src/googletest/googlemock/include/gmock/gmock-generated-actions.h:523:27: note: in instantiation of function template specialization 'CompleteContextActionP2<int, nullptr_t>::gmock_Impl<void (Context *)>::gmock_PerformImpl<Context *, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg>' requested here
    return impl->template gmock_PerformImpl<A0>(args, get<0>(args),
                          ^

tchaikov · 2017-01-05T15:49:16Z

@wjwithagen i will take a look at it tomorrow.

wjwithagen · 2017-01-05T17:36:37Z

@tchaikov

Probably it is due to NULL usuage on:
/usr/srcs/Ceph/work/ceph/src/test/librbd/test_mock_Journal.cc:268:29

But as usual I'm not able to fix it. nullptr/NULL is not quite my thing.

tchaikov · 2017-01-06T05:04:08Z

src/CMakeLists.txt

    rbd_types
    rbd_replay_types)
+  if(WITH_KRBD)


could you squash this commit into 6022d01 ?

tchaikov · 2017-01-06T05:51:42Z

src/librbd/CMakeLists.txt

  ${CMAKE_DL_LIBS}
  ${EXTRALIBS})
+if(HAVE_UDEV)


might want to remove udev from the linked libraries of librbd. as krbd links against it already.

@tchaikov
To start it should use the define for that:
./cmake/modules/Findudev.cmake:23:# UDEV_LIBRARIES - List of libraries when using udev.
But I feel uncomfortable removing it, since it was already there, and I try to change as little functionality as possible when adding FreeBSD/Clang specifics.
Old behaviour should be kept as much as possible for all others.

That said, you are right: librbd add udev.
I'll make a separate PR for that, and rebase this one.

I try to change as little functionality as possible when adding FreeBSD/Clang specifics.

i agree, but behaviorwise, removing libudev here does not change anything.

I'll make a separate PR for that, and rebase this one.

thanks.

tchaikov · 2017-01-06T05:53:30Z

src/test/cls_rbd/CMakeLists.txt

+endif()
+if(HAVE_UDEV)
+  target_link_libraries(ceph_test_cls_rbd
+    udev)


could just remove blkid and udev from ceph_test_cls_rbd's linked libraries.

@tchaikov
Yup, like above

tchaikov · 2017-01-06T05:55:11Z

CMakeLists.txt

@@ -233,6 +233,9 @@ option(WITH_KVS "Key value store is here" ON)
 # remote block storage
 option(WITH_RBD "Remote block storage is here" ON)

+# KERNEL remote block storage
+option(WITH_KRBD "Kernel Remote block storage is here" ON)


should send out a FATAL_ERROR message if WITH_KRBD=ON and WITH_RBD=OFF.

@tchaikov
I'll add some diganostics

~~this is not addressed.~~

tchaikov · 2017-01-06T06:16:25Z

src/include/compat.h

@@ -38,6 +38,9 @@
 #undef ENODATA
 #define ENODATA ENOATTR
 #endif
+#ifndef EREMOTEIO


could you consolidate this #define with the ones at line:79. so there is only one place that we define EREMOTEIO and HOST_NAME_MAX. also instead of HOST_NAME_MAX , FreeBSD has MAXHOSTNAMELEN. maybe we can define HOST_NAME_MAX like

#ifndef HOST_NAME_MAX # ifdef MAXHOSTNAMELEN) # define HOST_NAME_MAX MAXHOSTNAMELEN # else # define HOST_NAME_MAX 64 # endif #endif

BTW, 64 is a very conservative number. probably we can make it 255.

tchaikov · 2017-01-06T06:23:36Z

src/tools/rbd/action/Kernel.cc

@@ -163,6 +163,7 @@ static int parse_unmap_options(char *options)

 static int do_kernel_showmapped(Formatter *f)
 {
+#if defined(__linux__)


probably you can just ignore Kernel.cc and Nbd.cc in rbd_srcs in src/tools/rbd/CMakeLists.txt if !WITH_KRBD .

@tchaikov
I'm leaving this in for the moment, because I would like to not change the run-cli tests.
And that expects showmapped map unmap in the help output.
So for met that would make this the most transparent fix.

makes sense. it's non-trivial to conditionalize the run-cli tests, i guess.

Should it be based on WITH_KRBD?

@wjwithagen could you add WITH_KRBD to config-h.in.cmake and conditionalize on it instead?

tchaikov · 2017-01-06T06:27:42Z

src/test/librbd/test_mock_fixture.h

@@ -25,7 +25,11 @@ ACTION_P(CopyInBufferlist, str) {
 }

 ACTION_P2(CompleteContext, r, wq) {
+#if defined(__clang__)
+  ContextWQ *context_wq = static_cast<ContextWQ *>(wq);


although this is a little bit awkward, but this is the best we can have with minimal change, otherwise we need to change some of the callers of CompleteContext to cast the NULL parameter to the ContextWQ *.

@dillaman what do you think?

@tchaikov @dillaman
I'm not a real fan of this sollution either,
But it is the least invasive, and still very obvious as to why this is done.

Hmm --- it's pretty ugly so I might vote for just using a C-style cast if Clang doesn't like it.

... do you have any details on the compile failure? The only way I can see this failing is if somehow wq auto-resolved to nullptr_t, but nothing that I can find is passing in nullptr (unless Clang is somehow substituting nullptr for NULL -- which is wrong since they are not the same)

@dillaman @tchaikov
This is the full output of the build error when I use the original cast:

Building CXX object src/test/librbd/CMakeFiles/unittest_librbd.dir/test_mock_Journal.cc.o cd /usr/srcs/Ceph/work/ceph/build/src/test/librbd && ccache /usr/bin/CC -DCEPH_LIBDIR=\"/usr/local/lib\" -DCEPH_PKGLIBDIR=\"/usr/local/lib/ceph\" -DTEST_LIBRBD_INTERNALS -I/usr/srcs/Ceph/work/ceph/build/src/include -I/usr/srcs/Ceph/work/ceph/src -isystem /usr/local/include -isystem /usr/srcs/Ceph/work/ceph/build/include -I/usr/srcs/Ceph/work/ceph/src/xxHash -I/usr/srcs/Ceph/work/ceph/src/googletest/googlemock/include -I/usr/srcs/Ceph/work/ceph/src/googletest/googletest/include -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing -fsigned-char -Wno-unused-function -Wno-unused-local-typedef -Wno-varargs -Wno-gnu-designator -Wno-missing-braces -Wno-parentheses -Wno-deprecated-register -ftemplate-depth-1024 -Wno-invalid-offsetof -Wnon-virtual-dtor -Wno-inconsistent-missing-override -Wno-mismatched-tags -Wno-unused-private-field -fdiagnostics-color=auto -I/usr/local/include/nss/nss -I/usr/local/include/nspr -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -Wno-null-dereference -O0 -g -fPIE -DHAVE_CONFIG_H -D__CEPH__ -D_REENTRANT -D_THREAD_SAFE -D__STDC_FORMAT_MACROS -I/usr/srcs/Ceph/work/ceph/src/googletest/googlemock/include -I/usr/srcs/Ceph/work/ceph/build/src/googletest/googlemock/include -I/usr/srcs/Ceph/work/ceph/src/googletest/googletest/include -I/usr/srcs/Ceph/work/ceph/build/src/googletest/googletest/include -fno-strict-aliasing -std=c++11 -o CMakeFiles/unittest_librbd.dir/test_mock_Journal.cc.o -c /usr/srcs/Ceph/work/ceph/src/test/librbd/test_mock_Journal.cc In file included from /usr/srcs/Ceph/work/ceph/src/test/librbd/test_mock_Journal.cc:4: /usr/srcs/Ceph/work/ceph/src/test/librbd/test_mock_fixture.h:31:27: error: reinterpret_cast from 'nullptr_t' to 'ContextWQ *' is not allowed ContextWQ *context_wq = reinterpret_cast<ContextWQ *>(wq); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /usr/srcs/Ceph/work/ceph/src/googletest/googlemock/include/gmock/gmock-generated-actions.h:523:27: note: in instantiation of function template specialization 'CompleteContextActionP2<int, nullptr_t>::gmock_Impl<void (Context *)>::gmock_PerformImpl<Context *, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg, testing::internal::ExcessiveArg>' requested here return impl->template gmock_PerformImpl<A0>(args, get<0>(args), ^ /usr/srcs/Ceph/work/ceph/src/test/librbd/test_mock_fixture.h:27:1: note: in instantiation of function template specialization 'testing::internal::ActionHelper<void, CompleteContextActionP2<int, nullptr_t>::gmock_Impl<void (Context *)> >::Perform<Context *>' requested here ACTION_P2(CompleteContext, r, wq) { ^ /usr/srcs/Ceph/work/ceph/src/googletest/googlemock/include/gmock/gmock-generated-actions.h:1420:13: note: expanded from macro 'ACTION_P2' Perform(this, args);\ ^ /usr/srcs/Ceph/work/ceph/src/test/librbd/test_mock_fixture.h:27:1: note: in instantiation of member function 'CompleteContextActionP2<int, nullptr_t>::gmock_Impl<void (Context *)>::Perform' requested here /usr/srcs/Ceph/work/ceph/src/googletest/googlemock/include/gmock/gmock-generated-actions.h:1416:7: note: expanded from macro 'ACTION_P2' gmock_Impl(p0##_type gmock_p0, p1##_type gmock_p1) : p0(gmock_p0), \ ^ /usr/srcs/Ceph/work/ceph/src/test/librbd/test_mock_fixture.h:27:1: note: in instantiation of member function 'CompleteContextActionP2<int, nullptr_t>::gmock_Impl<void (Context *)>::gmock_Impl' requested here /usr/srcs/Ceph/work/ceph/src/googletest/googlemock/include/gmock/gmock-generated-actions.h:1436:39: note: expanded from macro 'ACTION_P2' return ::testing::Action<F>(new gmock_Impl<F>(p0, p1));\ ^ /usr/srcs/Ceph/work/ceph/src/test/librbd/test_mock_Journal.cc:268:29: note: in instantiation of function template specialization 'CompleteContextActionP2<int, nullptr_t>::operator Action<void (Context *)>' requested here .WillOnce(CompleteContext(0, NULL)); ^ 1 error generated.

@dillaman
Not quite.....
A nicer sollution might be to actually fix the typecaset for NULL:
CompleteContext(r, static_cast<ContextWQ *>(NULL))

That works for both FreeBSD/Clang and linux/GCC.
Need to fix that in about 8 locations.
Could do that in a separate PR and drop this commit from this PR?

That's fine as well -- long term would still need a Clang compiler in the Jenkins build set to ensure future PRs don't break things

@dillaman
And it needs to be running with a environment where NULL is defined to be nullptr !!
Will submit a PR tomorrow. It is past midnight here.

yeah, that's also what i suggested in the beginning. as NULL could also be defined as 0 where static_cast<>() does not work.

@tchaikov @dillaman
Gentleman,
If you could checkout #12880.
Once that is accepted, I'll rebase this PR to macht it.

wjwithagen · 2017-01-06T09:25:45Z

@tchaikov
Thanx for the review... I'll go over them and fix.

tchaikov · 2017-01-06T09:58:19Z

src/include/compat.h

 #define HOST_NAME_MAX 255
+#endif


see #12798 (comment), could use MAXHOSTNAMELEN instead hardwire HOST_NAME_MAX to 255.

and could move the #endif /* __APPLE__ */ before #ifndef EREMOTEIO and change it to #endif.

tchaikov · 2017-01-06T16:00:59Z

@wjwithagen needs rebase.

wjwithagen · 2017-01-06T16:13:35Z

@tchaikov
Yup, due to the UDEV commit. rebased.
I'm in and out of the datacenter with screwdriver and more, So this needs to go inbetween real work. :)

tchaikov · 2017-01-08T10:38:51Z

CMakeLists.txt

+option(WITH_KRBD "Kernel Remote block storage is here" ON)
+
+if(WITH_KRBD AND !WITH_RBD)
+  message(FATAL_ERROR "Cannot have WITH_KRBD with WITH_RBD.")


s/with/without/

@tchaikov
Every day something new to learn.... ;-)

tchaikov · 2017-01-08T10:45:19Z

src/tools/rbd/action/Kernel.cc

@@ -163,6 +163,7 @@ static int parse_unmap_options(char *options)

 static int do_kernel_showmapped(Formatter *f)
 {
+#if defined(__linux__)


@wjwithagen could you add WITH_KRBD to config-h.in.cmake and conditionalize on it instead?

wjwithagen · 2017-01-08T20:27:36Z

@tchaikov
I think I addressed your suggestions...

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

- and combined some FreeBSD defines with Apple's Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

…c_cast is sufficient Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

wjwithagen · 2017-01-12T15:20:44Z

@tchaikov @dillaman
Rebased for the NULL-cast if-def PR.
So this PR is now more or less how it should be as well.

dillaman

lgtm

wjwithagen mentioned this pull request Jan 5, 2017

rgw_file: radosgw-admin can be built under FreeBSD #12191

Merged

wjwithagen force-pushed the wip-wjw-krbd-optional branch from 2a6ed91 to 2f6a2aa Compare January 5, 2017 13:31

tchaikov self-assigned this Jan 5, 2017

wjwithagen force-pushed the wip-wjw-krbd-optional branch from 2f6a2aa to 37c909e Compare January 5, 2017 17:37

tchaikov requested changes Jan 6, 2017

View reviewed changes

tchaikov removed their assignment Jan 6, 2017

tchaikov added build/ops cleanup rbd labels Jan 6, 2017

tchaikov requested a review from dillaman January 6, 2017 06:31

wjwithagen force-pushed the wip-wjw-krbd-optional branch from 37c909e to 382117e Compare January 6, 2017 09:50

tchaikov reviewed Jan 6, 2017

View reviewed changes

wjwithagen force-pushed the wip-wjw-krbd-optional branch 2 times, most recently from 7861936 to d917dfb Compare January 6, 2017 10:47

wjwithagen force-pushed the wip-wjw-krbd-optional branch from d917dfb to 6150774 Compare January 6, 2017 16:11

tchaikov requested changes Jan 8, 2017

View reviewed changes

wjwithagen force-pushed the wip-wjw-krbd-optional branch from 6150774 to fb38fac Compare January 8, 2017 20:27

wjwithagen force-pushed the wip-wjw-krbd-optional branch from fb38fac to 3ebd88a Compare January 8, 2017 20:54

wjwithagen added 5 commits January 12, 2017 16:12

CMakeLists.txt: Introduce WITH_KBRD for kernel RBD

78c04b1

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

./do_freebsd.sh: Do not use KRBD

a491ccd

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common/event_socket.h: include errno.h to use errno

0521483

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

include/compat.h: define EREMOTEIO for FreeBSD

e30c27f

- and combined some FreeBSD defines with Apple's Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

include/rbd_types.h: include/types.h for __le*

915bd4a

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

wjwithagen added 3 commits January 12, 2017 16:12

tools/rbd/action/Kernel.cc: create stubs for FreeBSD

3d28554

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

test/librbd/Clang: Clang complains about reinterpret_cast where stati…

c605903

…c_cast is sufficient Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

test/run-cli-tests: enable RBD testing

f12c8eb

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

wjwithagen force-pushed the wip-wjw-krbd-optional branch from 3ebd88a to f12c8eb Compare January 12, 2017 15:19

dillaman approved these changes Jan 12, 2017

View reviewed changes

tchaikov approved these changes Jan 12, 2017

View reviewed changes

tchaikov self-assigned this Jan 12, 2017

dillaman changed the title ~~RBD: enable rbd on FreeBSD (without KRBD)~~ rbd: support rbd on FreeBSD (without KRBD) Jan 16, 2017

dillaman merged commit 7c55be9 into ceph:master Jan 16, 2017

wjwithagen deleted the wip-wjw-krbd-optional branch April 28, 2017 09:22

rbd: support rbd on FreeBSD (without KRBD) #12798

rbd: support rbd on FreeBSD (without KRBD) #12798

Conversation

wjwithagen commented Jan 5, 2017

wjwithagen commented Jan 5, 2017

tchaikov commented Jan 5, 2017

wjwithagen commented Jan 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tchaikov Jan 6, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dillaman Jan 6, 2017 • edited

Choose a reason for hiding this comment

dillaman Jan 6, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tchaikov Jan 11, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wjwithagen commented Jan 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tchaikov commented Jan 6, 2017

wjwithagen commented Jan 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wjwithagen commented Jan 8, 2017

wjwithagen commented Jan 12, 2017

dillaman left a comment

Choose a reason for hiding this comment

tchaikov Jan 6, 2017 •

edited

dillaman Jan 6, 2017 •

edited

dillaman Jan 6, 2017 •

edited

tchaikov Jan 11, 2017 •

edited