go get errors

[jupiturliu]

go get github.com/ceph/go-ceph

github.com/ceph/go-ceph/rbd

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to NSS_CMSMessage_ContentLevel' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSDigestContext_Update'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to PORT_SetError' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSSignerInfo_GetSignerCommonName'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to PK11_CipherOp' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSDecoder_Start'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to NSS_CMSSignedData_VerifySignerInfo' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSSignedData_ImportCerts'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to PORT_Free' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toPK11_ParamFromIV'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to PK11_FreeSlot' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toCERT_GetDefaultCertDB'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to NSS_ShutdownContext' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toPK11_CreateContextBySymKey'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to PK11_DestroyContext' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toPK11_ImportSymKey'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to NSS_CMSDecoder_Update' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toPK11_FreeSymKey'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to NSS_CMSSignedData_GetSignerInfo' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSSignedData_HasDigests'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to PORT_NewArena' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSContentInfo_GetContentTypeTag'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to NSS_CMSMessage_Destroy' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_InitContext'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to NSS_CMSSignedData_VerifyCertsOnly' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSMessage_ContentLevelCount'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to NSS_CMSContentInfo_GetContent' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSDigestContext_FinishMultiple'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to NSS_CMSDecoder_Cancel' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSSignerInfo_GetVerificationStatus'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to PK11_DigestFinal' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSDigestContext_StartMultiple'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to NSS_CMSUtil_VerificationStatusToString' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSSignedData_SetDigests'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to PR_GetError' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSMessage_GetContent'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to PORT_FreeArena' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toNSS_CMSSignedData_SignerInfoCount'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to SECITEM_FreeItem' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toPK11_GetBestSlot'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to NSS_CMSDecoder_Finish' /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference toSECMOD_RestartModules'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librbd.so: undefined reference to `NSS_CMSSignedData_GetDigestAlgs'
collect2: error: ld returned 1 exit status

[jupiturliu]

@noahdesu any thoughts?
I tried two machines. both of them failed to get the code compiled. I do check and verify all of library necessary are installed.

[dotnwat]

Hi @jupiturliu thanks for this report. I've been on vacation and can look at this now. How did you install Ceph (e.g. source, stable/dev?), and had you previously installed Ceph and/or go-ceph on the same machine? Since go-ceph just links against the C libraries, this doesn't appear to be a go-ceph issue, but it might be related to a missing library path.

[jupiturliu]

Hi @noahdesu Happy new year. thanks for your reply. I am actually compiling the ceph by myself. Here is the scripts I created to compile Ceph @ https://github.com/omnicube/utils/blob/master/ceph_compiling.sh
I developed several features inside of Ceph. I just want to have a quick try with any example written in go. I do installed librbd and librados2 library. What else do I need to install? By the way, I am using Ubuntu 14.04.
thanks,
James

[dotnwat]

It seems like Ceph was compiled against libnss but that libnss (or a different version) was used when compiling go-ceph. I'll try to confirm this bug. Have you tried to compile go-ceph against Ceph installed from pre-built packages?

[jupiturliu]

@noahdesu I did not get a chance to try pre-built ceph package. I will find a machine to have a try tonight if allowed.

[jupiturliu]

@noahdesu with standard Ceph release,I did not get this error.

[dotnwat]

Ok thanks for the update, that's good news. I looked at your script that compiles Ceph. Would you mind sending me the parameters you are using for configure? I built from source recently and everything worked for me so I am having a little trouble reproducing this. Can you compile other programs against librados or librbd? One thing that you might want to check is that cgo can find libnss. Perhaps libnss is installed in a path that cgo won't look for automatically.

[jupiturliu]

@noahdesu I just enabled rocksdb with ./configure --with-librocksdb-static. What kind of other programs I supposed to build. I do build fio with librbd support. I am able to run the fio test with librbd engine against new ceph build.

[jupiturliu]

@noahdesu I switched to another machine and tried to configure the ceph with like this one and see how it goes. ./configure --with-nss

[dotnwat]

@jupiturliu ok let me know how that goes. if that doesn't fix it, i will create post step-by-step instructions that works for me so you can compare and maybe we can narrow down where the problem is. thanks for your patience.

[jupiturliu]

@noahdesu
Just recompiled the code with ./configure --with-nss. I still did not get luck to make it work. Using ldd to check dependence of librbd that go-ceph rely on. I do see the my own build libraods was linked to librbd correctly.
ldd /usr/lib/librbd.so
linux-vdso.so.1 => (0x00007fff76ecb000)
librados.so.2 => /home/jamesliu/WorkSpace/ceph-install/lib/librados.so.2 (0x00007f8317a6a000)
libboost_thread.so.1.54.0 => /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0 (0x00007f8317839000)
libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f8317612000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f83173f4000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f83171ec000)
liblttng-ust.so.0 => /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 (0x00007f8316fa1000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8316d9d000)
libboost_system.so.1.54.0 => /usr/lib/x86_64-linux-gnu/libboost_system.so.1.54.0 (0x00007f8316b99000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8316894000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f831658e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f83161c8000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8315fb1000)
libcrypto++.so.9 => /usr/lib/libcrypto++.so.9 (0x00007f83158e6000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f83156e1000)
/lib64/ld-linux-x86-64.so.2 (0x00007f831c6de000)
liblttng-ust-tracepoint.so.0 => /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 (0x00007f83154c6000)
liburcu-bp.so.1 => /usr/lib/liburcu-bp.so.1 (0x00007f83152bf000)
liburcu-cds.so.1 => /usr/lib/liburcu-cds.so.1 (0x00007f83150b8000)

Whenever I removed my own built rados library in the library path and use standard one, everything works fine.
Pretty sure the problem come from ceph build.

[dotnwat]

Do you have multiple version of Ceph installed (e.g. one via a package
manager, and the one you built)? If so, does it work if you remove the one
that is installed via the package manager? It might be that there is cross
linking going on.

On Wed, Jan 6, 2016 at 5:10 PM, jupiturliu notifications@github.com wrote:

@noahdesu https://github.com/noahdesu
Just recompiled the code with ./configure --with-nss. I still did not get
luck to make it work. Using ldd to check dependence of librbd that go-ceph
rely on. I do see the my own build libraods was linked to librbd correctly.
ldd /usr/lib/librbd.so
linux-vdso.so.1 => (0x00007fff76ecb000)
librados.so.2 => /home/jamesliu/WorkSpace/ceph-install/lib/librados.so.2
(0x00007f8317a6a000)
libboost_thread.so.1.54.0 =>
/usr/lib/x86_64-linux-gnu/libboost_thread.so.1.54.0 (0x00007f8317839000)
libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f8317612000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f83173f4000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f83171ec000)
liblttng-ust.so.0 => /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
(0x00007f8316fa1000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8316d9d000)
libboost_system.so.1.54.0 =>
/usr/lib/x86_64-linux-gnu/libboost_system.so.1.54.0 (0x00007f8316b99000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(0x00007f8316894000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f831658e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f83161c8000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8315fb1000)
libcrypto++.so.9 => /usr/lib/libcrypto++.so.9 (0x00007f83158e6000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f83156e1000)
/lib64/ld-linux-x86-64.so.2 (0x00007f831c6de000)
liblttng-ust-tracepoint.so.0 =>
/usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 (0x00007f83154c6000)
liburcu-bp.so.1 => /usr/lib/liburcu-bp.so.1 (0x00007f83152bf000)
liburcu-cds.so.1 => /usr/lib/liburcu-cds.so.1 (0x00007f83150b8000)

Whenever I removed my own built rados library in the library path and use
standard one, everything works fine.
Pretty sure the problem come from ceph build.

—
Reply to this email directly or view it on GitHub
#5 (comment).

[jupiturliu]

@noahdesu It is likely, I will make sure the librbd and librados libary are from same sources. I will let you know soon.

[dotnwat]

Is this still an issue? I've not heard of this problem and haven't been able to reproduce it.

[phlogistonjohn]

Hello, I'm automatically closing all issues filed prior to 2018 that have not been commented on recently. If this issue is still relevant to you please reopen the issue and restart the discussion. Thank you for your understanding.