ARM cross compiler/linker cannot locate certain libraries #3902
Comments
|
What's surprising about that?
|
|
However, I could not install ARM version of libnetfilter-queue-dev on my AMD hardware. |
|
Yes, it is a package for an Arm64 host, not for a cross-compilation environment. Since it seems you are using the OP-TEE QEMU environment, which uses Buildroot, I think you should update the Buildroot configuration to install the libraries you need. Adding |
|
@jforissier do you have any idea about #3901 (comment) ? |
Hi @geneva1. Here buildroot should help you the same way as Jérôme showed for libnetfilterqueue support. To add package foo in the target, you should start by checking if package foo is supported by Buildroot and if so you can enable For example, to add busybox tool Another example: to embed ALSA library in the tested rootfs, add Doing so, you will get the expected files already installed in the root filesystem used by the Qemu setup. No need to manually copy them into the embedded filesystem. Check the packages supported by Builrdoot (.../buildroot/package/), there are quite a lot available. But maybe the software you want is not already supported by Buildroot... |
|
@etienne-lms @jforissier you were discussing about adding package inside QEMU buildroot. However, adding package inside buildroot does not help me to compile code (that needs to be running on ARM platform) on AMD platform using ARM cross-compiler. Note that there is no compiler inside QEMU. |
|
Buildroot will take care of cross-compiling the packages you enable through See for example GBD. The package is enable in br-ext/configs/gdbserver.conf ( (edited) |
@etienne-lms Adding package to buildroot does not just help to cross-compile. I think you are missing some more steps during your explanation above. What did you exactly mean by |
|
Here is the picture when you fetch the OP-TEE/Qemu repo manifest and use
When you run Therefore, if you enable a package in Buildroot, it is cross-compiled and embedded in rootfs.cpio.gz. This is the basic setup. @jbech-linaro details in #3901 thread how share of filetree between your host and the runtime Linux OS running in Qemu, using 9p virtio support. This is an extra (and useful feature) but not mandated to add a package in your tested target. I hope the above is clear but don't hesitate to ask for details if something is confusing. |
|
I suppose rootfs.cpio.gz contains the additional ARM packages. @etienne-lms So, how do I tell ARM cross-compiler to use rootfs.cpio.gz ? |
Buildroot takes care of that. For me to understand your concerns: do you have a specific, maybe private, package you would like to cross-compile and embed in the rootfs? or is it generic packages already supported by Buildroot that you want to add in the tested rootfs? |
|
@etienne-lms See #3902 (comment) Even after rootfs contains the additional packages, ARM cross compiler still throws me error. I think you missed out on steps to do cross-compilation for ARM after adding additional packages in buildroot |
|
Ok , lets see what happens when we ask Buildroot to enable libnfnetlink and libnetfilter_queue. As a start point I fetched a vanilla OP-TEE/qemu repo and build it: ubuntu> repo init -u https://github.com/OP-TEE/manifest.git -m default.xml
ubuntu> repo sync
ubuntu> make toolchains
ubuntu> make all
ubuntu> make run-onlyFrom here, from the Linux console, i've listed all files found by embedded tested Linux OS that are named buildroot login: root
# find / -name lib\*net\*
# Nothing found. That is what I expected. Now let's add the above lines in common.mk: BR2_PACKAGE_LIBNFNETLINK ?= y
BR2_PACKAGE_LIBNETFILTER_QUEUE ?= yLet's rebuild (from the host, not from Qemu!). From the build traces, I can see some new packages being built (by Buildroot): ubuntu> make all
(...)
>>> libmnl 1.0.4 Downloading
>>> libmnl 1.0.4 Extracting
>>> libmnl 1.0.4 Patching
(...)
>>> libmnl 1.0.4 Building
>>> libmnl 1.0.4 Installing to staging directory
>>> libmnl 1.0.4 Fixing libtool files
(...)
>>> libnfnetlink 1.0.1 Downloading
(...)
>>> libnfnetlink 1.0.1 Building
(...)
>>> libnfnetlink 1.0.1 Installing to target
(...)
>>> libnetfilter_queue 1.0.3 Downloading
(...)
>>> libnetfilter_queue 1.0.3 Building
(...)
>>> libnetfilter_queue 1.0.3 Installing to target
(...)No build error as the one you report in #3902 (comment). Now let's run Qemu again: ubuntu> make run-onlyAgain, from the test Linux console, i've listed again all file found by embedded tested Linux OS that are named buildroot login: root
# find / -name lib\*net\*
/usr/lib/libnfnetlink.so.0.2.0
/usr/lib/libnetfilter_queue.so.1
/usr/lib/libnfnetlink.so
/usr/lib/libnetfilter_queue.so
/usr/lib/libnfnetlink.so.0
/usr/lib/libnetfilter_queue.so.1.4.0
# That's fine: the libraries we expected have been copied to the tested rootfs. |
@etienne-lms are you implying that I could use these libraries found inside QEMU (ARM emulation platform) to do cross compilation on actual AMD platform ? |
|
hmm, I would say yes, but i'm not sure I properly understand your question. |
|
@etienne-lms The AMD platform is the ubuntu host OS that I used to build QEMU, OP-TEE, etc..... So, I will just copy those libraries from inside QEMU to the host OS for external linking during cross-compilation |
|
Then, ok. |
@etienne-lms why all the |
|
I can't tell. libmnl is installed in the target rootfs: ubuntu> find ../out-br/target -name libmnl\*
../out-br/target/usr/lib/libmnl.so.0.2.0
../out-br/target/usr/lib/libmnl.so.0
../out-br/target/usr/lib/libmnl.so
ubuntu> Maybe add argument |
|
Why the difference in output result for the same code in https://github.com/irontec/netfilter-nfqueue-samples ? on QEMU ARM emulation platform : on actual AMD Ubuntu PC platform : |
|
I don't think I can help you here. Maybe Linux kernel lacks required configuration switches? |
|
on actual AMD Ubuntu PC platform : no output on QEMU ARM emulation platform |
|
Someone told me that it does not seem like https://optee.readthedocs.io/en/latest/building/devices/qemu.html runs a kernel for me. Please correct me if wrong. |
|
I assume here you fetched OP-TEE repo On your AMD Ubuntu PC platform, the shell command
All in one: with OP-TEE Qemu/armv7 device [1], you DO run a cross-compiled Linux kernel in Qemu. [1] https://optee.readthedocs.io/en/latest/building/devices/qemu.html#qemu-v7 |
|
So, what causes #3902 (comment) that it seems no socket is listening within QEMU ? |
|
Sorry, I lack of knowledge to answer your question. |
Why should that be reported there? |
|
@jenswi-linaro because it works well in AMD Ubuntu host PC platform, but not inside QEMU ARM emulation platform |
|
Try running the command inside the emulator then. |
|
|
I don't see how this is OP-TEE related. |
|
the QEMU buildroot is from OP-TEE repo. I suspect something might have went wrong during QEMU buildroot compilation. |
|
@geneva1, this looks like dependencies needed by |
|
Sure, but why is it related to this specific package |
|
Because it needs special kernel options? |
|
It is not specific to this package. Many packages have dependencies. For example libnetfilter_queue has dependencies. There are listed in the native Buildroot [1]. Maybe the package(s) you need to enable in Buildroot are missing some dependencies declaration. If so, this should be addresses inside Buildroot component, not in OP-TEE. Unless of course if it is a dependency on a Linux kernel configuration, in which case you will need to update the Linux config for the OP-TEE/Qemu build in [2]. [1] https://git.buildroot.net/buildroot/tree/package/libnetfilter_queue/libnetfilter_queue.mk?h=2020.02.x |
the buildroot rootfs already had Could you advise about How do I include host-pkgconf into QEMU buildroot for OP-TEE ? Note: I had already had |
|
Inside QEMU, it seems like So, does this imply that nfq_open() error is not due to |
|
The more significant difference between running this netfilter code on actual AMD x64 platform and ARM emulation qemu platform is char buf[4096] __attribute__ ((aligned)); which is really system or architecture-dependent. Please correct me if wrong. |
|
Have you checked |
|
@jforissier Which I have and I have |
🤦♂️ nevermind
...
So, as suggested by @jenswi-linaro and @etienne-lms you are missing a kernel option. You need to add
|
|
@jforissier Someone told me the following : the kernel inside QEMU doesn't have netlink socket support, or sock_raw or the other parm aren't valid for netlink sockets. |
|
@jforissier I have already added |
|
@jforissier someone told me that I need some other kernel parameters (such as |
Yes, probably. We have been telling you this for a couple of days. Now would you please work with this knowledgeable person and stop posting here? I don't want to be rude, but your questions are now unrelated to OP-TEE and we can't spend time debugging everyone's application. Thank you. |
|
@jforissier sorry. I just reused the entire kernel config given from intel qemu repo which contains all the related netfilter kernel configs, but I still have the same strace error. |
|
Why the following compilation error ? |
|
After removing and recompiling the cross-compiled libraries, I am able to eliminate the "file format not recognized" errors. However, now I have the following errors. Why ? |
|
The issue is detailed in the build trace:
At line 55 of sample-helloworld.c, you use |
|
@etienne-lms you had only pinpointed the warning, but what about the 3 linker errors below : |
|
From the line you show, you compile manually sample-helloworld.c:
So you don't rely here on common.mk and other makefiles from OP-TEE/build/. Another way is that you create OP-TEE/build/br-ext/package/netfilter-nfqueue-samples/ (check how other OP-TEE/build/br-ext/package/* are defined) and let OP-TEE generic build get it built from Buildroot. |
No, I have already done So, why |
|
I have added $ ls -go ../out-br/target/usr/lib/
(...)
lrwxrwxrwx 1 15 Jun 19 12:30 ../out-br/target/usr/lib/libmnl.so -> libmnl.so.0.2.0
lrwxrwxrwx 1 15 Jun 19 12:30 ../out-br/target/usr/lib/libmnl.so.0 -> libmnl.so.0.2.0
-rwxr-xr-x 1 13736 Jun 19 12:31 ../out-br/target/usr/lib/libmnl.so.0.2.0
lrwxrwxrwx 1 27 Jun 19 12:30 ../out-br/target/usr/lib/libnetfilter_queue.so -> libnetfilter_queue.so.1.4.0
lrwxrwxrwx 1 27 Jun 19 12:30 ../out-br/target/usr/lib/libnetfilter_queue.so.1 -> libnetfilter_queue.so.1.4.0
-rwxr-xr-x 1 13384 Jun 19 12:31 ../out-br/target/usr/lib/libnetfilter_queue.so.1.4.0
lrwxrwxrwx 1 21 Jun 19 12:30 ../out-br/target/usr/lib/libnfnetlink.so -> libnfnetlink.so.0.2.0
lrwxrwxrwx 1 21 Jun 19 12:30 ../out-br/target/usr/lib/libnfnetlink.so.0 -> libnfnetlink.so.0.2.0
-rwxr-xr-x 1 17392 Jun 19 12:31 ../out-br/target/usr/lib/libnfnetlink.so.0.2.0
(...)
$ |
|
@etienne-lms I followed your advice above ( Here is full log output of |
ARM cross compiler/linker cannot locate certain libraries for https://github.com/irontec/netfilter-nfqueue-samples, but normal system-wide gcc compiler/linker can.
Why ?
The text was updated successfully, but these errors were encountered: