Linux arm64

[Nadav-Ruskin]

Howdy. I've spent the past three weeks compiling my project, which uses OpenCV, to Linux-arm64.

It hasn't been easy. My current solution uses hacks- manually soft linking header files from /usr/include to my cross compiler's sysroot's /include path (in my case, /opt/x-tools/aarch64-unknown-linux-gnueabi/include/.

I don't know why that is required, either I'm missing something or something's wrong with pkg-config.

This branch is my attempt to add my solution to dockcross, I doubt it's perfect in its current shape, but I'm starting the pull request so you can review it right now.

[Nadav-Ruskin]

We had an unexpected error preparing a VM for this build, potentially due to our infrastructure or cloud provider.  Please retry the build in a few minutes
Unexpected environment preparation error: Put https://vm-service.infra.circleci.com/tasks/5ae05141e40f570007f991f0-0-build%2F3D096AA6: dial tcp 172.16.6.26:443: getsockopt: connection refused

How can I request a new test?

[thewtex]

Nice! Looking good @Nadav-Eyesight

I restarted the CI build.

[Nadav-Ruskin]

I'm honestly surprised. So you don't think there's anything missing? If so, maybe this image should come with a warrant so people know hacks like I did might be needed. If this pull request gets approved, I'll still need to extend it with an image that does the soft linking.

[thewtex]

I disabled the linux-arm64 build in .circleci/config.yml -- could you please enable it again now that we have a replacement.

Regarding the hacks, CMake is preferred for cross-compiling, so if your build system can use it, less hacks should be required.

[Nadav-Ruskin]

OK, I enabled linux-arm64.

I use OpenCV. OpenCV couldn't find the packages' headers. OpenCV in turn uses pkg-manager to find the headers, AFAIK. I don't have this problem in any other platform using dockcross: I use linux-armv7, android-arm and android-arm64.

[Nadav-Ruskin]

I'm still getting the following error, not sure why:

Building /work/test/C/hello.c by calling /opt/x-tools/aarch64-unknown-linux-gnueabi-gcc...
/opt/x-tools/aarch64-unknown-linux-gnueabi-gcc /work/test/C/hello.c -o a.out
Traceback (most recent call last):
  File "test/run.py", line 178, in <module>
    debug=args.debug) != 0)
  File "test/run.py", line 149, in run_tests
    debug)
  File "test/run.py", line 132, in test_language
    debug)
  File "test/run.py", line 118, in test_build_system
    linker_flags, exe_suffix, debug)
  File "test/run.py", line 83, in test_source
    linker_flags, exe_suffix)
  File "test/run.py", line 38, in test_none_build_system
    return subprocess.call(build_cmd)
  File "/usr/lib/python2.7/subprocess.py", line 522, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Makefile:160: recipe for target 'linux-arm64.test' failed
make: *** [linux-arm64.test] Error 1

[thewtex]

OSError: [Errno 2] No such file or directory

Does /opt/x-tools/aarch64-unknown-linux-gnueabi-gcc exist in the image?

[Nadav-Ruskin]

Oh, so that's what it meant! Yah, that's not the right path, I made a mistake in the dockerfile. Problem is, I'm running the created container, and it doesn't seem like crosstool-ng installed anything...

In fact, I can't even find a mention of crosstool-ng in the log. I thought I was just supposed to use #include "common.crosstool"?

[Nadav-Ruskin]

The deed is done.

[thewtex]

Well done @Nadav-Eyesight !

[andre-richter]

Hi all,

this pull request moved the cross-binaries to a place that is not in PATH.

E.g. /usr/bin/${CROSS_TRIPLE}-gcc went to /opt/x-tools/${CROSS_TRIPLE}-gcc. Hence, I cannot execute gcc directly with ${CROSS_TRIPLE}-gcc, but always need to prepend the full path starting from /opt.

For my projects, this is a breaking change, and I currently cannot see sense in it. Toolchain binaries should never need the full path for invoking. Other dockcross containers also don't use this path as far as I can see.

Additionally, the ARM64 triple is changed from aarch64-linux-gnu to aarch64-linux-unknown-gnueabi. I am not sure if this is technically correct. From the top of my head I cannot recall seeing eabi in llvm aarch64 triples.

I think with this pull request, some user-customizations slipped in that should be reverted, maybe?

Thanks,
Andre

[marcoabreu]

I agree, we're also encountering issues in our project. Specifically, the fortran compiler now being missing. See apache/mxnet#10837 for reference

[Nadav-Ruskin]

@andre-richter @marcoabreu Sorry for any inconvenience.
@andre-richter I didn't know moving the compiler's location would affect anyone. The default location for the toolchain created by crosstool-ng is ~/x-tools which caused trouble, so I changed it to /opt/x-tools. It can be easily changed to /bin if you think that's a good idea.
Likewise, aarch64-linux-unknown-gnueabi was crosstool-ng's default. It can also be changed if it causes anyone trouble.

@marcoabreu I don't use fortran but the settings on crosstool-ng state that it should be installed, plus I thought the circleci tests for fortran compilations on images that have them. If it doesn't, it should be added as a test.

[marcoabreu]

I can't find it anywhere in the container. How would you propose to move forward? This is blocking us :/

[Nadav-Ruskin]

First of all, if you're blocked, I suggest you simply use a revision prior to the pull request's approval. I don't have access to fix anything right now.

Secondly, for the changes to the toolchain's location, one needs to edit linux-arm64/crosstool-ng.config and then reflect these changes in the toolchain and environmental variables.

[marcoabreu]

Thank you! Since the releases are not tagged, we created a mirror and published a cached version of the image as a temporary fix. To prevent problems like this in future, I've created a feature request at #223.

I think the problem is not the location if the toolchain. Rather, fortran is entirely missing inside the container (see https://github.com/dockcross/dockcross/pull/221/files#diff-6e12c3f1624f19bdd1a4370b48ce9292L11 for reference).

[Nadav-Ruskin]

Fortran isn't supposed to be installed through apt-get anymore. The whole point of this pull request is to replace the deprecated embedian PPA. It simply doesn't exist anymore.

This is the line that SHOULD install Fortran:
https://github.com/dockcross/dockcross/pull/221/files#diff-131fa95b6ee505301d520c0ce86d479fR409

Perhaps, however, this line needs to be set to CT_CC_LANG_FORTRAN=y?
https://github.com/dockcross/dockcross/pull/221/files#diff-131fa95b6ee505301d520c0ce86d479fR420

[andre-richter]

Hi,

Thanks for the fast replies. I think I am indifferent how things are solved, as long as the binaries will be in PATH so no absolute paths are needed to invoke them, and maybe change the target triple back to what it was before to stay backwards compatible in that regard (aka keep the original binary names).

Thx,
Andre

[Nadav-Ruskin]

If you don't want to use absolute paths, why not use $CC? Anyway, I'm working on a fix for you guys.

[andre-richter]

In my case I'm using objcopy as well, which doesn't have an env variable set for it. Ofc this is also viable, but some more env variables are needed I guess. Currently only the usual suspects like gcc et al. have one.

[Nadav-Ruskin]

#224 compiled on my machine, and has gfortran.