New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Official ARM support #2103

Open
blitzcode opened this Issue May 5, 2016 · 105 comments

Comments

Projects
None yet
@blitzcode

blitzcode commented May 5, 2016

The current process of getting a Stack Haskell application build on ARM is rather... tricky:

https://github.com/blitzcode/hue-dashboard/#raspberry-pi

I'd love for Stack to have an official ARM bindist and be able to install GHC/LLVM with stack setup.

@borsboom

This comment has been minimized.

Contributor

borsboom commented May 5, 2016

The main thing we'd need to start to start generating ARM binaries for releases is a Vagrantfile that sets up a build environment similar to the ones found in https://github.com/commercialhaskell/stack/tree/master/etc/vagrant. I guess that Vagrantfile would have to launch a VM on an Arm-supporting VPS provider (or in an emulator, but that might be really slow).

From your link, it sounds like we may also want to add support for installing LLVM to stack setup. Although I guess if you already have LLVM, stack setup might already be able to take care of installing GHC (at least once we add a link to the correct GHC bindist for ARM to https://github.com/fpco/stackage-content/blob/master/stack/stack-setup-2.yaml).

@borsboom borsboom added this to the P3: Optional milestone May 5, 2016

@blitzcode

This comment has been minimized.

blitzcode commented May 6, 2016

I initially tried building on QEMU, but had little success with it. The Raspberry Pi with its 1GB of RAM might be problematic for reliable builds. As you suggest, a VPS provider that offers server-grade ARM systems with enough memory seems like the best option.

I filed a issue #1742 a while ago to suggest stack should install LLVM as it's often difficult to pick the right version and different GHC versions need different LLVMs. Wasn't there talk of GHC8 bundling the right LLVM anyway?

@quyse

This comment has been minimized.

Contributor

quyse commented May 8, 2016

@blitzcode Just a few notes, I have set up an ARM machine on Scaleway few months ago for CI purposes, and it's doing great. I would advise to use Arch Linux distribution instead of Debian/Ubuntu, as it has up-to-date binary packages, in particular GHC 7.10.3 (on ARM too) and LLVM, which work together flawlessly. As I remember the setup was relatively straightforward, just install ghc, llvm35, cabal-install from packages, add some swap space (2Gb RAM will not be enough), compile stack from sources, and that's it. Don't do parallel builds, as it will run out of memory. Unfortunately I don't know other good ARM cloud providers, I would love to hear about any providing bigger ARM machines in terms of memory.

@blitzcode

This comment has been minimized.

blitzcode commented May 9, 2016

@quyse Yes, I was checking them out, seems really affordable. 2GB really wasn't enough? I got everything to build with the 1GB RPi and 100MB swap, I figure 2GB should be smooth sailing. Maybe only with -j1 / -j2, though. I'm a bit wary of Arch simply because I don't like the idea of being forced to upgrade and I want reproduceability. If 1 year from now the SD card in a RPi I deployed dies, I'd want to be able to set up the exact same system again. If Arch has already moved to GHC8 by then, I might be in trouble with my LTS 5.x code. At least that's my worry without ever having tried Arch. It seems otherwise much easier to get going than Debian Jessie with its dated packages...

@quyse

This comment has been minimized.

Contributor

quyse commented May 9, 2016

@blitzcode I've checked again what I'm doing there, actually I'm not specifying -j1, so I guess it's actually -j4 by default (because of 4 cores), no wonder it was running out of memory :) With additional swap space it's fine though. -j1 is just too slow.
Good point about Arch, yes, you are usually forced to upgrade. You can freeze your system's state by simply not upgrading, you also can make a backup of your pacman cache, so you can restore packages in the future without contacting distribution servers. But, yes, probably easier to stick with Debian in your case. I'm just not worrying too much about reproduceability at the moment, as I'm upgrading regularly to the latest LTS.

@borsboom

This comment has been minimized.

Contributor

borsboom commented May 20, 2016

I'm trying to build a stack binary on a Scaleway VM running Debian Jessie (to match the GHC bindist), but am having trouble with LLVM. I downloaded an LLVM bindist as recommended in here, but when I try to use it I get:

# opt --version
Illegal instruction

Any ideas where I can easily get an LLVM that will work?

@blitzcode

This comment has been minimized.

blitzcode commented May 21, 2016

@borsboom That's puzzling, last time I checked their servers were armv7 / 32 bit, same as the Raspberry Pi with Raspbian. So it's really strange that the LLVM bindist would not work.

Could you maybe do a lscpu, uname -a and maybe run file or whatever on the LLVM bindist and see if the architectures etc. match? I also sometimes hear that SoC vendors have a lot of discretion on what to include in their ARM cores, like NEON, 32/64 bit, etc, so maybe there's something different about the server-grade SoCs Scaleway uses. No idea.

@borsboom

This comment has been minimized.

Contributor

borsboom commented May 21, 2016

@blitzcode:

root@scw-476b18:~# uname -a
Linux scw-476b18 4.3.5-std-1 #1 SMP Fri Feb 19 11:52:18 UTC 2016 armv7l GNU/Linux

root@scw-476b18:~# lscpu
Architecture:          armv7l
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
Model name:            ARMv7 Processor rev 2 (v7l)
CPU max MHz:           1333.0000
CPU min MHz:           666.5000

root@scw-476b18:~# /usr/local/bin/opt --version
Illegal instruction

root@scw-476b18:~# file /usr/local/bin/opt
/usr/local/bin/opt: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=f99b13805d70643bc8a3485026fb7ab9f6a37f04, stripped

Oddly enough, the file output matches that for ghc (which does run):

root@scw-476b18:~# /opt/ghc-7.10.3/lib/ghc-7.10.3/bin/ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.3

root@scw-476b18:~# file /opt/ghc-7.10.3/lib/ghc-7.10.3/bin/ghc
/root/opt/ghc-7.10.3/lib/ghc-7.10.3/bin/ghc: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=c2002709f062e00b92536c9c6770e7fa9d96f7d8, not stripped
@varosi

This comment has been minimized.

varosi commented Jun 9, 2016

@borsboom, you need at least LLVM 3.5.2 or greater. There is a fix related to GHC in it.

@varosi

This comment has been minimized.

varosi commented Jun 9, 2016

@blitzcode, I'm running GHC 7.10.2 with LLVM 3.5.2 on ARMv7 (32bit) and it is working well.

@blitzcode

This comment has been minimized.

blitzcode commented Jun 9, 2016

@borsboom I'm terribly sorry, I said 3.5.2 in my document and then linked 3.5.1, dammit...

@mgsloan

This comment has been minimized.

Collaborator

mgsloan commented Jun 9, 2016

Note that another thing that will likely need to be resolved is support for byte alignment in store - fpco/store#37 - as ARM cares about this.

@varosi

This comment has been minimized.

varosi commented Jun 9, 2016

@varosi

This comment has been minimized.

varosi commented Jun 9, 2016

This temporarily workarounds this problem:

cabal install --dependencies-only --constraint=cryptonite==0.15

@andreyk0

This comment has been minimized.

andreyk0 commented Jun 10, 2016

Jumping here from #1332

I've finally taken some time to untangle haskell/arm stuff from my personal files, here's the result:
https://github.com/andreyk0/haskell-on-arm

Hope it helps somebody.

@varosi cryptonite used to be an issue for me at some point but haven't had any problems recently. Something got fixed, I imagine.

@varosi

This comment has been minimized.

varosi commented Jun 11, 2016

@andreyk0, fix is not yet pushed in Hackage and I still have this problem on my ARM.

@andreyk0

This comment has been minimized.

andreyk0 commented Jun 11, 2016

@varosi hmm, could be my environment then. I don't remember everything but (it's in the readme in that repo and my images are on dockerhub) I've installed LLVM from the project's downloads page and hacked 7.10.2 to be a symlink to 7.10.3, so there's no 7.10.2 in my system.

@varosi

This comment has been minimized.

varosi commented Jun 13, 2016

with cryptonite-0.15 it is working fine on IMX.53 board. So I'm building project successfully with Stack. It'll be great if Stack have official distro for ARM and support stack setup.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Jun 15, 2016

I did some more experimenting with this. I got Debian Jessie for armhf running in qemu (note: the LLVM bindists are not compatible with armel or with Wheezy for armhf which I tried first) and things seem to be working, but very slow. I'm building cabal-install and it's been running for about 10 hours now, and looks like it might be finished in the next hour or so. At that rate, Stack will take at least a couple of days to build, and I expect the integration tests to take another day or so to run. At that point I'll at least know whether Stack works properly on ARMv7.

The situation with LLVM bindists is annoying though; they only seem to support fairly specific versions of distros (unlike GHC bindists, which tend to work on a wide range). And since each GHC version needs a specific version of LLVM, we can't leave it up to the Linux distro to install LLVM for us. So we could officially support ARMv7 on Jessie armhf, but not necessarily on any other distributions or versions.

@blitzcode

This comment has been minimized.

blitzcode commented Jun 15, 2016

@borsboom Good to hear, but I'm surprised that the build times are so much worse than on the RPi. I've read several times about good results with QEMU. Hope you build Cabal with EXTRA_CONFIGURE_OPTS="" ./bootstrap.sh, it helps quite a bit!

In the meantime, Stack v1.1 has been working absolutely without issues for me on ARM after the initial setup pains.

Debian Wheezy only seems to have LLVM 3.1 available, Wheezy only goes to 3.5, had to download the LLVM binaries. Where on the LLVM page does it actually say with which distros the binaries have been tested?

@borsboom

This comment has been minimized.

Contributor

borsboom commented Jun 15, 2016

Thanks for the tip about EXTRA_CONFIGURE_OPTS; I didn't know that!

I didn't see anywhere on the LLVM page where it mentioned the distros. But when I tried clang+llvm-3.7.1-armv7a-linux-gnueabihf (I'm using GHC 8.0.1, so the newer LLVM is required) on Wheezy, this happened:

manny@debarm:~$ clang
clang: /usr/lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by clang)
clang: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.15' not found (required by clang)
@blitzcode

This comment has been minimized.

blitzcode commented Jun 15, 2016

Oh, it's gated by glibc, that makes sense. That's an unfortunate problem...

Once you have Cabal be sure to use --disable-library-stripping --disable-executable-stripping and / or set executable-stripping: False + library-stripping: False in Cabal's ~/.cabal/config. I had to start over and rebuild all of Stack's dependencies on my first try :/

@varosi

This comment has been minimized.

varosi commented Jun 15, 2016

I have tested latest cryptonite-0.17 on ARMv7 and the compilation issue from 0.16 is fixed (haskell-crypto/cryptonite#90) there. So Stack will be building alright now.

@andreyk0

This comment has been minimized.

andreyk0 commented Jun 15, 2016

@varosi yeah, I must've hacked flags long time ago and forgot about it, sorry for the noise, you were right. Had the same problem with a clean ghc8.0.1 build, luckily it's fixed now!

@andreyk0

This comment has been minimized.

andreyk0 commented Jun 15, 2016

@borsboom " ARMv7 on Jessie armhf" is probably the way to go, too much pain to get this stuff to work anywhere else for now.
Regarding QEMU build times: I only used cabal-install once, about a year ago to build stack itself. Since then using stack to build packages and newer versions of stack itself. Most packages and my tiny projects compile in a few minutes, which is reasonable. There are outliers though, e.g. compiling regex-tdfa the other day with ghc8.0.1 took over 2h. Don't know why. At least stack will reuse the binaries, which helps a lot.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Jun 15, 2016

Replying to myself here, regarding #2103 (comment): I've seen some issues in other projects that need ARM CPUs with NEON support and have the same Illegal Instruction error on Scaleway (where the CPU doesn't have NEON support), so I'm guessing that's what is going on with the LLVM bindist. Annoying, because this would be much faster/easier if I could build on Scaleway.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Jul 20, 2017

Has anyone tried getting in touch with the Debian Haskell Group about updating that package? In theory, it should be fairly straightforward to do so.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Jul 20, 2017

I'm giving a try to building a stack-1.5.0 arm binary using a slightly updated version of my previous process. It'll take a few days and I'm going to be really tired of hearing my computer fan by the time it's done, but hopefully it'll work.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Jul 21, 2017

I'm hitting composewell/unicode-transforms#10 in my build. I'll try building with GHC 7.10.3 instead.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Jul 25, 2017

Thanks to @brent80's PR and @harendra-kumar's quick release of unicode-transforms-0.3.2 to work around the LLVM failure on long lists, we now have a stack-1.5.0 ARM binary: https://github.com/commercialhaskell/stack/releases/download/v1.5.0/stack-1.5.0-linux-arm.tar.gz

@varosi

This comment has been minimized.

varosi commented Jul 26, 2017

Great! I have tried it on my Raspberry Pi 3 and Debian Jessie and it worked just great! I used stack upgrade and it worked! Then I have tested some compilation with GHC 7.10.2 (LTS-3.0). I didn't have that compiler on my machine, so I tried stack setup and it worked! Cool!
So, great job you have done!

Which compilers could be installed now on ARMv7 with stack setup?

@borsboom

This comment has been minimized.

Contributor

borsboom commented Mar 2, 2018

Sounds like the Scaleway LLVM issue (see #2103 (comment)) may be fixed, according to #3840 (comment).

@varosi-chaosgroup

This comment has been minimized.

varosi-chaosgroup commented Mar 2, 2018

@borsboom

This comment has been minimized.

Contributor

borsboom commented Mar 2, 2018

@varosi 96 cores might be a bit of overkill for building Stack :)

In any case, I was able to build stack-1.6.5 on Scaleway in a very reasonable amount of time (and for a very reasonable cost): https://github.com/commercialhaskell/stack/releases/download/v1.6.5/stack-1.6.5-linux-arm.tar.gz

@varosi

This comment has been minimized.

varosi commented Mar 4, 2018

Great! Thanks! Could you add it to Stack shell script installation so it be installed on ARM systems automatically? Currently shell script from the main Stack site is getting incompatible executable.

@varosi

This comment has been minimized.

varosi commented Mar 4, 2018

hm, strange. On Packet cloud it gave me this error with your executable:

-bash: ./stack: cannot execute binary file: Exec format error

@borsboom

This comment has been minimized.

Contributor

borsboom commented Mar 5, 2018

I don't know ARM very well, but isn't ARMv8 (which Packet cloud offers) a different architecture than ARMv7? This bindist was built on and for ARMv7. I tested the bindist (and get.haskellstack.org script) on a different ARMv7 system from where I built it, and it seemed to work there.

@mcmayer

This comment has been minimized.

mcmayer commented Mar 5, 2018

ARMv8 is about as different to ARMv7 as 64 bit is to 32 bit.
(ARMv8 is actually sort of backward compatible but as far as I know ARMv7 executables do not run on ARMv8 just like that. At least executables requiring any dynamically linked libraries pretty much don't.)

@borsboom

This comment has been minimized.

Contributor

borsboom commented Mar 5, 2018

@mcmayer How do ARMv8 and AArch64 relate? GHC 8.2.2 has AArch64 support, so I wonder if that would work on ARMv8 systems. If so, we could potentially start building compatible bindists for stack-1.7.0 (which will probably be built using GHC 8.2.2).

@andreyk0

This comment has been minimized.

andreyk0 commented Mar 5, 2018

FWIW https://www.debian.org/ports/arm/ has a good description of ARM-specific permutations and descriptions of various options they've chosen for compilers. Particularly AArch64 and ARMv8 seem to be the same: https://wiki.debian.org/Arm64Port

@mayeranalytics

This comment has been minimized.

mayeranalytics commented Mar 6, 2018

@borsboom For practical purposes: ARMv8 is AArch64. (I'm sure that there are there are nuances to the nomenclature that totally elude me).

What I know for sure is that llvm needs the correct compiler flags to produce the desired executable. Between llvm versions these flags vary, and personally I handle any issues in a totally ad-hoc fashion without having obtained any deeper knowledge, so far.

I find the file command useful, it reveals with good detail what kind of executable you have. For example, on scaleway's machine:

> file .local/bin/stack          
.local/bin/stack: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=4e8818d9ec8c236b102b26f66346402f5f1cf0f5, stripped

More information about the executable can be obtained with objdump -x.

Going deeper into the realm of wild guesses: Installing binfmt-support, qemu, qemu-user-static and such things might help to run ARMv7 executables on ARMv8 metal, although an ARMv8 (or should I say AArch64?) executable is preferable, of course.

So, if GHC 8.2.2 has AArch64 support then that's perfect ... (unless, and I'm speaking from the practitioner's point of view again, if the llvm flags are/were somehow screwed up then you may get an executable that is AArch64 but still doesn't run on your machine). GHC allows to pass flags to llvm (if llvm is the chosen backend), so it should be possible to get this right.

@varosi-chaosgroup

This comment has been minimized.

varosi-chaosgroup commented Mar 6, 2018

This explains everything. As colleague of mine explained it to me - Aarch64 architecture support both 32 and 64bit, ARM64 (or A64 more correctly) support only 64bit and A32 is 32bit only.

@blu

This comment has been minimized.

blu commented Mar 6, 2018

Just to toss in my .02eur (hey, Varosi!)

arm64 is available only under aarch64, but the latter is not limited to arm64 only -- armhf (ARMv7 Hard Float) is another binary that aarch64 can execute, via some, erm, persuasion.

On debian et al (e.g. ubuntu) you need to enable armhf as a valid architecture in your dpkg multi-arch support (same way with 386 on amd64):

$ dpkg --add-architecture armhf

Once you have done that (and the mandatory apt-get update) you can start tossing in the needed libraries for launching armhf executables (the exact set of which I don't currently recall, but I can check later tonight):

$ apt-get install package:armhf

(all of the above discussed here: https://wiki.debian.org/Multiarch/HOWTO)

An important clarification: there do exist aarch64 implementations that are arm64-only -- one such is Qualcomm's Centriq2400.

@varosi

This comment has been minimized.

varosi commented Mar 6, 2018

@borsboom

This comment has been minimized.

Contributor

borsboom commented Mar 6, 2018

I don't think Stack currently has the ability to differentiate between different ARM architectures (armv7 vs aarch64). What does stack setup say when you try to have it install GHC on aarch64?

@varosi

This comment has been minimized.

varosi commented Mar 6, 2018

@borsboom

This comment has been minimized.

Contributor

borsboom commented Mar 7, 2018

@varosi Did you specify an arch in your ~/.stack/config.yaml or somewhere else? Or is that what it's picking up automatically?

That cassava error is unrelated and due to running an old version of Stack. That was fixed in stack-1.6.3, so can you try upgrading to that? 1.6.3 doesn't use the new syntax in its stack.yaml so you should be able to upgrade to it from 1.5.x (and, once you have 1.6.3, you should be able to upgrade to 1.6.5).

Once you have stack-1.6.[35], please show me the output of stack --resolver=ghc-8.2.1 setup

@varosi

This comment has been minimized.

varosi commented Mar 7, 2018

@borsboom It's picked automatically. I don't have arch anywhere.

@varosi

This comment has been minimized.

varosi commented Mar 7, 2018

@borsboom with Stack 1.5.0:

varosi@arm:~/stack$ stack --resolver=ghc-8.2.1 setup
Warning: Unknown value for architecture setting: "aarch64"
I don't know how to install GHC for (Linux,OtherArch "aarch64"), please install manually

I'm still trying to build 1.6.3. With Cabal 1.22.6, GHC 8.0.1, Aarch64 - I get that message:

Configuring store-core-0.2.0.2...
    Cabal-simple_mPHDZzAJ_1.24.0.0_ghc-8.0.1: No executables, libraries, tests, or
    benchmarks found. Nothing to do.
@varosi

This comment has been minimized.

varosi commented Mar 7, 2018

hm, I have updated my GHC to 8.2.2 and Cabal to 1.24. Is it possible to build Stack 1.6.3 with it somehow? Currently I get same Aeson error.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Mar 7, 2018

@varosi I just checked and Scaleway had an aarch64 server available (they're usually out of stock when I check), so I snagged one and am now trying to build Stack. I'm bootstrapping by installing ghc-8.2.2 from the official bindist and then using cabal-install to build Stack. Not sure yet whether it will work, but it's building Stack's dependencies now.

@infinity0

This comment has been minimized.

infinity0 commented Mar 7, 2018

For all those following, the meaning of "armv8" depends on context, usually it just means "aarch64" but sometimes it refers to a 32-bit arm kernel running on a 64-bit aarch64 CPU. uname -m will say armv8l instead of aarch64 in that case, and the GNU binary userland is also 32-bit. (Analogue of running 32-bit system on a x86-64 CPU.)

If stack doesn't handle this, or does something else, then it's a bug.

For example see rust's architecture detection https://github.com/rust-lang/rust/blob/master/src/bootstrap/bootstrap.py#L275

@borsboom

This comment has been minimized.

Contributor

borsboom commented Mar 10, 2018

@varosi That bootstrap build worked fine, and then it to rebuild itself. I've also made a small change (#3910) to add support stack setup installing bindists on aarch64. You can grab a stack executable that includes the change here: stack-1.6.6-be415545029d1bcafddb9fc13b8283e3d1b1abb1-linux-aarch64.gz

@varosi

This comment has been minimized.

varosi commented Mar 14, 2018

Great! Thanks! It works for GHC 8.2.2 on Scaleway arm64 machine!
Will this:

curl -sSL https://get.haskellstack.org/ | sh

will work on ARM machines for next Stack?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment