Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can Binary Releases be Done for aarch64 and armv7h #5450

Closed
stormdragon2976 opened this issue Apr 16, 2019 · 73 comments
Closed

Can Binary Releases be Done for aarch64 and armv7h #5450

stormdragon2976 opened this issue Apr 16, 2019 · 73 comments
Labels

Comments

@stormdragon2976
Copy link

Aarch64 and Armv7h machines are becoming more and more common place now days. There are Windows, Mac, and Linux binaries, but the Linux binaries do not cover these architectures. Please add binary releases for them.

@jgm
Copy link
Owner

jgm commented Apr 16, 2019 via email

@tarleb
Copy link
Collaborator

tarleb commented Apr 18, 2019

AFAIK, both @vmchale and @2i3r managed to cross-compile pandoc. Maybe they have any insights on how we could support these platforms?

See also hslua/hslua#73.

@2i3r
Copy link

2i3r commented Apr 18, 2019

I have my armv7h cross-compiler setup running. unfortunately currently I don't have access to an aarch64 platform to test, but as far as I can say, armv7h binaries will run on most chips providing aarch64 such as armv8. (link and link). so it might be enough to provide only armv7h binary.
I would love to share or help on this issue.

@2i3r
Copy link

2i3r commented Apr 18, 2019

On my tries direct compilation of pandoc and ghc, was so much slow. so I cross-compiled pandoc using cabal cross-compiler, which by itself needed binutils, libffi, ghc and pandoc dependecies cross-compiled, and then Simple build type in pandoc.cabal and disabling Template Haskell flag during compilation was needed.
Someone may manage to run Template Haskell in cross-compile mode, so Simple build type may not be mandatory and data files would be embedded in the output binary.

@mb21 mb21 added the package label Apr 21, 2019
@mb21
Copy link
Collaborator

mb21 commented Apr 21, 2019

@2i3r do you think it would be hard to set this up on CircleCI (or similar), which we currently use for the pandoc Linux builds?

@2i3r
Copy link

2i3r commented Apr 23, 2019

I don't have any experience with CircleCI or similar tools.
In the method I described above, what makes armv7h release is an special binary of cabal and ghc which configured to cross-compile binaries, and there are backend tools such as libraries, headers, utils and c compiler (multiple GBs) it uses. It can be configured inside a docker or a PC, then I think build tools such as CirckeCI might be set up to use those configured compilers and then if errors handled correctly then it might work as expected.

@ickc
Copy link
Contributor

ickc commented May 19, 2019

@2i3r, how much time such build would normally take? One concern to use CI to build is the time limit. In the past people has tried using QEMU in CI but it becomes too slow to be manageable.

Otherwise, especially when a docker can be set up, it should be easy to do it in CircleCI.

@rhenwood-arm
Copy link

I'm also interested in this activity. Drone.io have builders for AArch64 and ArmV7. If that is of interest, I could begin to work on this with a colleague.

@jgm
Copy link
Owner

jgm commented Oct 4, 2019

@rhenwood-arm - since pandoc is open source, nothing stops you from building it and making the binaries available. I think that would be appreciated!

Whether we want to provide officially endorsed ARM binaries here (on the GitHub releases page) is another question. For now, I'd be happy to have a third-party source for them.

@ickc
Copy link
Contributor

ickc commented Oct 4, 2019

My experience on directly compiling on ARM:

For armv7l, I'm trying to provide it as an unofficial effort here: https://github.com/ickc/pandoc-arm/releases. Probably will fall in pandoc-extras later. (By the way, it took Raspberry Pi 4 about 2.8 GiB of RAM and 3 hours to compile.)

For aarch64, I have trouble getting stack/cabal compiled. If I'm not mistaken it is not supported.

If someone can give me the exact command of cross compiling pandoc targeting the 2 platforms, I can try to build them and test it on those arch.

In the long run, if it doesn't take too much time to compile with CI, and if there's a way to run tests on the target arch (current tests are run using cabal/stack in the compiling stage), then may be it can be provided as an official release.

By the way, am I right that for the current released binaries, the Windows' come from CI, but he macOS/Linux builds come from @jgm directly? I think this might be true in the past but not sure any more.

@jgm
Copy link
Owner

jgm commented Oct 4, 2019

By the way, am I right that for the current released binaries, the Windows' come from CI, but he macOS/Linux builds come from @jgm directly? I think this might be true in the past but not sure any more.

Correct, but this may soon change. I've been working on getting releases built for all three architectures with GitHub Actions CI. See the rc/test branch.

@vmchale
Copy link
Contributor

vmchale commented Nov 30, 2019

@stormdragon2976 if you want binaries, I have some here: https://github.com/vmchale/ghc-cross/releases/tag/Binaries

I also have linux -> linux cross-compilers for anyone who wants to try on their own: https://github.com/vmchale/ghc-cross/releases/tag/8.6.5 (both armv7 and aarch64)

@stormdragon2976
Copy link
Author

stormdragon2976 commented Nov 30, 2019 via email

@jgm
Copy link
Owner

jgm commented Nov 30, 2019

One small problem that happened when I tried to convert a file:

Could not find data file /home/vanessa/.cabal/share/aarch64-linux-ghc-8.6.5/pandoc-2.7.3/data/abbreviations

Set the embed_data_files flag when you compile; this will ensure that the data files are included in the binary.

@stormdragon2976
Copy link
Author

stormdragon2976 commented Nov 30, 2019 via email

@cstrouse
Copy link

I work on Chromebrew and we'd like to have ARMv7 binaries as well since Google chose to ship all ARMv8 Chromebooks with 32-bit user ABIs due to only having 4GB of memory.

@annProg
Copy link

annProg commented Feb 22, 2020

Any progress for this issue?

I try to build pandoc-2.9.2 for raspberry pi 4b, and got error

[133 of 161] Compiling Text.Pandoc.Readers.LaTeX
ghc: internal error: getMBlock: mmap: Invalid argument
    (GHC version 8.4.4 for arm_unknown_linux)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
cabal: Leaving directory '/tmp/cabal-tmp-8306/pandoc-2.9.2'

@ickc
Copy link
Contributor

ickc commented Feb 22, 2020

I have exactly the same issue on both aarch64 and armv7l on Raspberry Pi 4 4GB model with pandoc 2.9+. 4GB was just enough to compile pandoc briefly after Raspberry Pi 4 launched but then pandoc's need soon exceeds that.

If you can't wait there's a build in https://github.com/ickc/pandoc-arm/releases, with the last version compiled successfully on RPi4, I think 2.8.

I should emphasize that you can always add more swap. Long time ago I think I successfully compiled pandoc with 1GB RAM (forgot if it was a RPi original or another low mem. laptop.) Just add a ton of swap and it will work, albeit very slowly and your swap device's life shortened if it is flash (e.g. microSD.)

I was about to experiment with zram if I have time and see if it helps.

(As other has noted, the proper way to do it is to cross-compile and do it in CI, and distribute it officially. But until that happens...)

@annProg
Copy link

annProg commented Feb 22, 2020

@ickc the version is 2.7.3. It has problem on epub toc. So I am looking for 2.9+

@ickc
Copy link
Contributor

ickc commented Feb 22, 2020

Then build your own, add a ton of swap and wait for a few days.

@annProg
Copy link

annProg commented Feb 27, 2020

I successfully compiled 2.9.2 . see https://github.com/arm4rpi/pandoc-arm/releases

@ickc
Copy link
Contributor

ickc commented Feb 27, 2020

How? Just ton of swap?

Are you going to provide that continuously in the future? If so there's no need for 2 of us doing the same thing. Also, if you decided to continually provided that, you may want to think about security implication. e.g. as outlined in my repo I found that it is reproducible build so providing checksum should be suffices (may be this is the reason we still need to have 2 independent builds just for others to double check.)

As said above if people working on cross compiling has a solution then that would be preferred as doing it in the cloud in the open means less trust on us is needed. But before that it may be something you and I can do in the interim.

@annProg
Copy link

annProg commented Mar 4, 2020

@ickc I am trying to build using github actions. But got some errors, see https://github.com/arm4rpi/pandoc-arm/runs/483167534

failed to install.
base-compat-0.11.1-HgwyiOzKEQ9KmBXaEHlBfi failed during the configure step.
The exception was:
ExitFailure 1
base-orphans-0.8.2-3ZQM2pxPHbTALAIaG3rSz1 failed during the configure step.
The exception was:
ExitFailure 1
doctemplates-0.8.1-Iuw4He9PrsFAzb6E5YhZXp depends on doctemplates-0.8.1 which
failed to install.
ipynb-0.1-5azHA6CVSg970U03zfl88V depends on ipynb-0.1 which failed to install.
pandoc-2.9.2-5EHBhYTFz4u65WvZc3TiB5 depends on pandoc-2.9.2 which failed to
install.
pandoc-types-1.20-8EQHit822IcL8VC8fC6OZi depends on pandoc-types-1.20 which
failed to install.
skylighting-0.8.3.2-C2Li4Bm4LPW5TVFmKvF7GB depends on skylighting-0.8.3.2
which failed to install.
skylighting-core-0.8.3.2-BDFsVUriuS2BVnrNvKwqBs depends on
skylighting-core-0.8.3.2 which failed to install.
texmath-0.12.0.1-2htPgXHYu9jEzl2nKKanYP depends on texmath-0.12.0.1 which
failed to install.
time-compat-1.9.2.2-B4M2FpSJt3R4h2zQkxlzCq depends on time-compat-1.9.2.2
which failed to install.

##[error]Process completed with exit code 1.

do you know how to solve it?

@ickc
Copy link
Contributor

ickc commented Mar 4, 2020

Sorry no experience on GitHub Action.

What was the earlier errors? Do you see something fail to compile?

Could you bring me up to speed about the GitHub Action? e.g. do they provide armv7l and aarch64 or are you cross-compiling?

In the past experience when dealing with CI, is the "debug loop" can be troublesome. One way people often try to solve situation like this is to use the docker image of the CI instance itself (e.g. Travis CI) and debug it interactively.

@annProg
Copy link

annProg commented Mar 4, 2020

Seem no armv7l and aarch64 provided, I am using qemu-user-static

for debug. run

docker run -it --privileged --rm ann17/pandoc-ghc-arm:latest /bin/bash
cd rootfs
cp /etc/resolv.conf etc
mount -t devtmpfs devtmpfs dev
mount -t devpts devpts dev/pts
mount -t sysfs sysfs sys
mount -t tmpfs tmpfs tmp
mount -t proc proc proc

# chroot to arm
chroot . /bin/bash
uname -m
cabal update
cabal install -v time-compat-1.9.2.2

docker image ann17/pandoc-ghc-arm:latest was build from https://github.com/arm4rpi/ghc-arm

@annProg
Copy link

annProg commented Mar 14, 2020

Now I am trying to build for aarch64 and armv7l using Github Actions with qemu-user-static. Encountered two problems, can anyone help?

cabal resolving dependencies problem

Limited by execution time(6 hours), it is not possible to build successfull. So I want to pre build dependencies( see https://github.com/arm4rpi/pandoc-deps), but the problem is pandoc and pandoc-citeproc resolving deferent dependencies, libraries below need rebuild when build pandoc-citeproc, can not use prebuilt library for pandoc, the long suffix of library is different, for example: pandoc-2.9.2-cd37f71aada58e7c054fb0d9850c9c6facc99a5599ef764195fc2ac2b25bb988 and pandoc-2.9.2-028d873232afbbca4423e4e65d416bc237bf7809e93e4d4cd251f51fd681e9b8. So, now only pandoc will build successful.

In order, the following will be built:
 - time-compat-1.9.3 (lib) (requires download & build)
 - aeson-1.4.6.0 (lib) (requires download & build)
 - yaml-0.11.3.0 (lib) (requires download & build)
 - skylighting-core-0.8.3.2 (lib) (requires download & build)
 - pandoc-types-1.20 (lib) (requires download & build)
 - ipynb-0.1 (lib) (requires download & build)
 - doctemplates-0.8.1 (lib) (requires download & build)
 - aeson-pretty-0.8.8 (lib) (requires download & build)
 - HsYAML-aeson-0.2.0.0 (lib) (requires download & build)
 - skylighting-0.8.3.2 (lib) (requires download & build)
 - texmath-0.12.0.1 (lib) (requires download & build)
 - pandoc-2.9.2 (lib) (requires download & build)

armv7l always out of memory

I had add huge swap.

swapon: /mnt/swapfile: insecure permissions 0644, 0600 suggested.
swapon: /home/swapfile: insecure permissions 0666, 0600 suggested.
              total        used        free      shared  buff/cache   available
Mem:           6945         712         124          25        6108        5904
Swap:         30719          31       30688

alway out of memory when Compiling Text.Pandoc.Readers.LaTeX

[133 of 162] Compiling Text.Pandoc.Readers.LaTeX.Parsing
[134 of 162] Compiling Text.Pandoc.Readers.LaTeX
ghc: out of memory (requested 1048576 bytes)
CallStack (from HasCallStack):
  die', called at ./Distribution/Client/ProjectOrchestration.hs:977:55 in main:Distribution.Client.ProjectOrchestration
cabal: Failed to build pandoc-2.9.2 because it depends on pandoc-2.9.2 which
itself failed to build.
Failed to build
pandoc-2.9.2-a37414eb634713f674eb2c181278989d68c5c7af029fece038bbeefb7e04ca90.
The build process terminated with exit code 251

@tarleb
Copy link
Collaborator

tarleb commented Mar 15, 2020

I think the only solution for the out-of-memory error is to further break up Text.Pandoc.Readers.LaTeX into smaller modules.

Not sure about the dependency problems. The differences in hashes are probably due to different flags being used. Maybe try using more specific constraint --constraint='pandoc:embed_data_files instead of global --flag embed_data_files. You could also experiment with cabal.project.freeze files to ensure reproducible builds.

@benz0li
Copy link
Sponsor Contributor

benz0li commented Mar 6, 2021

Next steps on my part:

  1. Figure out the issue with gitlab/gitlab-runner for linux/arm64.
  2. Rebuild all Docker images and recreate all multi-arch manifests.

Then, the digests for all Docker images and multi-arch manifest are publicly available in the GitLab CI/CD Pipeline build logs.

@jgm
Copy link
Owner

jgm commented Mar 6, 2021

Sorry: I am pretty ignorant about docker.
I tried using just the tag originally, and it gave me the image for the wrong arch (amd64 instead of arm). Probably I'm doing something wrong: do I need to tell it explicitly which arch to use in docker run, and if so how?
[EDIT: It seems to be working now. I was probably doing something else wrong before!]

@znmeb
Copy link

znmeb commented Mar 6, 2021

@benz0li I've built multi-arch Docker images before. Unless the setup has changed, they run on an x86_64 server and build the other architectures using a static QEMU layer to emulate them. The upside is that you can build a multi-arch image for anything QEMU can emulate - arm64, PowerPC, IBM Z series, etc. So they'll work for anything where your base image - Ubuntu or Debian - has the binary toolchain to build pandoc.

I vaguely recall trying it (for just arm64) on my workstation last fall when I got my new one. And I got it to run on Travis CI but it couldn't complete a few steps within their 50 minute time limit.

@benz0li
Copy link
Sponsor Contributor

benz0li commented Mar 6, 2021

@jgm My fault! The manifest for tag 8.10.4 got overwritten while creating tag 9.0.1 for os/arch linux/amd64. This is fixed with commit 26c291eb.

No, you don't have to tell which arch to use. If the manifest consists of multiple os/arch images, Docker pulls the image for the arch Docker is running on.

@jgm
Copy link
Owner

jgm commented Mar 6, 2021

I ran out of memory on an 8 GB instance -- not even on one of the memory-hungry modules.

I'm a good way through a build now on a 16 GB t4g.xlarge AWS instance. top is showing a bit less than 6 GB free, 3-4 GB used, and 6-7 GB buffer/cache. On AMD, I can build on 4 GB, and even on 2 GB with some restarts. Is there an explanation of why we need so much more RAM to compile on ARM? (The 16 GB build machine costs 8 cents / hour more than the 8 GB -- not really a big deal if the build can complete in an hour or two -- but I'm still surprised by the memory needs.)

@znmeb
Copy link

znmeb commented Mar 6, 2021

@jgm I've never tried it on 8 GB - just 4, which fails, and 16, which works. My recollection is that the crashes are in the back end, and that I tried both the gcc and LLVM back ends. I don't remember if I ever got the native GHC back end to work or not, or which version of GHC I used - everything works in 16 GB with the GHC 8.0 that ships in Ubuntu 18.04 so I stopped experimenting.

@benz0li
Copy link
Sponsor Contributor

benz0li commented Mar 7, 2021

I can't explain the enormous memory needs while building pandoc on ARM. Tried on a t4g.xlarge (16 GB RAM) myself... and failed. Just started a new build with a 16 GB swapfile in addition.

I used some heavy machinery, t4g.2xlarge (8 vCPUS, 32 GB RAM + 16 GB swapfile), while building pandoc-2.11.4-1-arm64.deb as GHC Docker Images were built in parallel.

@jgm
Copy link
Owner

jgm commented Mar 7, 2021

Same experience here on t4g.xlarge. Trying now with t4g.2xlarge! It may mean that creating the release package on arm costs $1-2. We can afford that. But it's strange, the difference between ARM and non-ARM.
(The time it takes to compile is also much longer.)

@znmeb
Copy link

znmeb commented Mar 7, 2021

Interesting - my AGX Xavier has 16 GB of RAM, Ubuntu18.04 LTS for arm64, NVIDIA Docker runtime. The stock apt binaries are GHC 8.0 "unregisterised" - it doesn't have the native back end - and cabal-install 1.24. LLVM is 3.7

All I have to do to build Pandoc is upgrade cabal-install to 3.0 and do a cabal install pandoc. It takes overnight to do all that, even with the eight cores, but it does work. IIRC it works with either the GCC back end or the LLVM back end. And I can bootstrap up to GHC 8.2, or upgrade cabal-install and bootstrap up to GHC 8.4, etc. I haven't been able to get stack to work, but I don't need it.

My recollection from watching these things go by with top is that the memory hogs are the back ends, especially the gcc one, not GHC itself. If you've got connections in the GHC team and want me to capture any trace / debug info on this on the AGX Xavier, have them open an issue on https://github.com/edgyR/edgyR-containers/issues.

One other possibility for Ubuntu: there is a PPA on Launchpad with Haskell backports including Pandoc: https://launchpad.net/~savoury1/+archive/ubuntu/haskell-build/. It does not appear to be set up to build for arm64 but that may be changeable. That would use Ubuntu's build farm, I think.

@jgm
Copy link
Owner

jgm commented Mar 7, 2021

I've been able to build with the 2xl instance. I'm adding a script to automate this process, and I'll try to include an arm64 binary in the next release. (We can then get statistics to see how much use it gets.)

@benz0li
Copy link
Sponsor Contributor

benz0li commented Mar 7, 2021

I have been able to build it with a t4g.xlarge AWS EC2 instance:

  • vCPUS: 4
  • RAM: 16 GB
    • 16 GB swapfile in addition
  • Dockerfile used: registry.gitlab.b-data.ch/ghc/ghc4pandoc:8.10.1

Build time: 5 hours.

@benz0li
Copy link
Sponsor Contributor

benz0li commented Mar 7, 2021

The issue with the GitLab CI/CD Pipeline is resolved, too. I am going to rebuild all images and multi-arch manifests for registry.gitlab.b-data.ch/ghc/ghc4pandoc tomorrow.

I will continue to maintain these images and keep them publicly available. @jgm Let me know if you need a version in-between 8.10.1 and 8.10.4.

@znmeb
Copy link

znmeb commented Mar 7, 2021

The issue with the GitLab CI/CD Pipeline is resolved, too. I am going to rebuild all images and multi-arch manifests for registry.gitlab.b-data.ch/ghc/ghc4pandoc tomorrow.

I will continue to maintain these images and keep them publicly available. @jgm Let me know if you need a version in-between 8.10.1 and 8.10.4.

Hmm ... maybe I should move my Ubuntu build chain to GitLab and do my R and RStudio builds there on top of your Pandoc build. That would check off a huge box for me; I could go with R from the Ubuntu source repos instead of from upstream. I only need the Jetson base / NVIDIA tools for the last step in my processing.

@jgm jgm closed this as completed Mar 8, 2021
@benz0li
Copy link
Sponsor Contributor

benz0li commented Mar 8, 2021

@jgm pandoc-2.12-linux-arm64.tar.gz seems to be broken:

$ tar -xfz pandoc-2.12-linux-arm64.tar.gz

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

@jgm
Copy link
Owner

jgm commented Mar 8, 2021

Oh, weird! I've manually unpacked the deb and created a tarball from it, replacing the one on releases.
@benz0li can you try again?

I think I can guess what happened. My build script waits until the .tar.gz appears in artifacts, and then downloads it. Perhaps it downloaded it while it was incomplete? IN any case the build script needs some improvement.

@benz0li
Copy link
Sponsor Contributor

benz0li commented Mar 9, 2021

Upacking works fine now.

But I'm not sure, if this is the intended structure and content for the arm64 release:

$ tree pandoc-2.12-arm64 
pandoc-2.12-arm64
├── bin
│   └── pandoc
└── tmp
    └── usr
        ├── bin
        │   └── pandoc
        └── share
            ├── doc
            │   └── pandoc
            │       └── copyright
            └── man
                └── man1
                    └── pandoc.1.gz

9 directories, 4 files

Compared to the amd64 release:

$ tree pandoc-2.12-amd64 
pandoc-2.12-amd64
├── bin
│   └── pandoc
└── share
    └── man
        └── man1
            └── pandoc.1.gz

4 directories, 2 files

@jgm
Copy link
Owner

jgm commented Mar 9, 2021

No, that's not right. Let me fix it manually; I don't want to wait another 3 hours for a build to complete!

@jgm
Copy link
Owner

jgm commented Mar 16, 2021

@benz0li Good news.
I tried tweaking a few settings and now the build takes one hour 15 minutes.
What I did was increase heap size for the compiler and (more significantly) told it to use 4 cores instead of 1. (With this build machine we could probably go even higher.)

@znmeb
Copy link

znmeb commented Mar 16, 2021

My experiments didn't gain much from multiple cores. A bunch of small-ish libraries get built at the beginning, but once they're done, it's single-threaded for the large ones.

@benz0li
Copy link
Sponsor Contributor

benz0li commented Mar 18, 2021

@jgm More good news.

A linux/arm64-build on a Mac mini (Apple M1 chip, 16 GB memory) using the tech preview of Docker Desktop for Apple M1 takes just under an hour:

make_debpkg

Docker settings used:

Docker_preferences

ℹ️ Memory increased from 4 GB to 16 GB; Swap increased from 1 GB to 4 GB; CPUs left at 4 👉 the Mac mini M1 has 4 performance cores.

Bearing in mind that Docker Desktop for Apple M1 runs in a virtual machine, which uses the new Virtualization Framework, the performance is outstanding.

@jgm
Copy link
Owner

jgm commented Mar 18, 2021

Nice, so this produces a native M1 build? (See #6960)

@benz0li
Copy link
Sponsor Contributor

benz0li commented Mar 18, 2021

No. Docker Desktop for Apple M1 is effectively linux/arm64:

Virtualization

Create virtual machines and run Linux-based operating systems.


Overview

The Virtualization framework provides high-level APIs for creating and managing virtual machines on Apple silicon and Intel-based Mac computers. Use this framework to boot and run a Linux-based operating system in a custom environment that you define. The framework supports the Virtio specification, which defines standard interfaces for many device types, including network, socket, serial port, storage, entropy, and memory-balloon devices.

Developer > Documentation > Virtualization

@znmeb
Copy link

znmeb commented Mar 19, 2021

@benz0li Is the Apple operating system shim as light-weight as the Windows version of Docker Desktop, which uses WSL 2?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests