Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross compilation for iOs, Android etc #6266

Closed
vicuna opened this issue Dec 12, 2013 · 41 comments

Comments

Projects
None yet
2 participants
@vicuna
Copy link

commented Dec 12, 2013

Original bug ID: 6266
Reporter: strobegen
Assigned to: @damiendoligez
Status: closed (set by @damiendoligez on 2015-05-12T14:46:59Z)
Resolution: fixed
Priority: normal
Severity: feature
Target version: 4.02.2+dev / +rc1
Fixed in version: 4.02.2+dev / +rc1
Category: ~DO NOT USE (was: OCaml general)
Tags: patch
Related to: #4303 #5887 #6613
Parent of: #6861
Monitored by: @whitequark @gasche @dbuenzli gerd

Bug description

I guess that out of box compiling support for mobile platforms like iOs and Android might be a significant stimulus for popularization of language.
Currently mobile market is huge in stlll growing (for instance http://www.asymco.com/2012/01/17/the-rise-and-fall-of-personal-computing/).

I think that creating lot of bindings for native platforms APIs (like Xamarian did for mono, for instance) is not so important - that is really necessary that:

  • cross compilation
  • out of box ways to easily build library binary which can be linked to native application code with original platform tools/IDEs.

File attachments

@vicuna

This comment has been minimized.

Copy link
Author

commented Dec 12, 2013

Comment author: @gasche

I know nothing at all about cross-compilation, but I think that cross-compilation to android is already possible today, as explained by Jonathan Protzenko in the following blog post. I've also heard about iOS applications implemented in OCaml, see Psellos for example.

@vicuna

This comment has been minimized.

Copy link
Author

commented Dec 13, 2013

Comment author: strobegen

Yes it all possible now, I also know examples of few games written on Ocaml in iOs App Store (I guess, I was read some time ago article about small team which shipped few game titles to AppStore (not psellos)) but this functionally doesn't included to default compiler, currently only short way to build this kinds of apps for iOs is OCamlXARM (http://psellos.com/ocaml/compile-to-iphone.html), I'm not sure but I guess that OCamlXARM contain few patches to original Ocaml distribution related to ARM assembler (& crosscompiling) parts. Therefore my opinion that it will be very useful for Ocaml community if this functionally will be available in default distribution and officially supported:
on OSX: cross compiling to iOs: ARM, ARM64 & Android: ARM.
on Windows: cross compiling to Android ARM,
on Linux: cross compiling to Android ARM.

@vicuna

This comment has been minimized.

Copy link
Author

commented Oct 17, 2014

Comment author: @whitequark

I have started working on proper cross-compilation to Android, and shortly after that I will probably switch to iOS. My goal is to produce small, self-contained patches that can be merged one by one; I will post a patchset shortly. In fact, the amount of work required is not as large as one would expect.

@vicuna

This comment has been minimized.

Copy link
Author

commented Oct 17, 2014

Comment author: @whitequark

I have uploaded three small patches, which allow Android cross-compilation to proceed (both world and world.opt), given that config/{Makefile,m.h,s.h} are provided.

This can be easily tested using:

opam repo add android https://github.com/whitequark/opam-android
opam install ocaml-android
ocamlfind -toolchain android ocamlopt helloworld.ml

I will follow up these patches with ones that properly detect the target environment features without running target binaries.

@vicuna

This comment has been minimized.

Copy link
Author

commented Oct 17, 2014

Comment author: @whitequark

Actually, maybe not. I have just realized that my approach is fundamentally flawed while writing out some detailed instructions. Please wait for an updated patchset.

@vicuna

This comment has been minimized.

Copy link
Author

commented Oct 17, 2014

Comment author: strobegen

I will try to test when it be ready

@vicuna

This comment has been minimized.

Copy link
Author

commented Dec 21, 2014

Comment author: @whitequark

@gasche, I've attached a patch that brings cross-builds one step closer. Specifically this patch ensures that the variable CAMLRUN, which is already set by the configure script, is actually used when running boot/ binaries, freshly built binaries, and is also embedded in the shebang line in all the built tools.

It also replaces some usage of gcc with $(CC).

@vicuna

This comment has been minimized.

Copy link
Author

commented Dec 27, 2014

Comment author: @gasche

Applying the patch makes "make world.opt" fail on my machine at the following point:

make ocamlc.opt
make[2]: Entering directory `/home/gasche/Prog/ocaml/github-trunk'
/home/gasche/Prog/ocaml/github-trunk/boot/ocamlrun ./ocamlopt -nostdlib -I stdlib -I otherlibs/dynlink -use-runtime /home/gasche/Prog/ocaml/github-trunk/boot/ocamlrun -ccopt "-Wl,-E" -o ocamlc.opt
compilerlibs/ocamlcommon.cmxa compilerlibs/ocamlbytecomp.cmxa
driver/main.cmx -cclib "-lm -ldl -lcurses -lpthread"
./ocamlopt: unknown option '-use-runtime'.

@vicuna

This comment has been minimized.

Copy link
Author

commented Dec 28, 2014

Comment author: @whitequark

I've updated the patch. Notable changes:

  • All tools and all Makefile.nt files are also updated. (Some were missing in original patch.)
  • ocamlyacc, being the only other native component except ocamlrun, is also detected with ./configure and required to be present on host for cross-builds.
  • ocamlbuild/Makefile.noboot is removed, as it was never referenced and also ocamlbuild/Makefile does not require ocamlbuild anymore.

make world world.opt now passes.

@vicuna

This comment has been minimized.

Copy link
Author

commented Feb 8, 2015

Comment author: @gasche

The patch seems okay in principle and I was about to merge it, but:

  • the patch conflicts with Damien's change to use $(CC) more consistently in trunk@15784
    66af647
  • Damien only patched trunk, but we may want to apply this one change to 4.02 as well

whitequark, would you mind preparing two versions of the patch, one that applies cleanly to trunk and the other for 4.02. I feel sorry to ask for such tedious work, I could do it but I'll rather try to merge the other patches on my list during my weekend time window.

Damien, could you give your opinion on whether the present change and trunk@15784 are eligible for 4.02+dev?

@vicuna

This comment has been minimized.

Copy link
Author

commented Feb 9, 2015

Comment author: @damiendoligez

@whitequark if you don't want to bother with two versions of the patch, let's go for 4.02 and I'll deal with the conflicts when I merge it into trunk (after the 4.02.2 release). Or split it into two patches, one for CAMLRUN and one for CC.

I'm a bit nervous with the use of -use-runtime because it precludes using libraries that have C object files, but I guess it's OK for building the compiler.

There is a small problem with the patch: as far as I can tell, it gives the -use-runtime option twice to the compiler (but only in Makefile, not in Makefile.nt). You have to decide whether to add it to OCAMLC or to give it on the command line.

I'm OK for integrating this and trunk@15784 into 4.02.2.

@vicuna

This comment has been minimized.

Copy link
Author

commented Feb 12, 2015

Comment author: @whitequark

I've attached diffs for 4.02 and trunk.

I've fixed the double -use-runtime problem.

@vicuna

This comment has been minimized.

Copy link
Author

commented Feb 15, 2015

Comment author: @gasche

Damien, I just tried to cherry-pick trunk@15784 in 4.02, but:

  • it will also conflict (slightly) with #5887 so I'd rather wait for #5887 to be tested and hopefully picked as well
  • I don't understand the interaction between this change and 4.02@15762 ( ca53e95 ); why is -DPROFILING not used in trunk, only in 4.02? Why does byterun/Makefile not use it as well?

My guess would be that -DPROFILING is the right thing to do both in trunk and in byterun/. I can make the change but I'd rather double-check it.

@vicuna

This comment has been minimized.

Copy link
Author

commented Feb 27, 2015

Comment author: @whitequark

Ping?

1 similar comment
@vicuna

This comment has been minimized.

Copy link
Author

commented Mar 6, 2015

Comment author: @whitequark

Ping?

@vicuna

This comment has been minimized.

Copy link
Author

commented Mar 9, 2015

Comment author: @damiendoligez

Re: -DPROFILING

I fixed it in 4.02 and not trunk because it will get merged eventually, and I don't expect it to cause problems in the meantime.

It's in asmrun and not byterun because byterun doesn't have a set of object files annotated for profiling.

PS. If commit 15762 is bothering you, feel free to overwrite it, it's not important.

@vicuna

This comment has been minimized.

Copy link
Author

commented Apr 30, 2015

Comment author: @damiendoligez

I cherry-picked trunk@15784.

Then I tried to apply this patch (the "trunk" one applies cleanly to 4.02) but there is a problem: it breaks the bootstrap. The way you use -use-runtime ensures that the compiler always compiles code that runs on the same runtime as the compiler itself. But the bootstrap needs to make an intermediate compiler that runs on the old runtime and generates code for the new runtime.

I think the solution is to replace -use-runtime ${OCAMLRUN} with -use-prims byterun/primitives for the bootstrap executables (ocamlc, ocamllex, ocamldep) [see new patch]. I checked that it fixes the problem with bootstrapping, but I'm not 100% sure it does what is needed for cross-compilation. Can you check?

@vicuna

This comment has been minimized.

Copy link
Author

commented May 2, 2015

Comment author: @whitequark

No, this of course does not work. The cross-compiling patch uses -use-runtime in order to set the #! field correctly.

With your patch, the /bin looks like this:
$ for i in *; do echo $i head -n1 $i; done
ocaml #!/home/whitequark/.opam/4.02.1+32bit/arm-linux-androideabi/bin/ocamlrun
ocamlc #!/home/whitequark/.opam/4.02.1+32bit/arm-linux-androideabi/bin/ocamlrun
ocamlcp #!/home/whitequark/.opam/4.02.1+32bit/bin/ocamlrun
ocamldep #!/home/whitequark/.opam/4.02.1+32bit/arm-linux-androideabi/bin/ocamlrun
ocamllex #!/home/whitequark/.opam/4.02.1+32bit/arm-linux-androideabi/bin/ocamlrun
ocamlmklib #!/home/whitequark/.opam/4.02.1+32bit/bin/ocamlrun
ocamlmktop #!/bin/sh
ocamlobjinfo #!/home/whitequark/.opam/4.02.1+32bit/bin/ocamlrun
ocamlopt #!/home/whitequark/.opam/4.02.1+32bit/arm-linux-androideabi/bin/ocamlrun
ocamloptp #!/home/whitequark/.opam/4.02.1+32bit/bin/ocamlrun
ocamlprof #!/home/whitequark/.opam/4.02.1+32bit/bin/ocamlrun

In order for cross-compilation to be possible, all of these should point to #!/home/whitequark/.opam/4.02.1+32bit/bin/ocamlrun. (I'm especially puzzled by ocamlopt, which is not required for bootstrapping...)

@vicuna

This comment has been minimized.

Copy link
Author

commented May 5, 2015

Comment author: @damiendoligez

For ocamlopt (and ocaml): it's because they are made by the same Makefile as ocamlc.

I think the solution is to use an as-yet undocumented feature of these two options: if you specify both -use-runtime and -use-prims, the list of primitives will come from -use-prims but the runtime path will come from -use-runtime. It works because the bootstrap doesn't care what's in the header, and when you're cross-compiling the lists of primitives are the same so you care only about the header.

Here is a new diff. Please check if it works for cross-compilation.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 5, 2015

Comment author: @whitequark

Yup, this works completely! I've imported it in opam-android and I have built the compiler and a few packages, with C extensions and not.

Amazing. Thank you so much for your work on this.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 6, 2015

Comment author: @damiendoligez

Patch applied to branch 4.02 (rev 16093).

@vicuna

This comment has been minimized.

Copy link
Author

commented May 6, 2015

Comment author: @damiendoligez

Except that it doesn't work... I've reverted the patch while we're investigating.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 7, 2015

Comment author: @damiendoligez

Mark Shinwell and I brainstormed about this issue, here are our
conclusions:

Here is why it doesn't work: in normal mode, it sets the absolute path
of ocamlrun in the headers of the files that will be installed as
executables. But this absolute path points to the build directory, not
to the final binary directory, so this fails. In the case of
cross-compilation, it sets the path to ocamlrun, which is still
incorrect, but happens to work if you have the binary directory in your
PATH.

What we need is to set the header to the absolute path of the
installed ocamlrun as seen on the target machine. And that's what
the makefiles are doing by default: they set it to
${BINDIR}/ocamlrun. You just need to make sure that BINDIR is a
path on the target machine, not on the host.

Then, if you want to make install on the host, you can specify
a pre-prefix for the install with the DESTDIR variable.

For example, assuming you want to install in /usr/local on the target
and that the target's filesystem is mounted under /mnt/target on the
host, you would have:

binary directory on target = /usr/local/bin
on host = /mnt/target/local/bin

and you need to use:
PREFIX=/usr/local (i.e configure -prefix /usr/local)
DESTDIR=/mnt/target (i.e make install DESTDIR=/mnt/target)

Then the bytecode file headers will have the correct value:
#!/usr/local/bin/ocamlrun

If the target's filesystem is not mounted on the host, you can choose
any directory you want for DESTDIR, and then build a tarball from
the tree installed in that directory.

This is all from analyzing the build procedure and the previous
discussion: we don't have a cross-compilation setup on hand, so I've
prepared a new patch for you to test (camlrun-fixed-3.diff).

@vicuna

This comment has been minimized.

Copy link
Author

commented May 7, 2015

Comment author: @whitequark

No, that's wrong. I'm not cross-building the compiler itself! The newly built compiler will be executed on the same machine, and from the same filesystem, as the host compiler.

The reason I need to change ocamlrun at all is that by default, the #! in the newly built compiler points to /its own/ ocamlrun, which is built, like the runtime library is, for the target, and so it cannot be executed on host.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 7, 2015

Comment author: @whitequark

Essentially the problem we're running into is that the compiler and the runtime library (and thus ocamlrun) built in the same build tree will be executed on different architectures.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 7, 2015

Comment author: @damiendoligez

Ah ok I understand now.

Here is a new plan:

  • It does not make sense to install the (target) ocamlrun in the same directory as the cross-compiler, so we should install (copy) the host's pre-existing ocamlrun into ${BINDIR} at installation time.

  • The compiler's configuration (the camlheader file installed into ${LIBDIR}) should refer to the path of ocamlrun on the target.

  • The compiler itself must be built with another version of camlheader, which will refer to ${BINDIR} on the host.

  • When cross-compiling, the location of ocamlrun on the target must be specified by the user because there is no other way to determine it.

Does this sound workable?

@vicuna

This comment has been minimized.

Copy link
Author

commented May 7, 2015

Comment author: @damiendoligez

Can you try this new version (camlrun-fixed-4.diff) ?

@vicuna

This comment has been minimized.

Copy link
Author

commented May 8, 2015

Comment author: @whitequark

It seems to build correctly but fails at installation:

cd byterun; make install
make[1]: Entering directory '/home/whitequark/.opam/4.02.1+32bit/build/ocaml-android32.4.02.2/byterun'
cp type -p /home/whitequark/.opam/4.02.1+32bit/bin/ocamlrun /home/whitequark/.opam/4.02.1+32bit/arm-linux-androideabi/bin/ocamlrun
cp: invalid option -- ':'
Try 'cp --help' for more information.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 8, 2015

Comment author: @whitequark

After removing the type -p around the ocamlrun path, everything installs and works, though target ocamlrund and ocamlyacc are still left in the tree, falling short of a fix to PR6861.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 8, 2015

Comment author: @damiendoligez

The purpose of type -p is to get the absolute name of the ocamlrun in path. Apparently, type is not as portable as I hoped, but I'm curious: how did it get the full name of your ocamlrun? Did you configure it yourself?

For ocamlyacc, we should probably do the same as for ocamlrun, namely copy the one from the pre-existing install.

For ocamlrund, we should not install it on the host.

I'll make a new patch.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 8, 2015

@vicuna

This comment has been minimized.

Copy link
Author

commented May 8, 2015

Comment author: @dbuenzli

@doligez it's always a good idea to have a look at the posix spec. You'll see that the type utility has neither standarized argument options nor standarized a output.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/type.html

@vicuna

This comment has been minimized.

Copy link
Author

commented May 8, 2015

Comment author: @dbuenzli

command -p -v may be a better fit

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/command.html

@vicuna

This comment has been minimized.

Copy link
Author

commented May 8, 2015

Comment author: @damiendoligez

The OPAM package supplies a config/Makefile explicitly: https://github.com/whitequark/opam-android/blob/master/packages/ocaml-android32.4.02.2/files/config/Makefile.in#L9 [^]

I've left the install of ocamlrund for the moment, I'll probably later add a variable to config/Makefile (CROSS_COMPILER=true/false) to handle this.

Can you please check this new version (camlrun-fixed-5.diff)?

@dbuenzli thanks for the advice but I'd already extended the searchpath script...

@vicuna

This comment has been minimized.

Copy link
Author

commented May 9, 2015

Comment author: @whitequark

The new patch does not apply at all to the 4.02 branch.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 11, 2015

Comment author: @damiendoligez

I'm sorry, the branch moved under my feet. Here's one that applies cleanly.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 11, 2015

Comment author: @mshinwell

whitequark: does the new patch work?

@vicuna

This comment has been minimized.

Copy link
Author

commented May 11, 2015

Comment author: @whitequark

No, it does not apply.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 11, 2015

Comment author: @damiendoligez

I think I know what's going on. 4.02 is changing, and you're getting the version from the mirror, which is up to 4 hours behind the version I get.

I'll make a GitHub PR, it'll be easier to work with that.

@vicuna

This comment has been minimized.

Copy link
Author

commented May 11, 2015

Comment author: @damiendoligez

Let's continue this discussion (and debugging) at #183 .

@vicuna

This comment has been minimized.

Copy link
Author

commented May 12, 2015

Comment author: @damiendoligez

applied to branch 4.02 (rev 16114)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.