Skip to content

Compiler Setup

Xiaoguang Wang edited this page Jul 18, 2018 · 4 revisions

Overview of Compiler

The Popcorn compiler is built on top of clang+LLVM and builds binaries suitable for migration between heterogeneous-ISA systems. It requires a modified version of clang & LLVM and several supporting tools/libraries for building these binaries. These will be installed and used by a Makefile included with the compiler to generate multi-ISA binaries. For more information, see the READMEs in the repository (and sub-folders), which describe how the compiler, libraries and tools operate. We assume in this document the Popcorn cluster is comprised of two nodes; node 0 is running x86 at, and node 1 is running ARM at

Install the Compiler

  • Firstly, we'll install several prerequisites:
$ sudo apt-get install build-essential flex bison subversion cmake zip
  • Should have gcc installed for both x86-64 and ARM64. If you have followed from installing VM, the compiler suites will be already installed on the host. Otherwise, refer to Internet documetns for installing the crossbuild suite.

  • Should install g++ compiler on Ubuntu

$ sudo apt-get install x86_64-linux-gnu-g++
  • Download following files and place them in the same directory on the host.
Item Filename URL
Compiler toolchain
Example - NPB
  • Untar the compiler repository and look at the contents
$ tar -xzf popcorn-compiler.tar.gz
$ cd popcorn-compiler
$ ls
common  INSTALL  ISSUES  lib  patches  README  tool  util
  • Install the Popcorn compiler toolchain. /usr/local/popcorn is the default path where the toolchain is installed. Change <MYID> with your login ID:
$ sudo mkdir -p /usr/local/popcorn
$ sudo chown <MYID> /usr/local/popcorn
$ ./ --install-all --threads 8
  • Or install into a directory of your choosing (denoted as <POPCORN PATH>)
$ ./ --install-path <POPCORN PATH> --install-all --threads 8
  • The compiler, including supporting libraries and tools, is now installed at <POPCORN PATH>. You can use the the provided Makefile in "popcorn-compiler/util/Makefile.pyalign.template" to build applications. The makefile template expects that the entire application's source is contained in a single directory.

Example: NPB

We provide a port of NPB benchmark suite modified to run core computation on a remote node. The modified benchmark is comprised of 9 programs (bt, cg, ep, ft, is, lu , mg, sp, and ua) and three workloads (S, A, and B).

  • Untar the source code and have a look at the contents:
$ tar -xzf npb-popcorn.tar.gz
$ cd npb-popcorn
$ ls
bt  dc  ft  lu        mg              ua
cg  ep  is  Makefile  migration_points.patch  sp
  • Set the workload to run using make make {S | A | B}
$ make A    # Set the benchmark suite to run using workload A
  • Or {S | A | B}
$ ./ S  # Set to use workload S
  • Build the benchmark suite
$ make
  • You will find <PROGRAM>_aarch64 and <PROGRAM>_x86-64 on each <PROGRAM> directory. To run the programs, place them in the Popcorn nodes. Taking "ep" as an example:
$ cd ep
$ scp ep_aarch64 ep_x86-64 popcorn@
$ scp ep_aarch64 ep_x86-64 popcorn@
  • Set executables for each ISA.
popcorn@x86~$ cp ep_x86-64 ep
popcorn@arm~$ cp ep_aarch64 ep
  • DO NOT use soft-link nor hard-link for setting up the executable. Popcorn is so fragile for now that doing that may disallow the thread migration.

  • Now, you can run the program on node 0:

popcorn@x86:~$ ./ep
  • By default, kernels print out many debug messages to ensure the program execution and to help debugging. You can turn off them by unsetting CONFIG_POPCORIN_DEBUG_PROCESS_SERVER, CONFIG_POPCORN_DEBUG_PAGE_SERVER, and CONFIG_POPCORN_DEBUG_VMA_SERVER from the kernel configuration (e.g., .config) or Popcorn Distributed Execution Support/Log debug messages for Popcorn from the menuconfig.

Initiate thread migration

Programmers can instruct threads to be relocated to a specified node by putting migration points. In the NPB example above, such migration points can be found from migration_points.patch file.

The migration can be triggered by calling the following function:

#include "migrate.h"

void migrate(int nid, void (*callback)(void *), void *callback_data);

callback is invoked with callback_data as the argument after the thread is migrated. You can pass NULL if you are not interested in the callback. The execution is resumed at the next line of the migrate function on node nid.


  • Bring back migrated-away threads to their original node before exiting from the program.

  • A migrated thread should be brought back to the original node to be relocated to other node. For example, to migrate a thread which was created at node 0 and is running at node 1, the thread should be migrated back to node 0 before it is migrated to node 2 again.

  • You can do file I/O at any location, but file descriptors are not synchronized between migrated threads. This means, console is not migrated and printf() might not work neither. To check the execution progress, we recommend to open() a file at an absolute path, write() to the file, and close() for each print-out. File operations (e.g., fopen(), fwrite()) might not work neither.

  • You cannot create a thread while the thread is running at a remote node.

  • Networking is not supported.

Build your own applications

The compiler toolchain provides a template Makefile for building the multi-ISA binary. The template file can be found at popcorn-compiler/util/Makefile.pyalign.template. Copy the file to your project directory and edit accordingly.

  • In most cases, it is sufficient to put source files into SRC and set the executable name to BIN. For example:
BIN := test

SRC := hello.c world.c
  • Build multi-ISA binaries
$ make
  • You will find $(BIN)_aarch64 and $(BIN)_x86-64 in the directory, which are the executables.
$ ls
... test_aarch64   test_x86-64  ...
  • The executable should be at the same absolute path arcoss the machines.
$ scp test_x86-64 test_aarch64 popcorn@
$ scp test_x86-64 test_aarch64 popcorn@
  • Make test point to the proper-ISA executable on each machine:
popcorn@x86:~$ cp test_x86-64  test
popcorn@arm:~$ cp test_aarch64 test
  • Now, you can run the executable with ${BIN} which is test in this example.
You can’t perform that action at this time.