INSTALL.cross

                      MPE (Multi-Processing Environment)
                      ----------------------------------

                          Version 2.5.0. Oct, 2009

                    Mathematics and Computer Science Division
                           Argonne National Laboratory

II. CONFIGURATION
-----------------

Please read the CONFIGURATION section II. a) and II. b) in INSTALL to get
yourself familar with the Configuration Model of MPE.

III. INSTALLATION INSTRUCTIONS
------------------------------

This file describes the Cross-Compilation build instructions for MPE.
If you don't need cross-compilation support or you don't know what
cross-compilation is, you probably don't need to read this.  INSTALL
is all you need to know.

**** The following is NO longer true in this MPE release ****
This release of MPE has better support of cross-compilation and does
not require a 2-step process, i.e. no cross-spec file is needed.
Cross compilation is similar to native compilation except the addition
of --host and --build as required by autoconf(i.e. configure).
In short, cross-compile requires specification of CC/F77/MPI_CC/MPI_F77,
--host and --build as part of configure command.
More documentation will be added later.
**************************************************************

For those who need cross-compilation support, 7 to 10 variables are
needed.  These variables are determined through some runtime tests on
cross-compiled host (i.e. the host where the libraries/executable are
being built for).  The exact number of variables depends on if the
MPE's internal Fortran to C MPI profiling wrapper library needs to be
built.  7 of the 10 environmental variables are related to the C compiler
of the cross-compiled platform.

CROSS_SIZEOF_CHAR      : sizeof(char), e.g. 1
CROSS_SIZEOF_SHORT     : sizeof(short), e.g. 2
CROSS_SIZEOF_INT       : sizeof(int), e.g. 4
CROSS_SIZEOF_LONG      : sizeof(long), e.g. 8
CROSS_SIZEOF_LONG_LONG : sizeof(long long), e.g. 8
CROSS_SIZEOF_VOID_P    : sizeof(void *), e.g. 8
CROSS_BIGENDIAN        : true if it is a big endian machine, false otherwise.

If your MPI implementation does NOT provide any Fortran to C MPI profiling
wrapper library, then you need to specify 3 more environmental variables.
The following environmental variables specify properties of MPI
implementation if F2CMPI_LIBS is defined, i.e MPE's internal Fortran
to C MPI profiling library is used.  These variables are related to
how Fortran primitive type looks like in C.

CROSS_MPI_STATUS_SIZE  : value of MPI_STATUS_SIZE defined in mpif.h, i.e.
                         sizeof(MPI_STATUS_SIZE)/sizeof(MPI_Fint), e.g. 4
CROSS_FORTRAN2C_TRUE   : value of fortran's .TRUE. in C program, e.g. 1
CROSS_FORTRAN2C_FALSE  : value of fortran's .FALSE. in C program, e.g. 0



III. a) Cross-Compile Specification
-----------------------------------

In order to simplify and avoid any ambiguity in the determination of these
values, a small subpackage called "cross_spec" within MPE can be used.
This goal of this subpackage is to generate a cross-compile specification
file which will contain all the relevant CROSS_xxx variables listed above.
The subapackage has its own independent configure and accepts a similar set
of configure variables and options as MPE's configure.  If you are not
familar with MPE's configure variables and options, please see INSTALL file's
section II. a) and II. b).  Or you can do "mpe2/cross_spec/configure --help"
for a list of the cross_spec's options and environment variables.
The default cross-compile specification file generated by cross_spec
is called cross_spec.txt which can be overriden by option --with-spec.

Let say we are cross-compiling MPE for x86_64 (64bit linux) backend nodes
on a 32bit linux frontend box which has the following cross-compilers:

    the serial C cross-compiler :        /usr/local/bin/lans64_cc,
    the serial Fortran cross-compiler :  /usr/local/bin/lans64_fc,
    the MPI C cross-compiler :           /usr/local/bin/lans64_mpicc,
    the MPI Fortran cross-compiler :     /usr/local/bin/lans64_mpif90

Usually the frontend will have a command to run executable in the backend,
let say this is called "/usr/local/bin/lans64_run -np 1", where "-np 1"
means the executable will be run on 1 node.

Now pick a build directory <build_spec_dir> to build cross_spec, be sure
not set <build_spec_dir> as <mpe2_src_dir>.  Doing so will cause conflict
between cross_spec's Makefile and mpe2's main Makefile.

One can configure cross_spec as follows:

    mkdir <build_spec_dir>
    cd <build_spec_dir>
    <mpe2_src_dir>/cross_spec/configure --host=x86_64-linux                   \
                                        AR=ar RANLIB=ranlib                   \
                                        CC=/usr/local/bin/lans64_cc           \
                                        F77=/usr/local/bin/lans64_fc          \
                                        MPI_CC=/usr/local/bin/lans64_mpicc    \
                                        MPI_F77=/usr/local/bin/lans64_mpif90  \
                                        MPERUN="/usr/local/bin/lans64_run -np 1"
    make
    make run

Where --host=x86_64_linux informs configure we are building for 64bit linux.
Notice there isn't a --prefix option in the configure command, as it isn't
necessary to install this subpackage anywhere.  Here AR and RANLIB are used
in the configure command to highlight these variables can be added for
platform which has specific AR and RANLIB for the backend nodes.  The MPERUN
allows users to run the tests on the backend through Makefile, i.e. through
the "make run" commad.  Without MPERUN or there isn't a command to launch
executable to run on the backend, the "make run" will execute the tests
locally on the frontend.  In that case, one has to launch the tests separately
by hand.

There are 2 executables and 1 script created by cross_spec.  They are
cross_cc, cross_f2c and cross_env.

    cross_cc:  an executable that specifies various C compiler features
               in the backend.  (Has to run on the backend)
    cross_f2c: an executable that specifies various Fortran to C features
               in the backend. (Has to run on the backend)
    cross_env: a script that specifies the configure options used in
               cross_spec and these options will be used in cross-compiling
               MPE again.  (run on the frontend)

If running of the cross_cc, cross_f2c and cross_env returns successfully,
a cross_spec.txt file will be generated, an example of file is like

# C compiler runtime characteristics...
CROSS_SIZEOF_CHAR=1
CROSS_SIZEOF_SHORT=2
CROSS_SIZEOF_INT=4
CROSS_SIZEOF_LONG=8
CROSS_SIZEOF_LONG_LONG=8
CROSS_SIZEOF_VOID_P=8
CROSS_BIGENDIAN=false
# Fortran to C runtime characteristics...
CROSS_MPI_STATUS_SIZE=5
CROSS_FORTRAN2C_TRUE=1
CROSS_FORTRAN2C_FALSE=0
# C and Fortran development tool variables...
MPI_CC=/usr/local/bin/lans64_mpicc
CC=/usr/local/bin/lans64_cc
MPI_F77=/usr/local/bin/lans64_mpif90
F77=/usr/local/bin/lans64_fc
AR=ar
RANLIB=ranlib

Any other configure variables, e.g. CFLAGS, for MPE configure
can be appended in the cross_spec.txt, as they will be proprogangated
down to other subpackages within MPE configure.



III. b) Cross-Compilation With "cross_spec.txt"
-----------------------------------------------

With cross_spec.txt generated by cross_spec subpackage, cross_compile
MPE becomes easy.  One can do it as follows:

    mkdir <build_mpe2_for_backend>
    cd <build_mpe2_for_backend>
    <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                       \
                             --host=x86_64_linux                               \
                             --with-cross-spec=<build_spec_dir>/cross_spec.txt \
                             MPERUN="/usr/local/bin/lans64_run -np 4"
    make
    make install
    make installcheck-all

Here AR and RANLIB are added for illustration purposes.

Where --host is used to inform configure what type of host to be built for
the backend.  Again, MPERUN is optional here.  With MPERUN, the 
"make installcheck-all" command will launch runtime tests to the backend
to check if every selected components are built correctly.  Notice that
the MPERUN command here is different from the one used in cross_spec, since
we want to run multiple processes job in the "installcheck-all".


    instructions for BlueGeneLight (BG/L)
    -------------------------------------

    **** Cross-Spec build ****
    mkdir <build_spec_dir>
    cd <build_spec_dir>
    <mpe2_src_dir>/cross_spec/configure  CC=blrts_xlc F77=blrts_xlf         \
                                         MPI_CC=mpixlc MPI_F77=mpixlf77     \
                                         --host=powerpc-unknown-linux-gnu   \
                                         --build=powerpc-pc-linux-gnu
    make
    cqsub -n 1 -t 5 cross_cc
    cqsub -n 1 -t 5 cross_f2c
    cross_env

    Here --host is set to powerpc-unknown-linux-gnu even BG/L is not linux
    Since we don't know what is the correct host-type for BG/L, --host
    is arbitary set to  powerpc-unknown-linux-gnu and should be different
    from --build value so the configure knows we are doing cross-compilation.
    Also make sure that "cqsub ... cross_cc" is finished __before__
    "cqsub ... cross_f2c", otherwise the cross_spec.txt may have overlapped
    output from both cross_cc and cross_f2c at the same time.

    **** MPE2 build ****
    mkdir <build_mpe2_for_backend>
    cd <build_mpe2_for_backend>
    <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                       \
                             --host=powerpc-unknown-linux-gnu                  \
                             --build=powerpc-pc-linux-gnu                      \
                             --with-cross-spec=<build_spec_dir>/cross_spec.txt \
                             --with-java=/usr/lib/IBMJava2/jre                 \
                             --disable-misc --disable-graphics                 \
                             --disable-sample --disable-rlog
    make
    make install
    make installcheck

    instructions for BlueGene P (BG/P)
    -------------------------------------

    **** Cross-Spec build ****
    mkdir <build_spec_dir>
    cd <build_spec_dir>
    <mpe2_src_dir>/cross_spec/configure  CC=bgxlc_r F77=bgxlf_r             \
                                         MPI_CC=mpixlc_r MPI_F77=mpixlf77_r \
                                         --host=powerpc-unknown-linux-gnu   \
                                         --build=powerpc-pc-linux-gnu

    ** the "_r" is used to enable thread support in the compiler **
    ** mpixlc_r/mpixlf_r are in /bgsys/drivers/ppcfloor/comm/bin **

    make
    cqsub -n 1 -t 5 cross_cc
    cqsub -n 1 -t 5 cross_f2c
    cross_env

    Here --host is set to powerpc-unknown-linux-gnu even BG/P is not linux
    Since we don't know what is the correct host-type for BG/P, --host
    is arbitary set to  powerpc-unknown-linux-gnu and should be different
    from --build value so the configure knows we are doing cross-compilation.
    Also make sure that "cqsub ... cross_cc" is finished __before__
    "cqsub ... cross_f2c", otherwise the cross_spec.txt may have overlapped
    output from both cross_cc and cross_f2c at the same time.

    **** MPE2 build ****
    mkdir <build_mpe2_for_backend>
    cd <build_mpe2_for_backend>
    <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                      \ 
                             --host=powerpc-unknown-linux-gnu                 \
                             --build=powerpc-pc-linux-gnu                     \
                             --with-cross-spec=<build_spec_dir>/cross_spec.txt\
                             --disable-safePMPI                               \
                             --disable-misc --disable-graphics                \
                             --disable-sample --disable-rlog
    make
    make install
    make installcheck


    instructions for SiCortex (SC5812)
    -------------------------------------

    On the i686 linux cross-compiling frontend (bblogin)

    **** Cross-Spec build ****
    mkdir <build_spec_dir>
    cd <build_spec_dir>
    <mpe2_src_dir>/cross_spec/configure                                      \ 
                              CC="scpathcc -n32"                             \
                              F77="scpathf95 -fno-second-underscore -n32"    \
                              MPI_CC="scmpicc --pathscale -n32"              \
                              MPI_F77="scmpif90 -fno-second-underscore -n32" \
                              LIBS=-lpthread                                 \
                              --host=mips64el-gentoo-linux-gnu               \
                              --build=x86_64_linux        
    make

    The above build is for n32 binary.  For 64-bit binary, replace -n32 by -64.
    Now log onto sicortex's login node (fd-login)
    to launch programs with srun to get the cross_spec.txt

    srun -n 1 cross_cc
    srun -n 1 cross_f2c
    srun -n 1 cross_env

    **** MPE2 build ****
    mkdir <build_mpe2_for_backend>
    cd <build_mpe2_for_backend>
    <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                      \
                             --host=mips64el-gentoo-linux-gnu                 \
                             --build=x86-linux-gnu_linux                      \
                             --with-cross-spec=<build_spec_dir>/cross_spec.txt\
                             --disable-safePMPI
    make
    make install
    make installcheck


    instructions for Cray X1E (Phoenix)
    ----------------------------------

    On the i686 linux cross-compiling frontend (robin)

    **** Cross-Spec build ****
    mkdir <build_spec_dir>
    cd <build_spec_dir>
    <mpe2_src_dir>/cross_spec/configure CC=cc F77="ftn -s real64 -dp"         \
                                        MPI_CC=cc MPI_F77="ftn -s real64 -dp" \
                                        --host=craynv-cray-unicosmp3.0.X      \
                                        --build=i686-pc-linux-gnu
    make

    # Now create a cross_tests.cmd as follows:
    cat > cross_tests.cmd <<EOF
#!/bin/sh
#PBS -l walltime=5:00,mppe=1
#PBS -j oe
#PBS -m abe
#PBS -A xxxxx

# job starts in home directory, cd to the submission directory
cd ${PBS_O_WORKDIR}

echo "running cross_cc"
aprun -n 1 ./cross_cc

echo "running cross_f2c"
aprun -n 1 ./cross_f2c

EOF

    qsub cross_tests.cmd
    # After the job is done, then run cross_env on the front-end
    ./cross_env


    Now make sure that java (SUN or IBM's jdk) is in your path.

    **** MPE2 build ****
    mkdir <build_mpe2_for_backend>
    cd <build_mpe2_for_backend>
    <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                       \
                             --host=craynv-cray-unicosmp3.0.X                  \
                             --build=i686-pc-linux-gnu                         \
                             --with-cross-spec=<build_spec_dir>/cross_spec.txt \
                             --disable-misc --disable-graphics                 \
                             --disable-threadlogging

    --disable-threadlogging disables the pthread logging support since Cray
    backend does not seem to support pthread calls.

    make
    make install
    make installcheck


    instructions for Cray XT4 (jaguar)
    ----------------------------------

    Compiling for the catamount(no thread support) on the frontend

    **** Cross-Spec build ****
    mkdir <build_spec_dir>
    cd <build_spec_dir>
    <mpe2_src_dir>/cross_spec/configure CC="cc --target=catamount"            \
                                        F77="ftn --target=catamount"          \
                                        MPI_CC="cc --target=catamount"        \
                                        MPI_F77="ftn --target=catamount"      \
                                        --host=x86_64-catamount-linux         \
                                        --build=x86_64-linux
    make

# allocate an interactive PBS job slot to run cross_cc and cross_f2c
    qsub -A XXXXX -V -I -lwalltime=00:15:00,size=2

# cd to where cross_cc and cross_f2c re
    cd <mpe2_src_dir>/cross_spec

# run cross_cc and cross_f2c
    yod -sz 1 cross_cc
    yod -sz 1 cross_f2c

# Now make sure that java (SUN or IBM's jdk) is in your path if it is available

    **** MPE2 build ****
    mkdir <build_mpe2_for_backend>
    cd <build_mpe2_for_backend>
    <mpe2_src_dir>/configure --prefix=<mpe2_install_dir>                       \
                             --host=x86_64-catamount-linux                     \
                             --build=x86_64-linux                              \
                             --with-cross-spec=<build_spec_dir>/cross_spec.txt \
                             CC="cc --target=catamount"                        \
                             F77="ftn --target=catamount"                      \
                             MPI_CC="cc --target=catamount"                    \
                             MPI_F77="ftn --target=catamount"
    make
    make install
                      




III. c) Cross-Compilation Without "cross_spec.txt"
--------------------------------------------------

The section contains some examples of cross-compiling MPE on older platforms:

For cross-compilation with an existing version of MPICH(or LAM). e.g. ASCI-Red.

*** If the Fortran to C MPI profiling wrapper library exists (it is called
    libfmpich.a in MPICH and liblamf77mpi.a in LAM), do

    setenv CC cicc
    setenv F77 cif77
    setenv CROSS_SIZEOF_CHAR 1
    setenv CROSS_SIZEOF_SHORT 2
    setenv CROSS_SIZEOF_INT 4
    setenv CROSS_SIZEOF_LONG_LONG 8
    setenv CROSS_BIGENDIAN false
    ${MPE_SRC_DIR}/configure --with-mpicc=${MPICH_INSTALL_DIR}/bin/mpicc \
                             --with-mpif77=${MPICH_INSTALL_DIR}/bin/mpif77 \
                             --with-f2cmpilibs=-lfmpich \
                             --disable-graphics
    make
    make install PREFIX=${MPE_INSTALL_DIR}

Here assume mpicc/mpif77 scripts exist.  If not, mpicc should include lines:

#! /bin/sh
cicc -I${MPICH_INSTALL_DIR}/include -L${MPICH_INSTALL_DIR}/lib $@ <-lmpi>


mpif77 can be composed by including the following line

#! /bin/sh
cicc -I${MPICH_INSTALL_DIR}/include -L${MPICH_INSTALL_DIR}/lib $@ <-lmpi>


<-lmpi> refers to various MPI libraries included in your MPI implementation.


*** If the Fortran to C MPI profiling wrapper library does NOT exist,
    then the MPE's own Fortran to C MPI profiling wrapper library needs
    to be built.  3 additional environment variables are needed.  They
    are CROSS_MPI_STATUS_SIZE, CROSS_FORTRAN2C_TRUE & CROSS_FORTRAN2C_FALSE

    setenv CC cicc
    setenv F77 cif77
    setenv CROSS_SIZEOF_CHAR 1
    setenv CROSS_SIZEOF_SHORT 2
    setenv CROSS_SIZEOF_INT 4
    setenv CROSS_SIZEOF_LONG_LONG 8
    setenv CROSS_BIGENDIAN false
    setenv CROSS_MPI_STATUS_SIZE 4
    setenv CROSS_FORTRAN2C_TRUE  1
    setenv CROSS_FORTRAN2C_FALSE 0
    ${MPE_SRC_DIR}/configure --with-mpicc=${MPICH_INSTALL_DIR}/bin/mpicc \
                             --with-mpif77=${MPICH_INSTALL_DIR}/bin/mpif77 \
                             --disable-graphics
    make
    make install PREFIX=${MPE_INSTALL_DIR}


For Cray X1E MPI implementation: cross-compilation at a linux front-end(robin)
(e.g. phoenix at Oak Ridge)

    ${MPE_SRC_DIR}/configure                                              \
    MPI_F77="ftn -s real64 -dp"                                           \
    F77="ftn -s real64 -dp"                                               \
    MPI_CC=cc                                                             \
    CC=$HOME/phoenix/bin/gcc-robin                                        \
    --host=craynv-cray-unicosmp3.0.X                                      \
    --build=i686-pc-linux-gnu                                             \
    CROSS_SIZEOF_CHAR=1                                                   \
    CROSS_SIZEOF_SHORT=2                                                  \
    CROSS_SIZEOF_INT=4                                                    \
    CROSS_SIZEOF_LONG_LONG=8                                              \
    CROSS_BIGENDIAN=true                                                  \
    CROSS_MPI_STATUS_SIZE=6                                               \
    CROSS_FORTRAN2C_TRUE=1                                                \
    CROSS_FORTRAN2C_FALSE=0                                               \
    --disable-misc --disable-graphics --disable-collchk                   \
    --disable-sample --disable-rlog