/
README.txt
158 lines (128 loc) · 6.24 KB
/
README.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
***********************************************************************************************
# What is collfs?
collfs is a user library and set of patches to glibc that enables file system calls to be handled collectively over an
MPI Communicator. collfs was written with the specific goal of providing scalable dynamic loading on supercomputers
with faster interconnect than i/o performance, but is a general-purpose tool that can be used either directly as a
library or implicitly by wrapping file system calls with collective versions.
The collfs patches are currently quite minimal. They insert an externally visible struct object into the run-time
dynamic linker containing void function hooks for collective MPI versions of the standard file system API. These function pointers
are then used (when non-NULL) to replace file system function calls within the standard dynamic loading routines in glibc. Additionally,
we provide a small static C library via LD_PRELOAD that initializes MPI (this is pre-main, so it passes NULL to the Init
routines) and activates the function hooks to point to our collective versions of the standard file system API,
shadowing the original file system functions (but utilizing them by default).
***********************************************************************************************
# Installing collfs on a Fedora Core 5 system
Fedora Core 5 is at this point, quite old, but it features a 2.4 version of glibc, which is very close to the 2.4 glibc
installed on the IBM Blue Gene/P.
This assumes you have an FC5 installation with the standard development RPMs installed. If you are unable to build
glibc-2.4 from the SRPM, you will probably need to install the missing RPMs from the DVD or a web repository.
Tips for setting up a non-root installation environment for SRPMs can be found here:
http://www.owlriver.com/tips/non-root//
mkdir -p ~/sandbox/glibc
cd ~/sandbox/glibc
wget http://dl.dropbox.com/u/65439/glibc-2.4-11-collfs.src.rpm
rpmbuild --nodeps --rebuild -bc glibc-2.4-11-collfs.src.rpm
This should leave you with a built glibc (good for testing) in the RPM build directory, which
was/var/tmp/glibc-2.4-root on my machine.
***********************************************************************************************
# Building glibc-2.4 for the compute nodes:
tmpdir=/tmp/aron-glibc-bgp-build
prefix=/home/aron/bgpsys
build=powerpc-linux-gnu
# download and patch glibc sources
cd $tmpdir
curl -O ftp://ftp.gnu.org/gnu/glibc/glibc-2.4.tar.gz
tar -zxvf glibc-2.4.tar.gz
# apply IBM patches to glibc
patch -p2 -E < /bgsys/drivers/ppcfloor/toolchain/glibc-2.4.diff
# temporary linux headers
mkdir -p $tmpdir/templinuxheaders-build/include && \
cp -f -r -L /usr/include/asm \
$tmpdir/templinuxheaders-build/include/asm && \
cp -f -r -L /usr/include/asm-ppc \
$tmpdir/templinuxheaders-build/include/asm-ppc && \
cp -f -r -L /usr/include/asm-generic \
$tmpdir/templinuxheaders-build/include/asm-generic && \
cp -f -r -L /usr/include/linux \
$tmpdir/templinuxheaders-build/include/linux
# configure using default-shared gcc specs
mkdir -p $tmpdir/glibc-2.4-build
cd $tmpdir/glibc-2.4-build && \
PATH=/usr/gnu/bin:/usr/bin:/bin:/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/bin/ LD_LIBRARY_PATH= \
CC="powerpc-bgp-linux-gcc -specs=/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/lib/gcc/powerpc-bgp-linux/4.1.2/specs.orig" \
libc_cv_ppc_machine=yes \
dd1_workarounds= \
libc_cv_forced_unwind=yes \
libc_cv_c_cleanup=yes \
libc_cv_slibdir="/lib" \
$tmpdir/glibc-2.4/configure \
--prefix=$prefix \
--sysconfdir="/etc" \
--libdir="/lib" \
--bindir="/bin" \
--datadir="/usr/share" \
--libexecdir="/libexec" \
--build=$build \
--host=powerpc-bgp-linux \
--enable-shared \
--enable-multilib \
--without-cvs \
--without-gd \
--with-elf \
--with-fp=yes \
--enable-add-ons=powerpc-cpu,nptl \
--with-cpu=450fp2 \
--with-tls \
--with-__thread \
--enable-__cxa_atexit \
--with-headers=$tmpdir/templinuxheaders-build/include
# patch makeconfig
# make
cd $tmpdir/glibc-2.4-build && PATH=/usr/gnu/bin:/usr/bin:/bin:/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/bin/ LD_LIBRARY_PATH= \
make 2>&1 | tee make.log
# make install
cd $tmpdir/glibc-2.4-build && PATH=/usr/gnu/bin:/usr/bin:/bin:/bgsys/drivers/V1R4M2_200_2010-100508P/ppc/gnu-linux/bin/ LD_LIBRARY_PATH= \
make install 2>&1 | tee install.log
***********************************************************************************************
See shaheen/build_glibc_2.4-sles-10.sh for the basic idea of how to patch the SLES 10
glibc with collfs. I am working from the patched source that you can obtain from the SLES
10 src.rpm by unpacking the distribution then applying all the patches (just before the
configure stage).
If you want to run an executable with the updated dynamic loader interpreter, you need to
launch your executable using the installed ld64.so.1 like this:
/home/aron/sys/lib/ld64.so.1 ./main
***********************************************************************************************
some random notes for building glibc Shaheen:
../configure --prefix=/home/aron/sys --libexecdir=/home/aron/sys/lib64 --infodir=/home/aron/sys/share/info --enable-add-ons=nptl,libidn,dceext --srcdir=.. --without-cvs --with-headers=/home/aron/src/rpm/BUILD/kernel-headers --build ppc64-suse-linux --host ppc64-suse-linux --with-tls --with-__thread --enable-kernel=2.6.4 2>&1 CC="gcc -m64" | tee configure.log
make 2>&1 | tee make.log
# make install wants a local ld.so.conf, so copy the one over from /etc or build it by hand
cp ~/sandbox/collfs.git/shaheen/ld.so.conf ~/sys/etc/ld.so.conf
make install
***********************************************************************************************
anticipated changes to:
dl-load.c
793: __close
837: __fxstat64
858: __close
887: __close
912: __close
975: __seek
976: __libc_read
1185: __mmap
1236: __mmap
1350: __munmap
1418: __close
1617: __open
1629: __libc_read
1727: __lseek
1728: __libc_read
1745: __lseek
1746: __libc_read
1761: __close
1872: __fxstat64
1878: __close
1899: __close
2118: __close
dl-close.c
498: DL_UNMAP
***********************************************************************************************