-
Notifications
You must be signed in to change notification settings - Fork 5
/
README
254 lines (195 loc) · 12 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
# ASKAP SDP HAS Moved!
The **ASKAP SDP project**, which is astronomy data reduction software **ASKAPsoft** and other things, is now being hosted at the CSIRO Bitbucket repository which is publicly accessible. The version of ASKAPsoft in this github repo is old and not supported in any way. Go to the CSIRO Bitbucket to obtain the latest version. Information and links (including Digital Object Identifiers (DOI) for referencing) to various versions can be found if you [search the CSIRO Data Access Portal (DAP)](https://data.csiro.au/dap/search?q=ASKAPsoft). The DAP will also have contact details for queries and issues.
We have moved from github to that system on CSIRO recommendation. This github repo was setup before CSIRO obtained their Bitbucket services.
---------
ASKAPsoft Science Data Processor
================================
These notes provide just a brief outline for building the ASKAPsoft Science Data
Processor codebase. More detailed instructions exist on the ASKAP Computing
Wiki.
Software Requirements
=====================
ASKAPsoft is designed to be built and executed in a standard Unix/Linux
environment. The current official supported environments are:
* Debian 7.x (Wheezy) Linux
* Mac OSX 10.9.x (Mavericks)
* Cray Linux Environment (XC30 platform)
Core dependencies must be fulfilled by the platform. These include, but are not
limited to, a C/C++/Fortran compiler, Make, Python 2.7, Java 7 and MPI. More
specific dependencies are downloaded by the ASKAPsoft build system and are
installed within the ASKAPsoft development tree.
Specific to the Debian platform, after a standard installation of Debian Wheezy
(7.x) the following packages will need to be installed with apt-get:
* g++
* gfortran
* openjdk-7-jdk
* python-dev
* flex
* bison
* openmpi-bin
* libopenmpi-dev
* libfreetype6-dev
* libpng12-dev
Checkout & Bootstrap
====================
The first step of building ASKAPsoft involved checking out a copy of the code
from the Subversion repository and then running the bootstrap script.
Note that during the bootstrap and build process a number of tarballs containing
third party software will be fetched. If Direct HTTP access is not available on
your build host then please refer the the section below titled "Archive
Repository".
The boostrap script build and configures the developer tools necessary for
building the rest of the ASKAPsoft codebase:
svn co https://svn.atnf.csiro.au/askapsdp/trunk ASKAPsoft
cd ASKAPsoft
/usr/bin/python2.7 bootstrap.py -n
The "-n" option to bootstrap says not to use "svn update" to update the
workspace prior to doing the bootstrap.
Building
========
Before building ASKAPsoft, certain environment variable need to be set. This is
accomplished by sourcing the initaskap.sh script, which will exist in the top
level directory after the bootstrap procedure described above has been
completed. Then the recursive builder (rbuild) can be used to build the entire
ASKAPsoft tree:
cd ASKAPsoft
. initaskap.sh
rbuild -S -a
The "-S" option to rbuild says not to do an "svn update" prior to building.
To build a release tarball for a specific package, or set of packages the
release target option must be set. For example to build the core data processing
applications (assuming initaskap.sh has already been sourced):
cd $ASKAP_ROOT
cd Code/Systems/cpapps
rbuild -S -t release
This will create a filenamed release-<svnrev>.tgz
Archive Repository
==================
ASKAPsoft depends on a number of 3rd party packages. Some of these must be
supplied by the operating platform. For example a compiler such as GCC and an
MPI implementation must be present.
Other software is downloaded during the ASKAPsoft build process, and as such
HTTP access is required. If your build host relies on a HTTP proxy to access the
internet then the http_proxy & https_proxy environment variables must be set in
your environment. Here is an example, where "webproxy.myorg.com" is the hostname
for the proxy server that is running on port 8080
export http_proxy=http://webproxy.myorg.com:8080
export https_proxy=$http_proxy
Alternatively, if access to download these files is not available at all on the
build host a local repository can be set up. In this case the bootstrap.ini must
be configured (prior to executing the bootstrap below) to refer to the local
cache via a "file" URL. For example:
remote_archive = file:///home/joe/askapsoft_dependencies
Documentation
=============
Included within the ASKAPsoft tree is both API documentation and user
documentation. The user documentation can be built like so:
cd $ASKAP_ROOT/Code/Doc/user/current
rbuild -n -t doc
The HTML documentation tree will be created at the following location:
$ASKAP_ROOT/Code/Doc/user/current/doc/_build/html
Rather than a single unified API documentation tree, each package hosts its own
documentation. A package can generally be identified by the fact it has a
build.py in its package root directory. The following example shows how to build
the "Synthesis" package API documentation:
cd $ASKAP_ROOT/Code/Components/Synthesis/synthesis/current
rbuild -n -t doc
The HTML documentation tree will be created at the following location:
$ASKAP_ROOT/Code/Components/Synthesis/synthesis/current/html
Release Build
=============
While the ASKAPsoft applications can be executed directly out of a development
tree, a deployment consisting of the binaries and libraries is support. A
release tarball can be created for each package separately if desired, however
super-packages are available as a convenient grouping. For example the "cpapps"
package provides the calibration, imaging & source-finding software. It can be
built like so:
cd $ASKAP_ROOT/Code/Systems/cpapps
rbuild -n -t release
This will create a file release-1234.tgz (where 1234 is the Subversion release
number) which contains bin & lib directories. This tarball can be extracted in a
convenient location, such as /opt or /usr/local, and PATH/LD_LIBRARY_PATH
environment variables can then be set to include the bin/lib directories
respectively.
In the typical HPC environment, the ASKAPsoft "cpapps" package would be deployed
as an [Environment Module].
[Environment Module]:http://modules.sourceforge.net/
rbuild
======
The 'rbuild' command is the main build command for developers. It has the
ability of updating from the subversion repository and also recursively resolve,
update and build dependencies.
To get help for 'rbuild' simply type
------------------------------------------------------------------------------
%% rbuild -h
Usage: rbuild [options] [<package_path>]
This is the main ASKAPsoft build command for developers. It can handle
dependencies, subversion updates and changes to the build system. There are
two types of build targets: recursive [depends, install, stage, release,
signature] and non-recursive [bclean, clean, doc, format, pylint, functest,
test, deploy]. The non-recursive targets only apply to the current package,
while recursive targets are applies to all dependencies of the current
package. The recursive behaviour may be overridden with appropriate flag. The
default <package_path> is the current directory.
Options:
-h, --help show this help message and exit
-a, --autobuild mode where dependencies/packages are computed once.
Additionally turns on no-update and no-recursion flags
i.e. -n -R
-f, --force force building of packages ignoring NO_BUILD files
-q, --quiet do not show all builder output [default=True]
-v, --verbose show all builder output
-n, --no-update no svn updates, rebuild of myself or Tools rebuild.
Equivalent to "-S -M -T"
-N, --no-recursive-update
no svn updates, rebuild of myself, Tools rebuild or
recursion. Equivalent to "-S -M -T -R -v" or
"python build.py TARGET"
-p EXTRAOPTS, --pass-options=EXTRAOPTS
pass on package specific build options, e.g. "nompi=1"
or specific functional tests e.g "-t functest -p
functests/mytest.py"
-t TARGET, --target=TARGET
select TARGET from: depends, install, stage, release,
signature, bclean, clean, doc, format, pylint,
functest, test, deploy [default=install]
--release-name=RELEASE_NAME
the name of the staging directory and the release
tarball
--deploy-targets=DEPLOY_TARGETS
the deployement targets to execute. Select any
DEPLOY_TARGETS from:
authenticate,deploy_local,deploy_remote,verify [defaul
t=authenticate,deploy_local,deploy_remote,verify]
Advanced Options:
Caution: Use these options at your own risk.
-D, --debug do not run the actual package building command
-M, --no-build-myself
do not rebuild myself (rbuild)
-P, --no-parallel do not do parallel builds of packages
-R, --no-recursion do not apply target recursively to dependencies
-S, --no-svn-update
do not perform subversion update
-T, --no-tools do not rebuild Tools
-U, --update-only Ignore any target options and just do svn update
-V, --no-virtualenv
do not include virtualenv in a release
-X, --no-exit-on-error
continue building ignoring any individual package
build failures
------------------------------------------------------------------------------
rbuild system
=============
The rbuild system is the infrastructure underlying the rbuild command.
At the package level it can be run via the rbuild script or calling the
build.py files directly. In this case there is no recursive building.
For example:
python build.py [-q|-x] <target>
It provides the following command-line options
-q suppress messages to stdout and just print errors
-x exit recursivebuild when an error is encountered
and targets as for rbuild.
Troubleshooting
===============
* Make sure that you don't have a ~/.pydistutils.cfg because this can
conflict with virtualenv.