-
Notifications
You must be signed in to change notification settings - Fork 1
/
README.in
260 lines (192 loc) · 11.7 KB
/
README.in
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
NAME/VERSION
@PACKAGE_NAME@ @PACKAGE_VERSION@
IN BRIEF
Gold is an open source accounting system that tracks and manages resource
usage on High Performance Computers. It acts much like a bank in which
resource credits are deposited into accounts with access controls designating
which users, projects and machines may access the account. As jobs complete
or as resources are utilized, accounts are charged and resource usage
recorded. Gold supports familiar operations such as deposits, withdrawals,
transfers and refunds. It provides balance and usage feedback to users,
managers, and system administrators.
Since accounting needs vary widely from organization to organization, Gold
has been designed to be extremely flexible, featuring customizable accounting
and supporting a variety of accounting models. Attention has been given to
scalability, security, and fault tolerance. Gold facilitates resource sharing
between organizations or within a Grid by providing distributed accounting
while preserving local site autonomy.
Gold was originally developed at Pacific Northwest National Laboratory (PNNL)
under the Scalable Systems Software (SSS) SciDAC project. It is now being
actively maintained and enhanced by Adaptive Computing.
STATUS
This is a general release of Gold; the second release from Adaptive Computing.
DESCRIPTION
In order for an organization to efficiently use its high performance
computers, it must be able to allocate resources to the users and projects
that need them most in a manner that is fair and according to mission
objectives. Tracking the historical resource usage allows for insightful
capacity planning and in making decisions on how to best mete out these
resources. It allows the funding sources that have invested heavily in a
supercomputing resource a means to show that it is being utilized efficiently.
Additionally, accounting and allocation management are critical to being able
to take advantage of the tremendous utilization gains afforded by
meta-scheduling.
The Gold Accounting and Allocation Manager tracks and manages job and resource
usage. Much like a bank, an allocation manager associates a cost to computing
resources and allows resource credits to be allocated to users and projects
and meted out in a fair and judicious manner. As jobs complete or as resources
are utilized, projects are dynamically charged and resource usage recorded.
Gold was originally developed at PNNL as open source software under the
Scalable Systems Software (SSS) SciDAC project. A standard communication
protocol has been designed to facilitate communication between resource
management and accounting components and to allow component substitution.
A flexible GUI exists to simplify use and the management of project and
accounting data.
FEATURES
· Rather than post-processing resource usage records on a periodic basis to
rectify project balances, acounts are updated immediately at job completion.
· A reservation is placed against the account for the estimated number of
resource credits before the job runs, followed by an appropriate charge at
the moment the job completes, thereby preventing projects from using more
resources than were allocated to them.
· A uniquely flexible account design allows resource credits to be allocated
to specific projects, users and machines.
· Resource credits may be restricted for use within a designated time period
allowing sites to implement a use-it-or-lose-it policy to prevent year-end
resource exhaustion and establishing a project cycle.
· The system can track and charge for composite resource usage (memory, disk,
CPU, etc) and custom charge multipliers can be applied (Quality of Service,
Node Type, Time of Day, etc).
· Users and resource brokers can determine ahead of time the cost of using
resources via a guaranteed quote mechanism.
· Accounts feature an optional credit limit allowing support for both debit
and credit models. This feature can also be used to enable overdraft
protection for specific accounts.
· Accounts may be nested creating an account hierarchy. This allows for the
delegation of management responsibilities, the establishment of automatic
rules for the distribution of downstream resource credits, and the option of
making higher level credits available to lower level accounts.
· Gold supports a powerful querying and update mechanism that facilitates
flexible reporting and streamlines administrative tasks.
· Gold allows the establishment of default projects, machines and users.
Additionally Gold can allow user, machines and projects to be automatically
created the first time they are seen by the resource management system.
These features allow job submitters to use the system without even knowing it.
· Gold provides fine-grained (instance-level) Role Based Access Control for
all operations.
· Sites can create or modify record types on the fly enabling them to meet
their custom accounting needs. Dynamic object creation allows sites to
customize the types of accounting data they collect without modifying the
code. This capability turns this system into a generalized information
service. This capability is extremely powerful and can be used to manage
all varieties of custom data.
· Gold implements a journaling mechanism that preserves the indefinite
historical state of all objects and records. This powerful mechanism allows
historical bank statements to be generated, provides an undo/redo capability
and allows commands to be run as if it were any arbitrary time in the past.
· A traceback mechanism can allow all parties of a transaction (resource
requestor and provider) to have a first-hand record of the resource
utilization and to have a say as to whether or not the job should be permitted
to run, based on their independent policies and priorities. A job will only
run if all parties are agreeable to the idea that the target resources can be
used in the manner and amount requested. Support for traceback debits will
facilitate the establishment of trust and exchange relationships between
administrative domains.
· The system was designed from the outset with goals of scalability, fault
tolerance, security, portability and ease of use in mind.
· A web-based graphical user interface can be used to simplify use of Gold's
powerful features.
· Being open source allows for site self-sufficiency, customizability and
promotes community development and interoperability.
LICENSE
Gold has an open source license under copyright of PNNL as expressed in
LICENSE.
DISTRIBUTION
TESTED CONFIGURATIONS
To date Gold has been successfully tested in the following environments:
OS Version Architecture Database Version Batch Interface
--------- -------- ------------- -------- --------- ---------------
RedHat 7.3 i386 Postgres 7.1 Maui
RedHat 9.0 i386 SMP Postgres 7.3 Maui
RedHat AS2.1 ia64 Postgres 7.4,8.0 LSF
Fedora Core 2 Athlon Postgres 7.4
Suse 9.0 Athlon i586 Postgres 7.3
AIX 5.1 RS/6K Power4 Postgres 7.3
Tru64 4.0d Alpha Postgres 7.4
OS X 7.5.0 Mac Powerpc Postgres 8.0
HP-UX 11.0 HP J5600 Postgres 7.3
IRIX 6.5 SGI O2 Postgres 7.3
SunOS 5.8 SUN Ultra10 Postgres 7.3
RedHat 7.3 i386 SQLite 3.0
RedHat 7.3 i386 MySQL 3.23,4.1
REQUIREMENTS
You will first need to build, test and install the following prerequisites:
Software Name and Version Where to Obtain Necessity
----------------------------------- -------------------------------- ---------
Perl 5.8 or higher w/ suidperl http://www.perl.com/ Required
a tested Database (Postgres 7.2+) ftp://ftp.postgresql.org/pub/ Required
LibXML2 (2.4.25 or higher) http://www.xmlsoft.org/ Required
Apache Httpd Server (2.0 or higher) http://httpd.apache.org/ Optional
OpenSSL (0.9.5a or higher) http://www.openssl.org/ Optional
mod_ssl (2.26 or higher) http://www.modssl.org/ Optional
Gnu ReadLine (2.0 or higher) ftp://ftp.gnu.org/gnu/ Optional
Please reference the hints in INSTALL.
INSTALLATION
Installation instructions are contained in INSTALL. The high level steps are:
1) Prepare to build and install Gold by unpacking the archive and creating
an appropriate user to manage Gold.
2) Install the prerequisites. These include Perl 5.6 (with suidperl), libxml2
and PostgreSQL 7. Apache Httpd server, OpenSSL and mod_ssl are needed
if you want to use the web GUI. Gnu readline is optional.
3) Perform pre-install configurations by running the "configure" script.
4) Compile the program with "make". Also use "make gui" if you intend to
install the web GUI.
4) Install the prerequisite Perl modules with "make deps".
5) Install Gold with "make install". Generate a secret key with
"make auth_key". Install the gui with "make install-gui" if necessary.
6) Perform general setup by editing Gold config files and setting
environment variables.
7) Database setup includes insuring postgres allows internet domain sockets,
authorizing postgres to connect to your host, defining users, creating
the gold database and initially populating the database.
8) Web-Server setup (optional) involves modifying the config file to allow
SSL connections, creating a certificate keystore, and getting httpd
started up.
9) Startup the gold server daemon (goldd).
10) Initialize Gold by defining users, projects, accounts, etc as necessary.
TESTING
There are no built-in tests for Gold at this time. Automated and manual
testing is conducted at Adaptive Computing Enterprises, Inc.
PROJECT WEB PAGE
The current project web page is at Adaptive Computing:
<http://www.clusterresources.com/products/gold>.
DOCUMENTATION
A Gold User's Guide is available in the docs directory both in pdf and html
formats. Each of the perl clients have built-in usage and man-page
documentation accessible via the --help and --man flags. Some documents, such
as a Requirements document, communication protocol docs, presentations, etc.
may be found at the Adaptive Computing Gold Project Documentation page at
<http://www.clusterresources.com/products/gold/docs/>.
COMPATIBILITY
For semantic purposes let's assume VRMF nomenclature (Version, Release,
Maintenance level, Fix level), such that Gold 2.7.3.2 represents
Version 2, Release 7, Maintenance level 3, Fix level 2.
Gold Release numbers are changed when a change to the database format
is made, and hence different Gold Releases are not compatible.
Different Gold Maintenance Levels should however be compatible and may
coexist and operate on a single database.
CHANGES
Changes are enumerated in the CHANGES file.
PROBLEM REPORTING
You may send comments and bug reports to <gold-support@adaptivecomputing.com>.
Please include as much detail as possible in order to expedite a resolution.
Be advised that this queue will be handled on a best-effort basis unless a
support contract is established with Adaptive Computing.
DISTRIBUTION LIST
A majordomo list has been setup for Gold contributors, administrators and
users in <gold-users@supercomputing.org>. Anyone may subscribe and then post
to the list.
AUTHORS
Gold has been developed by Scott Jackson <scottmo@adaptivecomputing.com> while
at Pacific Northwest National Laboratory. It continues to be enhanced and
maintained by Scott Jackson and other staff at Adaptive Computing.