-
Notifications
You must be signed in to change notification settings - Fork 2
/
README
286 lines (229 loc) · 13.9 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
miniAMR mini-application
--------------------------------------
Contents of this README file:
1. miniAMR overview
2. miniAMR versions
3. building miniAMR
4. running miniAMR
5. notes about the code
--------------------------------------
--------------------------------------
1. miniAMR overview
miniAMR applies a stencil calculation on a unit cube computational domain,
which is divided into blocks. The blocks all have the same number of cells
in each direction and communicate ghost values with neighboring blocks. With
adaptive mesh refinement, the blocks can represent different levels of
refinement in the larger mesh. Neighboring blocks can be at the same level
or one level different, which means that the length of cells in neighboring
blocks can differ by only a factor of two in each direction. The calculations
on the variables in each cell is an averaging of the values in the chosen
stencil. The refinement and coarsening of the blocks is driven by objects
that are pushed through the mesh. If a block intersects with the surface
or the volume of an object, then that block can be refined. There is also
an option to uniformly refine the mesh. Each cell contains a number of
variables, each of which is evaluated indepently.
--------------------------------------
2. miniAMR versions:
- miniAMR_ref:
reference version: self-contained MPI-parallel.
- miniAMR_serial
serial version of reference version
-------------------
3. Building miniAMR:
To make the code, type 'make' in the directory containing the source.
The enclosed Makefile.mpi is configured for a general MPI installation.
Other compiler or other machines will need changes in the CFLAGS
variable to correspond with the flags available for the compiler being used.
-------------------
4. Running miniAMR:
miniAMR can be run like this:
% <mpi-run-command> ./miniAMR.x
where <mpi-run-command> varies from system to system but usually looks something like 'mpirun -np 4 ' or similar.
Execution is then driven entirely by the default settings, as configured in default-settings.h. Options may be listed using
% ./miniAMR.x --help
To run the program, there are several arguments on the command line.
The list of arguments and their defaults is as follows:
--nx - block size in x
--ny - block size in y
--nz - block size in z
These control the size of the blocks in the mesh. All of these need to
be even and greater than zero. The default is 10 for each variable.
--init_x - initial blocks in x
--init_y - initial blocks in y
--init_z - initial blocks in z
These control the number of the blocks on each processor in the
initial mesh. These need to be greater than zero. The default
is 1 block in each direction per processor. The initial mesh
is a unit cube regardless of the number of blocks.
--reorder - ordering of blocks
This controls whether the blocks are ordered by the RCB algorithm
or by a natural ordering of the processors. The default is 1 which
selects the RCB ordering and the natural ordering is 0.
--npx - number of processors in the x direction
--npy - number of processors in the y direction
--npz - number of processors in the z direction
These control the number of processors is each direction. The product
of these number has to equal the number of processors being used. The
default is 1 block in each direction.
--max_blocks - maximun number of blocks per processor
The maximun number of blocks used per processor. This is the number of
blocks that will be allocated at the start of the run and the code will
fail if this number is exceeded. The default is 500 blocks.
--num_refine - number of levels of refinement
This is the number of levels of refinement that blocks which are refined
will be refined to. If it is zero then the mesh will not be refined.
the default is 5 levels of refinement.
--block_change - number of levels a block can change during refinement
This parameter controls the number of levels that a block can change
(either refining or coarsening) during a refinement step. The default
is the number of levels of refinement.
--uniform_refine - if 1, then grid is uniformly refined
This controls whether the mesh is uniformly refined. If it is 1 then the
mesh will be uniformly refined, while if it is zero, the refinement will
be controlled by objects in the mesh. The default is 1.
--refine_freq - frequency (in timesteps) of checking for refinement
This determines the frequency (in timesteps) between checking if
refinement is needed. The default is every 5 timesteps.
--target_active - target number of blocks per processor
--target_max - max number of blocks per processor
--target_min - min number of blocks per processor
These allow the user to control the number of blocks per processor.
If these are zero, then no adjustment is made. If target_active is
greater than zero than the code will adjust the number of blocks to
that target after the refinement step. If target_max is greater than
zero then the number of blocks will be reduced if it exceeds this
number. Likewise, if target_min is greater than zero, than the number
of blocks will be raised if there is less than that number after the
refinement step. The default for all of these is zero.
--inbalance - percentage inbalance to trigger inbalance
This parameter allows the user to set a percentage threshold above
which the load will be balanced amoung the processors. The value
that this is checked against is the maximum number of blocks on a
processor minus the minimum number of blocks on a processor divided
by the average. The default is zero, which means to always load
balance at each refinement step.
--lb_opt - (0, 1, 2) determine load balance strategy
If set to 0, then load balancing is not performed. The default is
set to 1 which load balances each refinement step. Setting the
parameter to 2 results in load balancing at each stage of the
refinement step. If a processor has a large number of blocks which
are refined several steps, this allows the work (and space needed)
to be shared amoung more processors.
--num_vars - number of variables (> 0)
The number of variables the will be calculated on and communicated.
The default is 40 variables.
--comm_vars - number of vars to communicate together
The number of variables that will communicated together. This will
allow shorter but more variables if it is set to something less than
the total number of variables. The default is zero which will
communicate all of the variables at once.
--num_tsteps - number of timesteps (> 0)
The number of timesteps for which the simulation will be run. The
default is 20.
--stages_per_ts - number of comm/calc stages per timestep
The number of calculate/communicate stages per timestep. The default
is 20.
--permute - (no argument) permute communication directions
If this is set, then the order of the communication directions will
be permuted through the six options available. The default is
to send messages in the x direction first, then y, and then z.
--blocking_send - (no argument) Use blocking sends in the communication
routine instead of the default nonblocking sends.
--code - change the way communication is done
The default is 0 which communicates only the ghost values that are
needed. Setting this to 1 sends all of the ghost values, and setting
this to 2 also does all of the message processing (refinement or
unrefinement) to be done on the sending side. This allows us to
more closely minic the communication behaviour of codes.
--checksum_freq - number of stages between checksums
The number of stages between calculating checksums on the variables.
The default is 5. If it is zero, no checks are performed.
--stencil - 7 or 27 point 3D stencil
The 3D stencil used for the calculations. It can be either 7 or 27
and the default is 7 since the 27 point calculation will not conserve
the sum of the variables except for the case of uniform refinement.
--error_tol - (e^{-error_tol} ; >= 0)
This determines the error tolerance for the checksums for the variables.
the tolerance is 10 to the negative power of error_tol. The default
is 8, so the default tolerance is 10^(-8).
--report_diffusion - (>= 0) none if 0
This determines if the checksums are printed when they are calculated.
The default is 0, which is no printing.
--report_perf - (0 .. 15)
This determines how the performance output is displayed. The default
is YAML output (value of 1). There are four output modes and each is
controlled by a bit in the value. The YAML output (to a file called
results.yaml) is controlled by the first bit (report_perf & 1), the
text output file (results.txt) is controlled by the second bit
(report_perf & 2), the output to standard out is controlled by the
third bit (report_perf & 4), and the output of block decomposition
at each refine step is controlled by the forth bit (report_perf & 8).
These options can be combined in any way desired and zero to four
of these options can be used in any run. Setting report_perf to 0
will result in no output.
--refine_freq - frequency (timesteps) of refinement (0 for none)
This determines how frequently (in timesteps) the mesh is checked
and refinement is done. The default is every 5 timesteps. If
uniform refinement is turned on, the setting of refine_freq does
not matter and the mesh will be refined before the first timestep.
--refine_ghosts - (no argument)
The default is to not use the ghost cells of a block to determine if
that block will be refined. Specifying this flag will allow those
ghost cells to be used.
--num_objects - (>= 0) number of objects to cause refinement
The number of objects on which refinement is based. Default is zero.
--object - type, position, movement, size, size rate of change
The object keyword has 14 arguments. The first two are integers
and the rest are floating point numbers. They are:
type - The type of object. There is 16 types of objects. They include
the surface of a rectangle (0), a solid rectangle (1),
the surface of a spheroid (2), a solid spheroid (3),
the surface of a hemispheroid (+/- with 3 cutting planes)
(4, 6, 8, 10, 12, 14),
a solid spheroid (+/- with 3 cutting planes)(5, 7, 9, 11, 13, 15),
the surface of a cylinder (20, 22, 24),
and the volume of a cylinder (21, 23, 25).
bounce - If this is 1 then an object will bounce off of the walls
when the center hits an edge of the unit cube. If it is
zero, then the object can leave the mesh.
center - Three doubles that determine the center of the object in the
x, y, and z directions.
move - Three doubles that determine the rate of movement of the center
of the object in the x, y, and z directions. The object moves
this far at each timestep.
size - The initial size of the object in the x, y, and z directions.
If any of these become negative, the object will not be used
in the calculations to determine refinement. These sizes are
from the center to the edge in the specified direction.
inc - The change in size of the object in the x, y, and z directions.
Examples of run scripts for a Cray XE6 that illustrate several of the options:
One sphere moving diagonally on 27 processors:
mpirun -np 27 -N 7 miniAMR.x --num_refine 4 --max_blocks 9000 --npx 3 --npy 3 --npz 3 --nx 8 --ny 8 --nz 8 --num_objects 1 --object 2 0 -1.71 -1.71 -1.71 0.04 0.04 0.04 1.7 1.7 1.7 0.0 0.0 0.0 --num_tsteps 100 --checksum_freq 1
An expanding sphere on 64 processors:
mpirun -np 64 miniAMR.x --num_refine 4 --max_blocks 6000 --init_x 1 --init_y 1 --init_z 1 --npx 4 --npy 4 --npz 4 --nx 8 --ny 8 --nz 8 --num_objects 1 --object 2 0 -0.01 -0.01 -0.01 0.0 0.0 0.0 0.0 0.0 0.0 0.0009 0.0009 0.0009 --num_tsteps 200 --comm_vars 2
Two moving spheres on 16 processors:
mpirun -np 16 miniAMR.x --num_refine 4 --max_blocks 4000 --init_x 1 --init_y 1 --init_z 1 --npx 4 --npy 2 --npz 2 --nx 8 --ny 8 --nz 8 --num_objects 2 --object 2 0 -1.10 -1.10 -1.10 0.030 0.030 0.030 1.5 1.5 1.5 0.0 0.0 0.0 --object 2 0 0.5 0.5 1.76 0.0 0.0 -0.025 0.75 0.75 0.75 0.0 0.0 0.0 --num_tsteps 100 --checksum_freq 4 --stages_per_ts 16
-------------------
5. The code:
block.c Routines to split and recombine blocks
check_sum.c Calculates check_sum for the arrays
comm_block.c Communicate new location for block during refine
comm.c General routine to do interblock communication
comm_parent.c Communicate refine/unrefine information to parents/children
comm_refine.c Communicate block refine/unrefine to neighbors during refine
comm_util.c Utilities to manage communication lists
driver.c Main driver
init.c Initialization routine
main.c Main routine that reads command line and launches program
move.c Routines that check overlap of objects and blocks
pack.c Pack and unpack blocks to move
plot.c Write out block information for plotting
profile.c Write out performance data
rcb.c Load balancing routines
refine.c Routines to direct refinement step
stencil.c Perform stencil calculations
target.c Add/subtract blocks to reach a target number
util.c Utility routines for timing and allocation
-- End README file.
Courtenay T. Vaughan
(ctvaugh@sandia.gov)