-
Notifications
You must be signed in to change notification settings - Fork 6
CODES
CODES ANL page: http://press3.mcs.anl.gov/codes/developer-access/
CODES repo: https://xgitlab.cels.anl.gov/codes/codes/wikis/home
ROSS GitHub repo: https://github.com/carothersc/ROSS
Network Workloads (NW): A simulated MPI Process
Logical Process (LP): a Terminal or Router
Terminal: a simulated burst buffer, compute node, or MPI process
Router: a simulated network device that connects terminals to other Routers
Kernel Process (KP): a group of Logical Processes, the default is 16
PE: physical MPI process
Event Rate: number of forward events processed per second
Efficiency: number of forward events / number of backward (rollback) events
The current approach to partitioning in CODES is a linear breakup of LPs. LPs are grouped with some number of NWs followed by an equal number of terminals and a router. KPs are built over several of these groups. Several of these KPs are given to each PE. The partition of these KPs is a linear partition based on IDs.
We will be partitioning the graph resulting from the application mapped to the network topology. This graph is defined at run time in CODES and thus, an in-memory approach will be taken to compute a static partitioning of their graph (we assume that CODES does not support migration). The construction of the CODES partition is done in the function int codes_mapping_setup(void).
The original idea is to use an application trace to define work associated to LPs and create weights off of this. Knowing this would require loading trace files (1s-10s of GB) as well as knowing the routing protocol (adaptive routing is based on congestion). If we can read and interpret the files in parallel the trace files wouldn't be too difficult. If we are dealing with adaptive routing we may need to use dynamic load balancing which would require migrating LPs/KPs.
It looks like the codes-workload-dump tool described here: https://xgitlab.cels.anl.gov/codes/codes/wikis/codes-dumpi-workload#inspecting-dumpi-traces can read DUMPI traces in serial and collect some basic info. Is there a parallel version of this tool, or something similar? From what I can tell the 'dumpi_trace_nw_workload_load' struct's loading method https://xgitlab.cels.anl.gov/codes/codes/blob/b8df46df548cc7e1e07c0d3c139f7c96497cf183/src/workload/methods/codes-dumpi-trace-nw-wrkld.c#L772 , and the higher level api 'codes_workload_load', support parallel loading.
The wiki page https://xgitlab.cels.anl.gov/codes/codes/wikis/Dragonfly-Plus appears to demonstrate a synthetic workload at the end, and this page https://xgitlab.cels.anl.gov/codes/codes/wikis/codes-dragonfly has some examples of running dumpi traces. A four process dragonfly example using a DUMPI trace also appears here: https://xgitlab.cels.anl.gov/codes/codes/wikis/codes-dumpi-workload#replaying-application-trace-on-codes-mpi-simulation-layer
Instrumentation of events (and other things) is done in ROSS.
The following pages discuss the collection of event traces:
https://github.com/carothersc/ROSS/blob/10d7a06b2d5a6864aba4ac779e1faf45d30ffd0d/README-vis.md
https://xgitlab.cels.anl.gov/codes/codes/blob/master/codes-vis-readme.md
http://carothersc.github.io/ROSS/instrumentation/instrumentation.html
http://carothersc.github.io/ROSS/instrumentation/event-tracing.html
- Q) Does CODES support migrating LPs? A) No.
-
How are KPs defined? If we assign LPs to MPI processes (the partition), will KPs be defined within each process or is something else done that will result in problems (i.e., a KP spread across multiple processes)?
-
How can we measure the imbalance of a partition in CODES? Would the # of (forward) events processed by each MPI process make sense? If so, we would define the imbalance as the
maximum(process events)/average(process events)? Here the thinking is that we want to eliminate nodes that process a lot of events. We may also want to look at the number of rollbacks a partition results in; i.e., a bad partition has lots of rollback.
First grab and install ROSS
git clone http://github.com/carothersc/ROSS.git
# if using 0.5.2 release: git checkout d3bdc07
cd ROSS
mkdir build
cd build
# note: other options for ARCH include i386 (for 32 bit machines),
# bgp, and bgq (for Blue Gene systems)
ARCH=x86_64 CC=mpicc CXX=mpicxx cmake -DCMAKE_INSTALL_PREFIX=../install ../
make -j4
make install
If you don't have pkg-config, flex, or bison. Install those now with your package manager. On Ubuntu:
sudo apt-get install pkg-config
sudo apt-get install flex bison
Grab and build DUMPI from
git clone https://github.com/sstsimulator/sst-dumpi.git
#Set these flags when using mpich>=3.0
CFLAGS="-DMPICH_SUPPRESS_PROTOTYPES=1 -DHAVE_PRAGMA_HP_SEC_DEF=1"
./bootstrap.sh
./configure --enable-libdumpi CC=mpicc --prefix=/path/to/INSTALL_PATH
make -j4
make install
Grab CODES and build it
git clone https://xgitlab.cels.anl.gov/codes/codes.git
cd codes
./prepare.sh
mkdir build
cd build
../configure --prefix=/path/to/codes/install CC=mpicc CXX=mpicxx PKG_CONFIG_PATH=/path/to/ross/install/lib/pkgconfig --with-dumpi=path/to/dumpi/install/
make -j4
make install
First setup the network configuration. To setup the edison dragonfly network:
cd scripts/gen-cray-topo/
##If you don't have numpy you must download it
#pip install numpy
python edison.py links-edison.txt /path/to/intra_file /path/to/inter_file
With the network files created, now you must edit the following file:
src/network-workloads/conf/dragonfly-custom/modelnet-test-dragonfly-edison.conf
Change these lines:
# network config file for intra-group connections
intra-group-connections="../src/network-workloads/conf/dragonfly-custom/intra-edison";
# network config file for inter-group connections
inter-group-connections="../src/network-workloads/conf/dragonfly-custom/inter-edison";
to point to your intra and inter files created in the previous step.
Now you can run a synthetic workload
mpirun -np 4 /path/to/install/bin/model-net-synthetic-custom-dfly --sync=3 -- src/network-workloads/conf/dragonfly-custom/modelnet-test-dragonfly-edison.conf
To run a DUMPI workload first grab a DUMPI trace from http://portal.nersc.gov/project/CAL/designforward.htm.
The below example uses the AMG 1728 tasks trace.
To run:
mpirun -np 4 install/bin/model-net-mpi-replay --sync=3 --disable_compute=1 --workload_type="dumpi" --workload_file=/path/to/DUMPI/trace/dumpi-2014.03.03.14.55.50- --num_net_traces=1728 -- src/network-workloads/conf/dragonfly-custom/modelnet-test-dragonfly-edison.conf
To get an event-trace in the run you must checkout the inst-model-bug-fix branch and add --event-trace=1 to the run command.
This will create a series of stats-output directories. A reader here may be able to help read the binary data in those directories.