Skip to content

Home

Jump to bottom
Bruno Chareyre edited this page Oct 8, 2018 · 13 revisions

This page is a draft outline of the ideas behind this yade-mpi branch.

Objective: implement domain decomposition for load distribution through mpi4py

Definitions and rules (can still be discussed and changed):

1- a subdomain is a set of bodies + the bounding box of this set

2- a body is owned by only one primary subdomain but it may have images in other subdomains

3- the time integration of one subdomain is assigned one mpi worker, it results in updating the motion of all bodies assigned to the subdomain

4- a subdomain 'A' can return subsets of bodies which are in potential interaction with another 'B' subdomain. The subset is obtained by intersecting the bounding box of B with the bounding boxes of all 'A'-bodies.

5- The above subset is called intersect(A,B). Note: intersect(A,B)!=intersect(B,A) since the intersection is always a subset of the first argument.

6- intersect(A,B) defines A-bodies which must have updated images in B, in order for B to compute all interactions on B-bodies (including B-B and A-B interactions).

7- the same is true for intersect(B,A): A computes all interactions on A-bodies (including A-A and A-B interactions).

8- as a (bad) consequence of 6-7 all A-B interactions are calculated twice (i.e. by two different different threads)

9- as (good) consequence of 6-7 there is only one synchronization per time iteration which corresponds to updating in B the image particles from A, and in A the image particles from B. There is no synchronization of forces, which would require an additional synchronization per time iteration.

10- remark: it is probably also possible to do the opposite: synchronize only forces and let the motion of image A-particles be integrated by B (same motion integrated by multiple threads).

Preliminary steps Preliminary steps listed below aim at making the manipulation of Yade objects and containers easy with mpi4py, for later implementation and optimization.

DONE:

  • check that boost::python/YADE_CLASS serialization can go through mpi messages transparently (example script below)

  • make bodyContainer a YADE_CLASS so that it can also go through mpi messages (https://github.com/bchareyre/yade-mpi/commit/6d822c71137fa7f84b2324e950a7c041d0108bc8), "O.bodies.rawBodies" gives read/write access to the underlying container

  • the scene (O._sceneObj) and O.interactions needs to go through messages as well

  • implement bound dispatching for subdomains (make subdomain a Body?)

  • implement intersect(A,B) based on ISCollider

  • set up a concrete example with arbitrary number of particle and subdomain

Next steps:

I- methods to define/update the subdomains efficiently (load balancing)

II- benchmark/improvement on HPC

III- make all the above transparent to the user so that instead of running 'mpiexec -n N python script.py' yade will spawn pools of mpi workers internally. Ultimately the difference between a single-thread script and a mpi script could just be the replacement of "O.run" by "O.mpiRun", triggering automatic domain decomposition.

Note: I+II will be a topic for a 'Hackaton' planned in early 2019, in relation with people handling the Grenoble HPC facility.

========== NAIVE EXAMPLE SCRIPT ==========

# mpiexec -n 2 python script.py

import sys
from os.path import expanduser
sys.path.append(expanduser('~')+'/yade/bin') # path where you have yadeimport.py
# yadeimport.py is generated by `ln yade-versionNo yadeimport.py`, where yade-versionNo is the yade executable

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()



if rank == 0:
    import yadeimport as yade
    s0=yade.sphere((0,0,0),666)
    yade.Omega().bodies.append(s0)
    print "sent sphere:",s0
    comm.send(yade.Omega().bodies[0], dest=1, tag=10)
    comm.send(yade.Omega().engines, dest=1, tag=11)
    comm.send(yade.Omega().bodies.rawBodies, dest=1, tag=12) 

elif rank == 1:
    import yadeimport as yade
    s1 = comm.recv(source=0, tag=10)
    print "received sphere:",s1
    engs = comm.recv(source=0, tag=11)
    yade.Omega().engines = engs
    bodies = comm.recv(source=0, tag=12)
    yade.Omega().bodies.rawBodies = bodies
    print "radius(s1)=", s1.shape.radius, "radius(bodies[0])=",yade.Omega().bodies[0].shape.radius

========== FUNCTIONAL EXAMPLE SCRIPT ==========

See:

Clone this wiki locally