-
Notifications
You must be signed in to change notification settings - Fork 122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can we use more than one processor when system='multithreaded' #38
Comments
Hi Chao, 'Multithreaded' is intended for small-scale applications in which each solver instance runs on a single core. Currently it is not possible to use it with solver executables that require more than core. That said, it might be possible to add this functionality. Could you describe what system/cluster you are running on? What workflow are you carrying out (inversion, migration, ...)? Ryan |
UPDATE: Actually, I believe there is a way to add this functionality by modifying only a single line. Since it's such a simple change, I'll go ahead and submit a pull request. |
UPDATE: I think it should be ready to go now. |
attached is the parameter.py |
Hi Chao, From the traceback it looks like an issue with the utility for smoothing kernels. (To double check, you could try running with SMOOTH=False.) As a workaround I would suggest commenting out completely the 'solver.specfem2d.smooth' method so that the parent class method 'solver.base.smooth' which uses the SPECFEM xcombine_sem utility will be invoked instead. Does that make sense? |
Hi Chao, Very useful, I'm not sure what's going on with this traceback actually, some type of regression? Let me take a look. |
Looks like our cluster here is having issues, I don't think I can debug it immediately, but I will as soon as it is back on. In the meantime, could you remind me
EDIT: |
I just use the 2D checkers example:
BTW, in the bug.log, there is a Warning: mesh_properties.nproc != PAR.NPROC, because now, mesh_properties.nproc=1,PAR.NPROC=4, does this cause the problem? |
Hi Chao, Did you remesh the model? To change the number of processors from 1 to 4, you would need to generate a new numerical mesh via xmeshfem2D. |
I also used the model provided in your examples. You means that if i want to change the number of processors from 1 to 4, i need to remesh the model using xmeshfem2d, that is mpiexec -n 4 ./xmeshfem2d , right? |
That's right, youd need to remesh and supply a new model. You can find information on this, I believe, on the SPECFEM2D manual or issues page. Good luck! |
Hi Chao, You need to create a new model in the form SPECFEM2D is able to On Mon, Aug 29, 2016 at 12:07 PM, CHAO ZHANG notifications@github.com
|
If it's alright I'll go ahead and close soon. |
Hi Chao, MPI parallelization is working fine the 3D case so I'm not sure what's wrong in the 2D case. Perhaps check the xcombine_sem for bugs (SPECFEM2D has never been a funded project so unfortunately there are bugs ). Also, check the xcombine_sem is being invoked with the proper mpiexec wrapper by overloading system.mpiexec if necessary. |
so, you also meet with the same problem in your computer, right? Buy the way, except system='multithreaded', if i want to use mpiexec, can i use the system='mpi'? For this case, can i set |
I mean that all the examples in your guide for 2D, are using nproc=1, do you have some examples which using nproc>1. I want if it is set as nproc=1, how can you use the mpiexec |
Hi Chao, It might help to step back a bit first. You're running an inversion and you want each individual solver instance to run on multiple cores. In 3D this is currently working well for us. Such an approach is not currently implemented in 2D, but it should be fairly straightforward if you are familiar with SPECFEM2D and seisflows. But let me ask, why do you want to do this for 2D? If your 2D model is so large that you can't fit as many copies of it in the memory available on a single node as you have processors available on that node, then it makes sense to have each solver instance run on multiple cores. If not, I can't think of any significant advantage in terms of speed or efficiency. |
Good question, let me explain the naming convention. The names of modules in seisflows/system reflects how parallelization over shots is implemented. For example, system/serial means that shots are carried out one at a time. system/multithreaded means that as many shots are run at a single time as allowed by the available number of processors. There is no connection here to whether or not individual solver instances run in a parallel, only to how parallelization over shots is handled. |
Hi, Ryan
so, do you try the NPROC>1 (parallel for each task) in one of the module in seisflows/system? |
|
Hi Ryan, Make sense, thank for your help |
Hope it was helpful. Good luck!
|
Hi Chao, Thank you for identifying these issues, which arise when the 2D kernel summation and smoothing routines are used with MPI models. I'm unable to address these issues now myself because my PhD defense is within a few weeks. If you wanted to look into it yourself, it would be a matter of debugging and fixing SPECFEM2D's xcombine_sem and xsmooth_sem utilities. If you want to, feel free to open a new issue either here or in the SPECFEM2D issues page, something along the lines of "SPECFEM2D's xcombine_sem and xsmooth_sem not working for MPI models". Thanks, |
Hi Ryan, |
As I was trying to explain on the September 1 post, I'm a little confused about your ultimate goals. If you're running on a cluster, 2D inversions are quite fast even with one core per solver instance--unless your 2D model is huge, there is no need to parallelize over model regions. On the other hand, if you're running on a desktop or laptop, the number of cores is the limiting factor so you'll likely see no speedup by parallelizing over model region. To answer your question anyway though, I ran those 2D experiments on a cluster, so I used the |
Hi Ryan so i think can we set the nproc>1, then it will evoke parallel solver (mpirun -n nproc ./xmeshfem2d and mpirun -n nproc ./xspecfem2d) for each task, so it will run faster when running. Does this make sense? |
If you're doing an inversion with only 16 cores, using more cores per solver instance doesn't get you much because it limits the number of shots you can run simultaneously. That's all I was trying to say. |
Hi, Ryan
I can run the package using system='multithreaded' on my computer, however,
we only use one processor for each task so that the runtime is rather long for some examples.
"NPROC=1 # processors per task"
Can we use the parallel ( mpiexec or mpirun) under the multithreaded system? that is,
can we set NPROC>1?
Thanks
The text was updated successfully, but these errors were encountered: