Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding support for Parallel distributed MPI based programs #342

Open
incardon opened this issue Jul 29, 2020 · 4 comments
Open

Adding support for Parallel distributed MPI based programs #342

incardon opened this issue Jul 29, 2020 · 4 comments

Comments

@incardon
Copy link

incardon commented Jul 29, 2020

Is your feature request related to a problem? Please describe.
Dear cs01 thanks for gdbgui it is a very interesting project. Parallel distribute MPI (Message Passing Intergace) debugger like Allinea DDT or TotalView are very expensive debuggers. DDT, despite being the best. does not support Mac OSX. I am at the moment exploring the possibility to extend gdbgui to work with MPI based programs. The approach is to use multiple remote gdb-servers with 1 GUI controlling by muliple gdb sessions connected to the servers. Here is my fork https://github.com/incardon/gdbgui. My idea is to arrive to a version that I can integrate in our parallel simulation environment to debug parallel distributed programs

Describe the solution you'd like
Now is early. But I would like to check if there would be interest in integrating such feature at some point once is finished. And in particular, to discuss the approach I am following and introducing regression tests.

@incardon
Copy link
Author

incardon commented Aug 2, 2020

... Actively working on it. After some analysis and prototype that make me understand how the project work I found its simplicity/multi-session/event-driven design very ideal to add such support. Was long searching for a base like that to do an opensource parallel debugger.

@incardon
Copy link
Author

incardon commented Aug 17, 2020

Ok I have it in preview in my fork experiment. Still have to create tests (jsx mainly) and docs. I have to check how tests suite is done in python and in particular on jsx (most change are there)

Screenshot from 2020-08-17 10-09-06

Graphically has been done two changes:

one is the processor buttons, to switch between processes and check the status by color ... running paused disconnected

The other is the source code view. Because we have multiple processes there could be divergences. The more marked line is the position of the process on focus the light marked line is the position of the other processors.

The MPI gdbgui version work creating multiple gdb session and connecting to each process (in one click not manually). The typical commands continue breakpoint etc. are sent to all sessions, while variables stacktrace right view are issued for each process when we change the process on focus.

An additional option in LoadBinary has been introduced that is Connect to mpi-gdbserver

.................. I will add more info and instructions soon

@incardon
Copy link
Author

For who want to try:

  1. Clone the fork at git clone https://github.com/incardon/gdbgui.git ... cd gdbgui ... and switch branch with git checkout extension_to_mpi_programs

  2. Install openmpi with your package manager (apt dnf yum) or compile mpi from source.
    Check that you have the command mpirun and mpic++ in your PATH

  3. Enter gdbgui-mpi folder and run ./compile.sh. This should create the program print_nodes a small utility to discover the nodes names on which we are running in case want to run our debugger on super-computers + because is an mpi program itself we can debug it

  4. return to the root directory of gdbgui and
    We run now the standard command to make gdbgui working:
    nox -s develop-3.7 (or your python version)
    source .nox/develop-3-7/bin/activate
    yarn install
    yarn build

Note: if you find problem with yarn build complaining about webpack ... TypeError: Cannot read property 'properties' of undefined
in package.json I used to fix
"webpack": "^4.44.1",
"webpack-cli": "^3.3.10",
rerun yarn install and than yarn build

  1. we are now ready to run our first debug session in parallel. Open a second shell in the gdbgui root directory
    and launch an mpi orogram with

    ./gdbgui-mpi/launch_mpi_debugger 6 gdbgui-mpi/print_nodes

    In particular here we have 6 the number of processes and gdbgui-mpi/print_nodes the mpi program we want to debug everything it follow is passed to the mpi program

  2. go on the previous shell where you activated develop-3.7 and start a gdbgui server with
    python -m gdbgui-mpi

  3. Connect with your browser to 127.0.0.1:5000

  4. switch from Load Binary to connect to mpi-gdbserver and fill the input with *:60000 and press connect

  5. program should load into main with 6 processors ... Note the button on top indicate the status of the processes red on pause green running grey disconnected or exited

  6. Set a breakpoint at line 40 and press continue. (Do not press step out command on MPI_Init, it seems to go in inifite loops ... it also happen with other parallel debuggers. A way to recover is press sigint and that press continue to hit the nect breakpoint)

  7. Note all processes go red and hit line 40 if you do a next step command right arrow 2 lines are going to be on evidence 43 and 60 the 43 is of the process on focus 60 is for the others processors. (this come by the line if world_rank == 0) only the first process enter the if (43) all the others go on the else (60). We can check this also changing the process on focus pressing on the processes buttons. To note that local variables and expressions change their value based on the process on focus.

  8. Have fun (If you want to restart. please restart both gdbgui-mpi/launch_mpi_debugger and the gdbgui server)

@cs01
Copy link
Owner

cs01 commented Aug 25, 2020

This is really interesting, nice work. I've never heard of MPI. I just looked it up and it seems like a really interesting domain that I would be really happy if gdbgui could server. I will add some comments on your pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants