Building backends using entry_points and NetworkX's parallel backend #279
Description
Title
nx-parallel : NetworkX's parallel backend
Abstract:
Hi! Join me as we explore parallel graph algorithms in nx-parallel, going over the implementation details, performance enhancements, the effects of chunking, and challenges of parallelizing functions that “yield” instead of “return”. We’ll start by exploring a social network analysis problem and see the need for a parallel backend. Then we’ll dive into the NetworkX’s backend dispatching architecture and walk through setting up a custom backend and testing it using the existing suite, and understand the significance of the dispachable decorator. We'll delve into the internals of parallel algorithms, observing process creation and CPU core utilization. Finally, we'll discuss future directions for handling even larger graphs. Experience how nx-parallel accelerates network analysis through parallel processing!
Description:
During the first 5-6 minutes of the talk, I will guide the audience through a comprehensive, but brief, journey into the world of network analysis through a social networks example. I’ll begin by performing the computations using NetworkX and then move on to the need for a parallel backend. The initial segment will unravel some of the core NetworkX algorithms and backend dispatching, shedding light on the automated testing framework and the significance of the dispatchable decorator.
Transitioning into the crux of the discussion, the subsequent 10-15 minutes will be dedicated to dissecting how parallel algorithms are implemented in nx-parallel. I will elucidate the internal workings of parallelization, showcasing the increase in processes and the activity distribution across different CPU cores while the parallel algorithms are running. A focal point of this section will be the role of chunking in nx-parallel and its impact on speedups. Furthermore, I may present benchmarking results that demonstrate the performance benefits derived from parallel computing. Moreover, I will also address challenges encountered while parallelizing functions that yield instead of return a value and what chunking means for this kind of function. By sharing my experiences of fiddling with these issues, attendees might gain some insights into navigating similar hurdles in their own projects.
In the closing minutes, we'll take a glimpse into the future of nx-parallel and NetworkX dispatching, by discussing some of the “bigger picture” questions about the pipeline connecting NetworkX to nx-parallel to joblib to joblib's different parallel backends and using something like distributed graph processing for a much more memory-efficient parallelization. And then this section will lead to a much more interactive 5-minute window for the audience to participate, ponder and raise more questions, and share their thoughts, knowledge, and perspectives!
Pre-requisites & reading material
None,
Not required but if you want to read: https://schefflera-arboricola.github.io/Schefflera-Arboricola/NetworkX-Internship-Working-on-nx-parallel
Time required for the talk
~ 30 mins
Link to slides/demos
No response
About you
Hi, I'm Aditi. I am currently an undergraduate student. I'm doing my bachelor's in Computer Science from Delhi University, and my bachelor's in Data Science and Application from the Indian Institute of Technology, Madaras. I've been working on the NetworkX project and the nx-parallel backend for quite a few months now, previously as an Independent contractor and currently as a GSoC contributor.
Blog: https://schefflera-arboricola.github.io/Schefflera-Arboricola/
GitHub: https://github.com/Schefflera-Arboricola
Availability
June'24-Aug'24
Illustration
nx-parallel github repo: https://github.com/networkx/nx-parallel