Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace MPI send/receive with asychronous sockets calls #3

Closed
ZacBlanco opened this issue Nov 3, 2016 · 3 comments
Closed

Replace MPI send/receive with asychronous sockets calls #3

ZacBlanco opened this issue Nov 3, 2016 · 3 comments

Comments

@ZacBlanco
Copy link
Owner

We need to stop using MPI, but rather UDP.

We've shown from some research that the algorithm can run faster if we use UDP rather than MPI communication and can be faster in terms of resiliency.

With this in mind we should create some kind of async socket interface in order to send and listen for data in the background while our algorithm is listening.
ZacBlanco added a commit that referenced this issue Nov 4, 2016
@ZacBlanco
[#3] - Adding in cloud_comm class with some tests
1a3a2b2
ZacBlanco added a commit that referenced this issue Nov 5, 2016
@ZacBlanco
[#3] - First steps towards sender implementation
ddc075f 
Working on a sender which can automatically chunk the data into packets
and send it with its own sequencing logic.

Added a new method as well to calculate the size

However I'm rethinking the JSON-style approach and might try to use a
more rigid approach where we just define bytes in the first X locations
to be metadata like sequence/tag/etc.. then the rest is the actual data.

Still working out the kinks but with the tests going coming along with
development it shouldn't be too bad.
@ZacBlanco
Copy link
Owner Author

@mariellej If you've got time check out the udp-communicator branch and look for cloud_comm.py. Let me know if you've got questions/suggestions/ideas. Hoping to have the implementation finished this weekend.

ZacBlanco added a commit that referenced this issue Nov 6, 2016
@ZacBlanco
[#3] - Methods for receiving and sending packets
01c889b
ZacBlanco added a commit that referenced this issue Nov 6, 2016
@ZacBlanco
[#3] - First run at sphinx documentation
c2106ec 
Build docs using `make html`
ZacBlanco added a commit that referenced this issue Nov 6, 2016
@ZacBlanco
[#3] - Move docs directory and add a readme to build docs
72cfc4f
ZacBlanco added a commit that referenced this issue Nov 6, 2016
@ZacBlanco
[#3] - Add real requirements to requirements file
f2e5730
ZacBlanco added a commit that referenced this issue Nov 7, 2016
@ZacBlanco
[#3] - Listen and Send tests
9b3bbf0 
Added more tests and squashed a couple bugs. Needed to set the listening
socket to be non-blocking (a 0 second timeout) as well as fix a couple
of typos by forgetting to reference `self`
ZacBlanco added a commit that referenced this issue Nov 7, 2016
@ZacBlanco
[#3] - Add Travis CI
11aa5d9
ZacBlanco added a commit that referenced this issue Nov 7, 2016
@ZacBlanco
[#3] - Fix travis build - fix unittest typo
f71193d
ZacBlanco added a commit that referenced this issue Nov 7, 2016
@ZacBlanco
[#3] - Attempt to fix hanging unittest
2065398
@ZacBlanco ZacBlanco mentioned this issue Nov 7, 2016
Merged
ZacBlanco added a commit that referenced this issue Nov 7, 2016
@ZacBlanco
[#3] - Fix bug where reused tag packets not overwritten
5d1e1ea
ZacBlanco added a commit that referenced this issue Nov 8, 2016
@ZacBlanco
[#3] - Improve on unstable unit test with threads
c9c5a14
ZacBlanco added a commit that referenced this issue Nov 8, 2016
@ZacBlanco
[#3] - Fixes for documentation
e131b19
ZacBlanco added a commit that referenced this issue Nov 8, 2016
@ZacBlanco
[#3] - Fix print methods for python3 compatibility
01eebec
ZacBlanco added a commit that referenced this issue Nov 8, 2016
@ZacBlanco
[#3] - Doc description for Consensus methods
a76c684
@ZacBlanco
Copy link
Owner Author

A good chunk of this work has been completed. cloud_comm.py seems somewhat stable. There are enough unit tests here to check a fair amount of the cases and I've also implemented travis builds.

Completed work:

  • full implementation of basic UDP-sequencing protocol
  • threaded listener to receive packets
  • unittests
  • CI
  • data storage scheme (this can still be improved upon)

TODO:

  • Implement the use of the new cloud_comm protocol in the consensus methods
  • Find a way to get IP addresses from MPI graph OR replace the MPI graph entirely with a graph library or other implementation.

@ZacBlanco
Copy link
Owner Author

Going to close this for now but I'm going to leave the last TODO bullet open

TODO:

  • Find a way to get IP addresses from MPI graph OR replace the MPI graph entirely with a graph library or other implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ZacBlanco