New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PulsarCast M.Sc Thesis - Scaling libp2p PubSub #266

Open
JGAntunes opened this Issue Oct 12, 2017 · 13 comments

Comments

Projects
None yet
6 participants
@JGAntunes
Member

JGAntunes commented Oct 12, 2017

Greetings everyone!

I’m João Antunes, a grad student (M.Sc candidate) at the University of Lisbon, a full-time Software Engineer at YLD and one of the co-organizers of SINFO for the last 3 years. David Dias introduced me to the IPFS project a little more than a year ago and since then I've been following its developments and contributing to the JavaScript implementation with the chunks of free time I could find.

Now that I finished all my courses, I'm thrilled to share that I managed to excite and convince my Thesis advisor to let me pick IPFS and libp2p PubSub for my research. My goal is to scale IPFS PubSub to hundreds, if not thousands, of millions of participants 🙌🏽 with the hopes of getting some massing real-time games fully distributed. I'll be fortunate enough to be advised by David Dias(@diasdavid - Protocol Labs) and Luís Veiga(INESC-ID + University of Lisbon).

The timeline for M.Sc will encompass two major deliveries:

  • The first happening in the beginning of January (5th), also known as the project checkpoint, where I deliver the state of the art, literature review and proposed solution for the architecture to be built.
  • The second to be delivered in October 2018, where I deliver the full thesis, report and result.

Over the last couple of months, I've getting more and more familiar with IPFS internals and its API. I've tested it with a just a small PoC, a cli based chat app. I've also been reading the literature that I could find about distributed PubSub. Thank you for the compreensive list available at research-pubsub, I'll share new papers I find and my notes in the end too.

I'm currently working on porting Curvatron to IPFS and exploring the open source game space to find the perfect candidate to test my PubSub topologies and strategies. So far, my search lead me to agar.io, but I'm open to more suggestions.

I would love to get to know everyone involved and interested in this particular piece of work, share any relevant research or even just thoughts! I'm sure we'll be able to generate some relevant and healthy discussion around the P2P PubSub topic.

@nicola

This comment has been minimized.

Show comment
Hide comment
@nicola

nicola Oct 13, 2017

Member

Hey @JGAntunes,
awesome to read your drive in this p2p space :)
Decentralized Pubsub is going to be fundamental in designing the next generation of p2p apps.
I really look forward to your work, of course providing help or resources if needed - I encourage the whole community to do so!

It would be great if you could document here your steps of your research, so that we can all follow what is going on :) (I tried to attemp something like this during my masters: https://github.com/nicola/decentralized-research/issues)

I will check in here often :)

Member

nicola commented Oct 13, 2017

Hey @JGAntunes,
awesome to read your drive in this p2p space :)
Decentralized Pubsub is going to be fundamental in designing the next generation of p2p apps.
I really look forward to your work, of course providing help or resources if needed - I encourage the whole community to do so!

It would be great if you could document here your steps of your research, so that we can all follow what is going on :) (I tried to attemp something like this during my masters: https://github.com/nicola/decentralized-research/issues)

I will check in here often :)

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Oct 15, 2017

Member

Hey @JGAntunes, Welcome! Take a look at the recent writeup by @vyzo here: libp2p/research-pubsub#17

Member

whyrusleeping commented Oct 15, 2017

Hey @JGAntunes, Welcome! Take a look at the recent writeup by @vyzo here: libp2p/research-pubsub#17

@JGAntunes

This comment has been minimized.

Show comment
Hide comment
@JGAntunes

JGAntunes Oct 27, 2017

Member

Thanks everyone!

@nicola so far I'm catching up on literature, while also integrating the current floodsub implementation on Curvatron (for those interested - https://github.com/JGAntunes/curvatron - it's still in early stage, I'll make sure to share some more tangible results once I have them 😄), but I'll make sure to share as much as possible here! Thanks for the support 👍

@whyrusleeping thanks for pointing that out! Haven't read the whole RFC but I'll give it a more thorough look the next few days 👍

Member

JGAntunes commented Oct 27, 2017

Thanks everyone!

@nicola so far I'm catching up on literature, while also integrating the current floodsub implementation on Curvatron (for those interested - https://github.com/JGAntunes/curvatron - it's still in early stage, I'll make sure to share some more tangible results once I have them 😄), but I'll make sure to share as much as possible here! Thanks for the support 👍

@whyrusleeping thanks for pointing that out! Haven't read the whole RFC but I'll give it a more thorough look the next few days 👍

@dasilvacontin

This comment has been minimized.

Show comment
Hide comment
@dasilvacontin

dasilvacontin Oct 28, 2017

Hi @JGAntunes! Very briefly: I'm interested in multiplayer games and game networking in general. I'm currently building ag-drift (not related to ipfs) on my free time, planning to release an MVP soon and move on.

I'm currently at IPFS day Barcelona trying to see how I could apply this technology. Looking forward to your progress! :)

dasilvacontin commented Oct 28, 2017

Hi @JGAntunes! Very briefly: I'm interested in multiplayer games and game networking in general. I'm currently building ag-drift (not related to ipfs) on my free time, planning to release an MVP soon and move on.

I'm currently at IPFS day Barcelona trying to see how I could apply this technology. Looking forward to your progress! :)

@JGAntunes

This comment has been minimized.

Show comment
Hide comment
@JGAntunes

JGAntunes Nov 13, 2017

Member

@dasilvacontin awesome! I've been working on my free time on - https://github.com/JGAntunes/curvatron - to understand the constraints around multiplayer games. Most of this is quite new to me, as I never guessed multiplayer synchronisation would turn out to be such a pain 🤦‍♂️

Been reading a bit about synchronising multiplayer games and other gamedev "lingo" (e.g. https://www.reddit.com/r/Overwatch/comments/3u5kfg/everything_you_need_to_know_about_tick_rate/ and https://gamedev.stackexchange.com/questions/6645/lag-compensation-with-networked-2d-games) and working around simple solutions for Curvatron, it has been mostly a work in progress but I'm getting somewhere (hopefully)

peek 2017-11-13 01-40
This is running locally but tested it out remotely with relative success synchronisation wise.

There's a lot of stuff though that I'm not even considering here nor have I investigated anything around it, such as consensus around game state (something that would be trivial with a server based solution).

Member

JGAntunes commented Nov 13, 2017

@dasilvacontin awesome! I've been working on my free time on - https://github.com/JGAntunes/curvatron - to understand the constraints around multiplayer games. Most of this is quite new to me, as I never guessed multiplayer synchronisation would turn out to be such a pain 🤦‍♂️

Been reading a bit about synchronising multiplayer games and other gamedev "lingo" (e.g. https://www.reddit.com/r/Overwatch/comments/3u5kfg/everything_you_need_to_know_about_tick_rate/ and https://gamedev.stackexchange.com/questions/6645/lag-compensation-with-networked-2d-games) and working around simple solutions for Curvatron, it has been mostly a work in progress but I'm getting somewhere (hopefully)

peek 2017-11-13 01-40
This is running locally but tested it out remotely with relative success synchronisation wise.

There's a lot of stuff though that I'm not even considering here nor have I investigated anything around it, such as consensus around game state (something that would be trivial with a server based solution).

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Nov 13, 2017

Member

I found this paper pretty insightful in the past: http://web.cs.wpi.edu/~claypool/courses/4513-B03/papers/games/bernier.pdf

Its by one of valves lead network engineers and discusses various strategies of interpolation for multiplayer realtime video games.

Member

whyrusleeping commented Nov 13, 2017

I found this paper pretty insightful in the past: http://web.cs.wpi.edu/~claypool/courses/4513-B03/papers/games/bernier.pdf

Its by one of valves lead network engineers and discusses various strategies of interpolation for multiplayer realtime video games.

@JGAntunes

This comment has been minimized.

Show comment
Hide comment
@JGAntunes

JGAntunes Jan 27, 2018

Member

Hey everyone! It's been a while since my last update (my bad...) so, here it is 🎉.

Over the past few months I've been working on the initial report due to this month where I tackled:

  • State of the art and literature review
  • Proposed solution

I've successfully delivered it and I'm now working on its presentation, due to the beginning of next month. Hopefully, after that, I'll be able to share and open source all the work I've done so far. Including the relevant systems I've addressed, proposed solution, etc. Something I'm really looking forward to 😄

After all of this, I'll start developing the solution itself 🙌 due to October.

Once again thanks for all the help and pointers everyone provided so far! Hope to be able to share all of this with you in the upcoming weeks 😄

Member

JGAntunes commented Jan 27, 2018

Hey everyone! It's been a while since my last update (my bad...) so, here it is 🎉.

Over the past few months I've been working on the initial report due to this month where I tackled:

  • State of the art and literature review
  • Proposed solution

I've successfully delivered it and I'm now working on its presentation, due to the beginning of next month. Hopefully, after that, I'll be able to share and open source all the work I've done so far. Including the relevant systems I've addressed, proposed solution, etc. Something I'm really looking forward to 😄

After all of this, I'll start developing the solution itself 🙌 due to October.

Once again thanks for all the help and pointers everyone provided so far! Hope to be able to share all of this with you in the upcoming weeks 😄

@JGAntunes

This comment has been minimized.

Show comment
Hide comment
@JGAntunes

JGAntunes Feb 16, 2018

Member

Hey everyone!

What's new

So, great news, I just open sourced the first part of my work (and consequently all that will follow) 🎉 You can check it out over here - https://github.com/JGAntunes/pulsarcast.

You can read the full report here and the presentation here. So far I've already collected valuable feedback from my presentation, listed here, but I would sure be glad to hear what everyone thinks and it would be definitely helpful for my final report due to October.

As for the solution itself (section 4 of the report), you'll notice we've moved the focus of it to a more reliable system, focusing on giving persistence, consistency and delivery guarantees. Over the next months we'll work on having a pub-sub PulsarCast module for IPFS, so stay tuned 😄 I'll try to leave regular feedback here so that everyone that's interested can follow the work accordingly.

Next steps

I've started looking into ways of building a solid test harness for PulsarCast as per #280 and JGAntunes/pulsarcast#26. This is probably what I'll be looking at for the next two weeks. Let me know if you have/know something that might help 😄

Help

Besides the report feedback and the test harness I'll be on the lookout for more recent relevant work in the pub-sub field, so if you have knowledge of papers or research that might help me, please, send it my way 🙌

And that's it! See you in a few weeks 👋

Member

JGAntunes commented Feb 16, 2018

Hey everyone!

What's new

So, great news, I just open sourced the first part of my work (and consequently all that will follow) 🎉 You can check it out over here - https://github.com/JGAntunes/pulsarcast.

You can read the full report here and the presentation here. So far I've already collected valuable feedback from my presentation, listed here, but I would sure be glad to hear what everyone thinks and it would be definitely helpful for my final report due to October.

As for the solution itself (section 4 of the report), you'll notice we've moved the focus of it to a more reliable system, focusing on giving persistence, consistency and delivery guarantees. Over the next months we'll work on having a pub-sub PulsarCast module for IPFS, so stay tuned 😄 I'll try to leave regular feedback here so that everyone that's interested can follow the work accordingly.

Next steps

I've started looking into ways of building a solid test harness for PulsarCast as per #280 and JGAntunes/pulsarcast#26. This is probably what I'll be looking at for the next two weeks. Let me know if you have/know something that might help 😄

Help

Besides the report feedback and the test harness I'll be on the lookout for more recent relevant work in the pub-sub field, so if you have knowledge of papers or research that might help me, please, send it my way 🙌

And that's it! See you in a few weeks 👋

@vyzo

This comment has been minimized.

Show comment
Hide comment
@vyzo

vyzo Feb 16, 2018

@JGAntunes you may be interested in this: https://github.com/vyzo/gerbil-simsub
This is gossipsub, the next iteration in libp2p pubsub implementation; this is work in progress, and the Go implementation should be complete before you start your protocol implementation (per your report schedule), so you may want to take it into account.

vyzo commented Feb 16, 2018

@JGAntunes you may be interested in this: https://github.com/vyzo/gerbil-simsub
This is gossipsub, the next iteration in libp2p pubsub implementation; this is work in progress, and the Go implementation should be complete before you start your protocol implementation (per your report schedule), so you may want to take it into account.

@diasdavid diasdavid changed the title from P2P PubSub M.Sc Thesis to PulsarCast M.Sc Thesis - Scaling libp2p PubSub Feb 19, 2018

@diasdavid diasdavid added the pubsub label Feb 20, 2018

@JGAntunes

This comment has been minimized.

Show comment
Hide comment
@JGAntunes

JGAntunes Feb 26, 2018

Member

Hey @vyzo! Really sorry for the late reply. Thanks for pointing that out, gossipsub looks quite interesting. I'll make sure to give it a thorough look the next couple of days. Btw, would love to know a bit more about libp2p/research-pubsub#17 (I assume this is a different proposal from gossipsub right?), has there been any implementation of it already?

@diasdavid has pointed out a number of times that I should speak with you, since you've been doing quite an awesome work on the pub-sub front of libp2p and ipfs 😄. So, any feedback on the solution itself, or relevant literature on pub-sub systems that could help me out (or even any generic advice to be honest 😄) would be very much appreciated.

Thank you very much once again!

Member

JGAntunes commented Feb 26, 2018

Hey @vyzo! Really sorry for the late reply. Thanks for pointing that out, gossipsub looks quite interesting. I'll make sure to give it a thorough look the next couple of days. Btw, would love to know a bit more about libp2p/research-pubsub#17 (I assume this is a different proposal from gossipsub right?), has there been any implementation of it already?

@diasdavid has pointed out a number of times that I should speak with you, since you've been doing quite an awesome work on the pub-sub front of libp2p and ipfs 😄. So, any feedback on the solution itself, or relevant literature on pub-sub systems that could help me out (or even any generic advice to be honest 😄) would be very much appreciated.

Thank you very much once again!

@vyzo

This comment has been minimized.

Show comment
Hide comment
@vyzo

vyzo Feb 26, 2018

Hi @JGAntunes !

The Go gossipsub implementation is progressing, there is an open pr now: libp2p/go-floodsub#67

The libp2p/research-pubsub#17 proposal is more of an endgame proposal, with the epidemic optimizations we would like to implement on top of gossipsub (with the next iteration, tentatively dubbed episub). Epidemic optimizations don't work well when you have many or random sources, so it's a more specialized protocol. Gossipsub is designed to be a general purpose, scalable protocol with good robustness and latency properties.

If I may offer some advice, I would point you to focus on the reliability/persistence part of pubsub, and not so much into building the overlay. gossipsub is a lot more scalable than floodsub, and it should serve you well as an overlay basis.

We can also arrange a call if you woud like to discuss.

vyzo commented Feb 26, 2018

Hi @JGAntunes !

The Go gossipsub implementation is progressing, there is an open pr now: libp2p/go-floodsub#67

The libp2p/research-pubsub#17 proposal is more of an endgame proposal, with the epidemic optimizations we would like to implement on top of gossipsub (with the next iteration, tentatively dubbed episub). Epidemic optimizations don't work well when you have many or random sources, so it's a more specialized protocol. Gossipsub is designed to be a general purpose, scalable protocol with good robustness and latency properties.

If I may offer some advice, I would point you to focus on the reliability/persistence part of pubsub, and not so much into building the overlay. gossipsub is a lot more scalable than floodsub, and it should serve you well as an overlay basis.

We can also arrange a call if you woud like to discuss.

@JGAntunes

This comment has been minimized.

Show comment
Hide comment
@JGAntunes

JGAntunes Mar 2, 2018

Member

Thanks for the overview @vyzo, would love to schedule a quick call to discuss this if you have time. Tomorrow my availability won't be that much, but after that almost any day will suit me. Feel free to reach me out via e-mail to me(at)jgantunes.com so we can arrange something 👍

Member

JGAntunes commented Mar 2, 2018

Thanks for the overview @vyzo, would love to schedule a quick call to discuss this if you have time. Tomorrow my availability won't be that much, but after that almost any day will suit me. Feel free to reach me out via e-mail to me(at)jgantunes.com so we can arrange something 👍

@JGAntunes

This comment has been minimized.

Show comment
Hide comment
@JGAntunes

JGAntunes Aug 6, 2018

Member

Greetings everyone!

I know I've been lacking a proper update for a while. So, to everyone following the work happening here, I hope these next few lines make up for my lack of feedback over the past couple of months 🙏

What's new

I've setup a test harness using containernet. It's in a very early stage but it works for my needs right now, as it allows me to run any kind of docker image, including my custom JS-IPFS and IPFS images. Hopefully in a near future I'll have some time to document, modularise and publish all of this in NPM accordingly, as I think it might be useful to more people 😄 meanwhile feel free to drop in any feedback you might have. I must confess I'll probably won't have much time to address it but I promise you won't be left without an answer 👍

More importantly, work as started on the actual javascript implementation of Pulsarcast. Again, it's a WIP and so far it has been just coding the spec. Has always, feel free to drop in and leave some feedback 🙌 I know it's a bit shy on documentation for now 🙈, but I promise to work on that SOON™.

Next steps

The focus right now is to Javascript the hell out of js-pulsarcast! 🙌 Once that is done (or mostly done, since "Javascripts" are never done), some adjustments will need to happen to reduce the memory footprint of our test-harness, as we're hopping to be able to spin up a few hundred IPFS nodes to be able to get some meaningful data.

And that's it, hopefully this time it won't be too long before I drop some lines here 🙈

Member

JGAntunes commented Aug 6, 2018

Greetings everyone!

I know I've been lacking a proper update for a while. So, to everyone following the work happening here, I hope these next few lines make up for my lack of feedback over the past couple of months 🙏

What's new

I've setup a test harness using containernet. It's in a very early stage but it works for my needs right now, as it allows me to run any kind of docker image, including my custom JS-IPFS and IPFS images. Hopefully in a near future I'll have some time to document, modularise and publish all of this in NPM accordingly, as I think it might be useful to more people 😄 meanwhile feel free to drop in any feedback you might have. I must confess I'll probably won't have much time to address it but I promise you won't be left without an answer 👍

More importantly, work as started on the actual javascript implementation of Pulsarcast. Again, it's a WIP and so far it has been just coding the spec. Has always, feel free to drop in and leave some feedback 🙌 I know it's a bit shy on documentation for now 🙈, but I promise to work on that SOON™.

Next steps

The focus right now is to Javascript the hell out of js-pulsarcast! 🙌 Once that is done (or mostly done, since "Javascripts" are never done), some adjustments will need to happen to reduce the memory footprint of our test-harness, as we're hopping to be able to spin up a few hundred IPFS nodes to be able to get some meaningful data.

And that's it, hopefully this time it won't be too long before I drop some lines here 🙈

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment