Skip to content

Conversation

@sneakers-the-rat
Copy link
Collaborator

@sneakers-the-rat sneakers-the-rat commented Oct 28, 2025

Behold: a p2p multiprocessing runner.

So we can't really just use a process pool or concurent futures or whatever because we
a) don't want to and
b) can't
pickle every node every time we want to call it.

we also want to eventually be able to run things on different machines.

so here's that: controlled p2p node runner.

  • one command node is basically the "tracker" that starts the other nodes and connects them to each other
  • then each node runs in its own process, subscribes to the other nodes that it consumed events, from, runs, then poops out more events.

it's pretty dang simple.

so this is still a WIP, listing todos:

  • outer ZMQRunner class that creates the CommandNode and spawns the NodeRunners
  • handling return nodes: subscribe to the returned events and yield as a generator
  • main loop inside of NodeRunner that just pulls from the event store as event combinations are ready for a given epoch
  • teardown
  • ser/des for non-jsonable events
  • testing baby
  • Allow EventStore.add() to take an already-created event, rather than creating the event.

caveats:

  • uses IPC so won't work on windows
  • intended for multiprocessing on one machine for now, will do multi-machine in another stage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants