Communication Protocols in RADICAL Pilot

This is work in progress, under discussion!

Communication Types

There are 3 different communication types present in RP:

1. moving stateful entities through a network of state-enacting components (see State Management in RADICAL-Pilot)

• this communication type strongly influences overall RP performance
• the mechanism may differ, depending if communicating components are co-threads (1.a), co-processes (1.b), or remote processes (1.c)

2. sending state change notifications to persistent storage (currently MongoDB), and to consumers (async callbacks)

• this can influence performance in some cases, when timely reactions on those events are needed (late binding scheduler)
• persistent storage of events (2.1) and communication of events (2.2) can in principle be decoupled.
• the latter again differs in implementation, depending on co-locality of producer and consumer, and depending if the consumer is known or not (1.a-d)

3. sending log and profile information to persistent storage (currently file system)

• this is not performance critical, as long as it does not significantly slow down operation otherwise, and as long as the data reach persistent storage eventually.
• this is also optional, ie. can be turned on and off per application instance.

Old Implementation (pre-refactor)

• 1.a: push/pull over mongoDB (module), Queues (Agent)
• 1.b: push/pull over mongoDB (module), Queues (Agent)
• 1.c: push/pull over mongoDB
• 2.1.a: piggy-packed on 1.a
• 2.1.b: piggy-packed on 1.b
• 2.1.c: piggy-packed on 1.c
• 2.1.d: piggy-packed on 1.c
• 2.2.a: piggy-packed on 1.a
• 2.2.b: piggy-packed on 1.b
• 2.2.c: piggy-packed on 1.c
• 2.2.d: piggy-packed on 1.c
• 3.a: write to disk, stdout/stderr
• 3.b: n/a

Target Implementation (post-refactor)

• 1.a: threading.Queue
• 1.b: multiprocessing.Queue
• 1.c: ZeroMQ.Queue
• 2.1.a: ZeroMQ bridge toward MongoDB
• 2.1.b: ZeroMQ bridge toward MongoDB
• 2.1.c: ZeroMQ bridge toward MongoDB
• 2.1.d: n/a
• 2.2.a: ZeroMQ pub/sub
• 2.2.b: ZeroMQ pub/sub
• 2.2.c: ZeroMQ pub/sub
• 2.2.d: ZeroMQ pub/sub
• 3.a: write to disk, stdout/stderr
• 3.b: store in memory, dump on completion via ZeroMQ bridge to disk or MongoDB