Repository navigation
Portability
A deployed system may have N targets talking over TCP. During development you may have one machine. gridlane lets you run the same binaries in both settings; only the configuration file changes.
The same job, 8 workers, run two ways:
| Deployment A | Deployment B | |
|---|---|---|
| Layout | 2 targets × 1 application × 4 workers | 1 target × 2 applications × 4 workers |
--layout |
0:0:4,1:0:4 |
0:0:4,0:1:4 |
| Between applications | TCP | POSIX shared memory |
| Result | agrees with the Python reference | agrees with the Python reference |
The final fields of A and B are equal byte for byte. scripts/portability_demo.sh runs both and compares them, and CTest runs it on every build.
flowchart LR
subgraph A["A: two targets"]
direction TB
a0["target 0<br/>app 0: ranks 0-3"]
a1["target 1<br/>app 0: ranks 4-7"]
a0 <-->|TCP| a1
end
subgraph B["B: one target"]
direction TB
b0["app 0: ranks 0-3"]
b1["app 1: ranks 4-7"]
b0 <-->|POSIX shm| b1
end
pushseg reads one configuration file shared by all processes. Deployment A, over TCP:
[apps]
coordinator = 1
worker = 2
[node.0]
tcp = 10.0.0.10:10000
[node.1]
tcp = 10.0.0.11:10000
[links]
default = tcpDeployment B, on one machine over shared memory:
[apps]
coordinator = 1
worker = 2
[node.0]
shm =
[links]
default = shm
[backend.shm]
name_prefix = /gridlaneThe backend is chosen per pair of nodes, so a system can mix them: shared memory inside a machine, TCP between machines, a device API where a device exists.
- The algorithm: it sees ranks, buffers and flags.
- The control messages: the same generated codecs, the same pairring channel.
- The binaries:
gridlane_workerandgridlane_coordinatorare not rebuilt.
Each cell of the computation goes through the same operations in the same order whatever the split, and sums are combined in member order. The transport only moves bytes, it never changes them. So any deployment of the same job gives the same bits; the distributed run is also bit-identical to a single-threaded run.
gridlane
Repositories