Found while giving graph-bench a zu column for the linkbench write operations.
zu now reports numbers for eight of the ten linkbench operations. It beats ladybug on every read by 2x to 8x, in process, same machine, same dataset. It loses on every write by 60x to 110x. p50 in milliseconds at smoke scale, 1000 objects and 5342 associations:
| query |
zu |
ladybug |
| lb-add-node |
446.487 |
4.056 |
| lb-add-link |
295.083 |
4.167 |
| lb-update-link |
294.136 |
5.003 |
| lb-delete-link |
318.127 |
5.027 |
That is not fsync and it is not the harness. It is CPU, and it is proportional to the number of bytes already in the store rather than to the size of the change.
Twenty bare inserts of one row each, timed with /usr/bin/time, against stores built by zu copy. First column varies the string payload width at a fixed 1000 nodes and 5000 edges, second varies the edge count at a fixed one byte payload. The figure is user CPU per insert:
| payload |
CPU per insert |
edges |
CPU per insert |
| 1 byte |
7.5ms |
1000 |
1.5ms |
| 8 bytes |
21ms |
2500 |
3.5ms |
| 128 bytes |
42ms |
5000 |
7.5ms |
| 512 bytes |
56ms |
10000 |
17.5ms |
Both dimensions are close to linear, and the same twenty inserts against a 50 node store cost 1ms each in total. Against the real linkbench store, which carries wide payloads on both tables and comes to about 10 MiB on disk, they cost 170ms each. That works out to roughly 60 MiB per second of rewrite, which is about what it costs to walk every property column and every CSR in the file.
So the shape is: a commit folds the appended rows in by rebuilding the tables they touch, whole. The cost of writing one row is the cost of reading and rewriting everything already there. On a 10 MiB store that is a fifth of a second. On a gigabyte store it is not a write path at all.
What would fix it is folding incrementally, or deferring the fold so a commit only has to make the appended rows durable and readable and the rebuild happens in the background or on a threshold. Either one turns write latency into a function of the change rather than a function of the database.
This is the single largest gap between zu and ladybug anywhere in graph-bench right now, and it is the one that keeps zu out of every write workload at any scale above smoke. graph-bench gate/budgets.go sets Write and InProc p99 at 2ms for SF1, which zu currently misses by two orders of magnitude.
Found while giving graph-bench a zu column for the linkbench write operations.
zu now reports numbers for eight of the ten linkbench operations. It beats ladybug on every read by 2x to 8x, in process, same machine, same dataset. It loses on every write by 60x to 110x. p50 in milliseconds at smoke scale, 1000 objects and 5342 associations:
That is not fsync and it is not the harness. It is CPU, and it is proportional to the number of bytes already in the store rather than to the size of the change.
Twenty bare inserts of one row each, timed with /usr/bin/time, against stores built by zu copy. First column varies the string payload width at a fixed 1000 nodes and 5000 edges, second varies the edge count at a fixed one byte payload. The figure is user CPU per insert:
Both dimensions are close to linear, and the same twenty inserts against a 50 node store cost 1ms each in total. Against the real linkbench store, which carries wide payloads on both tables and comes to about 10 MiB on disk, they cost 170ms each. That works out to roughly 60 MiB per second of rewrite, which is about what it costs to walk every property column and every CSR in the file.
So the shape is: a commit folds the appended rows in by rebuilding the tables they touch, whole. The cost of writing one row is the cost of reading and rewriting everything already there. On a 10 MiB store that is a fifth of a second. On a gigabyte store it is not a write path at all.
What would fix it is folding incrementally, or deferring the fold so a commit only has to make the appended rows durable and readable and the rebuild happens in the background or on a threshold. Either one turns write latency into a function of the change rather than a function of the database.
This is the single largest gap between zu and ladybug anywhere in graph-bench right now, and it is the one that keeps zu out of every write workload at any scale above smoke. graph-bench gate/budgets.go sets Write and InProc p99 at 2ms for SF1, which zu currently misses by two orders of magnitude.