High performance message collection and shema managment
- Collector is a high performance generic message collection service. It receives messages from a watch through a client-streaming grpc connection.
- Schematic Layer registers and maintains message schemas. It allows services to decode unknown messages at run time and verify their integrity.
- ConsumerPy is using generated code to consume the messages.
- ConsumeGo is not aware of the schema of consumed messages. It is able to deserialize and manipulate them by fetching their file descriptor from the schematic layer.
left: consumerGo, middle: watchExporter App, right: consumerPy
- Apple Watch app establishes client-sctreaming grpc connection with collector.
- Collector, consumer, and schematic layer establish connection with the kafka broker.
- Biometric message schema (file descriptor) is registered with the schematic layer.
- ConsumeGo fetches biometric schema.
- Apple Watch app begins streaming biometric data.
- ConsumerPy receives messages and deserializes them with pre-generated code.
- ConsumerGo receives messages and deserializes them with previously fetched schema.
Test Details:
- kafka: 1 partition, 2 replicas,
- 1 collector
- 0.5 sec collector flush rate
- collector waits for leader acknowledgement
- each kafka broker sits on independent n1-standard node
- there are 10 clients, each on their independent n1-standard-2, each with up to 2500 connections, thus in total there are 25 000 connections to collector
- collector runs on n1-standard-4 (4 vCPU, 15 GB memory) and n1-standard-8 (8 vCPU, 30 GB memory)
- time is measured to collect 10 million ~126-254 byte message
1 collector running on n1-standard-4 (GKE)
| Concurrent Connections | 2.5e2 | 2.5e3 | 2.5e4 |
|---|---|---|---|
| Test completion times | 45.2117 | 53.2022 | 63.6975 |
| (seconds) | 44.8457 | 52.2528 | 62.9623 |
| 45.3871 | 51.9387 | 61.9285 | |
| 46.3244 | 52.8396 | 63.6561 | |
| Average | 45.442225 | 52.558325 | 63.0611 |
1 collector running on n1-standard-8 (GKE)
| Concurrent Connections | 2.5e2 | 2.5e3 | 2.5e4 |
|---|---|---|---|
| Test completion times | 31.8510 | 32.1720 | 38.1253 |
| (seconds) | 30.3364 | 33.3108 | 36.5358 |
| 30.2292 | 32.1427 | 38.3180 | |
| 30.5378 | 32.3915 | 36.7096 | |
| Average | 30.7386 | 32.5042 | 37.4221 |