Jor-El is the service extends Google Pub/Sub with some new features:
-
Routing - the message can be republished in the different topics based on their meta information.
-
Scheduling - the message can be republished with some delay.
-
Debounce - messages with the same aggregation-id can be debounced. It means, will be republished ONLY the last message (or first) (not implemented yet).
-
Aggregation - messages with the same aggregation-id can be joined. It means, will be republished one aggregated message contains an array of original messages (not implemented yet).
-
create GCP project (or use existing) and download the service account key.
-
you need the following topics and subscriptions:
- ingress-topic topic with ingress-subs subscription;
- default-topic topic with default-topic-subs subscription;
- cancel-topic topic with cancel-topic-subs subscription;
-
activate Cloud SQL with PostgresSQL 11 (documentation https://cloud.google.com/sql/docs/postgres/quickstart)
-
create two databases: one for the production service, one for the tests
-
create tables using create_tables.sql script
-
set environment variables:
-
GOOGLE_APPLICATION_CREDENTIALS = path_to_the_account_key
-
JOR_EL_POSTGRES_URL - production database connection string (something like postgresql://user:password@host:port/database)
-
JOR_EL_POSTGRES_TEST_URL - test database connection string
-
run tests go test ./...
-
run test receiver make run-receiver
-
run jor-el make run-jorel (or run several jorel instances in different terminals windows)
-
run test publisher make run-publisher
config.yaml
---
project-id: jorel-test-project
ingress-subscription: ingress-subs
default:
topic-name: default-topic
aggregation: no
routing:
- message-type: cancel
topic-name: cancel-topic
aggregation: no
The first two lines are pretty obvious: GCP project name and jor-el ingress subscription ID.
default section is the configuration for the jor-el default egress topic. jor-el handles all messages without message type or with unknown message type according to this section rules. It's a mandatory section.
routing - section with routing rules. Optional.
The ‘Meta Message’ (or jor-el ingress message) wraps the ‘Target Message’ as jor-el only needs the bytes that will be sent and the name of the queue, the ‘Target Queue’. This way jor-el doesn’t need to know anything about the content of that ‘Target Message’.
Meta information is passing through message attributes:
- delay - message delay in seconds. Mandatory attribute. It can be 0, it means - republish immediately.
- message-type - using for the routing (if needed). Optional.
- aggregation-id - using for the debounce and aggregation. Optional. Not implemented yet.
- Create GKE cluster
- Install secrets:
kubectl create secret generic account-credentials --from-file=service-key.json=service-key.json
kubectl create secret generic db-credentials --from-literal=connection=<DB_CONNECTION_STRING>
Build Docker image & generate deployment:
./scripts/generate_deploy.py
Deploy Jor-El on Kubernetes (deployment is being generated on the previous step):
./run_deployment.sh
Check deployment, view logs, scale if needed:
kubectl get pods
kubectl logs -f jorel-deployment-6f8dd87f4f-msp5q jorel // you have to specify container because two containers deployed in one pod
kubectl scale deployment.v1.apps/jorel-deployment --replicas=3
Scale cluster into 5 pods:
kubectl scale deployment.v1.apps/jorel-deployment --replicas=5
Run two receivers locally in different terminals:
make run-receiver // receiver for the default topic
make run-receiver2 // receiver for the cancel topic
Open /cmd/publisher/main.go and update tests loop count:
for repeat := 0; repeat < 500; repeat++ { // 100 -> 500
for indx, d := range delays {
Run publisher and wait about 20 minutes:
make run-publisher2 // publish messages with two different types
Publisher & receivers are being run locally, so local configuration is important: MacBook Pro 2,3 GHz 8-Core Intel Core i9 32 GB 2400 MHz DDR4
Publisher sent in 10000 messages in 8m 23s.
Receiver 1:
- received 4986 messages, max deviation 2.5s, mean deviation 1.5s
Receiver 2:
- received 5014 messages, max deviation 3.3s, mean deviation 1.5s
All messages were delivered, the scheduling accuracy is acceptable.