Objective Decision Making

The Process

1. State the problem
2. Engage the other stakeholders
3. Define the qualification criteria
4. Define the selection criteria
5. Weight the selection criteria
6. Outline the candidate solutions
7. Outline the excluded solutions
8. Qualify the candidate solutions
9. Rate the candidate solutions
10. Apply the selection criteria weightings

State the problem

A clear description of the problem, and the benefits expected by solving it, e.g.

We need a general purpose message broker to support an event driven architecture that will improve decoupling, reliability and performance.

Engage the other stakeholders

Making a good decision usually requires considering the perspective of other people. Identify the key stakeholders and involve them in the processes. e.g.

• Head of Platform
• Head of SRE
• Heads of Engineering
• Lead Engineers

Define the qualification criteria

Qualify out solutions early by defining a set of qualification (yes/no) criteria. For example

• Must be compatible with Node.js and .Net
• Must be configurable (including queues & topics) via Terraform
• Must offer at-least-once message delivery
• Must offer at-most-once message delivery
• Must support broadcast to multiple consumers
• Must support local testing and development through Docker
• Must operate in AWS
• Must integrate with Datadog
• Must support secure messages while in transport

Define the selection criteria

Define a set of criteria that will be used to compare the relative merits of each candidate solution. If there are many, organise them into a hierarchy. For example

• Features
  • Timed message expiry
  • Capped redelivery
  • Max message size
  • Message filtering
  • Message validation
  • Message deduplication
  • Guaranteed ordering
  • Delayed delivery
  • Dead Letter Queues
• Cost
• Performance
  • Message throughput
• Support Experience
  • Documentation qualify
  • Issue responsiveness
  • Community engagement
  • Release frequency
  • Ease of configuration / deployment
  • Depth of observability
• Developer Experience
  • Learning curve
  • API Simplicity
  • Quality of client libraries
  • Local setup

Weight the selection criteria

Weight the importance of each acceptance criteria in relation to each other by created a matrix for each sub criteria. So if Message filtering was 20 times as important as Timed message expiry it would be weighted 2 but if it were only half as important it would be weighted 1/2. For example

Features

	Timed message expiry	Capped redelivery	Max message size	Message filtering	Message validation	Message deduplication	Guaranteed ordering	Delayed delivery
Timed message expiry	1.0	0.3	0.5	0.2	10.0	10.0	100.0	2.0
Capped redelivery	4.0	1.0	2.0	0.8	40.0	40.0	400.0	8.0
Max message size	2.0	0.5	1.0	0.4	20.0	20.0	200.0	4.0
Message filtering	5.0	1.3	2.5	1.0	50.0	50.0	500.0	10.0
Message validation	0.1	0.0	0.1	0.0	1.0	1.0	10.0	0.2
Message deduplication	0.1	0.0	0.1	0.0	1.0	1.0	10.0	0.2
Guaranteed ordering	0.0	0.0	0.0	0.0	0.1	0.1	1.0	0.0
Delayed delivery	0.5	0.1	0.3	0.1	5.0	5.0	50.0	1.0

Next calculate the product for each criteria Then calculate the geometric mean for each criteria by taking the N^th root of the product Sum the geometric means Calculate the weighting by dividing the geometric means by the sum

	Product	Geometric Mean	Weighting
Timed message expiry	500	2.2	7.87
Capped redelivery	32768000	8.7	31.47
Max message size	128000	4.3	15.74
Message filtering	0	10.9	39.34
Message validation	0	0.2	0.79
Message deduplication	0	0.2	0.79
Guaranteed ordering	0	0	0.08
Delayed delivery	2	1.1	3.93
		27.6

Repeat this exercise for each set of sub criteria, then repeat for the top level criteria. This should result in a matrix similar to the following

	Features	Cost	Performance	Support	Developer Experience	Product	Geometric Mean	Weighting
Features	1.0	2.0	4.0	1.0	0.5	4.0	1.2	21.69%
Cost	0.5	1.0	2.0	0.5	0.3	0.1	0.8	14.06%
Performance	0.3	0.5	1.0	0.3	0.1	0.0	0.5	9.12%
Support	1.0	2.0	4.0	1.0	0.5	4.0	1.2	21.69%
Developer Experience	2.0	4.0	8.0	2.0	1.0	128.0	1.8	33.45%
							5.4835

Outline the candidate solutions

List each of the candidate solutions. For example

1. RabbitMQ
2. Kafka
3. SNS + SQS
4. EventBridge

When listing a custom solution you should also provide an overview.

Outline the excluded solutions

If there are too many alternatives to evaluate and you will have a good reason to believe that some will be less attractive than others, then it makes sense to exclude them, but record which one were excluded and why.

1. ActiveMQ - Heavily based on JMS
2. IBM MQ - Expensive, no hosted service in AWS
3. Redis - Limited functionality

Qualify the candidate solutions

	Node.js	.Net	Terraform	at-least-once	at-most-once	Broadcast	Docker	AWS	Datadog	TLS
RabbitMQ	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y
Kafka	Y	Y	Y	Y	N	-	-	-	-	-
SNS+SQS	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y
EventBridge	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y

Rate the candidate solutions

Award each of the remaining candidates a score between 0 (bad) and 10 (good) for each the selection criteria. A candidate which over-achieves for one criteria should not reduce the score of another that also performs well. For example, both RabbitMQ and SNS+SQS may both receive a 10 for message throughput even though RabbitMQ is faster than SNS+SQS.

Features

	Timed message expiry	Capped redelivery	Max message size	Message filtering	Message validation	Message deduplication	Guaranteed ordering	Delayed delivery	Dead letter queues
RabbitMQ	7	5	10	10	0	0	10	10	10
SNS+SQS	5	10	3	10	0	10	5	5	10
EventBridge	5	10	3	10	10	10	5	5	10

Cost

	Cost
RabbitMQ	7
SNS+SQS	8
EventBridge	5

Apply the selection criteria weightings