The One with the Plan

[linki]

Background

When developing mate we found out that one of the more difficult parts was calculating the diff from a current state to a desired state of dns records. In mate, this responsibility was part of each consumer implementation although many things could have been shared between consumers and it was difficult to test.

Glossary:

• producer: source of desired dns records
• consumer: adapter to the specific dns provider
• controller: glue between producers and consumers

Proposal

We propose an intermediate object that's used to capture the transition that's needed to go from a current to a desired state of dns recors in a dns-provider-independent way. This object would then be used as input to the consumers and the logic is not part of each of them.

Hence, making the consumer implementations easier to write (less responsibilities) and allowing the intermediate object to be easier to test (less external dependencies). I would like to call this object Plan as it's a sort of execution plan.

A plan would be constructed from two input lists:

• a desired list of records and a current list of records.

And it would return three lists as output:

• a list of records to create
• a list of records to update (including old and new records)
• a list of records to delete

This plan object can be tested in isolation and many combinations and edge cases can easily be added to the test suite.

The plan would then be passed to a consumer object which would "just execute" the actions according to the dns provider. This also makes each consumer easier to test as one just needs to provide a Plan and see that the consumer runs the right actions against the dns provider. Any errors that may happen (e.g. because of records to delete that don't exist) can be considered errors and don't need to be handled necessarily.

The flow would be the following:

• controller asks producer to return a list of desired dns records
• controller asks consumer for the current list of dns records
• controller passes list from producer (desired) and list from consumer (current) to plan calculator and retrieves a Plan
• controller passes Plan to consumer for execution

We chose to represent the necessary actions as three lists as it seems to be the most compatible way of defining it between dns providers. For instance, AWS Route53 doesn't differentiate between creates and updates so it could just merge them into an upsert list before processing. On the other hand, Google CloudDNS doesn't support updates natively and one would have to convert the updates list to a combination of delete+create first. Other consumers will be different but we think those three lists capture all the information they would require.

Inspired by work we had planned for mate: linki/mate#46

[linki]

/cc @ideahitme @justinsb @iterion

[ideahitme]

awesome write-up! 👍

this approach will allow us to have a standardised way for managing dnsprovider state and make plugging new dnsprovider fairly simple (in theory :P)

[iterion]

I really like the idea.

I recall from a Slack convo with @justinsb (unfortunately that history is gone now) that one user has many thousands of dns records. So, in the back of our minds we should probably think about how we can do this efficiently at that scale. Perhaps we should target some specific SLA up front?

[Raffo]

+1 on the great write up and I like the idea.

[ideahitme]

@iterion yes this is definitely something we should consider. Testing for such scenarios is something we should do as well :)

[LappleApple]

+1 @linki, well done

[linki]

Thanks for the +1s :-)

We implemented the first working version of Plan in #26.