Skip to content

NEW FEATURE: diff2: A better "diff" mechanism#1852

Merged
tlimoncelli merged 38 commits into
masterfrom
tlim_newdiff2
Dec 11, 2022
Merged

NEW FEATURE: diff2: A better "diff" mechanism#1852
tlimoncelli merged 38 commits into
masterfrom
tlim_newdiff2

Conversation

@tlimoncelli
Copy link
Copy Markdown
Contributor

This PR adds an experimental diffing algorithm called "diff2" which will some day replace the original "diff" algorithm. For now this feature is hidden behind a flag. Once it is stable, it will become the default and the original will be removed.

The feature is hidden behind the flag --diff2 (or -diff2 when running integration tests).

Diff2 is a complete rewrite from scratch. The goals are (1) make IGNORE*() and related features more reliable, (2) make it easier to write providers by providing a more simple API, (3) add a general mechanism (UNMANAGED()) which replaces the IGNORE*() functions (the IGNORE*() functions will still exist, they'll just call UNMANAGED())

diff2 is significantly less complex than the original diff. How did we achieve this? The old system worked with the data as it exists in the models.Records structure, which is difficult to work with. diff2 starts by gathering all the DNS records into a structure that is more friendly to the work that needs to be done. Now the differencing algorithm becomes easier to write. In other words, instead of solving a hard problem, we first re-arrange the data so that the original problem is easier.

diff2 is significantly easier for the providers to use. The old system returned a list of adds, changes and deletes and expected the provider to re-arrange the data into groups that matched how the API expects updates to be done. The new system provides 4 entry-points covering the 4 types of APIs we've seen in the wild: ByRecord, ByRSet, ByLabel and ByZone. They all return a list of "instructions" that simply need to be translated into API calls.

Timeline:

Since the old pkg/diff and pkg/diff2 can co-exist, there's no rush to convert all providers. However, I don't want to delay too much. I will probably convert most providers myself or at least send draft PRs to provider-owners. There will be at least 1 release with the --diff2 flag defaulting off, so that all users can test the new system before it becomes the default. After `diff2 becomes the default, the old code will disappear shortly after. I hope to have this completed by February or March of 2023. Maybe sooner? We'll see!

@tlimoncelli tlimoncelli merged commit 54fc2e9 into master Dec 11, 2022
@tlimoncelli tlimoncelli deleted the tlim_newdiff2 branch December 11, 2022 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant