Future of octoDNS - Road to 1.0

[ross]

Using this issue to write out some thoughts/plans for octoDNS's path forward. There's no specific timeline on any of this, some of it will happen almost immediately other bits will come over time.

Separate Org - github.com/octodns

The biggest change is that I plan to move things from https://github.com/github/octodns/ to https://github.com/octodns/. That will put this repo at https://github.com/octodns/octodns/. The good news is that everything should pretty much redirect and work normally if you hit the old URL. There's a few reasons behind this change, the primary one being so that we can break up the core of octodns from the providers. This give us a better setup on a few fronts.

Maintainers, Access, & Testing

Having separate repos for each provider will allow outside contributors to be granted further access and own/drive specific providers. There are many providers where I don't have access to run or test with making it difficult for me to verify changes and making me/GitHub a bottleneck. Having things in separate repos that don't live under the /github org will allow us to grant control to the authors of the providers.

Release Schedules

Having repos like https://github.com/octodns/route53, https://github.com/octodns/azure, https://github.com/octodns/ns1, ... This will allow each provider to release on its own schedule as makes sense for that provider and structure things more cleanly in terms of issues and PRs.

Requirements & Dependancies

This will also result in a much better setup around requirements. With everything in a single repo we've chosen to only include the core requirements for octoDNS to avoid installing 100's of modules, most of which wouldn't be used in any given install. With separate modules each provider can spell out their exact requirements in their setup.py/requirements.txt and things will work as people expect.

The path from here to there

Again no specific timeline, the first few of these will likely happen over the next few weeks. The "api", testing, and breakup of providers will likely happen over time.

• Create the https://github.com/octodns/ org
• Grant access (owners/admins) to roughly match those on https://github.com/github/octodns
• Move https://github.com/github/octodns/ to https://github.com/octodns/octodns/
• Update docs, readmes, pypi etc to point to the new location
• Codify the existing Source & Provider structure as a formal API & write extensive behavior tests for that api to help ensure that changes won't break providers
• Create a new "template" provider to establish the structure and behavior of sources/provider sub-modules and to serve as a starting point for pulling existing providers out and for developing new providers
• Look into running tests for all the submodules against the octoDNS core on https://github.com/octodns/octodns/ PRs.
• Split out each provider into a separate repo & module.
• ...

[yeled]

@ross I'm happy to help out with as much open-source wrangling as possible

[ross]

• Move https://github.com/github/octodns/ to https://github.com/octodns/octodns/

See https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/transferring-a-repository for details of the process/results.

[ross]

• Update docs, readmes, pypi etc to point to the new location

If you have an existing checkout and would like to update it's url you can do the following. Things should generally work/redirect with the old one, but it doesn't hurt to update:

salmon:octodns ross$ git remote -v
origin	git@github.com:github/octodns.git (fetch)
origin	git@github.com:github/octodns.git (push)
salmon:octodns ross$ git remote set-url origin git@github.com:octodns/octodns.git
salmon:octodns ross$ git remote -v
origin	git@github.com:octodns/octodns.git (fetch)
origin	git@github.com:octodns/octodns.git (push)
salmon:octodns ross$ git pull
remote: Enumerating objects: 21, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 21 (delta 7), reused 14 (delta 6), pack-reused 0
Unpacking objects: 100% (21/21), done.
From github.com:octodns/octodns
   f9aa08b..16241d2  master     -> origin/master
Already up to date.

[kashike]

Hey @ross, just wondering if there has been any more progress towards this? 😅

[ross]

Hey @ross, just wondering if there has been any more progress towards this? 😅

Not much for a bit now. The checklist is up to date. Most of what's left is the breakup of providers into modules and associated work to get that to be safe/robust.

Anything in particular you're waiting/wanting for?

[ross]

• Create a new "template" provider to establish the structure and behavior of sources/provider sub-modules and to serve as a starting point for pulling existing providers out and for developing new providers

Templatized the skeleton bits of the work in octodns-powerdns into https://github.com/octodns/octodns-template which should make extracting providers (and creating new ones) fairly easy to get started.

PowerDnsProvider is being extracted as the test over in #822 & https://github.com/octodns/octodns-powerdns.

[istr]

Over the past few weeks, I've been rummaging through the tests. The implementation, quality and coverage of the respective provider APIs varies greatly. In terms of consistent support, I would like to see the providers in modules use at least common test cases. I would therefore suggest to leave test/config/{featureset}.tests.yaml in the central repo and organize the tests in such a way that each provider has to fulfil at least the test cases specified there. This would promote a uniform quality of the providers.

All providers should at least pass the supported records of test/config/unit.tests.yaml. If a provider claims to support dynamic, it should additionally pass at least all the supported records of test/config/dynamic.tests.yaml.

Apart from that I would like to see a "provides safe updates" flag in the README provider table, that is set to false if the provider simply uses the (unsafe) "delete, then recreate" pattern and set to true if it implements real (in-place / non-destructive) updates.

That is to say, all providers using the following implementation or a similar logic should be marked "unsafe" imho. ymmv.

    def _apply_Update(self, zone_id, change):
        # It's way simpler to delete-then-recreate than to update
        self._apply_Delete(zone_id, change)
        self._apply_Create(zone_id, change)

[ross]

Over the past few weeks, I've been rummaging through the tests. The implementation, quality and coverage of the respective provider APIs varies greatly.

Yeah. That's driven by a lot of things, including who implemented them, how much time & effort was available, and at least in my case what purpose it was going to be put to, e.g. just for fun vs. in production use at GitHub.

In terms of consistent support, I would like to see the providers in modules use at least common test cases.

Yeah. Something like that has always been on the wishlist and it'd definitely be nice to have. To some degree that's what is meant up above by:

Codify the existing Source & Provider structure as a formal API & write extensive behavior tests for that api to help ensure that changes won't break providers

That said I didn't/don't it's somewhat orthogonal to splitting things up. The same problems/bugs could exist whether the providers share a repo or they're separate so I don't want to block progress towards the splitting up of things and (eventually) 1.0 specifically on that.

octoDNS is more thoroughly tested than most/many OSS projects and it's safety is aided by the way things work in terms of planning, reviewing, and applying. If general/basic things are going wrong they'll be showing up in the plan output. The main potential source of bugs is then during the apply phase where the provider details live. That's a pretty tough to test point b/c the actual interactions with the providers have to be mocked/stubbed in most cases, but we've had fairly good luck over the past few years there.

Again, can definitely be improved, but in my opinion a general TODO and not specifically something related to or blocking 1.0.

I would therefore suggest to leave test/config/{featureset}.tests.yaml in the central repo and organize the tests in such a way that each provider has to fulfil at least the test cases specified there. This would promote a uniform quality of the providers.

Yeah. Biggest part of that would be to start including some of the test configs & fixtures as part of the released & installed packages, either as part of the main module or an octodns-test one.

That said it'll only go so far towards the uniform quality mostly b/c of the aforementioned stubbing/mocking. It's fairly easy to write passing tests with incorrect expectations, e.g. test mocks/stubs that behave differently than the provider actually does.

All providers should at least pass the supported records of test/config/unit.tests.yaml. If a provider claims to support dynamic, it should additionally pass at least all the supported records of test/config/dynamic.tests.yaml.

The tests would have to adapt to the provider's capabilities and omit/skip tests that don't apply.

Apart from that I would like to see a "provides safe updates" flag in the README provider table, that is set to false if the provider simply uses the (unsafe) "delete, then recreate" pattern and set to true if it implements real (in-place / non-destructive) updates.

The reality is that I don't think any of the providers 100% provide this due to shortcomings of the provider apis & semantics so it's not really a boolean thing so much as a continuum. I'm not sure how we'd effectively signify where each provider lies on that scale and TBH in a lot of cases there are assumptions made/required about how they'll behave since we don't know how the underlying provider actually implements things, e.g. eventual consistency, strict ordering of operations, ...

[istr]

Again, can definitely be improved, but in my opinion a general TODO and not specifically something related to or blocking 1.0.

I absolutely agree.

It's fairly easy to write passing tests with incorrect expectations, e.g. test mocks/stubs that behave differently than the provider actually does.

This is precisely what I can confirm, at least as far as the one provider that I took a closer look at is concerned.

[ross]

• Create a new "template" provider to establish the structure and behavior of sources/provider sub-modules and to serve as a starting point for pulling existing providers out and for developing new providers

This has been done and it works fairly well: https://github.com/octodns/octodns-template

• Split out each provider into a separate repo & module.

As of #859 this is done for providers. I plan to look over sources and figure out what if anything needs to happen with them soon.

[ross]

• Look into running tests for all the submodules against the octoDNS core on https://github.com/octodns/octodns/ PRs.

/cc #883

[ross]

• Codify the existing Source & Provider structure as a formal API & write extensive behavior tests for that api to help ensure that changes won't break providers

Calling this good enough. Going to start a rough clock on 1.0. Will probably aim to do 1-2 more point releases before kicking off the 1.x bump. Guessing we're looking at ~4-6 months out, but no specific timeline and it doesn't really matter. Going to close this though as it has served its purpose.