Kwil Release Process
This document describes an ideal development cycle for Kwil, good practices for shipping stable releases, and Go module tagging considerations.
Loosely speaking, a development cycle begins and ends with major software releases, where establishing requirements and design precedes most new development. There are also typically lagging activities following a major release, including deployments, additional documentation, and feedback gathering. This document will focus on the final stages leading up to the release of a major new version i.e. the release process.
The FreeBSD Release Engineering system sets a great example. Their release cycle refers to the later stages of development that I'm focusing on here. Although our products and team are very different, we will refer to a number of concepts used there. We also have a less nuanced process with a single main development track and only patches on release branches.
Two closely related points are timeline and scope.
Scope refers to what work and features or functions are intended to be developed. The development cycle starts with planning to establish scope for obvious reasons, with the ability to set a timeline and manage expectations at the top of the list.
Scope can evolve for a long portion of the development cycle, particularly when research and experimentation is involved. However, as the features and required work become more certain after significant development has occurred, it is important to take steps avoid "scope creep" and rushed releases.
The start of the actual release process should involve a freeze on features or certain types of changes well in advance of the final release. Of course, everything is still up for discussion after this point, but this is a necessary discipline for two main reasons:
- Establishing a realistic timeline for the release process.
- Stabilizing the product and avoiding quality issues by leaving ample time for testing and evaluation.
The second point above is particularly important for blockchain applications since consensus errors can be unforgiving.
Particularly in software development, setting hard release dates at the beginning of the development cycle is not practical. There are notable exceptions, such as the Ubuntu release schedule that is set far in advance, but it is rarely possible to anticipate every customer need and engineering hurdle, or even for a developer to accurately estimate the time required for anything but very small tasks let alone multiple major features.
The best approach is to keep scope under control and, once development has progressed sufficiently, only then set anticipated target dates for a series of freezes and pre-releases.
As is probably becoming clear, a large release needs a disciplined release process to ensure that when the "final" revision is tagged we are confident it will work as intended (and required). We cannot do this if we push major changes in the days leading up to the final release since it is not practical for everything to be thoroughly tested and deployed in internal staging or public pre-release environments.
Milestones we should consider prior to a major release:
- When we are ready, shift priorities from new features or large changes to testing and bug fixes. FreeBSD calls this a code "slush". This is mostly about prioritizing resources, and for Kwil this should mean informally deciding as a team that all large changes are nearing completion and we should start focusing on stability. In other words, a beta is imminent, and we should be cautious about adding to the list of big changes scoped for the release.
- A beta release. This can happen even while we are aware of some functional changes that have yet to occur, but the product should be fairly mature. This is most important if there is a consistent feedback loop with the public or stakeholders.
- Freeze on new features or large changes. We are a small enough organization that in practice everything is up for discussion, but it should be a shift in our stance on stability vs. functionality. Most resources should be testing, documenting, or working on features for the next cycle.
- At least one release candidate (RC), often already on a release branch. Although we can plan on more than one, the "candidate" terminology conveys that it may be the stable production release, meaning planned changes are completed and there are no known release-blocking bugs.
- Final release.
A formal slush or freeze status is not really required, but we should be thinking in these terms, as they characterize the subsequent beta and RC releases. That is, if we tag a beta, we're not going to add to the list of features although some may still be incomplete. If we tag an RC, we are only planning to fix bugs or make extremely minor functional changes.
It is a good idea to communicate with stakeholders at various points in the process, particularly before and after the first beta. Communicate our anticipated timeline, what features are included, and any notable or breaking changes. Request feedback.
Any very early pre-release would be more of an "alpha". Publicly available releases tagged as a beta or RC should be fairly stable and nearly feature complete.
NOTE: This process would apply to our "major" releases, which for Kwil refers to semver's MINOR version (MAJOR.MINOR.PATCH). For example, the first 0.6.0 version following 0.5.x.
Doing all of these steps might simply be too drawn out for Kwil at present. But at an absolute minimum, I think we should freeze at least two weeks out, and have at least one release candidate about one week before final. Large changes merging on the day of release should not happen (and I am guilty of this).
Time permitting, the ideal scenario would involve a public beta once there are no breaking changes planned, hopefully at least three weeks out, and multiple release candidates. This will become a minimum when there are more users and live networks that cannot break.
Most of this is already implied, but to be explicit about what were are doing between the milestones in this release cycle:
- Fixing known bugs.
- Testing, in our individual developer workspaces as well as our internal staging environments.
- Deploying to any public pre-release or beta networks.
- Simulating upgrade scenarios for existing networks, if applicable.
- Working with partners who can evaluate the beta and RC releases.
- Reading and revising our docs.
- Preparing the production environment for a smooth upgrade when the final release is ready.
If the release involves multiple repositories or products, we should also focus on full system testing to reconcile any incompatibilities or inconsistencies.
The preferred versioning used for tags on product releases (different from Go module tags!) should be:
- Beta releases: (
vX.Y.Z-beta.N) e.g.v0.6.0-beta.1. - Release candidates: (
vX.Y.Z-rc.N) e.g.v0.6.0-rc.2 - Final: (
vX.Y.X) e.g.v0.6.0
Here we are referring to both the version string in the software and the tag for the main module of kwil-db. This is because we have decided the main module will pertain to the applications (the cmd packages).
The main module of the kwil-db repository is used for the application versions, which represent our product releases that are deployed in productions systems. As such, plain semver tags such as v0.6.0 or even custom tags such as release-v0.6.0 can be used to fetch and install these apps. However, using semver tags has the helpful effect of allowing users to use @latest as the revision when doing go run or go install for one of the the applications such as kwild.
Other Go modules such as core and parse should be tagged in a very specific way that is required for them to be used via require entries in an application's go.mod.
For core, the tag must be formatted like core/vX.Y.Z, where the version numbers may be completely independent from the product version discussed above. For instance core/v0.1.0 or core/v1.0.0 could be required by our applications in a v0.6.0 product release. However, we would not be setting the major (X) value above 0 until we are prepared to bump the major version for any breaking API changes.
It is worth noting that we do not strictly need to tag our submodules until we intend for them to be used directly by third party software once we are open-sourced. This would mean that in the main module's go.mod it would be using a pseudo-version such as v0.0.0-20230811130428-ced1acdcaa24 to require a certain revision of that module.
The following is primarily for open source code, when our apps can be installed by go install, and our libraries used with require directives in a go.mod file.
A replace directive in a go.mod is ignored unless building that module directly from a source folder. They are ignored when present in an imported module's go.mod, and they cause errors in the module of an executable application when using go install or go run.
When using go install to install an executable, it will cause an error if the main module has a replace. For instance, if we do go install github.com/kwilteam/kwil-db/cmd/kwild@main (or go run), we would get an error:
$ go install github.com/kwilteam/kwil-db/cmd/kwild@v0.6.2
go: github.com/kwilteam/kwil-db/cmd/kwild@v0.6.2 (in github.com/kwilteam/kwil-db@v0.6.2):
The go.mod file for the module providing named packages contains one or
more replace directives. It must not contain directives that would cause
it to be interpreted differently than if it were the main module.
If a project requires another module for importable packages, any replace in that other module's go.mod is not applied. That is, if module A requires module B, any replace directives in the go.mod for module B are ignored when building module A. For instance, if we require some third party module that has replace directives, they are not applied. Conversely, if another project tries to use our core or parse modules, the replace directives in them will not be applied.
From the Go module documents:
The go.mod file can also specify replacements and excluded versions that only apply when building the module directly; they are ignored when the module is incorporated into a larger build.
In other words, only the top level module can specify replace directives.
For the above reasons, any replace directives must be removed before tagging any revision (at least once we are open source).
The modules should be tagged in steps, between which dependant module's go.mod files should be updated to refer to the newly tagged modules.
This process would begin by updating the "leaf" modules that do not require any other Kwil modules that are part of the release.
In the above dependency graph, the leaf modules are: core, sql-grammar-go, action-grammar-go, and kwil-extensions, so the process would be as follows:
- Tag the
coresub-module of thekwil-dbrepository. The format for a submodule tag is<submodule path>/v<semver>, such ascore/v0.2.3. - Tag the
sql-grammar-go,action-grammar-go, andkwil-extensionsmain modules. Since these are main modules defined at the root of a repository, the tags are plain semver likev1.2.3. We can also skip this step for these repos and use pseudo-versions in the dependent modules. - Update the
requiredirectives in thego.modfile of theparsesubmodule. - Tag the
parsesubmodule in thekwil-dbrepo. - Update the
requiredirectives in thekuneiformmodule. - Tag the
kuneiformrepository main module. - Update the
requiredirectives in thekwil-dbmain module.
The test module is never tagged. It is also not necessary to remove any replace directives or even update the requires, although it can't hurt to update the requires even with the replaces still there.
Before beginning the process, it is wise to review each module's go.mod file to ensure that the above dependency graph is still accurate.
In a perfect world, the go.mod files would not need replace directives with the invent of workspace mode (GOWORK and the go.work file). However, numerous official and unofficial tools and go subcommands still do not have first-class support for workspace mode. Until that happens, we are stuck with replace directives in go.mod, which is checked into source control.