dripip release tool

[jasonkuhrt]

Supports graphql-nexus/nexus-plugin-prisma#379

This diagram is a visual representation of the release workflow explained below.

---

dripip

dripip is a CLI for releasing npm packages. It bakes in the concepts of stable-version, pull-request-version, and next-version releases. It assumes the project adhears to semantic versioning and conventional commits. It is designed for extreme simplicty and CI/CD.

Principals:

• Releases should not show up as git commits, only tags.
• Every commit should be published.
• The release system should be simple enough to understand fully in a few minutes.

Release Flows

dripip supports three complementary release flows:

• $ dripip stable
• $ dripip preview
• $ dripip preview on pull-requests

dripip stable

This bumps the package by a conventional commit calculation against the commits since last stable release.

1. • Assert HEAD is not a stable release.
2. • Calculate new version:
   1. • Find the last stable version (git tag on the current branch). If none use 0.0.1.
   2. • Calculate the semver bump type. Do this by analyizing the commits on the branch between HEAD and the last stable git tag. The highest change type found is used.
   3. • Bump last stable version by bump type, thus producing the new version.
3. • Run publishing process:
   1. • Change package.json version field to be new version.
   2. • Run npm publish --tag latest.
   3. • Undo package.json change.
   4. • Run git tag {newVersion}.
   5. • Run git tag next (normally should already be present)
   6. • Run git push --tags.
   7. • Create a GH release titled by the version and pointing at the git tag version.

dripip preview

This publishes pre-releases like 1.2.3-next.1, 1.2.3-next.2, ... and so on that increment until finally resetting upon the next stable release.

1. • Assert HEAD has no release.
2. • Assert is not a PR
     • or --no-pr
3. • Calculate new version:
   1. • Find the last pre-release on the current branch. Take its build number. If none use 1.
   2. • Calculate the semver bump type. Do this by analyizing the commits on the branch between HEAD and the last stable git tag. The highest change type found is used. If no previous stable git tag use 0.0.1.
   3. • Bump last stable version by bump type, thus producing the next version.
   4. • Construct new version {nextVer}-next.{buildNum}. Example: 1.2.3-next.1.
4. • Run publishing process:
   1. • Change package.json version field to be new version.
   2. • Run npm publish --tag next.
   3. • Undo package.json change.
   4. • Run git tag {newVer}.
   5. • Run git tag next.
   6. • Run git push --tags.
   7. • Creates a GH pre-release called next at tag next
        If next release exists (it always will except for the first release ever) then do nothing (the force push updating next tag should be enough to make the GH next release pointing at a new SHA.

dripip preview on pull-requests

This publishes pre-releases like 0.0.0-pr.1.8c01648, 0.0.0-pr.2.8y2716f. "On pull-requests" is determined by dripip by looking for well known environment variables exported by CI providers. The CI providers currently supported are GitHub and CircleCI. This implies we are optimizing for running on CI systems. If you want to force this behaviour say from your machine add the --pr flag.

1. • Must be a PR and no --no-pr OR --pr present.
2. • Assert that not both --no-pr and --pr are present.
3. • Assert HEAD has no release.
4. • If --pr passed, use same algorithm used by hub show pr to infer pr number.
5. • Calculate new version:
   1. • Construct new version {0.0.0}-pr{prNum}.{shortSHA}. Example: 0.0.0-pr.121.8y2716f.
6. • Run publishing process (no git tags here):
   1. • Change package.json version field to be new version.
   2. • Run npm publish --tag pr.{prNum}.
   3. • Undo package.json change.

Version Branches

dripip supports version branches defined below. The three release flows receive special integrity checks ensuring sound coordination with version branches.

1. A version branch is a branch whose name is a valid non-release-candidate semver value with optional v prefix.

2. • On such a branch it is only possible to make releases that do not bump the part locked in the branch name. For example: Given v2 branch only patch and minor releases are possible; Given v3.3 branch only patch releases are possible.
3. • On the parent of such a branch it is only possible to make releases of a higher order than the locked part of the version branch. For example: Given a v2 branch all subsequent parent branch releases must be v3.0.0 or higher; Given a v3.3 branch all subsequent parent branch releases must be v3.4.0 or higher.
4. • It is important that releases on a version branch do not collide with releases anywhere else, such as on the parent branch. dripip version-branch <vb-ver> helps with this. It:
   1. • Creates a new git branch
   2. • tags the commit it branched off from with vb<vb-verb>

   The special tag vb<vb-verb> helps dripip do the right thing with respect to points 2 & 3 above.

   • If you opt to not use dripip CLI to create your version branch then manually create the tag as specified here.

   • If we find a reliable way in the future to correctly and effeciently rely on git CLI alone to enforce 2 & 3 we will, thus dropping the extra tag signal.

Release Notes

1. • Release notes will be added to the GitHub release.
2. • Release notes will be produced in Markdown format.
3. • Release notes will be exportable as JSON.
4. • Release notes for a single release will look like:

     <!-- @META { "checksum": "...", "dripipVersion": "..." } -->
   
     <!-- @CUSTOM { "side": "start" } -->
     <!-- @CUSTOM { "side": "end" } -->
   
     #### BREAKING CHANGES
   
       <!-- @CUSTOM { "side": "start" } -->
       <!-- @CUSTOM { "side": "end" } -->
   
       * [next.#] <short-sha>(link) <commit message> [ty <gh-user-name>(link)!]
         
         <commit message body>
   
       * ...
   
       <!-- @CUSTOM { "side": "start" } -->
       <!-- @CUSTOM { "side": "end" } -->
   
     #### Features
   
       <!-- @CUSTOM { "side": "start" } -->
       <!-- @CUSTOM { "side": "end" } -->
   
       * [next.#] <short-sha>(link) <commit message> [ty <gh-user-name>(link)!]
         
         <commit message body>
   
       * ...
   
       <!-- @CUSTOM { "side": "start" } -->
       <!-- @CUSTOM { "side": "end" } -->
         
     #### Fixes
   
       <!-- @CUSTOM { "side": "start" } -->
       <!-- @CUSTOM { "side": "end" } -->
   
       * [next.#] <short-sha>(link) <commit message> [ty <gh-user-name>(link)!]
         
         <commit message body>
   
       * ...
         
       <!-- @CUSTOM { "side": "start" } -->
       <!-- @CUSTOM { "side": "end" } -->
   
     #### Improvements
   
       <!-- @CUSTOM { "side": "start" } -->
       <!-- @CUSTOM { "side": "end" } -->
   
       * [next.#] <commit message> <short-sha>(link) [ty <gh-user-name>(link)!]
         
         <commit message body>
     
       * ...
   
       <!-- @CUSTOM { "side": "start" } -->
       <!-- @CUSTOM { "side": "end" } -->
   
     <!-- @CUSTOM { "side": "start" } -->
     <!-- @CUSTOM { "side": "end" } -->
   

   1. • The @META checksum is derived from the generated release notes contents. Later on dripip can check contents against the checksum to easily detect drift (manual edits).
   2. • The @META dripipVersion contains the same piece of data that would be in packge.json version snapshoted at the time of generation. This is useful to permit document format migrations from one version of dripip to the next.
   3. • The @CUSTOM directives permit users to customize their release notes while preserving the ability for dripip to update them later on.
   4. • There is an @CUSTOM directive in append/prepend position for the entire release notes as well as per sub-section therein.
   5. NOTE @CUSTOM directives may be useful for:
      1. next blocks which are edited in place frequently
      2. Refreshing release notes based on a new version of dripip that improves the layout somehow (e.g. adds a "Thanks to these users! ..." summary block)
      3. Perhaps/probably dripip has release notes style options sooner or later that users choose to adjust their settings on over time.
   6. • chore commits will not show up in the changelog.
   7. • feat commits will show up under Features section.
   8. • fix commits will show up under Fixes section.
   9. • All other commit types will show up under Improvements section.
   10. • Commits without commit types will show up in the changelog under Other Changes
   11. • Commits with BREAKING CHANGE will be copied to the BREAKING CHANGES section.
   12. • Commits from users not in the CODEOWNERS.txt file and not members of the org containing the repo will receive a special shoutout.
       • TODO Consider and/or being members on users in the same org as the repo
   13. • If a release section has no applicable commits then it is not rendered at all.
   14. • If this is a preview release then all commit entries are prefixed with next.<#>, the preview version they came from (we do this b/c pre-releases sum theirchanges since last stable)
5. • Preview release release notes on GitHub Releases
6. • Titled by the next version e.g. 1.2.0-next.5
7. • Use the next git tag thus permitting a permalink
8. • Every next release Updates-in-place the same GitHub release
9. • Are marked as a pre-release
10. • Contain the sum of changes since last stable
11. • Formatted like so:

      Latest is <preview-version> released on <date>
      Average time between stable releases is <x> days. Last one was on <date>.
    
      <release notes>  
    

12. • When there is no next releases following latest stable release then the
      next GitHub release is formatted like:

    No releases since last stable.
    

---

TODO

• Show release notes strategy
• show release notes custom edits strategy
• Show workflow for version branches. E.g. lib on 3.x but 2.x maintained
• update diagram with version branches Currently lacking an omnigraffle license
• Define good flows around handling undefined commits w.r.t. release note management
• Reconcile tradeoff between 1) clean git history 2) cicd 3) in-repo release notes
• finalize name

[jasonkuhrt]

Great catch by @janpio:

[jasonkuhrt]

Another principal:

Be judicious with developers' time. Stable releases should be batches of change. Users subscribing to previews opt-in for more noise. Everyone else, no.

[jasonkuhrt]

Spec Update

• 0.0.0-pr.#.# where 0.0.0 is a static major.minor.patch value now. This is chosen because on a pull-request:

  1. -pr.#.# is a pre-release so how does it make sense to say branching off release 1.2.3 starts to produce pre-releases of that version? It is nonsense. With 0.0.0 we're signalling "hey, ignore this".
  2. It can sometimes not be known if it will be just a fix, feat, or breaking change (usually the last we know though). Nor are commits in a PR often following conventional commit. All to say we cannot really address the above point by trying to do here what we do on trunk previews.
  3. A pull-request can live across releases (on trunk) and be getting rebased off it to stay up to date. That lends toward the major.minor.patch getting more and and more misleading if it stays based on what it was at time of branching off from.
  4. Calculating the trunk commit that a branch branched off of is surprisingly error prone in git. Doable in the simple cases but has gotchyas.

• updated diagram
• trunk previews now apply bump as per previous feedback.

[Weakky]

Could you provide a link to the liberate cli? I can't find anything when googling it

[jasonkuhrt]

@Weakky it doesn't exist, it would be our invention. Was also thinking of libre.

[jasonkuhrt]

Been following this flow manually on https://github.com/prisma-labs/nexus-prisma/pull/434/files and its working ok. nexus pr release installed into nexus-prisma and then nexus-prisma pr release and then both installed into nexus-prisma example blog. Due to the node_modules file system trickery used linking doesn't "just work".

Examples of commands I am running:

❯ yarn publish --tag next --new-version "0.5.0-next.2"

❯ yarn publish --tag pr434 --no-git-tag-version --new-version "0.0.0-pr.434.$(git show -s head --format='%h')"

[jasonkuhrt]

Added TODOs for

• Show workflow for version branches. E.g. lib on 3.x but 2.x maintained
• Show release notes strategy

[jasonkuhrt]

Spec Update

Came up with the following spec for release notes tonight. You can see below but it has been copied up to the opening comment too.

Release Notes

Release Notes

• Release notes will be added to a CHANGELOG.md file if present.
• Release notes will be added to a HISTORY.md file if present.
• Release notes will be added to a README.md Changelog section if present.
• Release notes will be added to the GitHub release.
• Release notes will be produced in Markdown format.
• Release notes will be exportable as JSON.
• Release notes for a single release will look like:

    #### BREAKING CHANGES
  
      * [next.#] <short-sha>(link) <commit message> [ty <gh-user-name>(link)!]
        
        <commit message body>
  
      * ...
  
    #### Features
  
      * [next.#] <short-sha>(link) <commit message> [ty <gh-user-name>(link)!]
        
        <commit message body>
  
      * ...
        
    #### Patches
  
      * [next.#] <commit message> <short-sha>(link) [ty <gh-user-name>(link)!]
        
        <commit message body>
    
      * ...
  

  • chore commits will not show up in the changelog.
  • feat commits will show up under Features section.
  • All other commit types will show up under Patches section.
  • Commits without commit types will not show up in the changelog.
  • Commits with BREAKING CHANGE will be copied to the BREAKING CHANGES section.
  • Commits from users not in the CODEOWNERS.txt file will receive a special shoutout.
  • If a release section has no applicable commits then it is not rendered at all.
  • If this is a preview release then all commit entries are prefixed with next.<#>, the preview version they came from (we do this b/c pre-releases sum theirchanges since last stable)
• Release notes on GitHub Releases
  • If the release is a preview then it is marked as a GitHub pre-release
  • preview releases will contain a sum of changes since last stable
• Release notes in files
  • will have all #<num> strings converted into links to the corresponding issue/pr.
  • will have the following template:

    # Changelog
    
    ## `next`
    
    Latest is <preview-version> released on <date>
    Average time between stable releases is <x> days. Last one was on <date>.
    
    <release notes (see above)>
    
    ## <stable-version>(link to repo at SHA) <date>
    
    <release notes (see above)>
    
    ## ...
    

    • preview releases will be grouped under next where a sum of changes since last stable accumulate.

    • When there is no next releases following latest stable release then the
      next section is formatted like:

      ## `next`
      
      No releases since last stable.
      

• Release notes in README.md
  • will begin at the header level next from of that found for Changelog heading
• Future features
  • It should be possible to manually change changelog files (for example with long form migration info under a breaking change) yet still accrue updates from automation.

[janpio]

Sound good. 2 notes:

• All other commit types will show up under Patches section.

Wording might be problematic because of MAJOR.MINOR.PATCH. "Changes" maybe?

• Commits without commit types will not show up in the changelog.

Maybe copy them to a "undefined" section by default that has to be cleaned out? Otherwise forgetting the type once might make something invisible.

[jasonkuhrt]

Wording might be problematic because of MAJOR.MINOR.PATCH. "Changes" maybe?

Agree. Its more consistent. For example feat: affects MINOR but we don't call the heading Minors.

Maybe copy them to a "undefined" section by default that has to be cleaned out? Otherwise forgetting the type once might make something invisible.

Thought about it. Wasn't sure. Can see that yes probably something has to be done. Some things we could do (none, some, all):

1. List all undefined commits in terminal
2. process exit 1
3. emit all undefined commits to undefined-commits.md
4. emit into file/gh-release-contents under undefined changes heading requiring user edits
5. choose [safe?] defaults and support flags to opt-in/out of behaviours

[jasonkuhrt]

Spec Update

+ * `fix` commits will show up under `Fixes` section.
+ * All other commit types will show up under `Other Improvements` section.

[pantharshit00]

Zeit has a similar release cli:
https://github.com/zeit/release

Maybe we can take some inspiration

[jasonkuhrt]

Description Update

Adding support for:

• format migrations
• custom edits
• retaining custom edits across format migrations

+    * The `@META` `checksum` is derived from the generated release notes contents. Later on `liberate` can check contents against the checksum to easily detect drift (manual edits).
+    * The `@META` `liberateVersion` contains the same piece of data that would be in `packge.json` `version` snapshoted at the time of generation. This is useful to permit document format migrations from one version of `liberate` to the next.
+    * The `@CUSTOM` directives permit users to customize their release notes while preserving the ability for `liberate` to update them later on.
+    * There is an `@CUSTOM` directive in append/prepend position for the entire release notes as well as per sub-section therein.
+    * NOTE `@CUSTOM` directives may be useful for:
+      1. `next` blocks which are edited in place frequently
+     1. Refreshing release notes based on a new version of `liberate` that improves the layout somehow (e.g. adds a "Thanks to these users! ..." summary block)
+     1. Perhaps/probably `liberate` has release notes style options sooner or later that users choose to adjust their settings on over time.

Note that a robust spec for how to perform format migrations is not done and likely not to be before implementation work starts. The important thing is to have a reasonable foundation in place to make it possible later.

[jasonkuhrt]

Description Update

• added ability to force pr preview with --pr boolean flag on preview command.

  + 1. Must be a PR and no `--no-pr` OR `--pr` present.
  + 1. Assert that not both `--no-pr` and `--pr` are present.
  + 1. If `--pr` passed, use same algorithm used by `hub show pr` to infer pr number.

• added a spec for version branches

  + ## Version Branches
  + 
  + `liberate` supports version branches defined below. The three release flows receive special integrity checks ensuring sound coordination with `version branches`.
  + 
  + 1. A version branch is a branch whose name is a valid non-release-candidate semver value with optional `v` prefix.
  + 1. On such a branch it is only possible to make releases that do not bump the part locked in the branch name. For example: Given `v2` branch only patch and minor releases are possible; Given `v3.3` branch only patch releases are possible.
  + 1. On the parent of such a branch it is only possible to make releases of a higher order than the locked part of the version branch. For example: Given a `v2` branch all subsequent parent branch releases must be `v3.0.0` or higher; Given a `v3.3` branch all subsequent parent branch releases must be `v3.4.0` or higher.
  + 1. It is important that releases on a version branch do not collide with releases anywhere else, such as on the parent branch. `liberate version-branch <vb-ver>` helps with this. It:
  + 
  +   1. Creates a new git branch
  +   2. tags the commit it branched off from with `vb<vb-verb>` 
  +
  +   The special tag `vb<vb-verb>` helps `liberate` do the right thing with respect to points 2 & 3 above.
  +
  +   If you opt to not use `liberate` CLI to create your version branch then manually create the tag as specified here.
  +
  +   If we find a reliable way in the future to correctly and effeciently rely on `git` CLI alone to enforce 2 & 3 we will, thus dropping the extra tag signal.

• Specify a safe fallback for undefined commit types in release notes. There are a lot of features we can explore around this nuanced topic later. Starting simple/safe.

  -   * Commits without commit types will not show up in the changelog.
  +   * Commits without commit types will show up in the changelog under `Other Changes`

• Adding one last TODO about the required tradeoffs between 1) clean git history 2) cicd 3) in-repo release notes. This will be the last thing to reconcile before breaking ground I think. We may see a significant change to the current release notes spec which affords considerable support for release notes but lives in a fantasy land where it wouldn't break the principal that releases shouldn't show up in the commit history. In fact they will so long as release notes are files in the repo. Worse yet is the workflow for editing those release notes. This needs to be directly addressed in the spec I've written so far.

[jasonkuhrt]

Description Updated

One of our principals is

Releases should not show up as git commits, only tags.

This is because we care about CICD but also about a clean git history. If every commit is released (CICD) but releases require commits that means 50% of the git history is noise.

To solve this we figured out how to hoist version numbers out of [committed] source code and into git tags. However with release notes, as files, we have brought the problem of git commits back upon us. Worse yet, release notes are complicated and often require fine tuning because release notes are a kind of interface/content designed human consumption. Such fine tuning implies even more git commits, more than 50% of the git history. This is not workable.

Therefore the spec affordances for release note files will be removed. In turn there will be a focus on GitHub releases. In the future we may also consider GitHub wiki as another release notes target. Further future maybe pluggable targets.

---

• Added better handling of next release in GitHub Releases

  +    1. Run `git tag next`.

  +    1. Run `git tag next` (normally should already be present)

  + * Preview release release notes on GitHub Releases
  +   * Titled by the next version e.g. `1.2.0-next.5`
  +   * Use the `next` git tag thus permitting a permalink
  +   * Every `next` release Updates-in-place the same GitHub release
  +   * Are marked as a pre-release  
  +   * Contain the sum of changes since last stable
  +   * Formatted like so:
  +     ```
  +       Latest is <preview-version> released on <date>
  +       Average time between stable releases is <x> days. Last one was on <date>.
  +   
  +       <release notes>  
  +     ```
  +   * When there is no `next` releases following latest stable release then the
  +     `next` GitHub release is formatted like: 
  +   
  +     ```
  +     No releases since last stable.
  +     ```

• Remove future feature note, we have it now afterall:

  - * Future features
  - * It should be possible to manually change changelog files (for example with long form migration info under a breaking change) yet still accrue updates from automation.

  • Remove spec about changelog files

    - - * Release notes in files
    -   * will have all `#<num>` strings converted into links to the corresponding issue/pr.
    -   * will have the following template:
    -     ```
    -     # Changelog
    - 
    -     ## `next`
    - 
    -     Latest is <preview-version> released on <date>
    -     Average time between stable releases is <x> days. Last one was on <date>.
    - 
    -     <release notes (see above)>
    - 
    -     ## <stable-version>(link to repo at SHA) <date>
    - 
    -     <release notes (see above)>
    - 
    -     ## ...
    -     ```
    -     * preview releases will be grouped under `next` where a sum of changes since last stable accumulate.
    -     * When there is no `next` releases following latest stable release then the
    -       `next` section is formatted like: 
    -     
    -       ```
    -       ## `next`
    - 
    -       No releases since last stable.
    -       ```
    - * Release notes in `README.md`
    -   * will begin at the header level next from of that found for `Changelog` heading

    - * Release notes will be added to a `CHANGELOG.md` file if present.
    - * Release notes will be added to a `HISTORY.md` file if present.
    - * Release notes will be added to a `README.md` `Changelog` section if present.

[jasonkuhrt]

Description Update

• all instances of liberate have been renamed libre.

Thought a bit more about naming. Liberate is somehow pompous or on the nose I feel, and not short making it functionally annoying to type on the command line. I considered a variety of alternatives but all the basic ones (cut scissor shear release ...) were taken on npm or had other problems, e.g.cut is a standard posix tool. I also felt going for something distinctive was worthwhile here. So going back to libre.

https://www.thoughtco.com/what-does-libre-mean-3079046

Instead, libre, related to words such as "liberate" and "liberty," usually refers to being free in the sense of being free of restraints or sometimes in the sense of being available.

This is a good metaphorical fit for this tool, since this tool is all about making work on a project easily available to users.

Like so many names, It is taken on npm too. But overall it is a distinct enough name that I'm going to give it a shot anyways. I considered a few publishing workarounds libre2 and librelibre. I think the latter is less evil than suffixing a 2. Numeric suffixes are unoriginal and suggest a continuity. Publishing under librelibre and stating the cli is exposed as just one utterance of the word seems intuitive enough.

[janpio]

#2 (comment) contains all the current information if I just want to get an overview?

(Try to contact both @livingbreathing and npm, they might be able to help with the npm package name - repo seems to be gone and package unused)

[jasonkuhrt]

#2 (comment) contains all the current information if I just want to get an overview?

Yep It is kept up to date with the latest thinking.

(Try to contact both @livingbreathing and npm, they might be able to help with the npm package name - repo seems to be gone and package unused)

Good point will do!

--update Email to Evan, the listed maintainer of libre, sent.

[jasonkuhrt]

I have updated the description with checkboxes. I am starting work over at https://github.com/prisma-labs/libre. I debated how to start tracking issues. My approach will be to use this as an epic to track the bulk of v1 development. I will not continue to put every new feature idea here though. Those will begin to get created as new issues on the libre repo. This issue has been useful to get the project off the ground, but was not intended to last forever.

So, as I make progress on libre I will come back here to update progress by simply checking the boxes off.

[janpio]

(Leaving a comment anyway will push this into my notifications, which might be beneficial - or not ;) )

[jasonkuhrt]

Thinking, rather than the --no-pr flag design, sub-commands.

$ libre preview           <-- automates selection between pr and trunk
$ libre preview pr
$ libre preview trunk

Possible alternatives to trunk:

$ libre preview stable
$ libre preview master
$ libre preview latest
...

[jasonkuhrt]

Renamed from libre to dripip. Detail: prisma-labs/dripip#13