Spec discussion: Updating or freezing pip URL when running lock --update

[aaronsteers]

There are approximately four ways we can pin a pip_url ref.

Pip pin methods

1.A. Default branch pin

This is a pin to the repo itself. Whatever the default branch is, that branch's latest commit will be used during installation.

1.B. Pin to a named branch

The named branch would be used instead of the default branch. (Otherwise same as "1.A".)

1.C. Pin to a release tag

Syntactically this is identical to "1.B", except that the pinned ref should fit the pattern for a semvar identifier.

1.D. Pin to a commit ref

Syntactically this is identical to "1.B", except that the pinned ref should fit the pattern for a SHA hash or partial SHA hash.

2.A. Pin to a PyPi library name

A bare library name (without / or /http(s) refs) is likely a PyPi library name.

2.B. Pin to a specific PyPi library version

Similar to "2.A" but with an "==" version constraint.

Challenges

1. The pip_url string can actually reference many libraries (space delimited), and these libraries may each have different bump strategies.
2. Without a separate 'constraint' entry, it's not clear what bump methods should be permitted. For example, should major version bumps be followed, should an explicit version tag be "upgraded" to an unreleased commit ref on main branch?

Applications

There are two applications for this functionality:

Locally: The user may want to run a version of meltano lock like meltano lock --update=pip_url which gets the latest pip url bump.

In hub.meltano.com: In an automated fashion, we may want to "bump and lock" all pip urls within the current yaml definition files. This would ensure that when users install or add a plugin, they are getting references that would be stable from a pip/python perspective, as well as from a meltano definition perspective.

EDIT: Separate of 'freeze' and 'update' behaviors

After logging the above, I realized that another approach would be to create distinct actions to 'freeze' the Pip URL (which takes a slippery ref and replaces it with a pinned one, versus the action to 'update' the ref, which tries to advance the ref using one or more promotion strategies.

It might be more feasible to first 'freeze' the existing volatile refs, by checking the slippery ref and comparing it with an equivalent pinned ref, replacing the pinned ref if the two are identical. Then, separately, we could handle bumps manually or in a semi-automated manner according to a few specific known patterns and/or according to maintainer requests.

A pragmatic approach

1. Freeze Hub Refs (safely):
   1. Replace bare PyPi refs with a pin to their current (latest) PyPi version number.
   2. Replace bare GitHub repo refs with a pin to the latest commit on their default branch. If a GitHub release tag is available that matches the commit's SHA ref, use the GitHub release tag.
   3. Replace GitHub named branch repo refs with a pin to the latest commit on the named branch. If a GitHub release tag is available that matches the commit's SHA ref, use the GitHub release tag.
2. Add bump functionality (each bump requires retest):
   1. Commit SHA refs are updated to the latest fast-forwardable commit from the prior ref.
   2. PyPi releases are updated to the latest released version on PyPi.
   3. Release tags (based on semvar pattern matching against git tag) are bumped to the latest available tag that also matches a compatible semvar promotion.