Proclaim that EXPERIMENTAL APIs can be changed after 1.0.0

[kazuho]

People often wonder if going to version 1.0.0 would slow the development. IMO this is because the specification / FAQ is silent about how to develop additional APIs after reaching to 1.0.0 while specifically stating that 0.x.y is a development release.

To tackle such (mis)understanding, I think adding a clause like below would make us more comfortable.

1. An API marked as experimental should not be considered as part of the public API. They can be changed anytime without incrementing the major version number.

[crazedsanity]

How would you suggest the API be marked as such? Are you suggesting the possibility of moving (a project) to 1.0.0 while still having an experimental API?

[kazuho]

How would you suggest the API be marked as such?

For example it could be documented somewhere, or be marked as such in the doc-comment. I do not think we would need to clarify how it should be marked, considering the fact that the current specification does not define how public API should be marked.

Are you suggesting the possibility of moving (a project) to 1.0.0 while still having an experimental API?

Yes.

In most cases, development continues after reaching 1.0.0. And in some cases (especially in large-scale projects) it is usual to have an unstable API that is provided through several minor releases.

But the current specification is silent in how such APIs should be provided, even though it specifically discusses about 0.x.y (which are unstable development releases) and a.b.c-(alpha/beta)d releases (which are pre-releases of specific versions).

To me it seems like confusing; so I decided to open this issue.

[crazedsanity]

I'm definitely interested. I'd definitely like to hear some more views on the topic.

[linkdd]

Hopefully, git provides branches.

For example :

• your master branch will contain your major versions with no experimental API (X.Y.Z | X > 0) ;
• your develop branch will contain your experimental API and is still in major version 0 ;
• pre-release should be in a release- which is merged into master ;
• when you consider that you can make public your experimental API, just merge develop in your pre-release branch ;
• new features / bugfixs should be developed in separated branches, so we can merge them in master, develop, and pre-release branches, to increment each branch's version individually.

[crazedsanity]

@linkdd Yes, git provides branches, as does all the other popular source control systems out there that I'm aware of. Separating experimental from development and/or pre-release branches doesn't really solve the problem, as each of them still should have a unique version.

That being said, it might be sufficient to declare a particular version (whether it is in a different branch or not) as experimental, and note in the README as such. Keeping that experimental branch private is all well and good until you want to have it tested more: once that happens, and it gets out in the world, it should definitely have a unique version attached to it, so anyone helping with the experiment can properly upgrade to non-experimental.

So, all that being said, it seems as though we've almost come full-circle. Keeping an experiment private and contained (never releasing it to anyone and storing it in a branch) is probably not a great way to go.

Have I missed something?

[kazuho]

@linkdd Thank you for the suggestion.

IMO, using two version numbers in parallel (1.a.b for a release containing only public APIs only and 0.c.d for a release containing experimental APIs as well) would make release engineering complicated.

And it also makes the user-side code complicated. The idea behind semver is to define a formal method for determining whether a certain feature is provided by a software. For example, with semver it is possible to write code like:

if (lib.version >= 1.2.0) {
   call_func_introduced_in_v1_2();
}

However the scheme would not work if you are to number the releases with experimental APIs with 0.c.d. If 0.c.d is the release corresponding to 1.a.0 with experimental APIs added, then the application code needs to me modified as follows so that it would work as expected.

if (lib.version >= 1.a.0 || lib.version == 0.c.d) {
   call_public_API_introduced_in_v1_a_0();
}

And what make it even worse is you cannot even use > for checking 0.c.d since versions before 1.0.0 are allowed to change the API any time. So the application needs to be changed like below every time a new release with experimental APIs that is upper-compatible with 1.a.0 is released.

if (lib.version >= 1.a.0 || (0.c.d <= lib.version || lib.version <= 0.e.f)) {
   call_public_API_introduced_in_v1_a_0();
}

As can be seen, all the merits for using semver has gone.

EDIT: expanded my understanding why it is not a good idea.

[alechenninger]

Semver doesn't mandate how the public API is documented. If your project says that "experimental" APIs are not part of the official public API, then you are following the spec.

For this system to work, you first need to declare a public API. This may consist of documentation or be enforced by the code itself. Regardless, it is important that this API be clear and precise.

[AloisMahdal]

This sounds too much like breaking semver.

Can't you just:

• have feature branch for the particular experiment,
• or maybe have one "experimental" branch for integration testing of these experiments,
• then set up your build scripts that any build coming from these branches must have meta-data refering to the branch or even particular commit in it, eg. 1.0.3+branch_name.g1234def,
• have release engineering follow simple rule: any builds with meta-data must not ever be released?

Then you can merge/rebase back and forth as you want to get yourself plenty of time to do your experiments before they mature enough to be considered for merging into 1.1.0 or 2.0.0.

If you want to expose the "future API" you can always share the actual repos, you can have separate document that includes also the experimental changes somewhere in notes/ sub-folder in your repo, or in a completely separate repo.

[zafarkhaja]

My 2 cents on this

1. I see there's a common misconception among people about the size of the major version numeric. Looks like many think that the major versions are limited by 1 digit or even 1 bit, so they are afraid to go beyond 1 and often stick to either 0 or 1. I think there's nothing wrong with frequently incrementing major versions, even if your version ends up being 256.0.0 very fast. Please refer to the first 4 questions of the FAQ.
2. On the other hand developing additional API, be it experimental or not, would only require incrementing the minor version because it's new and you don't break the existing API. Say, you added an experimental API and then the experiment failed, again you'd just mark the API as "deprecated" and increment the minor version (SemVer p.7). You can later remove the deprecated API in the next major release.

[jwdonahue]

@kazuho, is there some reason why you can't use prerelease tags on these experimental versions? The spec unambiguously defines what the prerelease tags means in terms of the semantics of the version string and it seems to be exactly what you need.

Unless you have further questions or comments, please close this issue at your earliest possible convenience.

[grv87]

I've seen this situation in Gradle. They have a lot of ongoing parallel projects to deliver new functionality.
If they tag each release with experimental API as prerelease they would never make a release. Something new is always cooking, and they try to deliver it ASAP to get fast and early feedback.

They instead follow @alechenninger opinion and mark new API as incubating so that changes in it are not considered as breaking.

[jwdonahue]

@grv87 , the main problem with that approach is your consumers automation is likely not constructing an AST of the API's and reasoning about the relative risks for production or test consumption. Their tooling is making decisions based on the contents of the SemVer string. It is very bad form to release experimental API's in packages that are marked as stable and also bad form to only ever publish prereleases. It's just bad hygiene from an engineering viewpoint and not conforming to SemVer. But then, SemVer is not required in many cases and echo-systems develop around common practice, however unsound they might be.

[AloisMahdal]

@jwdonahue I'm not big fan of what @grv87 mentioned, but is it really so bad to add API? Can addition break anything? IMO the real trouble starts if someone uses the experimental API, and if they do so in production environment. Now if the (in-)stability of the API is made clear, the onus is on the developer to read all the words in the reference and restrain from using it.

I agree it's not the best hygiene to mix experimental and stable; it's much cleaner to provide stone-stable releases with stone-set APIs. I just think it's not really matter of automation but of development. For this to cause a real problem requires even worse hygiene on consumer part.

(Of course it depends on what "addition" means: defining new Python method sure is addition. Inserting element into struct in C is probably not.)

[jwdonahue]

@AloisMahdal, the spec does leave a lot of wiggle room wrt exactly what is being versioned, so I can't really answer your question definitively. Adding API's is one thing, removing or altering it later simply because it had been tagged with a language specific keyword, and not bumping the appropriate version at that point, has a bad smell.

The spec says your documentation or code defines your API, and that would include language keywords such as EXPERIMENTAL. So technically, breaking changes in such code does not require a major version bump. But you've got developers busy breaking things in the code, how stable is the implementation behind the published non-experimental API?

@grv87 says "If they tag each release with experimental API as prerelease they would never make a release". While obviously not true, that suggests an unstable product to me, but I don't really know the details of how Gradle is developed. It is obvious from their published versions, they don't use SemVer. It's one thing to suggest that "this API is subject to change" and entirely different to add "without notice". Do they bump their major version when they make a change in an experimental API?

[grv87]

@jwdonahue, yes, Gradle doesn't use SemVer. I just chose it to demonstrate the concept of experimental (incubating) features. I believe this could work the same way with SemVer.
More detailed description of Gradle feature lifecycle is here.
No, they don't bump major version when experimental API is changed, see e.g. here.

@AloisMahdal, I agree that this is not the best option, but one of. Better way is to split the product to separate plugins/modules and version them separately.

Adding API in existing classes can break subclasses (in some languages). Also, adding API could create conflicts if other projects use the same class/method/… names. But I haven't seen an example when anybody would truly consider these as breaking changes.

[alexandrtovmach]

Thanks everyone for contributions, you're amazing 🎆 Did you find any consensus?

[MatheusRich]

Just adding my 2cents:

Ruby started to adding "experimental" features in minor release to gather feedback from the community. Not only bugs, but to really experiment how those new features feel. Experimental features may be removed or have breaking changes in between minor releases. This approach allowed maintainers to add new features and collaborate with the community without keeping a separated branch or releasing a major release every time.

More about Ruby's versioning here.

[ben-xD]

From my thoughts, this issue is completely solved by spec item 9, the solution being use pre-release versions (e.g. 1.0.0-alpha.1) if you want to keep breaking new APIs you have added. However, once you've released a stable version (non pre-release and 1.0.0 and above), you have to make sure you don't break the APIs already available in those stable releases.

A pre-release version MAY be denoted by appending a hyphen and a series of dot separated identifiers immediately following the patch version. Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading zeroes. Pre-release versions have a lower precedence than the associated normal version. A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.–.

What are your thoughts?

[davidmurdoch]

A discussion about semver has ensued because a library we are developing intends to ship a new (undocumented) property named __experimental_info. I think it is clear that developers who may choose use an undocumented property prefixed with not one, but two _s and the word experimental know that this property will change or be removed in the future.

Clarity on this in the not-too-distant-future would be much appreciated! :-D

[ljharb]

@davidmurdoch it's not clear, actually - in practice, people will grow to depend on this property, and breaking it will break them.

If you don't want something to be part of your API, don't make it reachable from your API.

[davidmurdoch]

@ljharb if that is the stance semver takes then we don't need semver at all; ever single change is breaking. Someone might come to rely on a bug, or might rely on a feature being missing...

[ljharb]

Making a slippery slope argument doesn't change anything. There is a huge difference between a bug and an undocumented property.

[davidmurdoch]

Dismissing my counter argument as a slippery slope is a straw man.

I noticed you work at Coinbase... if you are in NYC for the NFT NYC conference maybe we could debate these things in person 😁.

[jwdonahue]

@davidmurdoch

... if that is the stance semver takes then we don't need semver at all; ever single change is breaking.

Yes, SemVer is not always the correct approach and we should not try to make it the be-all/end-all scheme, that would just complicate it too much.

Someone might come to rely on a bug, or might rely on a feature being missing.

I don't see how anybody could take a dependency on a non-existent feature, but it is the case that developers take dependencies on bugs. I know of more than a few cases where publishers got locked-in to supporting a small set of them due to the sizeable fraction of their customers who came to depend on them, before said bugs were identified. They are mostly low severity bugs that don't have to get fixed, but in some cases, they represented security issues and then the publisher had to break everybody.

When SemVer is used in such cases, the major version bump, tends to slow adoption of the critical security fixes, and require more PR effort to drive adoption. It is less disruptive than releasing the breaking change(s) as a patch however. Annoying the hell out of your customers is not the best way to get them to accept your critical security patches. In many cases, the publishers were well within their rights of claiming that said bugs were not part of the defined API of the product, issuing patches for them and accepting that a small percentage of their user base might not be very happy about it.

A discussion about semver has ensued because a library we are developing intends to ship a new (undocumented) property named __experimental_info. I think it is clear that developers who may choose use an undocumented property prefixed with not one, but two _s and the word experimental know that this property will change or be removed in the future.

If your API docs clearly state your intention that __experimental_info is not part of your supported API, then you are well within your rights to remove it later, without the major version bump. However, the '_experimental' part of that name alone, is not sufficient to indicate that particular publicly available symbol is not part of the supported API, without additional documentation indicating that it is the case. The SemVer spec already allows for that. First it establishes that you define what your API is, then it defines the syntax and rules for bumping/resetting the various SemVer fields.

@kazuho 's proposed change should be rejected IMO. The -prerelease mechanism is already sufficient and I don't see how "marked as experimental" should necessarily be translated to "not part of my API".