Mark all features which are not supported in all SDKs as experimental & API unstable

[lucaswoj]

From @lucaswoj on January 10, 2017 19:20

Credit for this elegant design goes to @jfirebaugh 🙇

When we add new features, especially when we're in the process of porting features across platforms, we often need to revise our initial API design.

We currently do a poor job of communicating to our users which APIs are stable and which APIs are likely to change.

We should prominently mark that any feature which is not supported in all SDKs as "experimental" in the SDK documentation. We should explain that an experimental feature's API may change in a future release.

Copied from original issue: mapbox/mapbox-gl-style-spec#653

[lucaswoj]

From @1ec5 on January 17, 2017 21:26

We should prominently mark that any feature which is not supported in all SDKs as "experimental" in the SDK documentation. We should explain that an experimental feature's API may change in a future release.

How does this relate to the SDKs’ use of semantic versioning? The Android and iOS SDKs and node-mapbox-gl-native are all past v1.0, which means a change to an “experimental” API would require a major version bump, just like a change to any other API. The iOS SDK has carved out some wiggle room for undocumented APIs, but this issue seems to be about documented features.

[lucaswoj]

From @jfirebaugh on January 17, 2017 21:30

The proposal is to exclude features marked as experimental from semantic versioning requirements. They would be free to change in backward-incompatible ways without a major version change for the SDK as a whole.

[lucaswoj]

From @1ec5 on January 17, 2017 21:51

They would be free to change in backward-incompatible ways without a major version change for the SDK as a whole.

I’d be against a general strategy of marking new APIs as unstable. Backwards compatibility is a big deal for iOS and macOS applications. As far as I can tell, the culture around CocoaPods, Carthage, and Cocoa frameworks in general doesn’t allow for parts of the API to be exempted from semantic versioning requirements. There’s no such thing as an “alpha class” or “beta method”. We have three strategies for mitigating this inflexible interpretation, in order of preference:

• Providing migration paths as the API changes. In some cases, we may have to play semantic games or declare (via documentation or deprecation messages) that certain methods no longer have any effect. The SDK eventually becomes littered with “abandoned alignments” until we do spring cleaning in a major version.
• Leaving an experimental API undocumented, as I described above. Apple uses this strategy a lot for “experimental” APIs. This means we also avoid showing developers how to use the API through example code or step-by-step guides. Experienced iOS developers will treat any use of the API as a hack and be very wary of relying on it. We get no points on the board, so to speak, from shipping an undocumented API.
• Exposing only an informal API as part of a flexible container, such as an options dictionary, and leaving no constants for dictionary keys and values. This reduces type safety and increases our reliance on the developer reading and comprehending API documentation. An API that relies on loosely typed dictionaries is difficult to work with in Swift.

These strategies aren’t a replacement for striving to release a stable API for each new feature.

[lucaswoj]

From @1ec5 on January 17, 2017 22:10

I neglected to mention a very important fourth mitigation strategy: prereleases. Android and iOS SDK releases spend a lot of time in gestation. Alpha and beta releases allow us to finalize a feature’s design and implementation before we expect developers to rely on it in production code. The iOS SDK has specific criteria for when a build qualifies as an alpha, beta, or release candidate. With a prerelease process in place, there should be no need to mark a feature as “experimental” and “unstable” in a final release.

[lucaswoj]

I neglected to mention a very important fourth mitigation strategy: prereleases. Android and iOS SDK releases spend a lot of time in gestation. Alpha and beta releases allow us to finalize a feature’s design and implementation before we expect developers to rely on it in production code.

While a feature may be well designed and stable within the context of the iOS SDK, porting the feature to another platform often inspires / requires design changes.

Leaving an experimental API undocumented

This sounds like a solid approach.

1. A majority of the time, features are implemented in GL JS first and will have their "experimental" designation lifted by virtue of their implementation in the native SDKs.
2. The rest of the time, we create a healthy pressure on ourselves to get parity before doing a SDK release.
3. The rare times we can't get parity before an SDK release, we'll include an undocumented API to be honest about the API's stability

[lucaswoj]

From @1ec5 on January 18, 2017 18:25

To reiterate, undocumented APIs might as well not exist: hypothetically, no examples about alternative projections would be published until alternative projections are supported on all platforms, and any blog posts would avoid implying that developers can try out the API. We’d indeed create pressure to get parity on each platform before calling alternative projections done.

The downside is that Web developers would be highly unlikely to use the undocumented API, so there would be less feedback from developers that could help us refine the implementation. The solution is for GL JS to publish prereleases – that was my point in mapbox/mapbox-gl-style-spec#653 (comment). Ensuring feature parity across platforms entails slowing down GL JS’s release cycle (but adding prereleases mid-cycle). I think v1.0 would be a good opportunity to formalize a prerelease strategy for GL JS.

[lucaswoj]

From @jfirebaugh on January 18, 2017 18:46

I don't think it's tenable for GL JS to publish all releases as "prerelease" in periods where it has features that the mobile SDKs do not, and where the implementation in mobile SDKs is likely to require specification or API design changes. Concretely, that would mean that GL JS would have to be on "prerelease" status right now, and for the past year or more.

@1ec5 Can I kindly ask you to reconsider your resistance to at least trying out the proposal here? We have to introduce flexibility somewhere. I believe that it's worth trying out the flagged approach, even though it may deviate from certain cultural norms. As @lucaswoj notes, in most cases it's the GL JS implementation that will come first and be flagged as experimental, and where the reverse is true we can test the waters with small features first. If it doesn't work, we'll find out and try something else. But let's not be afraid to experiment (heh).

[lucaswoj]

From @1ec5 on January 18, 2017 19:24

Concretely, that would mean that GL JS would have to be on "prerelease" status right now, and for the past year or more.

Right, we’d obviously need to avoid a prolonged beta testing phase. This speaks to the “healthy pressure” @lucaswoj refers to.

Can I kindly ask you to reconsider your resistance to at least trying out the proposal here? We have to introduce flexibility somewhere. I believe that it's worth trying out the flagged approach, even though it may deviate from certain cultural norms.

Perhaps I’m misunderstanding the purpose of this ticket. I understood the idea to be that every SDK would at times have to introduce an unstable API, documented as such but otherwise identical to a stable API, until a compatible API is implemented across all six platforms. The iOS SDK has to be a good citizen on its platform, and part of that is abiding by semver as developers understand it. I’m just the messenger.

If we document a documented API in the iOS SDK and tell developers to use it (whether in examples, blog posts, or compatibility tables), the API isn’t exempt from semver no matter how much we claim the contrary. After all, if a developer requires version ~> 3.4 of our pod and follows our examples to the tee, using this unstable API, their code breaks as soon as we release v3.5.0 with a backwards-incompatible change to the unstable API. As soon as they run pod update, they wind up with v3.5.0 and a build failure or, worse, unexpected behavior at runtime.

The status quo of releasing a feature to GL JS first seems to work well regardless of whether we document the feature as being unstable. The reverse, where a feature comes to a native SDK first, requires us to try to anticipate future porting problems during beta testing. But we aren’t perfect at predicting the future, so that’s where the three strategies in mapbox/mapbox-gl-style-spec#653 (comment) come in, especially the first strategy of providing compatibility shims.

If prominently marking “experimental” APIs works for GL JS, that’s great. But I don’t think the proposed approach can be applied to the iOS SDK.

[zugaldia]

Only seeing this ticket right now (thanks @1ec5 for the tip).

With the release of MAS v2.0.0-beta.2 today mapbox/mapbox-java#349 Android has introduced a new @Experimental annotation. We're documenting it as follows:

/**
 * An experimental user-facing API. Experimental APIs are quickly evolving
 * and might change or be removed in minor versions.
 *
 * Inspired by org.apache.spark.annotation.Experimental
 */

and it can be attached to all common Java elements (ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER, ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE).

So basically this sets the annotated element explicitly outside of SEMVER. We're using this annotation carefully and for now, we're only attaching it to the new navigation SDK which we expect to iterate fast over the next few weeks breaking the current API.

For the rest of the Android APIs we're keeping the expected SEMVER stability. Having said that, I'm open to using this same annotation in other places, including to coordinate with the usecase described at the beginning of this ticket.

[colleenmcginnis]

@asheemmamoowala is this something we should consider doing?

[asheemmamoowala]

The @mapbox/gl-native has recently been asking about this as well, and I had a brief discussion with @chloekraw about it too.

A lot has changed since this ticket was opened and discussed, but some things have not.

When we add new features, especially when we're in the process of porting features across platforms, we often need to revise our initial API design.

This has rarely happened over the past year or so, possibly as a coincidence and partly due to the more stable state of the SDKs and their aligned architecture. It can surely happen again, but I think there is now a different rationale for providing experimental APIs:

When developing new features, it is useful to get customer and developer feedback on the RFC, PR, and demo, but also important to allow them to play with the feature in their application or style. This would be on a longer time scale than the 1-2 week window between a pre-release and final release, and should probably happen without landing the PR into master. For example, this may have been useful with text-variable-anchorandtext-writing-mode`.

For GL-JS, features that are in-progress/experimental can be released in special builds from the feature branch on an as needed basis. For the native SDKs this is not as simple, and I think they have different needs for experimental APIs.

I'm not in favor of adding experimental APIs to official GL-JS releases for a couple of reasons:

• The style-spec and GL SDKs are used in many places in the Mapbox stack - in Studio, for server compositing and shaving of tiles, Static tiles API and Static Images API among others. Experimental APIs might break the behavior or expected behavior of these services.
• Many customers want to test new styles in Studio, and it is unlikely that they would build UI for experimental style properties. This would be better suited for a playground-like environment.

[asheemmamoowala]

Closing as something we won't be introducing in a documented way