[Spike] Explore how to roll out experimental features

[dhmlau]

Capturing the discussion we had so far regarding this experimental feature and turning it in a discussion thread on GitHub for others to chime in.

Why experimental features

While we want to make sure high quality of code goes into our codebase which may take many iterations, releasing the features as experimental allow the community to try out and we can get early feedback, and improve the code incrementally.

---

Discussion Topic 1 - Where the experimental code resides or how to indicate as experimental

Proposal 1: Introduce experimental flag

The experimental flag means we release features with different stages like what Node.js does: https://nodejs.org/api/documentation.html#documentation_stability_index

Proposal 2: Encourage composition packages

Create a separate package within the loopback-next monorepo but release only as 0.x.y to indicate it's experimental

Proposal 3: Use master-experimental branch

Use master as the production ready branch (stay unchanged). Create a master-experimental branch for the "master" of experimental features.

Pros:

• easier for testing because everything is in one monorepo

Cons:

• need to sync up between master and master-experimental branch. (might be able to set up a CI job for rebase)
• might skew our code coverage data

Proposal 4: a separate monorepo loopback-labs

Pros:

• Clear separation between stable (loopback-next) and experimental (loopback-labs)
• All experiments are in a single place, easy to find and combine together
• Less overhead in repo setup, from CI & DCO checks to GitHub permissions. (when all the experimental projects are in one mono repo)

---

Discussion Topic 2 - Acceptance criteria and graduation criteria

We need to determine on:

• clear goal/acceptance criteria for the experimental features
  • explain the expectations and limitations
• graduation criteria/expectation

cc @strongloop/loopback-maintainers

[dhmlau]

@strongloop/loopback-next, thoughts?

[jannyHou]

@dhmlau The experimental flag means we release features with different stages like what Node.js does: https://nodejs.org/api/documentation.html#documentation_stability_index

(@bajtos could correct me if I am wrong) For "Encourage composition packages", my understanding is we treat features as separate modules that's not part of core.

[bajtos]

we'll create a separate package within the loopback-next monorepo but release only as 0.x.y to indicate it's experimental?
For "Encourage composition packages", my understanding is we treat features as separate modules that's not part of core.

+1, that's my idea too.

The experimental flag means we release features with different stages like what Node.js does: https://nodejs.org/api/documentation.html#documentation_stability_index

IMO, experimental flags is a suboptimal approach as it does not offer enough flexibility both to framework users (they cannot pick a specific version of the experimental feature) and framework maintainers (we have to keep experimental code in our main codebase).

Node.js core is in a different position than we are. Node.js is shipped as a single binary, it is not possible to build a custom Node.js version by combining multiple components. LoopBack is composed from multiple optional building blocks from the beginning, e.g. users can choose to exclude @loopback/service-proxy or pick a different version of @loopback/boot.

I'd like us to try to implement as many experimental features as possible in standalone packages.

Use master-experimental branch
a separate monorepo loopback-labs

These two approaches can be combined together:

• loopback-labs can be created as a fork of loopback-next
• in loopback-labs, most development will happen in the master branch
• the master branch can include code from loopback-next together with the experiments
• it will be possible to open pull requests from loopback-labs to contribute changes to main codebase in loopback-next

The more I think about this topic the more reluctant I am to keep experimental code in the main loopback-next monorepo.

The aspect most important to me: how to prevent experiments from negatively effecting our "main" work on production-grade codebase and our milestone plans.

• Code coverage. We are keeping high (~90%) code coverage for our production code and have a process in places to let us know when a pull request would reduce it. If we wanted to land experimental code into loopback-next then we would either have to require the same level of code coverage for experimental code too, or configure our code coverage tool to ignore experimental code. IMO, both options are pretty bad.

• Test robustness and speed. Even if the code is experimental, once it's landed to loopback-next, it must pass its tests. Not only that, we also need the test suite to be robust, e.g. we don't want to see CI failures caused by timing-dependent tests.

• Issue tracking and triage. We already have hard times keeping up with the number of incoming issues and supporting users of our production-grade code. We need a mechanism that will make it easy for us to treat issues for experimental code as low priority.

• GitHub notifications. This is similar to the previous point, we need an easy way how to distinguish notifications about issues and discussions related to production-grade code from notifications related to experimental code.

With a new monorepo hosting experimental code, my four concerns are solved:

• Code coverage is calculated independently for loopback-next and loopback-labs.
• Experimental code in loopback-labs is not affecting CI builds for loopback-next in any way
• Issues and pull requests for experimental code are living in a different issue tracker and are clearly separated.
• Notifications are triggered for a different GitHub repo, it's possible to use different subscription settings for them (e.g. "watch all" for loopback-next vs. "only mentions" for loopback-labs).

[dhmlau]

For experimental features, I think we want to:

• make it easily discoverable by our users
• make it as separate as possible with our non-experimental codebase, so that
  1. we can remove the experimental features with relatively small effort if they are no longer relevant/applicable/desirable;
  2. it won't affect our core codebase's quality, given that there might not be as much as test coverage in the experimental portion.

With the reasons above, I'd prefer proposal 4. The repo name could be something else, but that's implementation details. :)

@raymondfeng mentioned that setting up automated testing/CI for proposal 4 is more complicated. That might be something to consider too, but I don't have enough knowledge to comment on that.

[raymondfeng]

Let's start with the simplest case here - adding a new module such as @loopback/socketio, which has the following characteristics:

1. It's an independent module that product-ready modules don't depend on
2. Any changes required for product-ready modules must be submitted as separate PRs to be reviewed and landed first

To support this case, we can do the following:

1. Add experimental-packages folder under loopback-next and make it part of lerna
2. Add README.md for experimental-packages
3. Drop in experimental packages under experimental-packages and use versions such as 0.x.y
4. Add a disclaimer to README to set the expectation right.
5. Develop a simple script to check production-ready modules do not have dependency on experimental packages. Also make sure the experimental packages use 0.x.y versions.
6. Introduce something like stability: 2 - Stable (used by Node.js features) or even icons to make it clear

[bajtos]

@raymondfeng please expand your proposal to describe how do you envision to address the concerns I raised in my earlier comment.

The aspect most important to me: how to prevent experiments from negatively effecting our "main" work on production-grade codebase and our milestone plans.

• Code coverage. We are keeping high (~90%) code coverage for our production code and have a process in places to let us know when a pull request would reduce it. If we wanted to land experimental code into loopback-next then we would either have to require the same level of code coverage for experimental code too, or configure our code coverage tool to ignore experimental code. IMO, both options are pretty bad.

• Test robustness and speed. Even if the code is experimental, once it's landed to loopback-next, it must pass its tests. Not only that, we also need the test suite to be robust, e.g. we don't want to see CI failures caused by timing-dependent tests.

• Issue tracking and triage. We already have hard times keeping up with the number of incoming issues and supporting users of our production-grade code. We need a mechanism that will make it easy for us to treat issues for experimental code as low priority.

• GitHub notifications. This is similar to the previous point, we need an easy way how to distinguish notifications about issues and discussions related to production-grade code from notifications related to experimental code.

[nabdelgadir]

+1 to keeping the experimental code outside of loopback-next master branch.

These two approaches can be combined together:

• loopback-labs can be created as a fork of loopback-next
• in loopback-labs, most development will happen in the master branch
• the master branch can include code from loopback-next together with the experiments
• it will be possible to open pull requests from loopback-labs to contribute changes to main codebase in loopback-next

In this approach, it's easier to keep the experimental code up-to-date with the non-experimental code and it would be easier to open a PR to land them in the master loopback-next branch when they're ready and graduated.

We can add a link to the loopback-next README for visibility and create a blog post introducing it.

Overall, imo this approach mixes the best of having the code outside the main loopback-next master branch (Miroslav's pros) but also having the experimental code along with the non-experimental code if the non-experimental code is needed instead of having it in a completely isolated repo. Also this way we don't have to set up a new repo and a completely separate CI.

[b-admike]

From the points made above, and from the graduation criteria for experimental features, I think setting up a new repository that can have lighter CI (e.g. just Travis instead of Travis/appveyor) and standalone code coverage would be good. Issues coming in for experimental features will also be triaged there. Discussing the topic with the team here, one concern is migration of the experimental code from the standalone repo to loopback-next once we deem it production ready. In that case, keeping the experimental features within loopback-next might make sense since it's all in one repository (plus we can use new labels in loopback-next for experimental stuff). Another point made in our discussion is when users have code in their experimental packages that also needs some changes in our core modules, then how it becomes tricky to integrate/test alongside without having the experimental code in our monorepo. I think we agree on rolling out these features, but not quite sure which direction to follow, so perhaps we could conduct a spike on the approach to take using one of the in-progress experimental packages.

[emonddr]

I like the idea of a separate repo for experimental stuff.
I guess if someone from the community submits a PR that will affect a core module in LoopBack, it should land in loopback-next. If someone from the community submits a PR that introduces some new component that is in our Roadmap, it should land in loopback-next. If someone from the community submits a PR that introduces some new component that is not in our roadmap, then at this point it lands in loopback-labs? :)

[jannyHou]

For the replaceable/plugable functionalities, I prefer to have them in a separate monorepo like loopback-lab. An example of such feature would be the authentication extension point: @loopback/authentication works w/ or w/o it.

While for features that affect the core module, like Context class extends the EventEmitter, it's hard to have it in a separate module at the beginning then migrate it to loopback-next after the code is well tested and mutual.

So I would suggest we treat features differently.

[hacksparrow]

+1 for 0.x.y.

[dhmlau]

@hacksparrow, do you mean to have the experimental packages reside in loopback-next monorepo and versioning using 0.x.y convention?

[bajtos]

I guess if someone from the community submits a PR that will affect a core module in LoopBack, it should land in loopback-next. If someone from the community submits a PR that introduces some new component that is in our Roadmap, it should land in loopback-next. If someone from the community submits a PR that introduces some new component that is not in our roadmap, then at this point it lands in loopback-labs? :)

Personally, I'd like to use a slightly different criteria to decide where to land a new component.

• Is the component on our roadmap AND are we comfortable to commit to maintain it going forward AND does the proposed implementation meet our current high standards of quality?
• If the answer is 3x yes, then land it in loopback-next.
• Otherwise land it in loopback-labs.

One of the goals of loopback-labs is to allow new components to be developed incrementally, starting with small bits and pieces that may not even work together well. So for example, to develop GraphQL server layer, we can start by landing a small pull request adding the new package and perhaps an empty (dummy) server class. Then we can incrementally add more meat to this walking skeleton, keeping individual pull requests small and easy to review.

If we said that GraphQL should go directly to loopback-next because it's on our roadmap, then the first pull request would have to include a meaningfully-complete implementation with 90%+ code coverage, which may cause the pull request to be too large to be reviewed effectively.

[hacksparrow]

@dhmlau yes, everything in loopback-next monorepo and versioned 0.x.y.

[raymondfeng]

To get started, I have created https://github.com/strongloop/loopback-labs from https://github.com/strongloop/loopback-next. We can decide:

1. If we should track loopback-next modules in loopback-next so that we can always rebase to the master@upstream?
2. Should we create an experimental directory or just use packages?

[dhmlau]

Had a discussion with the team, most of us are more inclined for a separate repo to contain the experimental features. Here is the summary:

• New repo: loopback-labs
  • All the new packages inside will be under the labs folder, just like we have examples, packages in loopback-next repo
  • loopback-labs will be a mirror clone of loopback-next, cannot be an official fork because github doesn't allow us to create a fork within the same github org and it would be too much effort to set up teams/access/CI if we create a new org.
  • In loopback-labs, we need to mark all loopback-next packages as private so that it won't be published to npm
• README of loopback-labs needs to:
  - provide link to loopback-next
  - purpose of loopback-labs and the relationships with loopback-next
  - document the setup, the workflow and how it works between the 2 repos
• publish on npm
  • we will use 0.x.y convention to indicate that it is for experimental features
• Things to consider in the future:
  • set up CI to rebase loopback-labs from loopback-next
  • what if we want to have experimental feature that is part of the loopback-next repo and cannot be put in the loopback-labs

Questions:

• What if code changes required for enabling the experimental features are outside the package (e.g. changes required in the core, loopback-next)?
  • If changes in core are needed, we need to open a new PR, land it and rebase in loopback-labs.
• How to move code from new repo back to loopback-next?
  • set up upstreams and then cherry picks commits from loopback-labs back to loopback-next so that we can preserve the git history.
• Is loopback-labs open for community?
  • Definitely! That's why we need to make sure we well-documented the process so that community can contribute as well.

Acceptance Criteria

Based on the above conclusion, the spike should exercise the workflow to show how it works bi-directionally, with the use of a dummy experimental feature:

• loopback-next -> loopback-labs: how to rebase /merge with changes from loopback-next to loopback-labs
• loopback-labs -> loopback-next: when the experimental feature graduates, how we can move code (together with git history) back to `loopback-next.
• describe the setup and workflow using the options explored in the spike
• describe how to set up CI (or simply set up CI for the new repo) - CI configs are copied over from loopback-next, so describe those processes (same set of jobs, etc.)

@strongloop/loopback-maintainers, I've put the summary of our discussion here. Please edit directly if there's anything I missed or that's incorrect. Please also review the acceptance criteria that was generated from the meeting. Thanks!

[bajtos]

As I was reviewing #2834, I realized there is one more area to research: how can experimental packages contribute documentation (hand-written markdown files) to https://loopback.io/doc/en/lb4 and API documentation to http://apidocs.loopback.io or the new apidocs solution proposed in #2834

[jannyHou]

Update

The diagram and work flow of how loopback-next and loopback-labs interact with each other is documented in file https://github.com/strongloop/loopback-labs/blob/master/LABS.md (Thank you @raymondfeng )

Let's continue the spike based on it.

[jannyHou]

Answer acceptance criteria questions:

Having experimental features inside loopback-next

how to rebase /merge with changes from master to the feature branch

Rebase the experimental branch against master to pick up the latest change from master.

git checkout experimental/<my-feature>
git rebase -i master

Then rebase the experimental dev branch against experimental branch

 git checkout experimental-dev/<my-feature>
 git rebase -i experimental/<my-feature>

when the experimental feature graduates, how we can move code (together with git history) back to folder 'packages'.

• Go to the feature branch git checkout experimental/<my-feature>
• Move the entire experimental package folder from /labs to /packages
• Update the entries/links in https://loopback.io/doc/en/contrib/code-contrib-lb4.html#register-the-new-package from loopback-next/labs to loopback-next/packages
• Rebase against master and merge the PR into master. The commit histories will be preserved just like how a regular branch get merged into the master.
• Make a new release that bumps the feature package's version from 0.x.x to 1.0.0

describe the setup and workflow using the options explored in the spike

See PR #2916

describe how to set up CI

N/A because the feature branches are created inside loopback-next

Q&A

what if code changes required for enabling the experimental features are outside the package (e.g. changes required in the core, loopback-next)?

• Option 1:
  If changes in core are needed, we need to open a new PR, land it and rebase in loopback-labs.
• Option 2:
  Rebase the experimental branch against master to pick up the latest change from master.

  git checkout experimental/<my-feature>
  git rebase -i master
  

  Then rebase the experimental dev branch against experimental branch

   git checkout experimental-dev/<my-feature>
   git rebase -i experimental/<my-feature>
  

The other 2 questions are only valid when using a separate repository

---

Having experimental features inside loopback-labs

how to rebase /merge with changes from loopback-next to loopback-labs

• To get the latest RELEASED code from loopback-next, just update the dependencies' versions.
• Since loopback-labs is a mirror repository, the base branch for its PRs can only be its own branch. This results in we need a branch(e.g.loopback-labs/master) sync with loobpack-next/master all the time to make sure the feature branch can pick up the latest NON-RELEASED code from loopback-next/master.
  Otherwise do:

  git checkout experimental/<my-feature>
  git rebase -i next/master

  will introduce noisy commits from loopback-next/master, like https://github.com/strongloop/loopback-labs/pull/6/commits.

graduation

See the LABS.md file in PR strongloop/loopback-labs#10
And PR #2926 illustrates the graduation.

setup, workflow, CI

See PR strongloop/loopback-labs#10

[jannyHou]

Comparison

Guide for using loopback-labs: see PR strongloop/loopback-labs#10
Guide for using loopback-next: see comment #2916 (comment)

Con of using loopback-labs:

• If the feature has changes in production ready packages, rebasing against loopback-next@master is hard and complicated in loopback-labs, see my explanation in [Spike] Explore how to roll out experimental features #2676 (comment)

Since loopback-labs is a mirror repository, the base branch for its PRs can only be its own branch.
This results in we need a branch(e.g.loopback-labs/master) sync with loobpack-next/master all the time to make sure the feature branch can pick up the latest NON-RELEASED code from loopback-next/master.
Otherwise do:

git checkout experimental/
git rebase -i next/master

will introduce noisy commits from loopback-next/master, like https://github.com/strongloop/loopback-labs/pull/6/commits.

Pro of using loopback-labs:

• Better organization of issue tracking and triaging. Clean notifications. see Miroslav's comment in [Spike] Explore how to roll out experimental features #2676 (comment)
  • Solution: we can use tags to organize the experimental issues, and ideally we wouldn't have too many experimental packages. It would be good to graduate them actively.

[jannyHou]

Update

Had a meeting with loopback team to update our findings, here is a summary of key points:

loopback-labs is created for the purpose of keeping lab packages isolated from the production ready packages. During the spike, we realized that a better way to isolate each feature is creating and releasing each feature from its own branch lab/<my-feature-name>, so that features are independent from each other and have their own lifecycle. It also makes the release easier, no need to release multiple packages a t one time.

Given this branching strategy, adding lab packages in loopback-next would simplify everything, especially we don't need to maintain the mirror branch of loopback-next@master or synchronize loopback-lab@master with loopback-next@master in the lab repository.

A PR is created to update the work flow of adding feature in loopback-next: #2985

The work flow for the loopback-labs strategy is still open in strongloop/loopback-labs#10, and the comparison between the two strategies is still valid in #2676 (comment), if team agrees on taking the loopback-next strategy, I will close that PR.

Next to do:

• Finalize the loopback-next@labs branch, see PR Update labs #2985
• (Creating adapter for the plugged in passport strategies #2311 on going)Work on an experimental feature to verify the steps in the instruction
• After we have a released experimental package, publish its api doc to https://apidocs.strongloop.com/ (I wasn't able to figure out how to publish the apidocs in a reasonable time, so I suggest we do it in a separate story): [Lab feature] Figure out how to publish apidocs for lab features #3039

[bajtos]

After we have a released experimental package, publish its api doc

👍

Few more tasks to consider:

1. How to build API docs using the new workflow based on api-extractor (see feat(tsdocs): add integration with api-extractor/documenter #2834) /cc @raymondfeng
2. How to add hand-written documentation for experimental packages to https://loopback.io/doc/en/lb4/index.html.

Re 2: For example, we can set the convention that experimental packages contain all documentation in their README and then leverage README inclusion to show the docs on our site.

[jannyHou]

The work flow of experimental features is documented in https://github.com/strongloop/loopback-next/blob/labs/base/LABS.md

follow up story created as #3039.
I am closing this spike 🎉