Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Explore how to roll out experimental features #2676

Open
dhmlau opened this Issue Apr 2, 2019 · 6 comments

Comments

Projects
None yet
4 participants
@dhmlau
Copy link
Contributor

dhmlau commented Apr 2, 2019

Capturing the discussion we had so far regarding this experimental feature and turning it in a discussion thread on GitHub for others to chime in.

Why experimental features

While we want to make sure high quality of code goes into our codebase which may take many iterations, releasing the features as experimental allow the community to try out and we can get early feedback, and improve the code incrementally.


Discussion Topic 1 - Where the experimental code resides or how to indicate as experimental

Proposal 1: Introduce experimental flag

The experimental flag means we release features with different stages like what Node.js does: https://nodejs.org/api/documentation.html#documentation_stability_index

Proposal 2: Encourage composition packages

Create a separate package within the loopback-next monorepo but release only as 0.x.y to indicate it's experimental

Proposal 3: Use master-experimental branch

Use master as the production ready branch (stay unchanged). Create a master-experimental branch for the "master" of experimental features.

Pros:

  • easier for testing because everything is in one monorepo

Cons:

  • need to sync up between master and master-experimental branch. (might be able to set up a CI job for rebase)
  • might skew our code coverage data

Proposal 4: a separate monorepo loopback-labs

Pros:

  • Clear separation between stable (loopback-next) and experimental (loopback-labs)
  • All experiments are in a single place, easy to find and combine together
  • Less overhead in repo setup, from CI & DCO checks to GitHub permissions. (when all the experimental projects are in one mono repo)

Discussion Topic 2 - Acceptance criteria and graduation criteria

We need to determine on:

  • clear goal/acceptance criteria for the experimental features
    • explain the expectations and limitations
  • graduation criteria/expectation

cc @strongloop/loopback-maintainers

@dhmlau

This comment has been minimized.

Copy link
Contributor Author

dhmlau commented Apr 2, 2019

@jannyHou

This comment has been minimized.

Copy link
Contributor

jannyHou commented Apr 3, 2019

@dhmlau The experimental flag means we release features with different stages like what Node.js does: https://nodejs.org/api/documentation.html#documentation_stability_index

(@bajtos could correct me if I am wrong) For "Encourage composition packages", my understanding is we treat features as separate modules that's not part of core.

@bajtos

This comment has been minimized.

Copy link
Member

bajtos commented Apr 5, 2019

we'll create a separate package within the loopback-next monorepo but release only as 0.x.y to indicate it's experimental?
For "Encourage composition packages", my understanding is we treat features as separate modules that's not part of core.

+1, that's my idea too.

The experimental flag means we release features with different stages like what Node.js does: https://nodejs.org/api/documentation.html#documentation_stability_index

IMO, experimental flags is a suboptimal approach as it does not offer enough flexibility both to framework users (they cannot pick a specific version of the experimental feature) and framework maintainers (we have to keep experimental code in our main codebase).

Node.js core is in a different position than we are. Node.js is shipped as a single binary, it is not possible to build a custom Node.js version by combining multiple components. LoopBack is composed from multiple optional building blocks from the beginning, e.g. users can choose to exclude @loopback/service-proxy or pick a different version of @loopback/boot.

I'd like us to try to implement as many experimental features as possible in standalone packages.

Use master-experimental branch
a separate monorepo loopback-labs

These two approaches can be combined together:

  • loopback-labs can be created as a fork of loopback-next
  • in loopback-labs, most development will happen in the master branch
  • the master branch can include code from loopback-next together with the experiments
  • it will be possible to open pull requests from loopback-labs to contribute changes to main codebase in loopback-next

The more I think about this topic the more reluctant I am to keep experimental code in the main loopback-next monorepo.

The aspect most important to me: how to prevent experiments from negatively effecting our "main" work on production-grade codebase and our milestone plans.

  • Code coverage. We are keeping high (~90%) code coverage for our production code and have a process in places to let us know when a pull request would reduce it. If we wanted to land experimental code into loopback-next then we would either have to require the same level of code coverage for experimental code too, or configure our code coverage tool to ignore experimental code. IMO, both options are pretty bad.

  • Test robustness and speed. Even if the code is experimental, once it's landed to loopback-next, it must pass its tests. Not only that, we also need the test suite to be robust, e.g. we don't want to see CI failures caused by timing-dependent tests.

  • Issue tracking and triage. We already have hard times keeping up with the number of incoming issues and supporting users of our production-grade code. We need a mechanism that will make it easy for us to treat issues for experimental code as low priority.

  • GitHub notifications. This is similar to the previous point, we need an easy way how to distinguish notifications about issues and discussions related to production-grade code from notifications related to experimental code.

With a new monorepo hosting experimental code, my four concerns are solved:

  • Code coverage is calculated independently for loopback-next and loopback-labs.
  • Experimental code in loopback-labs is not affecting CI builds for loopback-next in any way
  • Issues and pull requests for experimental code are living in a different issue tracker and are clearly separated.
  • Notifications are triggered for a different GitHub repo, it's possible to use different subscription settings for them (e.g. "watch all" for loopback-next vs. "only mentions" for loopback-labs).

@bajtos bajtos added the discussion label Apr 5, 2019

@dhmlau

This comment has been minimized.

Copy link
Contributor Author

dhmlau commented Apr 6, 2019

For experimental features, I think we want to:

  • make it easily discoverable by our users
  • make it as separate as possible with our non-experimental codebase, so that
    1. we can remove the experimental features with relatively small effort if they are no longer relevant/applicable/desirable;
    2. it won't affect our core codebase's quality, given that there might not be as much as test coverage in the experimental portion.

With the reasons above, I'd prefer proposal 4. The repo name could be something else, but that's implementation details. :)

@raymondfeng mentioned that setting up automated testing/CI for proposal 4 is more complicated. That might be something to consider too, but I don't have enough knowledge to comment on that.

@bajtos bajtos added 2019Q2 p1 labels Apr 12, 2019

@raymondfeng

This comment has been minimized.

Copy link
Member

raymondfeng commented Apr 17, 2019

Let's start with the simplest case here - adding a new module such as @loopback/socketio, which has the following characteristics:

  1. It's an independent module that product-ready modules don't depend on
  2. Any changes required for product-ready modules must be submitted as separate PRs to be reviewed and landed first

To support this case, we can do the following:

  1. Add experimental-packages folder under loopback-next and make it part of lerna
  2. Add README.md for experimental-packages
  3. Drop in experimental packages under experimental-packages and use versions such as 0.x.y
  4. Add a disclaimer to README to set the expectation right.
  5. Develop a simple script to check production-ready modules do not have dependency on experimental packages. Also make sure the experimental packages use 0.x.y versions.
  6. Introduce something like stability: 2 - Stable (used by Node.js features) or even icons to make it clear
@bajtos

This comment has been minimized.

Copy link
Member

bajtos commented Apr 18, 2019

@raymondfeng please expand your proposal to describe how do you envision to address the concerns I raised in my earlier comment.

The aspect most important to me: how to prevent experiments from negatively effecting our "main" work on production-grade codebase and our milestone plans.

  • Code coverage. We are keeping high (~90%) code coverage for our production code and have a process in places to let us know when a pull request would reduce it. If we wanted to land experimental code into loopback-next then we would either have to require the same level of code coverage for experimental code too, or configure our code coverage tool to ignore experimental code. IMO, both options are pretty bad.

  • Test robustness and speed. Even if the code is experimental, once it's landed to loopback-next, it must pass its tests. Not only that, we also need the test suite to be robust, e.g. we don't want to see CI failures caused by timing-dependent tests.

  • Issue tracking and triage. We already have hard times keeping up with the number of incoming issues and supporting users of our production-grade code. We need a mechanism that will make it easy for us to treat issues for experimental code as low priority.

  • GitHub notifications. This is similar to the previous point, we need an easy way how to distinguish notifications about issues and discussions related to production-grade code from notifications related to experimental code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.