Skip to content
Permalink
Browse files

feat: Add feature toggles (DSP-910) (#1742)

  • Loading branch information
benjamingeer committed Nov 5, 2020
1 parent 060b627 commit 2e6db2e3f32fc2d6fffac25b982ae85fe354f834
Showing with 2,793 additions and 832 deletions.
  1. +78 −0 docs/03-apis/feature-toggles.md
  2. +2 −0 docs/03-apis/index.md
  3. +1 −1 docs/05-internals/design/api-v2/how-to-add-a-route.md
  4. +277 −0 docs/05-internals/design/principles/feature-toggles.md
  5. +1 −0 docs/05-internals/design/principles/index.md
  6. +1 −0 third_party/dependencies.bzl
  7. +17 −0 webapi/src/main/resources/application.conf
  8. +2 −9 webapi/src/main/scala/org/knora/webapi/app/ApplicationActor.scala
  9. +8 −2 webapi/src/main/scala/org/knora/webapi/exceptions/Exceptions.scala
  10. +20 −0 webapi/src/main/scala/org/knora/webapi/feature/BUILD.bazel
  11. +439 −0 webapi/src/main/scala/org/knora/webapi/feature/FeatureFactory.scala
  12. +1 −1 webapi/src/main/scala/org/knora/webapi/http/version/ServerVersion.scala
  13. +3 −4 webapi/src/main/scala/org/knora/webapi/messages/admin/responder/sipimessages/SipiMessagesADM.scala
  14. +1 −0 webapi/src/main/scala/org/knora/webapi/routing/BUILD.bazel
  15. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/HealthRoute.scala
  16. +59 −7 webapi/src/main/scala/org/knora/webapi/routing/KnoraRoute.scala
  17. +6 −8 webapi/src/main/scala/org/knora/webapi/routing/RejectingRoute.scala
  18. +20 −15 webapi/src/main/scala/org/knora/webapi/routing/RouteUtilADM.scala
  19. +50 −38 webapi/src/main/scala/org/knora/webapi/routing/RouteUtilV2.scala
  20. +4 −3 webapi/src/main/scala/org/knora/webapi/routing/SwaggerApiDocsRoute.scala
  21. +6 −7 webapi/src/main/scala/org/knora/webapi/routing/VersionRoute.scala
  22. +61 −51 webapi/src/main/scala/org/knora/webapi/routing/admin/GroupsRouteADM.scala
  23. +8 −246 webapi/src/main/scala/org/knora/webapi/routing/admin/ListsRouteADM.scala
  24. +64 −57 webapi/src/main/scala/org/knora/webapi/routing/admin/PermissionsRouteADM.scala
  25. +181 −138 webapi/src/main/scala/org/knora/webapi/routing/admin/ProjectsRouteADM.scala
  26. +10 −8 webapi/src/main/scala/org/knora/webapi/routing/admin/SipiRouteADM.scala
  27. +8 −6 webapi/src/main/scala/org/knora/webapi/routing/admin/StoreRouteADM.scala
  28. +155 −123 webapi/src/main/scala/org/knora/webapi/routing/admin/UsersRouteADM.scala
  29. +57 −0 webapi/src/main/scala/org/knora/webapi/routing/admin/lists/ListsRouteADMFeatureFactory.scala
  30. +293 −0 webapi/src/main/scala/org/knora/webapi/routing/admin/lists/NewListsRouteADMFeature.scala
  31. +290 −0 webapi/src/main/scala/org/knora/webapi/routing/admin/lists/OldListsRouteADMFeature.scala
  32. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v1/AssetsRouteV1.scala
  33. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v1/AuthenticationRouteV1.scala
  34. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v1/CkanRouteV1.scala
  35. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v1/ListsRouteV1.scala
  36. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v1/ProjectsRouteV1.scala
  37. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v1/ResourceTypesRouteV1.scala
  38. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v1/ResourcesRouteV1.scala
  39. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v1/SearchRouteV1.scala
  40. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v1/StandoffRouteV1.scala
  41. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v1/UsersRouteV1.scala
  42. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v1/ValuesRouteV1.scala
  43. +2 −1 webapi/src/main/scala/org/knora/webapi/routing/v2/AuthenticationRouteV2.scala
  44. +8 −3 webapi/src/main/scala/org/knora/webapi/routing/v2/ListsRouteV2.scala
  45. +8 −3 webapi/src/main/scala/org/knora/webapi/routing/v2/MetadataRouteV2.scala
  46. +54 −23 webapi/src/main/scala/org/knora/webapi/routing/v2/OntologiesRouteV2.scala
  47. +39 −19 webapi/src/main/scala/org/knora/webapi/routing/v2/ResourcesRouteV2.scala
  48. +26 −10 webapi/src/main/scala/org/knora/webapi/routing/v2/SearchRouteV2.scala
  49. +4 −1 webapi/src/main/scala/org/knora/webapi/routing/v2/StandoffRouteV2.scala
  50. +14 −5 webapi/src/main/scala/org/knora/webapi/routing/v2/ValuesRouteV2.scala
  51. +110 −5 webapi/src/main/scala/org/knora/webapi/settings/KnoraSettings.scala
  52. +2 −2 webapi/src/main/scala/org/knora/webapi/store/triplestore/TriplestoreManager.scala
  53. +8 −12 webapi/src/main/scala/org/knora/webapi/store/triplestore/http/HttpTriplestoreConnector.scala
  54. +3 −6 webapi/src/test/scala/org/knora/webapi/E2ESpec.scala
  55. +4 −1 webapi/src/test/scala/org/knora/webapi/R2RSpec.scala
  56. +18 −0 webapi/src/test/scala/org/knora/webapi/e2e/BUILD.bazel
  57. +341 −0 webapi/src/test/scala/org/knora/webapi/e2e/FeatureToggleR2RSpec.scala
  58. +4 −4 webapi/src/test/scala/org/knora/webapi/e2e/http/ServerVersionE2ESpec.scala
  59. +1 −1 webapi/src/test/scala/org/knora/webapi/http/version/ServerVersionSpec.scala
@@ -0,0 +1,78 @@
<!---
Copyright © 2015-2019 the contributors (see Contributors.md).
This file is part of Knora.
Knora is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
Knora is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public
License along with Knora. If not, see <http://www.gnu.org/licenses/>.
-->

# Feature Toggles

Some Knora features can be turned on or off on a per-request basis.
This mechanism is based on
[Feature Toggles (aka Feature Flags)](https://martinfowler.com/articles/feature-toggles.html).

For example, a new feature that introduces a breaking API change may first be
introduced with a feature toggle that leaves it disabled by default, so that clients
can continue using the old functionality.

When the new feature is ready to be tested with client code, the Knora release notes
and documentation will indicate that it can be enabled on a per-request basis, as explained
below.

At a later date, the feature may be enabled by default, and the release notes
will indicate that it can still be disabled on a per-request basis by clients
that are not yet ready to use it.

There may be more than one version of a feature toggle. Every feature
toggle has at least one version number, which is an integer. The first
version is 1.

Most feature toggles have an expiration date, after which they will be removed.

## Request Header

A client can override one or more feature toggles by submitting the HTTP header
`X-Knora-Feature-Toggles`. Its value is a comma-separated list of
toggles. Each toggle consists of:

1. its name
2. a colon
3. the version number
4. an equals sign
5. a boolean value, which can be `on`/`off`, `yes`/`no`, or `true`/`false`

Using `on`/`off` is recommended for clarity. For example:

```
X-Knora-Feature-Toggles: new-foo:2=on,new-bar=off,fast-baz:1=on
```

A version number must be given when enabling a toggle.
Only one version of each toggle can be enabled at a time.
If a toggle is enabled by default, and you want a version
other than the default version, simply enable the toggle,
specifying the desired version number. The version number
you specify overrides the default.

Disabling a toggle means disabling all its versions. When
a toggle is disabled, you will get the functionality that you would have
got before the toggle existed. Therefore, a version number cannot
be given when disabling a toggle.

## Response Header

Knora API v2 and admin API responses contain the header
`X-Knora-Feature-Toggles`. It lists all configured toggles,
in the same format as the corresponding request header.
@@ -29,3 +29,5 @@ The Knora APIs include:
administering projects that use Knora as well as Knora itself.
* The Knora [Util API](api-util/index.md), which is intended to be used for information retrieval
about the Knora-stack itself.

Knora API v2 and the admin API support [Feature Toggles](feature-toggles.md).
@@ -64,7 +64,7 @@ See the routes in that package for examples. Typically, each route
route will construct a responder request message and pass it to
`RouteUtilV2.runRdfRouteWithFuture` to handle the request.

Finally, add your `knoraApiPath` function to the `apiRoutes` member
Finally, add your route's `knoraApiPath` function to the `apiRoutes` member
variable in `KnoraService`. Any exception thrown inside the route will
be handled by the `KnoraExceptionHandler`, so that the correct client
response (including the HTTP status code) will be returned.
@@ -0,0 +1,277 @@
<!---
Copyright © 2015-2019 the contributors (see Contributors.md).
This file is part of Knora.
Knora is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
Knora is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public
License along with Knora. If not, see <http://www.gnu.org/licenses/>.
-->

# Feature Toggles

For an overview of feature toggles, see
[Feature Toggles (aka Feature Flags)](https://martinfowler.com/articles/feature-toggles.html).
The design presented here is partly inspired by that article.

## Requirements

- It should be possible to turn features on and off by:

- changing a setting in `application.conf`

- sending a particular HTTP header value with an API request

- (in the future) using a web-based user interface to configure a
feature toggle service that multiple subsystems can access


- Feature implementations should be produced by factory classes,
so that the code using a feature does not need to know
about the toggling decision.

- Feature factories should use toggle configuration taken
from different sources, without knowing where the configuration
came from.

- An HTTP response should indicate which features are turned
on.

- A feature toggle should have metadata such as a description,
an expiration date, developer contact information, etc.

- A feature toggle should have a version number, so
you can get different versions of the same feature.

- It should be possible to configure a toggle in `application.conf`
so that its setting cannot be overridden per request.

- The design of feature toggles should avoid ambiguity and
try to prevent situations where clients might be surprised by
unexpected functionality. It should be clear what will change
when a client requests a particular toggle setting. Therefore,
per-request settings should require the client to be explicit
about what is being requested.

## Design

### Configuration

### Base Configuration

The base configuration of feature toggles is in `application.conf`
under `app.feature-toggles`. Example:

```
app {
feature-toggles {
new-foo {
description = "Replace the old foo routes with new ones."
available-versions = [ 1, 2 ]
default-version = 1
enabled-by-default = yes
override-allowed = yes
expiration-date = "2021-12-01T00:00:00Z"
developer-emails = [
"A developer <a.developer@example.org>"
]
}
new-bar {
description = "Replace the old bar routes with new ones."
available-versions = [ 1, 2, 3 ]
default-version = 3
enabled-by-default = yes
override-allowed = yes
expiration-date = "2021-12-01T00:00:00Z"
developer-emails = [
"A developer <a.developer@example.org>"
]
}
fast-baz {
description = "Replace the slower, more accurate baz route with a faster, less accurate one."
available-versions = [ 1 ]
default-version = 1
enabled-by-default = no
override-allowed = yes
developer-emails = [
"A developer <a.developer@example.org>"
]
}
}
}
```

All fields are required except `expiration-date`.

Since it may not be possible to predict which toggles will need versions,
all toggles must have at least one version. (If a toggle could be created
without versions, and then get versions later, it would not be obvious
what should happen if a client then requested the toggle without specifying
a version number.) Version numbers must be an ascending sequence of
consecutive integers starting from 1.

If `expiration-date` is provided, it must be an [`xsd:dateTimeStamp`](http://www.datypic.com/sc/xsd11/t-xsd_dateTimeStamp.html). All feature toggles
should have expiration dates except for long-lived ops toggles like `fast-baz` above.

`KnoraSettingsFeatureFactoryConfig` reads this base configuration on startup. If
a feature toggle has an expiration date in the past, a warning is logged
on startup.

### Per-Request Configuration

A client can override the base configuration by submitting the HTTP header
`X-Knora-Feature-Toggles`. Its value is a comma-separated list of
toggles. Each toggle consists of:

1. its name
2. a colon
3. the version number
4. an equals sign
5. a boolean value, which can be `on`/`off`, `yes`/`no`, or `true`/`false`

Using `on`/`off` is recommended for clarity. For example:

```
X-Knora-Feature-Toggles: new-foo:2=on,new-bar=off,fast-baz:1=on
```

A version number must be given when enabling a toggle.
Only one version of each toggle can be enabled at a time.
If a toggle is enabled by default, and you want a version
other than the default version, simply enable the toggle,
specifying the desired version number. The version number
you specify overrides the default.

Disabling a toggle means disabling all its versions. When
a toggle is disabled, you will get the functionality that you would have
got before the toggle existed. A version number cannot
be given when disabling a toggle, because it would not
be obvious what this would mean (disable all versions
or only the specified version).

## Response Header

Knora API v2 and admin API responses contain the header
`X-Knora-Feature-Toggles`. It lists all configured toggles,
in the same format as the corresponding request header.

## Implementation Framework

A `FeatureFactoryConfig` reads feature toggles from some
configuration source, and optionally delegates to a parent
`FeatureFactoryConfig`.

`KnoraRoute` constructs a `KnoraSettingsFeatureFactoryConfig`
to read the base configuration. For each request, it
constructs a `RequestContextFeatureFactoryConfig`, which
reads the per-request configuration and has the
`KnoraSettingsFeatureFactoryConfig` as its parent.
It then passes the per-request configuration object to the `makeRoute`
method, which can in turn pass it to a feature factory,
or send it in a request message to allow a responder to
use it.

### Feature Factories

The traits `FeatureFactory` and `Feature` are just tagging traits,
to make code clearer. The factory methods in a feature
factory will depend on the feature, and need only be known by
the code that uses the feature. The only requirement is that
each factory method must take a `FeatureFactoryConfig` parameter.

To get a `FeatureToggle`, a feature factory
calls `featureFactoryConfig.getToggle`, passing the name of the toggle.
If a feature toggle has only one version, it is enough to test
whether test if the toggle is enabled, by calling `isEnabled` on the toggle.

If the feature toggle has more than one version, call its `getMatchableState`
method. To allow the compiler to check that matches on version numbers
are exhaustive, this method is designed to be used with a sealed trait
(extending `Version`) that is implemented by case objects representing
the feature's version numbers. The method returns an instance of
`MatchableState`, which is analogous to `Option`: it is either `Off`
or `On`, and an instance of `On` contains one of the version objects.
For example:

```
// A trait for version numbers of the new 'foo' feature.
sealed trait NewFooVersion extends Version
// Represents version 1 of the new 'foo' feature.
case object NEW_FOO_1 extends NewFooVersion
// Represents version 2 of the new 'foo' feature.
case object NEW_FOO_2 extends NewFooVersion
// The old 'foo' feature implementation.
private val oldFoo = new OldFooFeature
// The new 'foo' feature implementation, version 1.
private val newFoo1 = new NewFooVersion1Feature
// The new 'foo' feature implementation, version 2.
private val newFoo2 = new NewFooVersion2Feature
def makeFoo(featureFactoryConfig: FeatureFactoryConfig): Foo = {
// Get the 'new-foo' feature toggle.
val fooToggle: FeatureToggle = featureFactoryConfig.getToggle("new-foo")
// Choose an implementation according to the toggle state.
fooToggle.getMatchableState(NEW_FOO_1, NEW_FOO_2) match {
case Off => oldFoo
case On(NEW_FOO_1) => newFoo1
case On(NEW_FOO_2) => newFoo2
}
}
```

### Routes as Features

To select different routes according to a feature toggle:

- Make a feature factory that extends `KnoraRouteFactory` and `FeatureFactory`,
and has a `makeRoute` method that returns different implementations,
each of which extends `KnoraRoute` and `Feature`.

- Make a façade route that extends `KnoraRoute`, is used in
`ApplicationActor.apiRoutes`, and has a `makeRoute` method that
delegates to the feature factory.

To avoid constructing redundant route instances, each façade route needs its
own feature factory class.

### Documenting a Feature Toggle

The behaviour of each possible setting of each feature toggle should be
documented. Feature toggles that are configurable per request should be described
in the release notes.

### Removing a Feature Toggle

To facilitate removing a feature toggle, each implementation should have:

- a separate file for its source code

- a separate file for its documentation

When the toggle is removed, the files that are no longer needed can be
deleted.
@@ -26,3 +26,4 @@ License along with Knora. If not, see <http://www.gnu.org/licenses/>.
- [Triplestore Updates](triplestore-updates.md)
- [Consistency Checking](consistency-checking.md)
- [Authentication](authentication.md)
- [Feature Toggles](feature-toggles.md)
@@ -153,6 +153,7 @@ ALL_WEBAPI_MAIN_DEPENDENCIES = [
"//webapi/src/main/scala/org/knora/webapi/app",
"//webapi/src/main/scala/org/knora/webapi/core",
"//webapi/src/main/scala/org/knora/webapi/exceptions",
"//webapi/src/main/scala/org/knora/webapi/feature",
"//webapi/src/main/scala/org/knora/webapi/http/handler",
"//webapi/src/main/scala/org/knora/webapi/http/version",
"//webapi/src/main/scala/org/knora/webapi/http/version/versioninfo",

0 comments on commit 2e6db2e

Please sign in to comment.