Create feature flags based on JVM settings to enable/disable experimental features

[poikilotherm]

Overview of the Feature Request

For upcoming functionality like #9229 / #9268 / #8822 enabling and disabling this kind of features would benefit from a standardized mechanism based on JvmSettings. Usually this type of switch is called "feature flag" or "feature gate" and indicates something is early stage, experimental, opt-in behaviour.

What kind of user is the feature intended for?
Sysadmin

What inspired the request?
See issues above.

What existing behavior do you want changed?
Add additional configurability with flags under the scope of dataverse.feature.[xxx]. This scope shall be used for opting into using some experimental code or feature preview. Once a feature stabilizes and matures, the flag shall go away (maybe replacing with a dedicated configuration setting).

Any brand new behavior do you want to add to Dataverse?
It's arguable if we did not have feature flags before, as we have configuration mechanisms in place to enable/disable features. Yet, these configurations are usually meant to be for a stable, accepted functionality, while this "new" mechanism is dedicated to previews and experimentation.

Any related open or closed issues to this feature request?
#9229 / #9268 / #8822 and potentially more in the future.

[qqmyers]

Is it worth making a separate mechanism that is only for experimental items - presumably you'd then want to switch to a different setting name if/when it is no longer experimental? How about one dedicated to dev-only instead, i.e. not for all newish features but ones you definitely should not be using in production?

[poikilotherm]

In terms of code structure, it will simply reuse JvmSettings but make it a little easier to read, abstracting away the boolean stuff.

Seeing other projects using feature flags (the biggest one being K8s itself), they usually go for a flag per feature, not a "devmode" thing. We can of course create a flag to indicate such a mode - is there a particular use case on your mind?

[qqmyers]

I'm mostly concerned about ambiguity. Many of the settings we have (Globus, Public Install, CVoc, Embargoes, etc.) turn on specific features - do those move? If I create a new one, do I put it in as normal or now use a feature? Other experimental features (blocks, direct upload api) aren't configured by settings. Do we have to change how we talk about those?

In the case of the session key for the API, it would be bad if people turn that on in production whereas most features you can use it or not. That seems like a distinction worth making very clear, so I can see why some special designation makes sense, but it would be nice to avoid experimental, feature, etc. becoming technical terms that only apply to some subsets of the way we'd colloquially use those words. Hence I'm proposing something like devmode/devonly to capture this new case without (partially) redefining 'feature', 'experimental', 'preview', etc.

[poikilotherm]

I think we need to differ between configuration and feature flags. Some "application feature" we provide might be tunable to be off or on by configuration. This is not what "feature flags" are meant to be about.

Any feature flags scope is to provide an opt-in to experimental or feature preview things. As soon as a feature matures and maybe is not meant to be configurable to turn off again, the flag is dropped.

If the feature needs to be tunable for turning it off and on, you need to create a configuration setting, which will NOT live within the dataverse.feature.xxx scope, but maybe in some scope already present or created anew (e.g. dataverse.xxx)

I did try to emphasize on that a bit in the docs included in #9299, but please feel free to suggest stronger wording. Maybe this is one of the reasons why some people rather call them "gates" and not "flags", as the "gatekeeper" aspect of this is more obvious.

I'm all ears if we want to rename "feature flags" to some other term and maybe also change the scope from "dataverse.feature" to something else. It probably helps making things obvious.

@qqmyers - as there is some code avail now, maybe you can leave a review and make some inline code suggestions for a renaming?