Skip to content

Commit

Permalink
Configure Turing UI to use Standard Ensembler for Custom Experiment E…
Browse files Browse the repository at this point in the history
…ngines (#235)

* Update type ensembler options to allow std ensemblers for custom exp engines

* Make create and edit router consume standard ensembler component from exp engine ui

* Add route name path to standard ensembler config struct

* Clean up props to be passed to remote ensembler component

* Add routes prop to pass to the remote standard ensembler

* Update typeOptions to return correct variant of standard ensembler

* Add standard ensembler config view

* Rename prop for StandardEnsemblerLoaderComponent

* Make status colours use default EUI palette as much as possible

* Remove redundant empty tags

* Refactor components to use common RemoteLoaderComponent

* Replace colour hexes corresponding to subdued to default

* Add docs for standard ensembler

* Remove redundant description for std ensembler and custom exp engines

* Rename generic component loader as ExperimentEngineComponentLoader

* Remove redundant line from docs
  • Loading branch information
deadlycoconuts committed Sep 27, 2022
1 parent 67fe457 commit 296251d
Show file tree
Hide file tree
Showing 14 changed files with 269 additions and 72 deletions.
16 changes: 15 additions & 1 deletion docs/how-to/create-a-router/configure-ensembler.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,11 @@ It is not possible to select as the final response a route that has traffic rule

## Standard Ensembler

For experiment engines configured to work with standard ensemblers (i.e., Standard Experiment Engines with experiment selection enabled), the router will return a response from one of the routes based on the configured mapping between routes and experiment treatments. At run time, the treatment returned by the engine will be used to select the corresponding route’s response.
There are two types of Standard Ensemblers available, one that is available with Standard Experiment Engines that
have experiment selection enabled, and the other with Custom Experiment Engines.

### Standard Experiment Engines
For routers configured with Standard Experiment Engines that have experiment selection enabled, the router will return a response from one of the routes based on the configured mapping between routes and experiment treatments. At run time, the treatment returned by the engine will be used to select the corresponding route’s response.

In addition, a fallback route may be configured whose results will be used at runtime when the call to the experiment engine fails or if a route mapping for the treatment generated by the experiment engine does not exist.

Expand All @@ -25,6 +29,16 @@ In addition, a fallback route may be configured whose results will be used at ru
It is not possible to select as the fallback response a route that has traffic rules associated to it.
{% endhint %}

### Custom Experiment Engines
For routers configured with Custom Experiment Engines, the router will return a response from one of the routes
corresponding to the route name that is found within the treatment configuration returned by the experiment
engine. At run time, the router will attempt to access the route name within the treatment configuration received via a
user-configured path.

A fallback route also has to be configured in order to capture cases whereby the route name found in the treatment
configuration does not correspond to any of the routes configured, or when the user-configured path is invalid with
respect to the treatment configuration received.

## Docker

Turing will deploy the specified image as a post-processor and will send in the request payload the following, for
Expand Down
24 changes: 24 additions & 0 deletions engines/experiment/docs/developer_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,30 @@ type IExperimentConfig = { [key: string]: IJsonValue };
}) => React.ReactElement
```

In addition, custom experiment engines should also provide the following UI components to support the edit form and view
configuration components for Standard Ensemblers (see
[this](../../../docs/how-to/create-a-router/configure-ensembler.md) for more details). The `routeNamePath` field
should be configured to allow the Turing Router to access the route name from a treatment configuration received
from the experiment engine.

```javascript
// EditStandardEnsemblerConfig component signature
({
projectId: int,
routes: Array<Route>,
routeNamePath: string,
onChangeHandler: (React.ChangeEvent<HTMLInputElement>) => void,
errors: yup.ValidationError,
}) => React.ReactElement

// StandardEnsemblerConfigDetails component signature.
({
projectId: int,
routes: Array<Route>,
routeNamePath: string
}) => React.ReactElement
```

### Experiment Runner

Experiment runners are required to implement the methods in the `ExperimentRunner` interface. This interface contains a single method to retrieve the treatment for a given request.
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,10 @@ const LoadDynamicScript = ({ url, setReady, setFailed }) => {
};

// Dynamic Script Loading component wrapper
export const ExperimentEngineLoaderComponent = ({
export const ExperimentEngineComponentLoader = ({
FallbackView,
experimentEngine,
remoteUi,
componentName,
children,
}) => {
const [urlReady, setUrlReady] = useState(false);
Expand All @@ -28,30 +29,30 @@ export const ExperimentEngineLoaderComponent = ({
const [configFailed, setConfigFailed] = useState(false);

return urlFailed ? (
<FallbackView text={"Failed to load Experiment Engine"} />
<FallbackView text={`Failed to load ${componentName}`} />
) : configFailed ? (
<FallbackView text={"Failed to load Experiment Engine Config"} />
<FallbackView text={`Failed to load ${componentName} Config`} />
) : !urlReady || !configReady ? (
<>
{!!experimentEngine.url && !urlReady && (
{!!remoteUi.url && !urlReady && (
<LoadDynamicScript
setReady={setUrlReady}
setFailed={setUrlFailed}
url={experimentEngine.url}
url={remoteUi.url}
/>
)}
{!!experimentEngine.config && !configReady && (
{!!remoteUi.config && !configReady && (
<LoadDynamicScript
setReady={setConfigReady}
setFailed={setConfigFailed}
url={experimentEngine.config}
url={remoteUi.config}
/>
)}
<FallbackView text={"Loading Experiment Engine..."} />
<FallbackView text={`Loading ${componentName}...`} />
</>
) : (
children
);
};

export default ExperimentEngineLoaderComponent;
export default ExperimentEngineComponentLoader;
10 changes: 6 additions & 4 deletions ui/src/experiment/ExperimentsRouter.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import {
EuiPageTemplate,
} from "@elastic/eui";
import { RemoteComponent } from "../components/remote_component/RemoteComponent";
import { ExperimentEngineLoaderComponent } from "../components/experiments/ExperimentEngineLoaderComponent";
import ExperimentEngineComponentLoader from "../components/remote_component/ExperimentEngineComponentLoader";

import { useConfig } from "../config";

Expand Down Expand Up @@ -32,16 +32,18 @@ const RemoteRouter = ({ projectId }) => {
return (
<React.Suspense
fallback={<FallbackView text="Loading Experiment Engine config" />}>
<ExperimentEngineLoaderComponent
<ExperimentEngineComponentLoader
FallbackView={FallbackView}
experimentEngine={defaultExperimentEngine}>
remoteUi={defaultExperimentEngine}
componentName="Experiment Engine"
>
<RemoteComponent
scope={defaultExperimentEngine.name}
name="./ExperimentsLandingPage"
fallback={<FallbackView text="Loading Experiment Engine" />}
projectId={projectId}
/>
</ExperimentEngineLoaderComponent>
</ExperimentEngineComponentLoader>
</React.Suspense>
);
};
Expand Down
Original file line number Diff line number Diff line change
@@ -1,16 +1,15 @@
import React, { Fragment } from "react";
import { DockerConfigViewGroup } from "./docker_config_section/DockerConfigViewGroup";
import { FallbackRouteConfigSection } from "./standard_config_section/FallbackRouteConfigSection";
import { TreatmentMappingConfigSection } from "./standard_config_section/TreatmentMappingConfigSection";
import { ExperimentEngineContextProvider } from "../../../../providers/experiments/ExperimentEngineContextProvider";
import { NopConfigViewGroup } from "./nop_config_section/NopConfigViewGroup";
import { PyFuncConfigViewGroup } from "./pyfunc_config_section/PyFuncConfigViewGroup";
import { StandardConfigViewGroup } from "./standard_config_section/StandardConfigViewGroup"
import { EnsemblersContextContextProvider } from "../../../../providers/ensemblers/context";
import { EuiFlexGroup, EuiFlexItem } from "@elastic/eui";

export const EnsemblerConfigSection = ({
projectId,
config: {
routes,
ensembler,
experiment_engine: { type, config: experimentConfig },
},
Expand Down Expand Up @@ -38,22 +37,13 @@ export const EnsemblerConfigSection = ({
)}
{ensembler.type === "standard" && (
<ExperimentEngineContextProvider>
<EuiFlexGroup direction="column">
<EuiFlexItem>
<TreatmentMappingConfigSection
engine={type}
experiments={(experimentConfig || {}).experiments || []}
mappings={ensembler.standard_config.experiment_mappings}
/>
</EuiFlexItem>
<EuiFlexItem>
<FallbackRouteConfigSection
fallbackResponseRouteId={
ensembler.standard_config.fallback_response_route_id
}
/>
</EuiFlexItem>
</EuiFlexGroup>
<StandardConfigViewGroup
projectId={projectId}
routes={routes}
standardConfig={ensembler.standard_config}
experimentConfig={(experimentConfig || {}).experiments || []}
type={type}
/>
</ExperimentEngineContextProvider>
)}
</Fragment>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { EuiFlexGroup, EuiFlexItem, EuiPanel } from "@elastic/eui";

import { ConfigSectionPanel } from "../../../../components/config_section";
import { RemoteComponent } from "../../../../components/remote_component/RemoteComponent";
import { ExperimentEngineLoaderComponent } from "../../../../components/experiments/ExperimentEngineLoaderComponent";
import ExperimentEngineComponentLoader from "../../../../components/remote_component/ExperimentEngineComponentLoader";
import ExperimentEngineContext from "../../../../providers/experiments/context";

import { StandardExperimentConfigGroup } from "./experiment_config_section/StandardExperimentConfigGroup";
Expand All @@ -29,17 +29,19 @@ const CustomExperimentConfigView = ({ projectId, remoteUi, config }) => {
return (
<React.Suspense
fallback={<FallbackView text="Loading Experiment Engine config" />}>
<ExperimentEngineLoaderComponent
<ExperimentEngineComponentLoader
FallbackView={FallbackView}
experimentEngine={remoteUi}>
remoteUi={remoteUi}
componentName="Experiment Engine"
>
<RemoteComponent
scope={remoteUi.name}
name="./ExperimentEngineConfigDetails"
fallback={<FallbackView text="Loading Experiment Engine config" />}
projectId={projectId}
config={config}
/>
</ExperimentEngineLoaderComponent>
</ExperimentEngineComponentLoader>
</React.Suspense>
);
};
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
import React, { useContext } from "react";

import { EuiFlexGroup, EuiFlexItem, EuiSpacer } from "@elastic/eui";
import { TreatmentMappingConfigSection } from "./TreatmentMappingConfigSection";
import { FallbackRouteConfigSection } from "./FallbackRouteConfigSection";
import ExperimentEngineContext from "../../../../../providers/experiments/context";
import { Panel } from "../../../form/components/Panel";
import ExperimentEngineComponentLoader from "../../../../../components/remote_component/ExperimentEngineComponentLoader";
import { RemoteComponent } from "../../../../../components/remote_component/RemoteComponent";

const FallbackView = ({ text }) => (
<EuiFlexItem grow={true}>
<Panel title="Route Selection">
<EuiSpacer size="m" />
{text}
</Panel>
</EuiFlexItem>
);

const StandardEnsemblerWithCustomExperimentEngineConfigView = ({
remoteUi,
projectId,
routes,
routeNamePath
}) => {
// Load component from remote host
return (
<React.Suspense
fallback={<FallbackView text="Loading Standard Ensembler config for the selected Custom Experiment Engine" />}>
<ExperimentEngineComponentLoader
FallbackView={FallbackView}
remoteUi={remoteUi}
componentName="Standard Ensembler"
>
<RemoteComponent
scope={remoteUi.name}
name="./StandardEnsemblerConfigDetails"
fallback={<FallbackView text="Loading Standard Ensembler config for the selected Custom Experiment Engine" />}
projectId={projectId}
routes={routes}
routeNamePath={routeNamePath}
/>
</ExperimentEngineComponentLoader>
</React.Suspense>
);
};

export const StandardConfigViewGroup = ({
projectId,
routes,
standardConfig,
experimentConfig,
type
}) => {
const { getEngineProperties, isLoaded } = useContext(ExperimentEngineContext);

const engineProps = getEngineProperties(type);

return (
!!standardConfig && (
<EuiFlexGroup direction="column">
{engineProps?.type === "standard" ? (
<EuiFlexItem>
<TreatmentMappingConfigSection
engine={type}
experiments={experimentConfig}
mappings={standardConfig.experiment_mappings}
/>
</EuiFlexItem>
) : isLoaded ? (
<EuiFlexItem>
<StandardEnsemblerWithCustomExperimentEngineConfigView
remoteUi={engineProps.custom_experiment_manager_config.remote_ui}
projectId={projectId}
routes={routes}
routeNamePath={standardConfig.route_name_path}
/>
</EuiFlexItem>
) : (
<EuiFlexItem>
<FallbackView text={"Loading ..."} />
</EuiFlexItem>
)}

<EuiFlexItem>
<FallbackRouteConfigSection
fallbackResponseRouteId={
standardConfig.fallback_response_route_id
}
/>
</EuiFlexItem>
</EuiFlexGroup>
)
);
};
Loading

0 comments on commit 296251d

Please sign in to comment.