Amazon CloudWatch Evidently L2 Constructs

[ljuti]

Amazon CloudWatch Evidently L2 Construct Library

Amazon CloudWatch Evidently lets application developers conduct experiments and identify unintended consequences of new features before rolling them out for general use, thereby reducing risk related to new feature roll-out. Evidently allows you to validate new features across the full application stack before release, which makes for a safer release. When launching new features, you can expose them to a small user base, monitor key metrics such as page load times or conversions, and then dial up traffic. Evidently also allows you to try different designs, collect user data, and release the most effective design in production.

Description

I propose an L2 construct library that cloud engineers can use to conduct Evidently launches experiments with CDK. The construct library would bring the Evidently capabilities closer to the development workflow, enabling quicker and smoother iterations and experimentation.

Core Constructs

Project

A Project is an AWS Evidently project where you group and track features, launches and experiments.

Feature

A Feature is an AWS Evidently feature, a set of behaviour in your application that you wish to launch or test.

Launch

A Launch is an AWS Evidently launch which you use to schedule or incrementally release a feature.

Experiment

An Experiment is an AWS Evidently experiment that helps you make feature design decisions based on evidence and data.

API Walkthrough with a Use Case Example

In an AWS News Blog article, author @sebsto demonstrates how to use AWS Evidently with a web application example.

Let's see how we would use the construct library to replace AWS Console tasks of the example.

1. Create a Project

import * as evidently from 'aws-cdk-lib/aws_evidently';

const project = new evidently.Project(this, 'GuestbookWebApp', {
  projectName: 'AWSNewsBlog',
  description: 'AWSNewsBlogDemo',
});

2. Add a Feature

const editable = new evidently.Variation({
  variationName: 'editable',
  valueType: VariationObjectValueType.BOOLEAN,
  value: true,
});

const readonly = new evidently.Variation({
  variationName: 'readonly',
  valueType: VariationObjectValueType.BOOLEAN,
  value: false,
});

const feature = new evidently.Feature(this, 'EditableGuestbook', {
  project: project,
  featureName: 'Editing',
  variations: [
    editable,
    readonly,
  ],
  entityOverrides: [
    new EntityOverride({
      entityId: 'seb',
      variation: editable,
    });
  ],
});

3. Create a Launch

const editableGroup = new evidently.LaunchGroup({
  groupName: 'Editable',
  feature: feature,
  variation: editable,
});
const readonlyGroup = new evidently.LaunchGroup({
  groupName: 'Readonly',
  feature: feature,
  variation: readonly,
});

const launchConfiguration = new StepConfig({
  groupsToWeight: [
    {
      group: editableGroup,
      splitWeight: SplitWeight.50_PCT,
    }
  ],
  startTime: <timestamp>,
});

const launch = new evidently.Launch(this, 'EditableGuestbook', {
  launchName: 'EditableGuestbook',
  description: 'Launch the editable guest book feature',
  groups: [
    editableGroup,
    readonlyGroup,
  ],
  scheduledSplitsConfig: [launchConfiguration],
});

4. Start an Experiment

const goal = new evidently.MetricGoal({
  desiredChange: MetricGoalDesiredChange.INCREASE,
  entityIdKey: 'user',
  eventPattern: 'eventbridge-pattern',
  metricName: 'Edits',
  valueKey: 'edits',
});

const treatmentOne = new evidently.Treatment({
  feature: feature,
  treatmentName: 'editing',
  variation: editable,
});

const treatmentTwo = new evidently.Treatment({
  feature: feature,
  treatmentName: 'readonly',
  variation: readonly,
});

const config = new evidently.OnlineAbConfig({
  controlTreatmentName: treatmentOne.treatmentName,
  treatmentWeights: [
    new evidently.TreatmentToWeight({
      splitWeight: 20000,
      treatment: treatmentOne,
    }),
    new evidently.TreatmentToWeight({
      splitWeight: 80000,
      treatment: treatmentTwo,
    }),
  ],
});

const experiment = new evidently.Experiment({
  project: project,
  experimentName: 'EditableGuestBook',
  metricGoals: [goal],
  onlineAbConfig: config,
  treatments: [
    treatmentOne,
    treatmentTwo,
  ],
});

Roles

Role	User
Proposed by	@ljuti
Author(s)	@ljuti
API Bar Raiser	@alias
Stakeholders	@alias, @alias, @alias

See RFC Process for details

Workflow

• Tracking issue created (label: status/proposed)
• API bar raiser assigned (ping us at #aws-cdk-rfcs if needed)
• Kick off meeting
• RFC pull request submitted (label: status/review)
• Community reach out (via Slack and/or Twitter)
• API signed-off (label api-approved applied to pull request)
• Final comments period (label: status/final-comments-period)
• Approved and merged (label: status/approved)
• Execution plan submitted (label: status/planning)
• Plan approved and merged (label: status/implementing)
• Implementation complete (label: status/done)

---

Author is responsible to progress the RFC according to this checklist, and
apply the relevant labels to this issue so that the RFC table in README gets
updated.

[serithemage]

Is there a timeline for this proposal?

[ljuti]

@serithemage I haven't been assigned an API bar raiser, so this isn't likely moving forward unless there are sufficient backers for the RFC.

My PoC code is available here.

[serithemage]

@ljuti Thank you!

[awsmjs]

Closing this ticket. We believe the functionality is beneficial, but does not intersect with the core framework and should be vended and maintained separately.

[flochaz]

FYI, I published @ljuti work (with a small fix on projectName returned value) on npm: https://www.npmjs.com/package/aws-evidently-l2

[hoegertn]

Would that be something you consider contributing to https://www.npmjs.com/package/@open-constructs/aws-cdk

[flochaz]

Would that be something you consider contributing to https://www.npmjs.com/package/@open-constructs/aws-cdk

Nice, I didn't know about this initiative. Sure happy to open a PR there