Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pure functions to abstract and compose cloud resources #3481

Closed
fogfish opened this issue Jul 30, 2019 · 7 comments

Comments

@fogfish
Copy link
Contributor

@fogfish fogfish commented Jul 30, 2019

  • I'm submitting a ...

    • 🐞 bug report
    • πŸš€ feature request
    • πŸ“š construct library gap
    • ☎️ security issue or vulnerability => Please see policy
    • ❓ support request => Please see note at the top of this template.
  • Please tell us about your environment:

    • CDK CLI Version: 1.2.0 (build 6b763b7)
    • OS: OSX Mojave
    • Language: TypeScript

Background

Composition is the essence of programming...

The composition is a style of development to build a new things from small reusable elements. The composition is a key feature in functional programming. Functional code looks great only if functions clearly describe your problem. Usually lines of code per function is only a single metric that reflects quality of the code.

If your functions have more than a few lines of code (a maximum of four to five lines per function is a good benchmark), you need to look more closely β€” chances are you have an opportunity to factor them into smaller, more tightly focused functions.

This is because functions describe the solution to your problem. If your code contains many lines then highly likely you are solving few problems without articulating them. A critical thinking is the process of software development.

Of the shelf composition with AWS CDK

A recent release of AWS CDK not only made an opportunity to deliver Infrastructure as a true Code but also gave a capability to make it from decomposable elements. Unfortunately, the development kit ignores a pure functional approach. The abstraction of cloud resources is exposed using class hierarchy.

Constructs are the basic building blocks of AWS CDK apps. A construct represents a "cloud component" and encapsulates everything AWS CloudFormation needs to create the component.
-- AWS CDK

Use this class to define code blocks that solves a single problem - just create a single cloud resource.

import * as cdk from '@aws-cdk/core'

class MyResource extends cdk.Construct {
  constructor(scope: cdk.Construct, id: string) {
    super(scope, id)
    ...
  }
}

class MyStack extends cdk.Stack {
  constructor(app: cdk.App, id: string) {
    super(app, id)
    new MyResource(this, 'MyResource')
  }
}

const app = new cdk.App()
new MyStack(app, 'MyStack')
app.synth()

You'll be able to build reusable cloud component library and automate infrastructure patterns with real code. The usage of cdk.Construct only suffers from boilerplate code.

Pure functions

Let's try to define pure functions to solve embarrassingly obvious composition problem. We need to shift our perspective from category of classes to category of pure functions.

Let's abstract our Infrastructure as a Code in terms of pure functions. Functions that takes a parent cdk.Construct and creates a new element.

type IaaC<T> = (parent: cdk.Construct) => T

The composition is about building efficiently parent-children relations due to nature of AWS CDK framework.

type Compose = <T>(parent: cdk.Construct, children: IaaC<T>) => cdk.Construct

Finally, we are able to define _ as compose function

function _<T>(parent: cdk.Construct, fn: IaaC<T>): cdk.Construct {
  root instanceof cdk.App
    ? fn(new cdk.Stack(parent, fn.name))
    : fn(new cdk.Construct(parent, fn.name))
  return parent
}

The definition of our cloud resources becomes reflected in terms of pure functions

function MyResource(scope: cdk.Construct): cdk.Construct {
   return ...
}

function MyStack(stack: cdk.Construct): cdk.Construct {
   return _(stack, MyResource)
}

const app = new cdk.App()
_(app, MyStack)
app.synth()

Now, the infrastructure is decomposable using small function. Each obviously correct and self explainable. This example eliminates boilerplate code of class constructions.

Here are some of the benefits:

  • Infrastructure intent is clear and obvious
  • It becomes testable and easier to reason about
  • Easier to debug and maintain

See the gist of pure.ts.

I'd be glad to hear your opinion on this subject.

@eladb

This comment has been minimized.

Copy link
Contributor

@eladb eladb commented Jul 30, 2019

I love this and would love to figure out a way to provide an alternative/additional purely functional API for constructs.

A few notes:

  1. There are a few limitation in jsii that will make it hard to expose this in all supported CDK languages: (a) no generics, (b) no free-floating functions, (c) no lambdas.
  2. It is very important that construct id remains stable across deployments because resource logical IDs are based on the these IDs. We considered actually having the logical ID default to the type name (similar to using the function name) but realized we needed to make it more explicit in order to surface this fragility. Moreover, there's definitely a use case for multiple instances of the type construct type within a scope, so being able to provide the ID is required (even if it's optional).
  3. Any ideas how we can make class-based constructs interoperate naturally with functional constructs? For example, how would I define a stack that includes an s3.Bucket and an sns.Topic with this programming model? I am not sure I fully understand.
@fogfish

This comment has been minimized.

Copy link
Contributor Author

@fogfish fogfish commented Jul 31, 2019

This is very interesting subjects. Let's keep 1 and 2 aside. I need to think more about it. However, 3 is quite easy. Here is an example of stack with API GW, Lambda, S3 and SNS

import * as cdk from '@aws-cdk/core'
import * as api from '@aws-cdk/aws-apigateway'
import * as lambda from '@aws-cdk/aws-lambda'
import * as s3 from '@aws-cdk/aws-s3'
import * as sns from '@aws-cdk/aws-sns'
import { _, iaac } from './pure'

function HttpBinRest(parent: cdk.Construct): api.RestApi {
  const rest = iaac(parent, HttpBinRestApi)
  const method = new api.LambdaIntegration( iaac(parent, HttpBinRestApiMethod) )

  rest.root.addResource('delete').addMethod('DELETE', method)
  rest.root.addResource('get').addMethod('GET', method)
  rest.root.addResource('patch').addMethod('PATCH', method)
  rest.root.addResource('post').addMethod('POST', method)
  rest.root.addResource('put').addMethod('PUT', method)
  return rest
}

function HttpBinRestApi(parent: cdk.Construct): api.RestApi {
  const rest = new api.RestApi(parent, 'HttpBinRestApi',
    {
      deploy: true,
      deployOptions: {
        stageName: 'api'
      },
      failOnWarnings: true,
      endpointTypes: [api.EndpointType.REGIONAL]
    }
  )

  return rest
}

function HttpBinRestApiMethod(parent: cdk.Construct): lambda.Function {
  return new lambda.Function(parent, 'HttpBinRestApiGet',
    {
      runtime: lambda.Runtime.NODEJS_10_X,
      code: new lambda.AssetCode('../apps/method'),
      handler: 'index.main'
    }
  )
}

function HttpBinS3(parent: cdk.Construct): s3.Bucket {
  return new s3.Bucket(parent, 'HttpBinS3', {
    bucketName: 'http-bin',
    publicReadAccess: false
  })
}

function HttpBinTopic(parent: cdk.Construct): cdk.Construct {
  return new sns.Topic(parent, 'HttpBinTopic', {
    displayName: 'http-bin',
    topicName: 'http-bin'
  })
}

function HttpBin(stack: cdk.Construct): cdk.Construct {
  _(stack, HttpBinRest)
  _(stack, HttpBinS3)
  _(stack, HttpBinTopic)
  return stack
}

const app = new cdk.App()
_(app, HttpBin)
app.synth()
@fogfish

This comment has been minimized.

Copy link
Contributor Author

@fogfish fogfish commented Aug 7, 2019

I've been trying to experiment with this technique on real-life examples. Learn a few things, what I'd like to emphasis discuss here.

Some thoughts about your second point*

I do see your point about Logical IDs and they impacts on stack deployments. However, the re-deployment of resource is also impacted by the position of node in the graph. In other words, not only Logical ID important but Logical ID of its parents. This has to be surfaces explicitly for developers as well.

One of the challenge with logical id. You repeat class/function name to Logical ID. May be there should be default naming schema that is derived from class/function. As an example below, it hurt my eyes (may be it is only me πŸ˜„ ) just because name HttpBinRestApi is repeated over again.

//
class HttpBinRestApi extends cdk.Construct {
   constructor(parent: cdk.Construct) {
     super(...)
     const rest = new api.RestApi(parent, 'HttpBinRestApi'
     ...

//
function HttpBinRestApi(parent: cdk.Construct): api.RestApi {
  const rest = new api.RestApi(parent, 'HttpBinRestApi'
  ...

Then I've met a new composition challenge often you need to pass a reference to created object. It works perfectly fine if you have a huge function that creates all these object.

const rest = new api.RestApi(...)
const role  = new iam.Role(...)
const lambda = return new lambda.Function(..., {role})
const method = new api.LambdaIntegration( lambda )
rest.root.addResource('put').addMethod('DELETE', method)

My goal to define a function for each element. In this case composition becomes a direct calls to function. Would it confuse developers about usage of _ composition vs direct call.

const rest = MyRest( ... )
const role = MyRole( ... )
const lambda = return new lambda.Function(..., {role})
const method = new api.LambdaIntegration( lambda )
rest.root.addResource('put').addMethod('DELETE', method)

I would appreciate about your thought on this subject.

@fogfish

This comment has been minimized.

Copy link
Contributor Author

@fogfish fogfish commented Aug 8, 2019

After all investigation on this subject, I think we are missing Type-Classes support in typescript.
One of the benefits of existed model is that TypeScript compiler validates types of props. E.g. following case, developer get error is props are not FunctionProps type. I do not like to break this benefit with some super generic wrappers for composition.

new lambda.Function(...,  props)

I've played with few stacks and still hold my original proposal of pure.ts. Please see this stack for example.

@eladb I'd be glad to hear you comments. If you think that this feature is valuable I can craft PR to aws-core just with this feature and unit tests. I'd be glad to learn if I missed something!

@fogfish

This comment has been minimized.

Copy link
Contributor Author

@fogfish fogfish commented Aug 20, 2019

Here is my latest proposal to the interface

import { App } from './app';
import { Construct } from './construct';
import { Stack } from './stack';

/**
 * The namespace provides utilities to abstract Infrastructure as a Code
 * in terms of pure functions and solve embarrassingly obvious composition problem.
 */
export namespace pure {

  /**
   * abstract definition of IaaC code block
   */
  export type IaaC<T> = (node: Construct) => T;

  /**
   * lifts pure functional IaaC blocks to CDK nodes, attaches them to
   * parent node. The node's logical name is a function name
   * @param node a parent node to attach IaaC blocks
   * @param fns list of IaaC blocks
   */
  export function join<T>(node: Construct, ...fns: Array<IaaC<T>>): Construct {
    fns.forEach(
      fn => {
      node instanceof App
        ? fn(new Stack(node, fn.name))
        : fn(new Construct(node, fn.name));
      }
    );
    return node;
  }

  /**
   * lifts pure functional IaaC blocks to Stack node, then attaches then it to application.
   * The node's logical name is either name of function or given name
   * @param node
   * @param fn
   * @param name
   */
  export function add<T>(node: App, fn: IaaC<T>, name?: string): Construct {
    fn(new Stack(node, name || fn.name));
    return node;
  }
}

but I'd like to find a good way to reduce boiler plate on node creation. E.g. from this example

I really do not like a fact that developer need to type boilerplate new lambda.Function(node, parent, 'WebHook', ...

function WebHook(parent: cdk.Construct): lambda.Function {
  ...
  return new lambda.Function(parent, 'WebHook',
    {
      runtime: lambda.Runtime.NODEJS_10_X,
      code: new lambda.AssetCode('../apps/webhook'),
      handler: 'webhook.main',
     ...
   }
  )
}

I would like to see it as

function WebHook(parent: cdk.Construct): lambda.FunctionProps {
  return  {
      runtime: lambda.Runtime.NODEJS_10_X,
      code: new lambda.AssetCode('../apps/webhook'),
      handler: 'webhook.main',
     ...
   }
}

and then join will deduct a correct type using ad-hoc polymorphism

pure.join<lambda.Function>(stack, WebHook)

Thought!

@fogfish

This comment has been minimized.

Copy link
Contributor Author

@fogfish fogfish commented Aug 23, 2019

Hello Gentlemen,

I think I've crunched this issue. Finally, I do have a clear proposal about rules of compositions. Then I've followed @rix0rrr advices and created a separate library to demo it.

I'll be glad to work our with you and incorporate into AWS CDK but I need any kind of feedback on

Best Regards,
Dmitry

@eladb

This comment has been minimized.

Copy link
Contributor

@eladb eladb commented Aug 28, 2019

Hi @fogfish this is really awesome. I love your project! I'm closing this here for now. Let's continue this conversation over your project. I think it's evolving beautifully.

@eladb eladb closed this Aug 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.