MTL Lambda #60

armanbilge · 2021-11-22T20:09:39Z

Here we go again 😆

This takes the idea in #51 (comment) about MTL-based middlewares for a spin.

The magic in this abstraction is provided by:

trait LambdaEnv[F[_], Event] {
  def event: F[Event]
  def context: F[Context[F]]
}

As long as you have a LambdaEnv, you can access the event and context from anywhere in the lambda. I think this is fantastic. It improves composability yet one more time: for example, now the http4s and CloudFormation lambdas no longer have to explicitly pass around Context[F] objects so that their routes/handlers have access to them. In general, you can build up your middleware stack however you want without worrying if the inputs of your middlewares line up: every middleware independently has full access to the LambdaEnv.

Co-authored-by: Brian P. Holt <bholt+github@planetholt.com>

armanbilge · 2021-11-22T20:45:04Z

lambda/shared/src/main/scala/feral/lambda/IOLambda.scala

+          }
+      }
+
+  def handler(implicit env: LambdaEnv[IO, Event]): Resource[IO, IO[Option[Result]]]


I think this is the most confusing aspect of this change: The Resource is run once, to create an IO that will be run many times. The IO is the handler.

Hmm, this is unsafe, because it would let you access the event and context while building the resource stack, which is not right. I knew those nulls were a bad sign 😅 . Fixing.

bpholt · 2021-11-23T04:36:19Z

This looks interesting! I'll try to publish it locally and pull it into the two Lambdas I've been playing with (one regular one, and one custom CFN resource) and report back. (Hopefully tomorrow.)

armanbilge · 2021-11-23T05:15:39Z

Thanks! No rush, and grateful for your patience with all the churn while we figure this out 😁 having you try these designs out with your lambdas has been invaluable, much appreciated!

If it helps, I just published this branch as:

"com.armanbilge" %%% "feral-lambda" % "0.1-2f94f32" // and always friends!

bpholt · 2021-12-02T16:55:55Z

Finally got around to trying this, so I'm going to add some comments on the code in no particular order.

bpholt

Working with LambdaEnv[F, Input] => F[Option[Output]] can be a little awkward when you want to define an anonymous function but make the LambdaEnv[F, Input] available implicitly to the body of the function. I started working around it by defining it as a Kleisli, which makes me wonder if it should actually be a Kleisli in the library after all.

bpholt · 2021-12-02T17:00:43Z

lambda-natchez/src/main/scala/feral/lambda/natchez/TracedLambda.scala

+
+object TracedLambda {
+
+  def apply[F[_]: MonadCancelThrow, Event: KernelSource, Result](entryPoint: EntryPoint[F])(


I think if we tweak this a bit, we'll get better type inference downstream:

def apply[F[_] : MonadCancelThrow, Event, Result](entryPoint: EntryPoint[F]) (lambda: Span[F] => F[Option[Result]]) (implicit env: LambdaEnv[F, Event], KS: KernelSource[Event]): F[Option[Result]]

As written now, it seems to get confused about what the Event type is, I think because it binds the KernelSource[Event] (using apiGatewayProxyEventV2KernelSource) before looking for the LambdaEnv[F, Event], which, if Event is not ApiGatewayProxyEventV2, won't be available.

Explicitly putting the implicit LambdaEnv[F, Event] parameter first lets it use that to bind Event, and then it correctly finds the emptyKernelSource low-priority implicit value.

Aha! Very nice!

bpholt · 2021-12-02T17:16:37Z

lambda/shared/src/main/scala/feral/lambda/IOLambda.scala

 import io.circe.Decoder
 import io.circe.Encoder
+import cats.effect.IOLocal

 abstract class IOLambda[Event, Result](
    implicit private[lambda] val decoder: Decoder[Event],
    private[lambda] val encoder: Encoder[Result]


I noticed that the Encoder[INothing] is missing from this PR. Was that intentional? I had to add it manually to get things working with a Result of INothing.

Oh yeah, I didn't merge that branch into this one. We should probably start landing some of these 😅

Yeah, agreed. I would love to get an alpha version of this published under "org.typelevel" soon!

bpholt · 2021-12-02T18:56:13Z

lambda/shared/src/main/scala/feral/lambda/LambdaEnv.scala

+    }
+}
+
+object LambdaEnv {


I'd like to add a summoner and implicit instances for Kleisli and friends:

def apply[F[_], A](implicit env: LambdaEnv[F, A]): LambdaEnv[F, A] = env implicit def kleisliLambdaEnv[F[_] : Functor, A, B](implicit env: LambdaEnv[F, A]): LambdaEnv[Kleisli[F, B, *], A] = env.mapK(Kleisli.liftK) implicit def optionTLambdaEnv[F[_] : Functor, A](implicit env: LambdaEnv[F, A]): LambdaEnv[OptionT[F, *], A] = env.mapK(OptionT.liftK) implicit def eitherTLambdaEnv[F[_] : Functor, A, B](implicit env: LambdaEnv[F, A]): LambdaEnv[EitherT[F, B, *], A] = env.mapK(EitherT.liftK) implicit def writerTLambdaEnv[F[_] : Applicative, A, B : Monoid](implicit env: LambdaEnv[F, A]): LambdaEnv[WriterT[F, B, *], A] = env.mapK(WriterT.liftK[F, B])

The Kleisli instance comes in handy when tracing with Kleisli[F, Span[F], *]. Not sure if the others are needed but I've noticed they get added in a lot of TypeLevel projects and I don't see any reason not to include them.

👍 for sure! Good to have.

bpholt · 2021-12-02T18:58:34Z

...ustom-resource/src/main/scala/feral/lambda/cloudformation/CloudFormationCustomResource.scala

+      client: Client[F],
+      handler: CloudFormationCustomResource[F, Input, Output])(
+      implicit
+      env: LambdaEnv[F, CloudFormationCustomResourceRequest[Input]]): F[Option[Unit]] = {


Should F[Option[Unit]] be F[Option[INothing]]?

bpholt · 2021-12-02T19:14:22Z

FYI, the POC code using this branch is available here for a plain-ol' AWS Lambda and here for a custom CloudFormation resource.

armanbilge · 2021-12-02T19:15:17Z

Working with LambdaEnv[F, Input] => F[Option[Output]] can be a little awkward when you want to define an anonymous function but make the LambdaEnv[F, Input] available implicitly to the body of the function.

Do you have an example of this? I was thinking something like this, but maybe it doesn't work so well.

{ implicit env =>
  // use it implicitly
}

I started working around it by defining it as a Kleisli, which makes me wonder if it should actually be a Kleisli in the library after all.

Yes, that's the heart of the matter :) I assume you mean like we considered in #51 (comment)?

The Kleisli and MTL approach are not mutually exclusive. If you have a Kleisli with (Event, Context[F]) then you can derive a LambdaEnv[F, Event]. So that might be one option. Edit: no, that's not quite right, the kleisli would become the effect type. In any case, this wouldn't work with IOLocal but maybe that's moot.

So overall, how do you like this MTL approach vs the approach in #51?

bpholt · 2021-12-02T19:50:24Z

Do you have an example of this? I was thinking something like this, but maybe it doesn't work so well.
{ implicit env =>
 // use it implicitly
}

That works fine if the return type is LambdaEnv[F, Input] => F[Option[Output]] directly, but I found that it required an explicit type on env when used e.g. in a for-comprehension returning Resource[F, LambdaEnv[F, Input] => F[Option[Output]]] like this one:

  private def resources[F[_] : Async : Console]: Resource[F, LambdaEnv[F, CloudFormationCustomResourceRequest[DatabaseMetadata]] => F[Option[Unit]]] =
    Resource.eval(Slf4jLogger.create[F]).flatMap { implicit logger =>
      for {
        client <- httpClient[F]
        entryPoint <- Resource.eval(Random.scalaUtilRandom[F]).flatMap { implicit r => XRay.entryPoint[F]() }
        secretsManager <- SecretsManagerAlg.resource[F, Kleisli[F, Span[F], *]]
      } yield Kleisli { implicit env: LambdaEnv[F, CloudFormationCustomResourceRequest[DatabaseMetadata]] =>
        TracedLambda[F, CloudFormationCustomResourceRequest[DatabaseMetadata], Unit](entryPoint) { span =>
          val tracedClient = NatchezMiddleware.client(client.translate(Kleisli.liftK[F, Span[F]])(Kleisli.applyK(span)))
          CloudFormationCustomResource(tracedClient, PostgresqlDatabaseInitHandlerImpl(secretsManager)).run(span)
        }
      }.run
    }

armanbilge · 2021-12-03T15:11:46Z

Thanks for that example!

So after playing with your RabbitMQ lambda a bit in Dwolla/rabbitmq-topology-backup#77 I think we can avoid issues like above by binding to IO just a bit earlier. So for example that snippet you posted above I think should just be the handler, and everything there should be in IO.

I think it is very reasonable/realistic/practical to acquire resources and patch together the lambda handler directly in IO. The important thing is that the handler logic and resources themselves are all written in F, just needs a big of glue at the end :)

What do you think?

Co-authored-by: Brian P. Holt <bholt@dwolla.com>

bpholt · 2021-12-12T23:38:18Z

I'm not sure what I was doing differently before, but I don't seem to have that problem doing

} yield { implicit env =>

with handlerF defined as

def handlerF[F[_] : Async]: Resource[F, LambdaEnv[F, RabbitMQConfig] => F[Option[INothing]]]

now, so that's good news.

(I still don't think I would want to move the resource acquisition stuff into IO, but that's just a personal preference—I don't think it's actually a dealbreaker.)

bpholt · 2021-12-12T23:41:25Z

Unless anyone sees any blockers, I think we should run with this design, cut an alpha release, and let people start playing with it. As early in the project as we are, I think we have flexibility to change things quite a bit if we discover this approach doesn't work for some reason.

kubukoz · 2021-12-13T00:45:22Z

@armanbilge bring it on :)

armanbilge and others added 19 commits November 9, 2021 08:27

Define Feral

8ffa2da

Redefine IOLambda in terms of Feral

1e2163b

Or maybe, just use Resource?

f36fdc0

Rework http4s lambda

ac17850

Rework cloudformation lambda

8fda33f

make Context generic in its effect type

1dc61d8

Create lambda-natchez module

7860c4d

Add arn/requestId tags to trace

6271e7a

Span-based TracedLambda

29e2c73

Rename run -> handler

f0f584c

Organize imports

269567a

Merge branch 'topic/agnostic-lambda' into topic/traced-lambda

e4fce73

Remove LiftTrace concept

463cb04

Make stackTraceLines private

d483705

Co-authored-by: Brian P. Holt <bholt+github@planetholt.com>

Create KernelSource/Sink abstraction

13dcb39

Rename Tags -> AwsTags

9c773d2

KernelSink doesn't make sense

3be3484

More idiomatic CloudFormationCustomResource lambda

0188819

Try out MTL-lambda concept

9ffbd30

armanbilge requested a review from bpholt November 22, 2021 20:09

armanbilge mentioned this pull request Nov 22, 2021

Composable, effect-agnostic lambdas #51

Closed

armanbilge added 3 commits November 22, 2021 20:17

mapK for LambdaEnv

4ca7cea

Merge branch 'topic/traced-lambda' into topic/mtl-lambda

11f9ec2

MTL-style TracedLambda

ce33cf2

armanbilge commented Nov 22, 2021

View reviewed changes

LambdaEnv is not available during resource acquisition

5a99f02

armanbilge mentioned this pull request Nov 29, 2021

Natchez integration for Lambda #55

Closed

bpholt reviewed Dec 2, 2021

View reviewed changes

armanbilge and others added 3 commits December 3, 2021 15:48

Reorder LambdaEnv/KernelSource implicits

10ebe93

Co-authored-by: Brian P. Holt <bholt@dwolla.com>

cats.data summoners for LambdaEnv

586379d

Co-authored-by: Brian P. Holt <bholt@dwolla.com>

Organize imports

3781d55

kubukoz approved these changes Dec 9, 2021

View reviewed changes

armanbilge mentioned this pull request Dec 9, 2021

MVP / 0.1.0 #69

Closed

6 tasks

bpholt approved these changes Dec 12, 2021

View reviewed changes

armanbilge mentioned this pull request Dec 13, 2021

Generic effects, take 2, and more? #45

Closed

armanbilge added 2 commits December 13, 2021 02:29

Merge branch 'main' into topic/mtl-lambda

05f90f5

Regenerate workflow

777126e

armanbilge merged commit dcbf652 into typelevel:main Dec 13, 2021

This was referenced Dec 13, 2021

Add a regression test for INothing implicit resolution #73

Open

Is LambdaEnv powerful enough? #85

Closed

armanbilge added the lambda label Jan 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MTL Lambda #60

MTL Lambda #60

armanbilge commented Nov 22, 2021

armanbilge Nov 22, 2021

armanbilge Nov 22, 2021

bpholt commented Nov 23, 2021

armanbilge commented Nov 23, 2021

bpholt commented Dec 2, 2021

bpholt left a comment

bpholt Dec 2, 2021

armanbilge Dec 2, 2021

bpholt Dec 2, 2021

armanbilge Dec 2, 2021

bpholt Dec 2, 2021

bpholt Dec 2, 2021

armanbilge Dec 2, 2021

bpholt Dec 2, 2021

bpholt commented Dec 2, 2021

armanbilge commented Dec 2, 2021 •

edited

Loading

bpholt commented Dec 2, 2021

armanbilge commented Dec 3, 2021

bpholt commented Dec 12, 2021

bpholt commented Dec 12, 2021

kubukoz commented Dec 13, 2021


		object TracedLambda {

		def apply[F[_]: MonadCancelThrow, Event: KernelSource, Result](entryPoint: EntryPoint[F])(

MTL Lambda #60

MTL Lambda #60

Conversation

armanbilge commented Nov 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bpholt commented Nov 23, 2021

armanbilge commented Nov 23, 2021

bpholt commented Dec 2, 2021

bpholt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bpholt commented Dec 2, 2021

armanbilge commented Dec 2, 2021 • edited Loading

bpholt commented Dec 2, 2021

armanbilge commented Dec 3, 2021

bpholt commented Dec 12, 2021

bpholt commented Dec 12, 2021

kubukoz commented Dec 13, 2021

armanbilge commented Dec 2, 2021 •

edited

Loading