Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom Lambda Runtime #276

Open
wants to merge 59 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 55 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
7ea939b
Bump base version
armanbilge Sep 25, 2022
26b1622
Shuffle build for custom runtime
armanbilge Sep 25, 2022
95efe18
http4s and cloudformation only depend on kernel
armanbilge Sep 25, 2022
be96317
WIP: added runtime skeleton, needs error handling
scott-thomson239 Sep 26, 2022
ebf5e97
fixed context creation from environment variables
scott-thomson239 Sep 27, 2022
6f990ad
added comments to unimplemented parts
scott-thomson239 Sep 28, 2022
f6a904b
used new cats effect Env instead of custom env var solution
scott-thomson239 Sep 29, 2022
defb679
added function to obtain LambdaRequest from response and added error …
scott-thomson239 Oct 2, 2022
c95cecb
renamed module and fixed handler error handling
scott-thomson239 Oct 3, 2022
2de1748
changed hander fiber to not raise error after cancellation
scott-thomson239 Oct 4, 2022
b7e6054
created custom invocation response header models
scott-thomson239 Oct 6, 2022
b07406c
tests work in progress
scott-thomson239 Oct 12, 2022
0eb162e
fixed all but one test
scott-thomson239 Oct 13, 2022
24c4261
added test to verify runtime can handle invocation errors correctly
scott-thomson239 Oct 15, 2022
602460b
fixed failing tests, although tests need refactored
scott-thomson239 Oct 16, 2022
25403d0
refactored tests
scott-thomson239 Oct 17, 2022
a65d045
added init error handling and tests
scott-thomson239 Oct 19, 2022
206ec6a
fixed failing test
scott-thomson239 Oct 20, 2022
aa06238
general clean up
scott-thomson239 Oct 20, 2022
39562d6
fix missing import
scott-thomson239 Oct 20, 2022
108c2cb
ran scalafmt
scott-thomson239 Oct 20, 2022
016df57
ran sbt githubWorkflowGenerate
scott-thomson239 Oct 20, 2022
7ce4f85
added file headers
scott-thomson239 Oct 20, 2022
fe73a65
added .settings(commonSettings) to lambdaRuntime and lambdaKernel
scott-thomson239 Oct 22, 2022
ed8cb7d
Merge branch 'main' into feature/custom-lambda-runtime
scott-thomson239 Oct 22, 2022
f2b7151
ran sbt githubWorkflowGenerate
scott-thomson239 Oct 22, 2022
0cf6033
fix type erasure warning
scott-thomson239 Oct 22, 2022
e3f8553
Merge branch 'main' into feature/custom-lambda-runtime
armanbilge Oct 22, 2022
7c69d9e
fix for potential type inference problem in expect method and correct…
scott-thomson239 Oct 23, 2022
cfd8f0a
modified sbtLambda in build.sbt to include lambdaKernel.js in scripte…
scott-thomson239 Oct 23, 2022
accb4b6
forgot to format buld.sbt oops
scott-thomson239 Oct 23, 2022
6acf432
updated natchez-core in lambdaKernel and updated scalajs plugin to 1.…
scott-thomson239 Oct 23, 2022
2449d3e
fix for 'withFilter is not a member of IO' error
scott-thomson239 Oct 23, 2022
9d79651
added snapshot resolver to sbt-lambda build.sbt
scott-thomson239 Oct 23, 2022
3c4974c
added lambdaKernel and lambdaRuntime to build.sbt unidocs
scott-thomson239 Oct 23, 2022
2548296
test to see if removing rethrow from IOSetup solves compiler error
scott-thomson239 Oct 23, 2022
35d797f
added MatrixExclude for 2.12/native
scott-thomson239 Oct 23, 2022
93d55aa
changed cats-effect-kernel to cats-effect in lambdaKernel and removed…
scott-thomson239 Oct 23, 2022
af3b527
Make lambda headers package-private
armanbilge Nov 30, 2022
f3022e4
Parse/decode in single op
armanbilge Nov 30, 2022
d630597
More package private models
armanbilge Nov 30, 2022
366248a
Build error body from `Throwable`
armanbilge Nov 30, 2022
ab6fa05
Define `EntityEncoder` for `LmabdaErrorBody`
armanbilge Nov 30, 2022
54c3032
Reduce noise
armanbilge Dec 4, 2022
b5e3019
Simplifications
armanbilge Dec 4, 2022
d6de562
Use `Uri` for `lambdaRuntimeApi`
armanbilge Dec 4, 2022
13aa8e9
Running lambda returns `Nothing`
armanbilge Dec 4, 2022
3517cb7
Refactor `processEvents`
armanbilge Dec 4, 2022
ffa7bb3
`deadlineTimeInMs` -> `deadlineTime`
armanbilge Dec 4, 2022
f1f7a2c
Header names are ci strings
armanbilge Dec 4, 2022
f83a9de
Do not leak input in sanitized error msg
armanbilge Dec 4, 2022
c700129
Fix compile
armanbilge Dec 4, 2022
9a49c7d
Use `embedError`
armanbilge Dec 6, 2022
d53f8be
refactored to have environment variables read in the initialization p…
scott-thomson239 Dec 6, 2022
f284899
readded Nothing instead of Unit in LambdaRuntime
scott-thomson239 Dec 6, 2022
3da24a9
fixed failing tests from refactoring initialization phase
scott-thomson239 Dec 6, 2022
8cdfc9b
remove accidental encoder import
scott-thomson239 Dec 6, 2022
541704e
reverted to using F[Nothing] as output type
scott-thomson239 Dec 7, 2022
89d1c69
changed init error test env to better reflect real behaviour
scott-thomson239 Dec 7, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 31 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,16 +30,20 @@ jobs:
os: [ubuntu-latest]
scala: [2.12.17, 3.2.0, 2.13.10]
java: [corretto@8, corretto@11]
project: [rootJS, rootJVM]
project: [rootJS, rootJVM, rootNative]
exclude:
- scala: 2.12.17
java: corretto@11
- scala: 3.2.0
java: corretto@11
- project: rootJS
java: corretto@11
- project: rootNative
java: corretto@11
- project: rootJS
scala: 2.12.17
- project: rootNative
scala: 2.12.17
runs-on: ${{ matrix.os }}
steps:
- name: Checkout current branch (full)
Expand Down Expand Up @@ -108,6 +112,10 @@ jobs:
if: matrix.project == 'rootJS'
run: sbt 'project ${{ matrix.project }}' '++${{ matrix.scala }}' Test/scalaJSLinkerResult

- name: nativeLink
if: matrix.project == 'rootNative'
run: sbt 'project ${{ matrix.project }}' '++${{ matrix.scala }}' Test/nativeLink

- name: Test
if: matrix.scala != '2.12.17'
run: sbt 'project ${{ matrix.project }}' '++${{ matrix.scala }}' test
Expand All @@ -126,11 +134,11 @@ jobs:

- name: Make target directories
if: github.event_name != 'pull_request' && (startsWith(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/main')
run: mkdir -p target lambda-cloudformation-custom-resource/.js/target lambda-http4s/.jvm/target unidocs/target .js/target core/.js/target examples/.js/target lambda-http4s/.js/target core/.jvm/target .jvm/target lambda/js/target .native/target examples/.jvm/target lambda/jvm/target sbt-lambda/target lambda-cloudformation-custom-resource/.jvm/target project/target
run: mkdir -p lambda-kernel/.jvm/target target lambda-cloudformation-custom-resource/.js/target lambda-http4s/.jvm/target unidocs/target .js/target core/.js/target examples/.js/target lambda-http4s/.js/target core/.jvm/target .jvm/target lambda/js/target lambda-kernel/.js/target .native/target lambda-kernel/.native/target examples/.jvm/target lambda/jvm/target lambda-runtime/.jvm/target sbt-lambda/target lambda-runtime/.native/target lambda-cloudformation-custom-resource/.jvm/target lambda-runtime/.js/target project/target

- name: Compress target directories
if: github.event_name != 'pull_request' && (startsWith(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/main')
run: tar cf targets.tar target lambda-cloudformation-custom-resource/.js/target lambda-http4s/.jvm/target unidocs/target .js/target core/.js/target examples/.js/target lambda-http4s/.js/target core/.jvm/target .jvm/target lambda/js/target .native/target examples/.jvm/target lambda/jvm/target sbt-lambda/target lambda-cloudformation-custom-resource/.jvm/target project/target
run: tar cf targets.tar lambda-kernel/.jvm/target target lambda-cloudformation-custom-resource/.js/target lambda-http4s/.jvm/target unidocs/target .js/target core/.js/target examples/.js/target lambda-http4s/.js/target core/.jvm/target .jvm/target lambda/js/target lambda-kernel/.js/target .native/target lambda-kernel/.native/target examples/.jvm/target lambda/jvm/target lambda-runtime/.jvm/target sbt-lambda/target lambda-runtime/.native/target lambda-cloudformation-custom-resource/.jvm/target lambda-runtime/.js/target project/target

- name: Upload target directories
if: github.event_name != 'pull_request' && (startsWith(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/main')
Expand Down Expand Up @@ -229,6 +237,16 @@ jobs:
tar xf targets.tar
rm targets.tar

- name: Download target directories (3.2.0, rootNative)
uses: actions/download-artifact@v2
with:
name: target-${{ matrix.os }}-${{ matrix.java }}-3.2.0-rootNative

- name: Inflate target directories (3.2.0, rootNative)
run: |
tar xf targets.tar
rm targets.tar

- name: Download target directories (2.13.10, rootJS)
uses: actions/download-artifact@v2
with:
Expand All @@ -249,6 +267,16 @@ jobs:
tar xf targets.tar
rm targets.tar

- name: Download target directories (2.13.10, rootNative)
uses: actions/download-artifact@v2
with:
name: target-${{ matrix.os }}-${{ matrix.java }}-2.13.10-rootNative

- name: Inflate target directories (2.13.10, rootNative)
run: |
tar xf targets.tar
rm targets.tar

- name: Import signing key
if: env.PGP_SECRET != '' && env.PGP_PASSPHRASE == ''
run: echo $PGP_SECRET | base64 -di | gpg --import
Expand Down
85 changes: 66 additions & 19 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

name := "feral"

ThisBuild / tlBaseVersion := "0.1"
ThisBuild / tlBaseVersion := "0.2"
ThisBuild / startYear := Some(2021)

ThisBuild / developers := List(
Expand All @@ -26,8 +26,10 @@ ThisBuild / developers := List(
)

ThisBuild / githubWorkflowJavaVersions := Seq("8", "11").map(JavaSpec.corretto(_))
ThisBuild / githubWorkflowBuildMatrixExclusions +=
MatrixExclude(Map("project" -> "rootJS", "scala" -> Scala212))
ThisBuild / githubWorkflowBuildMatrixExclusions ++=
Seq(
MatrixExclude(Map("project" -> "rootJS", "scala" -> Scala212)),
MatrixExclude(Map("project" -> "rootNative", "scala" -> Scala212)))

ThisBuild / githubWorkflowBuild ~= { steps =>
val scriptedStep = WorkflowStep.Sbt(
Expand Down Expand Up @@ -61,9 +63,11 @@ val circeVersion = "0.14.3"
val fs2Version = "3.3.0"
val http4sVersion = "0.23.16"
val natchezVersion = "0.1.6"
val munitVersion = "0.7.29"
val munitCEVersion = "1.0.7"
val scalacheckEffectVersion = "1.0.4"
val munitVersion = "1.0.0-M6"
val munitCEVersion = "2.0.0-M3"
val scalacheckEffectVersion = "2.0.0-M2"

ThisBuild / resolvers ++= Resolver.sonatypeOssRepos("snapshots")

lazy val commonSettings = Seq(
crossScalaVersions := Seq(Scala3, Scala213)
Expand All @@ -72,6 +76,8 @@ lazy val commonSettings = Seq(
lazy val root =
tlCrossRootProject.aggregate(
core,
lambdaKernel,
lambdaRuntime,
lambda,
sbtLambda,
lambdaHttp4s,
Expand All @@ -91,35 +97,69 @@ lazy val core = crossProject(JSPlatform, JVMPlatform)
)
.settings(commonSettings)

lazy val lambda = crossProject(JSPlatform, JVMPlatform)
.in(file("lambda"))
lazy val lambdaKernel = crossProject(JSPlatform, JVMPlatform, NativePlatform)
.crossType(CrossType.Pure)
.in(file("lambda-kernel"))
.settings(
name := "feral-lambda",
name := "feral-lambda-kernel",
libraryDependencies ++= Seq(
"org.tpolecat" %%% "natchez-core" % natchezVersion,
"org.typelevel" %%% "cats-effect" % catsEffectVersion,
"org.tpolecat" %%% "natchez-core" % "0.1.6-269-2b28bcd-SNAPSHOT",
"io.circe" %%% "circe-scodec" % circeVersion,
"io.circe" %%% "circe-jawn" % circeVersion,
"org.scodec" %%% "scodec-bits" % "1.1.34",
"org.scalameta" %%% "munit-scalacheck" % munitVersion % Test,
"org.typelevel" %%% "munit-cats-effect-3" % munitCEVersion % Test,
"org.typelevel" %%% "munit-cats-effect" % munitCEVersion % Test,
"io.circe" %%% "circe-literal" % circeVersion % Test
)
)
.settings(commonSettings)
.jsSettings(
.platformsSettings(JSPlatform, NativePlatform)(
libraryDependencies ++= Seq(
"io.circe" %%% "circe-scalajs" % circeVersion,
"io.github.cquiroz" %%% "scala-java-time" % "2.4.0"
)
)

lazy val lambdaRuntime = crossProject(JSPlatform, JVMPlatform, NativePlatform)
.crossType(CrossType.Pure)
.in(file("lambda-runtime"))
.settings(
name := "feral-lambda-runtime",
libraryDependencies ++= Seq(
"org.scalameta" %%% "munit-scalacheck" % munitVersion % Test,
"org.typelevel" %%% "munit-cats-effect" % munitCEVersion % Test,
"org.http4s" %%% "http4s-dsl" % http4sVersion % Test,
"org.http4s" %%% "http4s-client" % http4sVersion,
"org.http4s" %%% "http4s-circe" % http4sVersion
)
)
.settings(commonSettings)
.dependsOn(lambdaKernel)

lazy val lambda = crossProject(JSPlatform, JVMPlatform)
.in(file("lambda"))
.settings(
name := "feral-lambda",
libraryDependencies ++= Seq(
"org.scalameta" %%% "munit-scalacheck" % munitVersion % Test,
"org.typelevel" %%% "munit-cats-effect" % munitCEVersion % Test,
"io.circe" %%% "circe-literal" % circeVersion % Test
)
)
.settings(commonSettings)
.jsSettings(
libraryDependencies ++= Seq(
"io.circe" %%% "circe-scalajs" % circeVersion
)
)
.jvmSettings(
libraryDependencies ++= Seq(
"com.amazonaws" % "aws-lambda-java-core" % "1.2.1",
"co.fs2" %%% "fs2-io" % fs2Version,
"io.circe" %%% "circe-fs2" % "0.14.0"
)
)
.dependsOn(core)
.dependsOn(core, lambdaKernel)

lazy val sbtLambda = project
.in(file("sbt-lambda"))
Expand All @@ -134,7 +174,12 @@ lazy val sbtLambda = project
scriptedLaunchOpts := {
scriptedLaunchOpts.value ++ Seq("-Xmx1024M", "-Dplugin.version=" + version.value)
},
scripted := scripted.dependsOn(core.js / publishLocal, lambda.js / publishLocal).evaluated
scripted := scripted
.dependsOn(
core.js / publishLocal,
lambda.js / publishLocal,
lambdaKernel.js / publishLocal)
.evaluated
)

lazy val lambdaHttp4s = crossProject(JSPlatform, JVMPlatform)
Expand All @@ -147,7 +192,7 @@ lazy val lambdaHttp4s = crossProject(JSPlatform, JVMPlatform)
)
)
.settings(commonSettings)
.dependsOn(lambda % "compile->compile;test->test")
.dependsOn(lambdaKernel % "compile->compile;test->test")

lazy val lambdaCloudFormationCustomResource = crossProject(JSPlatform, JVMPlatform)
.crossType(CrossType.Pure)
Expand All @@ -164,15 +209,15 @@ lazy val lambdaCloudFormationCustomResource = crossProject(JSPlatform, JVMPlatfo
"org.http4s" %%% "http4s-circe" % http4sVersion,
"org.http4s" %%% "http4s-dsl" % http4sVersion % Test,
"org.scalameta" %%% "munit-scalacheck" % munitVersion % Test,
"org.typelevel" %%% "munit-cats-effect-3" % munitCEVersion % Test,
"org.typelevel" %%% "munit-cats-effect" % munitCEVersion % Test,
"org.typelevel" %%% "scalacheck-effect" % scalacheckEffectVersion % Test,
"org.typelevel" %%% "scalacheck-effect-munit" % scalacheckEffectVersion % Test,
"com.eed3si9n.expecty" %%% "expecty" % "0.16.0" % Test,
"io.circe" %%% "circe-testing" % circeVersion % Test
)
)
.settings(commonSettings)
.dependsOn(lambda)
.dependsOn(lambdaKernel)

lazy val examples = crossProject(JSPlatform, JVMPlatform)
.crossType(CrossType.Pure)
Expand Down Expand Up @@ -203,7 +248,9 @@ lazy val unidocs = project
core.jvm,
lambda.jvm,
lambdaHttp4s.jvm,
lambdaCloudFormationCustomResource.jvm
lambdaCloudFormationCustomResource.jvm,
lambdaKernel.jvm,
lambdaRuntime.jvm
)
}
)
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ import org.http4s.Request
import org.http4s.Uri
import org.http4s.headers.Cookie
import org.http4s.headers.`Set-Cookie`

object ApiGatewayProxyHandler {
def apply[F[_]: Concurrent: ApiGatewayProxyLambdaEnv](
routes: HttpRoutes[F]): F[Option[ApiGatewayProxyStructuredResultV2]] = httpRoutes(routes)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ final class Context[F[_]] private[lambda] (
f(remainingTime))
}

object Context extends ContextCompanionPlatform
object Context

final class CognitoIdentity(
val identityId: String,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
/*
* Copyright 2021 Typelevel
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package feral.lambda.runtime

import io.circe.Encoder
import io.circe.Printer
import org.http4s.EntityEncoder
import org.http4s.circe._

private[runtime] final class LambdaErrorBody(
val errorMessage: String,
val errorType: String,
val stackTrace: List[String])

private[runtime] object LambdaErrorBody {

def apply(errorMessage: String, errorType: String, stackTrace: List[String]) =
new LambdaErrorBody(errorMessage, errorType, stackTrace)

def fromThrowable(ex: Throwable): LambdaErrorBody =
LambdaErrorBody(
ex.getMessage,
ex.getClass.getSimpleName,
ex.getStackTrace().toList.map(_.toString)
)

implicit val encoder: Encoder[LambdaErrorBody] =
Encoder.forProduct3("errorMessage", "errorType", "stackTrace")(e =>
(e.errorMessage, e.errorType, e.stackTrace))

implicit def entityEncoder[F[_]]: EntityEncoder[F, LambdaErrorBody] =
jsonEncoderWithPrinterOf(Printer.noSpaces.copy(dropNullValues = true))
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
/*
* Copyright 2021 Typelevel
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package feral.lambda.runtime

import cats.effect.kernel.Concurrent
import io.circe._
import cats.syntax.all._
import feral.lambda._
import org.http4s.circe._
import org.http4s._
import java.util.concurrent.TimeUnit
import scala.concurrent.duration.FiniteDuration
import feral.lambda.runtime.headers._

private[runtime] final class LambdaRequest(
val deadlineTime: FiniteDuration,
val id: String,
val invokedFunctionArn: String,
val identity: Option[CognitoIdentity],
val clientContext: Option[ClientContext],
val body: Json
)

private[runtime] object LambdaRequest {
def fromResponse[F[_]](response: Response[F])(implicit F: Concurrent[F]): F[LambdaRequest] = {
implicit val jsonDecoder: EntityDecoder[F, Json] = jsonDecoderIncremental
for {
id <- response
.headers
.get[`Lambda-Runtime-Aws-Request-Id`]
.liftTo(new NoSuchElementException(`Lambda-Runtime-Aws-Request-Id`.name.toString))
invokedFunctionArn <- response
.headers
.get[`Lambda-Runtime-Invoked-Function-Arn`]
.liftTo(new NoSuchElementException(`Lambda-Runtime-Invoked-Function-Arn`.name.toString))
deadlineTimeInMs <- response
.headers
.get[`Lambda-Runtime-Deadline-Ms`]
.liftTo(new NoSuchElementException(`Lambda-Runtime-Deadline-Ms`.name.toString))
clientIdentity <- response.headers.get[`Lambda-Runtime-Client-Identity`].pure
clientContext <- response.headers.get[`Lambda-Runtime-Client-Context`].pure
body <- response.as[Json]
} yield {
new LambdaRequest(
FiniteDuration(deadlineTimeInMs.value, TimeUnit.MILLISECONDS),
id.value,
invokedFunctionArn.value,
clientIdentity.map(_.value),
clientContext.map(_.value),
body
)
}
}
}
Loading