Support for golang lambda functions

[wolfgangmeyers]

AWS now offers official support for golang lambda functions: https://aws.amazon.com/blogs/compute/announcing-go-support-for-aws-lambda/

Any idea on how to make these work with localstack? If they're not yet supported, where would be the best place to look if I wanted to add support?

[HoneyryderChuck]

+1, I'd be definitely interested in this

[wolfgangmeyers]

In case in helps anyone else, I've worked around this for now by implementing a webhook wrapper around the lambda handler that simulates a direct sns->lambda invocation. It handles subscription confirmation requests, but as far as I've seen a confirmation request isn't ever sent by localstack. This example uses the gin framework for http handling:

// SNSPayload represents the expected JSON formation from
// SNS that would be delivered to a webhook
type SNSPayload struct {
	Type         string // SubscriptionConfirmation | Notification
	SubscribeURL string
	Message      string
}

// Start up a replication webhook to fill in for lack of go lambda support in localstack
func webhook() {
	// Set environment variables that lambda task will expect
	os.Setenv("AWS_REGION", *awsRegion)
	os.Setenv("AWS_HOST", *awsHost)
	os.Setenv("AWS_PREFIX", *awsPrefix)

	r := gin.New()
	// GET / supports a check from the subscription tool to make sure
	// that the webhook is up before proceeding with the sns subscription
	r.GET("/", func(ctx *gin.Context) {
		ctx.String(http.StatusOK, "UP!")
	})
	r.POST("/", func(ctx *gin.Context) {
		// This is either subcription confirmation or a replication event is being sent
		snsPayload := &SNSPayload{}
		err := ctx.BindJSON(snsPayload)
		if snsPayload.Type == "SubscriptionConfirmation" {
			logger.Info("Handling subscription confirmation")
			_, err = http.Get(snsPayload.SubscribeURL)
			if err != nil {
				logger.Errorf("Error confirming subscription: %v", err.Error())
			}
		} else {
			// Simulate a normal SNS invocation of lambda
			err = replication.Handler(events.SNSEvent{
				Records: []events.SNSEventRecord{
					events.SNSEventRecord{
						SNS: events.SNSEntity{
							Message: snsPayload.Message,
						},
					},
				},
			})
			if err != nil {
				logger.Errorf("Error invoking lambda function: %v", err.Error())
			}
		}
	})
	addr := fmt.Sprintf(":%v", *appPort)
	logger.Infof("Starting up on %v", addr)
	http.ListenAndServe(addr, r)
}

[whummer]

lambci/docker-lambda recently added support for Go Lambda functions, see this example: https://github.com/lambci/docker-lambda/blob/master/examples/go1.x/handler.go .

Since we're already using the lambci Docker images for Java/Node/Python, it should be fairly straightforward to add support for Go. @wolfgangmeyers @HoneyryderChuck would you be able to look into into this? I'd be happy to help out where needed. This is a good starting point: lambda_executors.py

[wolfgangmeyers]

@whummer Thanks for the help on this. I looked at lambda_executors.py and lambda_api.py, and after some digging, it seems that the runtime that is passed to lambci comes from the create_function api:

localstack/localstack/services/awslambda/lambda_api.py

Line 453 in 2be88f5

func_details.runtime = data['Runtime']

It appears that lambci is downloaded automatically by docker, so I think that should just work if the runtime is configured when creating the function. We might need to configure the correct executor here when the function code is updated:

localstack/localstack/services/awslambda/lambda_api.py

Line 310 in 2be88f5

def set_function_code(code, lambda_name):

[wolfgangmeyers]

I'll try to see if I can get this working when I have time

[wolfgangmeyers]

@whummer I've made some progress. I've made some local changes so that the handler doesn't get a .py added to it, set up the LAMBDA_EXECUTOR=docker. The lambda executor seems to be attempting to use lambci now to run the task, but I'm hitting an error:

localstack_1  | 2018-02-16T19:27:12:WARNING:localstack.services.awslambda.lambda_api: Error executing Lambda function: Lambda process returned error status code: 1. Output:
localstack_1  | Unable to find image 'lambci/lambda:go1.x' locally
localstack_1  | go1.x: Pulling from lambci/lambda
localstack_1  | 5be106c3813f: Pulling fs layer
localstack_1  | e240967675e1: Pulling fs layer
localstack_1  | 5448b9eca4b0: Pulling fs layer
localstack_1  | 3bc1caac25ba: Pulling fs layer
localstack_1  | 3bc1caac25ba: Waiting
localstack_1  | e240967675e1: Verifying Checksum
localstack_1  | e240967675e1: Download complete
localstack_1  | 5448b9eca4b0: Verifying Checksum
localstack_1  | 5448b9eca4b0: Download complete
localstack_1  | 3bc1caac25ba: Verifying Checksum
localstack_1  | 3bc1caac25ba: Download complete
localstack_1  | 5be106c3813f: Verifying Checksum
localstack_1  | 5be106c3813f: Download complete
localstack_1  | 5be106c3813f: Pull complete
localstack_1  | e240967675e1: Pull complete
localstack_1  | 5448b9eca4b0: Pull complete
localstack_1  | 3bc1caac25ba: Pull complete
localstack_1  | Digest: sha256:d77adf847c45dcb5fae3cd93283447fad3f3d51ead024aed0c866a407a206e7c
localstack_1  | Status: Downloaded newer image for lambci/lambda:go1.x
localstack_1  | START RequestId: 7519261c-3b1f-1f6d-3d12-5144c6c5d86b Version: $LATEST
localstack_1  | END RequestId: 7519261c-3b1f-1f6d-3d12-5144c6c5d86b
localstack_1  | REPORT RequestId: 7519261c-3b1f-1f6d-3d12-5144c6c5d86b  Duration: 3.96 ms       Billed Duration: 100 ms Memory Size: 1536 MB   Max Memory Used: 6 MB
localstack_1  | {
localstack_1  |   "errorMessage": "fork/exec /var/task/my-go-task: permission denied",
localstack_1  |   "errorType": "PathError"
localstack_1  | }
localstack_1  |  Traceback (most recent call last):
localstack_1  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_api.py", line 255, in run_lambda
localstack_1  |     event, context=context, version=version, async=async)
localstack_1  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 130, in execute
localstack_1  |     result, log_output = self.run_lambda_executor(cmd, environment, async)
localstack_1  |   File "/opt/code/localstack/localstack/services/awslambda/lambda_executors.py", line 67, in run_lambda_executor
localstack_1  |     (return_code, log_output))

[wolfgangmeyers]

Similar issue reported here on a different project - aws/aws-sam-cli#274

[wolfgangmeyers]

I printed out the command that is running in order to launch docker:

localstack.services.awslambda.lambda_executors: CONTAINER_ID="$(docker create  -e AWS_LAMBDA_EVENT_BODY="$AWS_LAMBDA_EVENT_BODY" -e LOCALSTACK_HOSTNAME="$LOCALSTACK_HOSTNAME" -e HOSTNAME="$HOSTNAME" "lambci/lambda:go1.x" "my-go-task")";docker cp "/tmp/localstack/zipfile.1c5caf40/." "$CONTAINER_ID:/var/task";docker start -a "$CONTAINER_ID";

Going to see if I can run this outside of localstack and get more details

[wolfgangmeyers]

So I was able to take one of the stopped containers and recreate it with docker commit and launched it with bash as the entrypoint. I've confirmed that the executable permissions of the handler are gone, and that's why the patherror is being returned. Somewhere in the process of unzipping/copying the permissions are being lost.

[wolfgangmeyers]

Docs for docker cp state that file permission are preserved, so I'm going to look over the unzipping code - https://docs.docker.com/engine/reference/commandline/cp/#extended-description

[wolfgangmeyers]

Looks like the problem is with the behavior of ZipFile.extractall - I found a solution, will update the code: https://www.burgundywall.com/post/preserving-file-perms-with-python-zipfile-module

[wolfgangmeyers]

I think it worked...? I can't seem to get any kind of output from the task, just a statuscode. But the process seems to be executing normally now.

$ aws lambda create-function --endpoint="http://localhost:4574"  --function-name="hello" --runtime="go1.x"--handler="hello" --role="foobar" --zip-file=fileb://hello.zip
{
    "FunctionName": "hello",
    "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:hello",
    "Runtime": "go1.x",
    "Role": "foobar",
    "Handler": "hello",
    "VpcConfig": {
        "SubnetIds": [
            null
        ],
        "SecurityGroupIds": [
            null
        ]
    },
    "Environment": {
        "Variables": {},
        "Error": {}
    },
    "TracingConfig": {}
}
$ aws lambda --endpoint="http://localhost:4574" invoke --function-name=hello --payload='{"Name": "Wolfgang"}' test
{
    "StatusCode": 200
}

[wolfgangmeyers]

@whummer I've submitted a PR, any feedback or help with testing would be appreciated - #617

It might be worth documenting exactly how to get this working, and mention go1.x in the error that is thrown when LAMBDA_EXECUTOR=docker isn't set.

[whummer]

Initial support for Go Lambda functions added in #617

[zbuc]

I'm having trouble with this feature, and was wondering if you'd encountered it or if there could be any clarity added to the documentation.

I am using localstack inside a Docker container:

LAMBDA_EXECUTOR=docker localstack start --docker

and a very simple golang lambda handler:

$ cat main.go
package main

import (
        "fmt"
        "context"
        "github.com/aws/aws-lambda-go/lambda"
)

type MyEvent struct {
        Name string `json:"name"`
}

func HandleRequest(ctx context.Context, name MyEvent) (string, error) {
        return fmt.Sprintf("Hello %s!", name.Name ), nil
}

func main() {
        lambda.Start(HandleRequest)
}

I compile the lambda function:

go build

and then create a zip file:

zip lambda.zip lambda

I create the function and upload the zip file:

LAMBDA_EXECUTOR=go1.x aws --endpoint-url=http://localhost:4574 lambda create-function --function-name=test2 --runtime="go1.x" --role=r1 --handler=lambda --zip-file fileb://lambda.zip

And then try to execute it:

aws lambda --endpoint-url=http://localhost:4574 invoke --function-name test2 --payload='{"Name": "Test"}' result.log

And then it hangs for a while, nothing appears in the logs, and I see the message:

HTTPConnectionPool(host='localhost', port=4574): Read timed out. (read timeout=60)

after 60 seconds.

I'm able to run Python-based lambdas in localstack fine.

[wolfgangmeyers]

@whummer I'm not sure when I will have time to dig into this, hopefully soon. I'd like to automate a simple test case like this to test the handler. Any thoughts on having the example script compile during the tests, or would it make sense to check in the binary, or host it somewhere else?

[whummer]

Thanks @wolfgangmeyers 👍

Any thoughts on having the example script compile during the tests, or would it make sense to check in the binary, or host it somewhere else?

Setting up a Go environment in CI seems like a bit of an overkill. The sample zipped Go program is roughly 2.9MB - I'd prefer not to check in files of this size into the main repo.

I've created a new repo for test artifacts - can you please create a PR to add the binary here: https://github.com/localstack/test-artifacts

[wolfgangmeyers]

@zbuc I was able to get your scenario to work in localstack.

$ aws --endpoint-url=http://localhost:4574 lambda create-function --function-name=task --runtime="go1.x" --role=r1 --handler=task --zip-file fileb://task.zip --region=us-east-1
{
    "FunctionName": "task",
    "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:task",
    "Runtime": "go1.x",
    "Role": "r1",
    "Handler": "task",
    "VpcConfig": {
        "SubnetIds": [
            null
        ],
        "SecurityGroupIds": [
            null
        ]
    },
    "Environment": {
        "Variables": {},
        "Error": {}
    },
    "TracingConfig": {}
}

$ aws lambda --endpoint-url=http://localhost:4574 invoke --function-name task --payload='{"Name": "Test"}' --region=us-east-1 result.log{
    "StatusCode": 200
}

$ cat result.log
"Hello Test!"

Based on everything you listed, I think think this should work. What is your host operating system?
You might also try using docker-compose to launch localstack. This is the docker-compose.yml file I've been using on a recent project:

version: "3"

services:
  localstack:
    image: "localstack/localstack"
    ports:
      - "4569:4569" # dynamodb
      - "4574:4574" # lambda
      - "8080:8080" # dashboard
    environment:
      - SERVICES=dynamodb,lambda
      - LAMBDA_EXECUTOR=docker
      - DOCKER_HOST=unix:///var/run/docker.sock
      - DEBUG=1
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

[wolfgangmeyers]

@whummer PR on the test artifacts repo - localstack-samples/localstack-pro-samples#1

[wolfgangmeyers]

Source code for the test application posted here - https://bitbucket.org/wolfgang_meyers/go_hello_lambda/src/master/

[vineetvermait]

I am still facing this issue...response hangs and later response is

('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))

Any idea why would it happen??

here are the below Localstack logs:

2018-08-02T12:51:28:DEBUG:localstack.services.awslambda.lambda_executors: Running lambda cmd: CONTAINER_ID="$(docker create -e AWS_LAMBDA_FUNCTION_INVOKED_ARN="$AWS_LAMBDA_FUNCTION_INVOKED_ARN" -e AWS_LAMBDA_FUNCTION_NAME="$AWS_LAMBDA_FUNCTION_NAME" -e AWS_LAMBDA_EVENT_BODY="$AWS_LAMBDA_EVENT_BODY" -e HOSTNAME="$HOSTNAME" -e AWS_LAMBDA_FUNCTION_VERSION="$AWS_LAMBDA_FUNCTION_VERSION" -e LOCALSTACK_HOSTNAME="$LOCALSTACK_HOSTNAME" "lambci/lambda:go1.x" "pricing_occupancy_subscriber")";docker cp "/tmp/localstack/zipfile.25f94e16/." "$CONTAINER_ID:/var/task";docker start -a "$CONTAINER_ID";

the above line keeps on printing till the error appears

please help!!!

[johnpfeiffer]

I too wasn't sure how to configure the YAML for a Golang lambda, found this very helpful write-up, and ran into a similar issue...

localstack_1  | 2020-03-06T05:36:42:INFO:localstack.services.awslambda.lambda_executors: Running lambda cmd: CONTAINER_ID="$(docker create -i   -e DOCKER_LAMBDA_USE_STDIN="$DOCKER_LAMBDA_USE_S
TDIN" -e HOSTNAME="$HOSTNAME" -e LOCALSTACK_HOSTNAME="$LOCALSTACK_HOSTNAME" -e _HANDLER="$_HANDLER" -e AWS_LAMBDA_FUNCTION_TIMEOUT="$AWS_LAMBDA_FUNCTION_TIMEOUT" -e AWS_LAMBDA_FUNCTION_NAME="$
AWS_LAMBDA_FUNCTION_NAME" -e AWS_LAMBDA_FUNCTION_VERSION="$AWS_LAMBDA_FUNCTION_VERSION" -e AWS_LAMBDA_FUNCTION_INVOKED_ARN="$AWS_LAMBDA_FUNCTION_INVOKED_ARN"  --rm "lambci/lambda:go1.x" "task"
)";docker cp "/tmp/localstack/zipfile.342c8e02/." "$CONTAINER_ID:/var/task"; docker start -ai "$CONTAINER_ID";

I was able to successfully 😃 resolve this by first downloading the Docker container that is being used to execute Go binaries...
sudo docker pull lambci/lambda:go1.x

aws lambda --endpoint-url=http://localhost:4574 invoke --function-name task --payload='{"Name": "world"}' --region=us-east-1 myout.log

{ "StatusCode": 200 }

localstack_1  | 2020-03-06T05:45:22:DEBUG:localstack.services.awslambda.lambda_executors: Lambda arn:aws:lambda:us-east-1:000000000000:function:task result / log output:

[ypwu1]

aws lambda --endpoint-url=http://localhost:4574 invoke --function-name task --payload='{"Name": "world"}' --region=us-east-1 myout.log

I have pulled docker-image, but I still can't get it working ...
My localstack setup:

services:
  localstack:
    image: localstack/localstack
    ports:
      - "4566-4599:4566-4599"
      - "${PORT_WEB_UI-8888}:${PORT_WEB_UI-8080}" 
    environment:
      - SERVICES=${SERVICES-lambda,apigateway,sqs}
      - DEBUG=${DEBUG- }
      - DEFAULT_REGION=ap-southeast-2
      - DATA_DIR=${DATA_DIR- }
      - PORT_WEB_UI=${PORT_WEB_UI- }
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR-docker-reuse }
      - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
      - DOCKER_HOST=unix:///var/run/docker.sock
      - LAMBDA_CONTAINER_REGISTRY=lambci/lambda:build-go1.x
    volumes:
      - ".localstack:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"