Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VPC lambda support #281

Closed
siegesmund opened this Issue Aug 28, 2017 · 24 comments

Comments

Projects
None yet
@siegesmund
Copy link

commented Aug 28, 2017

I need up to create its function inside a VPC. It's easy to do with Apex, and I've tried using the Apex style configuration, but my functions are still being deployed without a VPC configuration. Is there a way to do this?

@tj

This comment has been minimized.

Copy link
Member

commented Aug 28, 2017

Ahh not yet, I don't think it's possible to deploy API Gateway to a VPC right now (only Lambdas), but keep an eye on #14

@tj tj closed this Aug 28, 2017

@siegesmund

This comment has been minimized.

Copy link
Author

commented Aug 28, 2017

I'm not trying to put the API Gateway inside the VPC; #14 is a different question. The goal is to put the lambda in the VPC, so it can access resources like an RDS instance that's only accessible inside the VPC. It can still function as a handler for the API Gateway endpoint(s). This is a feature of the Serverless framework, configurable in serverless.yml.

Perhaps my initial question was a little inarticulate. Does this make sense? And is it possible/or on the roadmap? And BTW, thanks TJ :)

@tj

This comment has been minimized.

Copy link
Member

commented Aug 28, 2017

ahh ok cool thanks, I'll reopen! I definitely wouldn't mind getting that in there


  • implement
  • documentation (listing subnet ids etc, link to AWS docs)
  • QA with and without

@tj tj reopened this Aug 28, 2017

@tj tj added the Feature Accepted label Aug 28, 2017

@tj tj changed the title Configure up.json to create function in VPC? VPC lambda support Aug 28, 2017

@siegesmund

This comment has been minimized.

Copy link
Author

commented Aug 29, 2017

Not sure the complexity of this one, but If you can point me in the right direction, I'd be happy to see if I can't make the changes in an pull request.

@choonkeat

This comment has been minimized.

Copy link

commented Nov 26, 2017

I'll be happy if I can use the same json as apex in my up.json

{
  "vpc": {
    "name": "vpc-...",
    "securityGroups": ["..."],
    "subnets": ["subnet-..."]
  }
}
@vietbui

This comment has been minimized.

Copy link

commented Dec 20, 2017

It's a deal breaker as in lots of user cases, the lambda function needs access to VPC resources (e.g. RDS, internal load balancer...). One option is to change back to deploying the service in ECS and access via public load balancer but it's an overhead and I really enjoyed using up so far.

By the way, thanks for your great work TJ.

@a-golovanov

This comment has been minimized.

Copy link

commented Dec 20, 2017

@vietbui as a temporary workaround you can manually specify VPC for the already created lambda and it will stay configured across deployments.

@vietbui

This comment has been minimized.

Copy link

commented Dec 20, 2017

Nice workaround @a-golovanov! Thank you.

@vespertilian

This comment has been minimized.

Copy link

commented Jan 15, 2018

Just FYI for anyone else following @a-golovanov temporary workaround. I had to add the extra credentials from this page of Apex before deploys would work for me:

https://github.com/apex/apex/blob/master/docs/aws-credentials.md

@cubabit

This comment has been minimized.

Copy link

commented Jan 19, 2018

Does anyone know if you have to open up any ports in the security group as I get this error in up logs:

  01:58:53pm FATA error: initializing relay: waiting for http://127.0.0.1:46113 to be in listening state: timed out after 15s

I'm not sure what is trying to connect to what and as the error is from the logs of the lambda function and its using the localhost ip I'm not sure if it is a security group issue

@naartjie

This comment has been minimized.

Copy link
Contributor

commented Jan 19, 2018

Hi @cubabit, I don't have too much context or experience using apex/up, but I didn't have to set up anything extra to get the logs (only to access RDS, that required a manual step on my part).

Looking at your error, is that what you get invoking up logs in your terminal? I would venture to say that might be your actual lambda log output. Have you tried going to CloudWatch and looking at the logs there to confirm that's not the case? For example if your app is not listening on port number in process.env.PORT i.e. 46113 in your case, that could be the problem. Just thinking out loud though, I could be wrong.

@cubabit

This comment has been minimized.

Copy link

commented Jan 19, 2018

Thanks @naartjie for replying.

Turns out my app was not listening on process.env.PORT like it should, but instead on another differently configured port. Once I changed it to use that env var everything worked!

@keshavab

This comment has been minimized.

Copy link

commented Feb 22, 2018

+1. It would be awesome to have support for deploying lambda in given VPC and subnet since we would have backend resources accessible only within VPC.

@simap

This comment has been minimized.

Copy link

commented Apr 4, 2018

+1
adding the lambda to a vpc didn't work for me, seemed to prevent it from reading env vars.
would rather not make my RDS public.

@cubabit

This comment has been minimized.

Copy link

commented Apr 4, 2018

I can see why this is probably not going to be a priority because web apps in Lambda experience a huge performance penalty if they are in a VPC (see this article).

Apex supports VPC, as it is aimed at non-user facing functions, unlike Up.

@simap

This comment has been minimized.

Copy link

commented Apr 4, 2018

@cubabit and that may or may not kill my use-case assuming warming doesn't work. If there was a way to lock down a public rds, I'm game to try that.
Sounds like eventually aws will fix that performance problem, or serverless Aurora becomes generally available presumably with some kind of iam role-based access.
It could be up isn't the best fit for this particular app at this time.

@cubabit

This comment has been minimized.

Copy link

commented Apr 4, 2018

@simap You can lock down RDS to the IP range of AWS Lambda IPs, which is a better than nothing solution but obviously not great for information security.

http://blog.rowanudell.com/updating-security-groups-with-lambda/

Also, the problem with warming is it is not always effective. Lambda may according to traffic scale up and create a new instance to handle your request at any time, which will be cold. API Gateway will dispatch to that instance and therefore suffer from a slow response time.

@simap

This comment has been minimized.

Copy link

commented Apr 5, 2018

@cubabit thanks for those links, thats close to what I was looking for and at least reduces the attack vector.

I'm not sure if the performance problem is there anymore. I just set up a basic lambda in a vpc consuming an simple private http endpoint on an ec2 and the performance looks great. First curl was ~.5s. cold load test w/ 200 connections:

wrk -t10 -c200 -d10s  --timeout 10 --latency  $url
Running 10s test @ $url
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   695.33ms    1.23s    9.73s    85.82%
    Req/Sec   125.19     53.29   240.00     70.13%
  Latency Distribution
     50%  103.71ms
     75%  687.64ms
     90%    2.48s 
     99%    5.46s 
  10402 requests in 10.10s, 4.72MB read
Requests/sec:   1029.74
Transfer/sec:    478.67KB

then warm:

wrk -t10 -c200 -d10s  --timeout 10 --latency  $url
Running 10s test @ $url
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   117.38ms   93.80ms   3.65s    98.62%
    Req/Sec   159.94     38.36   280.00     76.45%
  Latency Distribution
     50%  108.62ms
     75%  124.30ms
     90%  152.60ms
     99%  228.24ms
  15464 requests in 10.06s, 7.02MB read
Requests/sec:   1537.87
Transfer/sec:    714.87KB

lambda code:

var http = require('http');
exports.handler = function(event, context) {
  http.get("http://172.31.29.158:8080/test", function(res) {
    let resp =  {
        statusCode: 200,
        body: "hello world"
    }
    context.succeed(resp);
  }).on('error', function(e) {
    context.done(null, 'FAILURE');
  });
}

the internal endpoint is just http-server with a tiny static file.
to compare I have the basic hello world lambda not in a VPC.

var http = require('http');
exports.handler = function(event, context) {
  let resp =  {
      statusCode: 200,
      body: "hello world4"
  }
  context.succeed(resp);
}

first curl is also .5s and benchmark is not that much better considering it has 1 less hop to go through to service the request:

wrk -t10 -c200 -d10s  --timeout 10 --latency  $url
Running 10s test @ $url
  10 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   381.86ms  834.26ms   5.44s    90.13%
    Req/Sec   200.44     74.42   373.00     69.00%
  Latency Distribution
     50%   67.13ms
     75%  115.36ms
     90%    1.20s 
     99%    4.26s 
  18642 requests in 10.06s, 8.46MB read
Requests/sec:   1853.30
Transfer/sec:    861.49KB
@dusty

This comment has been minimized.

Copy link

commented Apr 6, 2018

I just manually added VPC support to a test app and I just had to attach this policy to the role to make it work. AWSLambdaVPCAccessExecutionRole

When I first tried to do it, the Save button was going from a gray disabled state and returning back to orange, without any error notification. I had to inspect the ajax payload in chrome to realize it wanted to tell me I didn't have the VPC role, then went googling.

@arxpoetica

This comment has been minimized.

Copy link

commented May 3, 2018

I tried doing a manual VPC on the Lambda, which worked great, but when I upped some changes to the Lambda again it overwrote all my VPC changes. 😬😩

@arxpoetica

This comment has been minimized.

Copy link

commented May 3, 2018

@tikotzky

This comment has been minimized.

Copy link

commented May 18, 2018

Are you sure you'd need a network load balancer in order to put a lambda in a VPC?

I have a few lambda functions managed by apex currently running in a VPC. Im put them in a VPC to

  1. have their outbound traffic egress via a static IP
  2. Give the access to resources available only on the VPC
    All I had to do to set it up was edit the VPC setting on the lambda function in the AWS interface.

I am then able to continue deploying via up and the VPC configuration is left untouched...

@dusty

This comment has been minimized.

Copy link

commented May 18, 2018

Me too. Here is what I had to do.

First I went into IAM and found the Role for that particular function. I added the AWSLambdaVPCAccessExecutionRole policy to it.

Then I went into the Lambda down to the Network section, picked a VPC, then pick the subnets, and the security group I wanted.

It gave me a warning about not having outbound access, but that was ok for me because I didn't need it. My function just communicates over the VPC to my elasticsearch instance.

I have no problems re-deploying that function with up.

PS, I can still access it fine externally.

@tj

This comment has been minimized.

Copy link
Member

commented May 18, 2018

I was reading the wrong documentation, my bad :D. I'll do some more reading, mine seems to apply fine but it just times out.

EDIT: routing in my default VPC was messed up apparently hahah

@tj tj closed this in bbf4071 May 18, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.