Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loopback Server and AWS Lambda hard limit #2

Closed
PerfectedApp opened this issue Jan 2, 2019 · 3 comments
Closed

Loopback Server and AWS Lambda hard limit #2

PerfectedApp opened this issue Jan 2, 2019 · 3 comments
Assignees
Labels
question Further information is requested

Comments

@PerfectedApp
Copy link

Lovely work, botbits!

Just curious how one may handle a loopback server that is more than 250 megs due to dependencies and etc. AWS Lambda has a hard limit of 250 megs.

@marcelobern
Copy link
Contributor

Happy New Year @PerfectedTech !

Supposedly the AWS Lambda storage limit can be increased, as referenced in the corresponding documentation. That said, I never asked for a storage limit increase so I cannot confirm it is possible or how high it might go.

If I had to tackle this issue, here are the steps I would experiment with (in no particular order):

  • Use webpack or similar approach to minify the project run-time dependencies.
  • Break down the loopback server into smaller chunks (i.e. multiple Lambda functions) so that each Lambda function fits the storage limit. In this case, it might be worth to stick common dependencies as Lambda layers.
  • Use the AWS Support Center console to request a Lambda storage increase.

Please let me know in case you run into issues with the process above and I would be willing to help (time permitting).

In particular I have been considering adding the Lambda layers example to this repo but did not want to make it overly complicated for those uninitiated in AWS Lambda.

So if it helps you I could look into adding a layer sample to this repo or creating a new repo for the layer sample (and maybe even use the dependencies for your current project if they are public).

@marcelobern marcelobern added the question Further information is requested label Jan 2, 2019
@marcelobern marcelobern self-assigned this Jan 2, 2019
@PerfectedApp
Copy link
Author

Hi @marcelobern,

Thank you very much for your detailed reply.

I reached out to AWS asking them to increase the limit, but it turns out that the application unzipped cannot be more than 250MB. They suggested breaking the application into smaller parts and invoking lambda from one another. Not sure that would be the best practice with the structure that Loopback provides.

If I come by any other approach to the problem, I will certainly update here.

All the best and happy new year,

Alexander

@marcelobern marcelobern pinned this issue Jan 6, 2019
@marcelobern
Copy link
Contributor

@PerfectedTech thanks for the update!

Breaking the application definitely makes sense to me. The one word of advise is to make sure you are using a cost effective way to do that, so your Lambda/API Gateway/etc. costs do not end up being higher than running a loopback server straight out of EC2 (or the like).

Here are some thoughts that come to mind related to loopback, Lambda, and serverless in the context of solving this issue:

  • Break down large logical chunks that can be run as different groups of API invocations and share the same dependencies (staying under the stated 250MB limit) as different Lambda functions. This means having multiple serverless functions and should be pretty straightforward to implement. The challenge here will be to manage the shared dependencies so serverless will avoid packaging unnecessary ones. I used serverless package and artifact to accomplish this along with gulp to control the dependencies I wanted packaged for each function. I tried using exclude/include but it did not work very well for me.
  • For specific workflows, I used SNS, DynamoDB, etc. so one Lambda function triggers another. Let us say I have a table update and something else that needs to happen after that. You can then break down the code so the before and after pieces of the workflow happen in different Lambda functions. This might get very weird when using loopback as a loopback app may be bundling the code for before/after in the same files (and the dependencies might be the same, so no space saved there).
  • Another way to tackle this is to invoke a Lambda function using the AWS-SDK from your Lambda function (note that aws-sdk does not need to be added to your uploaded dependencies as Amazon makes it available inside Lambda). I have not used this approach as the synchronous alternative means double Lambda charges and the asynchronous one comes across as increasing the troubleshooting complexity, so I would use this as a last resort.
  • While researching on Lambda functions triggering other Lambda functions I came across this article. While I do not believe this makes much sense in a loopback app, as I do not know your specific use cases/architecture, I figured it was worth sharing it ;-)

I will close the issue and pin it so others with a similar issue can easily benefit as you post updates ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants