Support Custom Lamby Events. Task Runner.

[metaskills]

Supporting migrations is the goal of this work but I decided to do something more abstract. The idea here is rather than get caught up in the semantics of how Rails v5.0 and up changes the way you could do this from code (https://stackoverflow.com/questions/7287250/run-migrations-from-rails-console) it is best to stick with the CLI interface of db:migrate and friends. So this change introduces a simple runner that someone could send a as message in the AWS Console using the following format.

{
  "lamby": {
    "runner": "pwd"
  }
}

Since app.rb is in the root of the project at /var/task you can put anything you need in the runner string. For example, a migration would be something like this.

{
  "lamby": {
    "runner": "./bin/rails db:migrate"
  }
}

By supporting this interface we can not care what the task is or supporting different migrate semantics. It just becomes normal Rails commands. For example:

{
  "lamby": {
    "runner": "./bin/rails db:migrate:down VERSION=20100905201547"
  }
}

[coding-red-panda]

What will prevent me from sending these messages:

{
   "lamby": {
       "runner": "./bin/rails db:drop"
   }
}

or

{
   "lamby": {
       "runner": "rm -rf /"
   }
}

Since you seem to be piping pretty much anything directly into the Open3 command without verifying what is going in.
Might as well just inject raw AWS CLI command and start reading out all your tokens and secrets.

Since you're using the runner which has access to Ruby, what's preventing me from sending any kind of Ruby payload, and hijacking your runners to basically do whatever I want that needs heavy calculation work and making you pay for it?

[jakub-roman]

This is one of the moments we need to evaluate usability vs. security and from my standpoint this open a lot of security concerns.

This basically adds a code to any lambda that uses Lamby (= almost all our lambdas) that enables to run anything inside that instance - I can for example open reverse shell to that lambda instance and start playing around while having access to secrets that lambda container has.

At the same time there's no auditing mechanism for this so you'd never found what I did.
This seems too dangerous to me and I think with Lambda infrastructure we should rather tighten up our security mechanism rather than loosen them.

[metaskills]

What will prevent me from sending these messages

Thanks y'all. Can you explain how you have permission to invoke your or any Lambda in this way? Please provide details?

At the same time there's no auditing mechanism for this so you'd never found what I did.

Is CloudTrail logging all AWS Console access enough? If not I can add an embedded metric here too.

[metaskills]

What will prevent me from sending these messages
"runner": "./bin/rails db:drop"

Same thing that prevents it on EC2, K8s, etc. First IAM permission to invoke the function this way. Second, rails does not let you do that command in Production envs.

[coding-red-panda]

What will prevent me from sending these messages

Thanks y'all. Can you explain how you have permission to invoke your or any Lambda in this way? Please provide details?

You're building a service that uses Lamby as infrastructure.
If this is being shared with anyone, or used by a system, it takes just a payload forwarding of your consumer to trigger whatever is needed inside the Lambda build on this change request.

I don't want to dig in our infrastructure on a public discussion, but sufficient to say, any compromised consumer of a Lambda/Service build on this puts the entire infrastructure at risk.
Can't write it any clearer than that.

[coding-red-panda]

What will prevent me from sending these messages
"runner": "./bin/rails db:drop"

Same thing that prevents it on EC2, K8s, etc. First IAM permission to invoke the function this way. Second, rails does not let you do that command in Production envs.

{
   "lamby":
      "runner": "RAILS_ENV=production ./bin/rails db:drop DISABLE_DATABASE_ENVIRONMENT_CHECK=1"
}

Bye DB

[metaskills]

RAILS_ENV=production ./bin/rails db:drop DISABLE_DATABASE_ENVIRONMENT_CHECK=1

Thanks. Correct, same is true if you give someone exec access to a container now. I'd like to focus more on your other points tho not because this one is not important but rather because it sits under some IAM event capabilities that I think need discussion first.

If this is being shared with anyone, or used by a system, it takes just a payload forwarding of your consumer to trigger whatever is needed inside the Lambda build on this change request.

Thanks, I think it is very easy to speak in secure abstract terms. Keep the ideas coming.

So, I think I hear what your saying but the terms are a bit confusing and there may be some assumptions on IAM capability. When you say "consumer" I think you mean there are situations where "Service A" is specifically given IAM capability to "Lamby Service X" to invoke it. In such, there is likely an established contract on event payloads. As such... Service A can become untrusted and change the event payload. Does that sound right?

If so, I think this is a good point for abstract functions but Lamby (Rack/HTTP) applications only have IAM invoke permissions from one source, API Gateway. It is physically impossible for API Gateway to send an invoke payload formed this way. Or are you more concerned with what could happen if Service A was setup? I'm having a hard time understanding why any Lamby (Rack/HTTP) function would be given IAM invoke permissions outside of an official AWS Resource which simply can not talk to the handler in this fashion. Maybe when I say "Service A" you are really talking about "Person A" who has IAM capabilities?

[coding-red-panda]

You're thinking too small. This is an open-source project, meaning your code is going to be used in many more systems and infrastructure setups than just ours.

You're fairly spot on with your interpretation. Even with the API Gateway, it is still possible to have request content forward to the invoked Lambda. Requires a REST API Gateway configuration that's used to accept payloads and forwards.
And yes, it can even be another service/web-app that's hooked up to communicate with your Lambda directly to invoke it through the AWS CLI or its endpoints. It all depends on how the infrastructure is configured.

Doesn't take away that this approach is extremely dangerous, as you're basically on the level of just accepting any form input, and running it through a system eval on your "server"

[metaskills]

You're thinking too small. This is an open-source project, meaning your code is going to be used in many more systems and infrastructure setups than just ours.

I know you did not mean that in a negative way but I took it as one. Literally never thinking small or myopic on any open source work I have done over many years. In fact I have already solicited feedback from other Lamby users I personally know.

Even with the API Gateway, it is still possible to have request content forward to the invoked Lambda. Requires a REST API Gateway configuration that's used to accept payloads and forwards.
... or its endpoints

This is new to me. Are you saying it is possible for API Gateway (REST or HTTP) or even an ALB with Lambda integration to be configured in such a way that the exact event payload above will be received? My assumption is that is impossible.

[coding-red-panda]

https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html#input-variable-reference

Perfectly possible to set up a Gateway with template that just takes and maps it forward.

[metaskills]

Thanks... my assumption was impossible given how Lamby uses an API Gateway (REST), like HTTP and ALB... as simple proxy for standard HTTP requests. So your point here is that if someone works hard enough with their own admin IAM capabilities they can configure API Gateway REST API as a foot gun.

In theme, if I give "Person A" the capability to migrate a DB forward, they also have capability to drop the DB... technically even tho that can be true and false at the same time. Pulling further, the capability to expose DB data via HTTP for any web framework means there are easy ways to foot gun exposing the whole database. I am pointing out there is a balance here.

I feel strongly that the nature of this feature is secure behind IAM Capabilities. In no way does exposing a HTTP service behind a proxy remotely open up this payload exploit without working hard to foot gun yourself. And even then the "pseudo console" access is still more secure than other container workloads given 1) lambda is read only and 2) you do not have an interactive REPL 3) subsequent commands are stateless and likely never reach the same target twice.

That said. I do realize the visuals of this look scary and should be considered seriously. I'll do some followup conversation in and outside the org first.

[h-m-m]

I wonder if

• Defense in depth is relevant to the security concerns here
• An allow list would help. For example:
  • A migration process that only allows x count of migrations and will fail in staging if you have more than that
  • A migration and rollback process (or more general command execution processes) that only allows reverting the last y deploys at most, or x * y migrations total

This might achieve a compromise that prevents the worst case scenario of being too permissive as well as the most common frustrations around being not permissive at all

[metaskills]

Thanks @h-m-m for the link and thoughts. Perhaps you might like this post too? It is a concept Hunter often shared with me around these types of security talks which I think fits well here too. Escaping Dark Age Cybersecurity Thinking; “Motte and Bailey” Thinking

The process you outlined was close to my first attempt but I wanted to hold back on that and gauge opinions to this approach first. Why? Early on I developed concerns that Lamby would be too coupled to Rails specific versions of things like ActiveRecord::Migrator being renamed & locations/names of bin files. Along with all the combinations of how to call those. I will admit, plucking off a few idomatic db commands would certainly be an option too. I'm going to close this PR and do wither one of two things:

1. A very focused invoke pattern to run idiomatic migrate up/down scripts.
2. A full blown ECR Docker pattern with Cloud9 mimicking Rails console and task runners.

[avinash-vllbh]

👍 to A full blown ECR Docker pattern with Cloud9 mimicking Rails console and task runners.

[metaskills]

From our good friend friend @mnapoli and Bref https://github.com/brefphp/bref with links to https://bref.sh/docs/runtimes/console.html and https://bref.sh/docs/web-apps/cron.html

[mnapoli]

Hey, since I got pinged I couldn't help but read the discussion.

Allowing to run a command via an invoke is like allowing someone to SSH into a container. I feel like reactions are a bit overblown. SSH is not a security issue. Nor is allowing to run a command.

The question is how you secure all of that.

I won't make assumptions about Lambda, so I'll talk about how it works in Bref:

1. you have to create the "task runner" Lambda function
2. you have to authorize someone to invoke the function

How to secure this thing?

Step 1: you are not obligated to create the lambda function. If you fear it's dangerous, don't create it. Just like nobody is forcing you to create a SSH access to your server.

Step 2: be careful who you authorize to invoke the function. Just like you would be careful who you authorize to SSH into your server.

If you mess around with API Gateway to allow anyone on the internet to invoke your "SSH-equivalent" Lambda function, you definitely did it on purpose.

🤷