New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Production AWS permissions #588

Closed
austinrivas opened this Issue Feb 9, 2016 · 21 comments

Comments

Projects
None yet
@austinrivas
Contributor

austinrivas commented Feb 9, 2016

In the docs you mention that the scope of IAM admin user permissions is not locked down due to the rapid dev pace of the framework. However using a super admin in production is just the kind of thing this framework was intended to solve by enforcing AWS best practices from the start.

Has there been any recent work done to nail down the scope of permissions needed to use serverless in a prod environment? Using a super user is a non starter at most shops, and a huge security vulnerable everywhere else.

@ac360 ac360 added the enhancement label Feb 10, 2016

@ac360 ac360 added this to the v0.5 milestone Feb 10, 2016

@dherault

This comment has been minimized.

Contributor

dherault commented Mar 3, 2016

👍 #726

@eahefnawy

This comment has been minimized.

Member

eahefnawy commented Mar 4, 2016

Serverless is now at a Beta status, so we haven't really made a thorough investigation on that front.

@doapp-ryanp @Nopik and @erikerikson I think you guys are working in an Enterprise environment. If you guys have time could you please share some thoughts on how you handle production permissions so we can add it to the docs? 😊

@eahefnawy eahefnawy modified the milestones: v0.6, v0.5 Mar 4, 2016

@swennemans

This comment has been minimized.

swennemans commented Mar 4, 2016

Also very interested in this. At the very least I would be interested to get a quick explanation how to change user permissions. From the documentation:

This should be done from the AWS CloudFormation UI, behind a user that has 2FA enabled and a secure password. All the Serverless tooling has a -c, --noExeCf option that will simply update your CloudFormation file, which can then be executed in the UI.

This is a lack of knowledge from my end, but I'm not sure how I would achieve this. Would I upload the created template from _meta/resources/. in the CloudFormation UI and change the policy in this UI? Specifically in this part:

image

Taken for granted that the PowerUser policy would be sufficient, I've added the according policy. However, this will result in an error: Action not allowed by stack policy.

Could someone give me quick pointer?

The role of s-resources-cf.json?

I have specific rules in my s-resources-cf.json eg:

 "Statement": [
            .....
            {
              "Effect": "Allow",
              "Action": [
                "dynamodb:Query"
              ],
              "Resource": "arn:aws:dynamodb:${region}:*:table/${project}-videos-${stage}/index/*"
            },
            {
              "Effect": "Allow",
              "Action": [
                "dynamodb:GetItem",
                "dynamodb:PutItem"
              ],
              "Resource": "arn:aws:dynamodb:${region}:*:table/${project}-users-${stage}"
            },
            .....

But from what i understand, this in combination with Administrator keys isn't a safe solution. Is this correct?

@erikerikson

This comment has been minimized.

Member

erikerikson commented Mar 4, 2016

@swennemans the requirement for Administration rights is the broad IAM permissions granted via the CFT. Once those roles are create then PowerUser is sufficient for deployment.

@erikerikson

This comment has been minimized.

Member

erikerikson commented Mar 4, 2016

@eahefnawy We handle permissions in a much finer grained and targeted manner than the default CFT. We have roles for each lambda and have very narrow lambda responsibilities (i.e. each distinct resource/method is a separate lambda).

That said, we have IAM rights broadly (nearly iam:*) that are overridden by custom privilege elevation detection and rejection code that triggers on IAM activity. The detection depends upon naming conventions that identify the creator/owner of the resource. This forces specificity of permissions to those resources we create and control but we go beyond that minimum bar to meet best practices by specifying in detail only the rights that a lambda needs to execute each of usage scenarios.

To enable this to be a not terrible PITA, we specify each role in a region/environment agnostic manner using Fn:Join in the same manner as the default CFT does so that we specify unique privileges only once. Actually, the CFT is crafted by code so that we cut duplication down even further. In the specification we use naming standards so that we can templatize the customName and customRole attributes in our s-function files for our lambdas.

I'd expect that it would be easy to generalize the specificity pattern but it wouldn't be a trivial development effort to do so and provide a management interface over top of it. Regardless, it would be desirable and useful.

@erikerikson

This comment has been minimized.

Member

erikerikson commented Mar 4, 2016

Feel free to ask for clarification if I have created more questions than answers.

@eahefnawy

This comment has been minimized.

Member

eahefnawy commented Mar 5, 2016

@erikerikson thanks a lot for the clarification. Do you think this setup is tied to some rules set up by your company in particular? Because it seems that there's lots of ways to handle production permissions and it all depends on the company policies. In that case I'm not sure what to recommend in our docs, and users will have to create their own policy for managing permission by having some balance between simplicity and security.

@erikerikson

This comment has been minimized.

Member

erikerikson commented Mar 5, 2016

@eahefnawy I'd say "yes and no", or "it depends", or something else non committal. I know that the IAM rights set up we have is not off the shelf and as such, its probably not terribly common. That said, I think the least privilege approaches to lambda rights is something we did as an important best practice as opposed to being a reaction to company policy. It could be important for Serverless to consider enabling smarter more fine grained permissions management options if only to reduce the liability for the tool of being viewed as insecure by design. If you end up deciding that it is, I'd say this isn't your highest priority. That said, given the per stage rights "bubble" that is created (and punting on external resource access management to users) I can see a reasonable argument for not enabling fine grained control from a "is it necessary/wasted resources" standpoint.

Stepping back a little, this all assumes that we're talking about the IAM roles as provided in the CFT and applied to our systems. The other IAM roles (or at least policy given rights) involved here are the deployment permission rights. I believe that the narrowing of the declaration of the necessary rights for each of the various aspects of a Serverless project's deployment to the cloud would be wonderful and fundamentally pragmatic. Again though, I'm not sure this is the highest priority concern.

To take that perspective on your original question:
I deploy under the rights I have been assigned by our central cloud management group. Those rights more expansive than some teams but realistically you can expect a lot of variance across companies. That said, if the developers using the tools are not empowered to make the IAM rights changes, then you have to allow them to hand off the smallest portion of highest privilege CFT to the team that has those rights. We had more of a subtle situation in that the ApiGateway rights are harder to control in our scheme and so we have to get another team to create our API that are thereafter allowed to modify (and thereby deploy against). Since we could add the ApiGateway ID in manually, we were okay. As a second example, the pattern might generalize and imply the flexibility that you need to enable: manual and separated processes for doing any of the things that Serverless does to accomplish a full deployment. That is, at least in theory. If a team doesn't have the rights to alter Lambdas and ApiGateways, then... maybe this isn't the tool for them.

@erikerikson

This comment has been minimized.

Member

erikerikson commented Mar 5, 2016

@eahefnawy for all my assertions of lower priority on these items, part of that is that I've been in a state of facing blocking issues and because these are not blocking in the "impossible" sense for my organization, I have deprioritized them. I can imagine a world wherein organizational constraints impinge in order to make some of these issues "blockers" by organizational decision. I imagine you know what I mean: security and privacy organizations and the like with gate passing or design review approval powers.

@flomotlik

This comment has been minimized.

Contributor

flomotlik commented May 11, 2016

As part of V1 we definitely want to make this better so you can rely on existing IAM Users and roles easily

@c-knowles

This comment has been minimized.

c-knowles commented May 12, 2016

Not sure if it helps but I started to investigate the permissions required with serverless v0.5.5. So far I have the below which covers the basic function and endpoint deployments. I didn't restrict the account ID or region since in my case it didn't matter so much. I'm using the ${project} variable so it has to be in a CF template read by serverless.

"PolicyDocument": {
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "cloudformation:Describe*",
      "cloudformation:List*",
      "cloudformation:Get*",
      "cloudformation:PreviewStackUpdate"
    ],
    "Resource": "*"
  }, {
    "Effect": "Allow",
    "Action": [
      "cloudformation:CreateStack",
      "cloudformation:UpdateStack",
      "cloudformation:DeleteStack"
    ],
    "Resource": "arn:aws:cloudformation:*:*:stack/${project}*"
  }, {
    "Effect": "Allow",
    "Action": [
      "lambda:Get*",
      "lambda:List*",
      "lambda:CreateFunction"
    ],
    "Resource": "*"
  }, {
    "Effect": "Allow",
    "Action": [
      "lambda:AddPermission",
      "lambda:CreateAlias",
      "lambda:DeleteFunction",
      "lambda:InvokeFunction",
      "lambda:PublishVersion",
      "lambda:RemovePermission",
      "lambda:Update*"
    ],
    "Resource": "arn:aws:lambda:*:*:function:${project}*"
  }, {
    "Effect": "Allow",
    "Action": [
      "iam:PassRole"
    ],
    "Resource": "arn:aws:iam::*:role/${project}*"
  }, {
    "Effect": "Allow",
    "Action": [
      "apigateway:GET"
    ],
    "Resource": "arn:aws:apigateway:*::/restapis"
  }, {
    "Effect": "Allow",
    "Action": [
      "apigateway:*"
    ],
    "Resource": "arn:aws:apigateway:*::/restapis/GATEWAY_ID/*"
  }]
}

I haven't looked at all the different event kinds yet as I don't use a lot of those. The API Gateway permissions require the specific gateway ID used (looks something like 1f4mhc9s5c) but that's something serverless could grab and generate the CF template for setting up these permissions. I guess the biggest sticking point here is that this is a bit of catch 22 since the ID won't be generated until the deployment happens but the deployment can't happen without the permissions so perhaps an initial version need to be generated or a specific command would be needed to initialise the API Gateway.

@flomotlik flomotlik modified the milestones: 1.0, v0.6 May 12, 2016

@c-knowles

This comment has been minimized.

c-knowles commented May 17, 2016

I've done a few fixes to the above on an actual production deploy with restricted permissions. It was made a bit more difficult than expected because some errors are caught and swallowed by the existing code which means the deployment produces an error elsewhere and I had to work back from that to determine what went wrong.

One example is that if lambda:removePermission is not allowed, the error is swallowed here which in turn means the deployment fails later on addPermission. I'm sure there may have been a reason to swallow these errors but maybe it would be good to expose them more across the code to make it easier to lock down the permissions.

@DanielRuskin1

This comment has been minimized.

DanielRuskin1 commented May 30, 2016

@c-knowles Thanks for posting. I modified your policy on my deployment to be a bit more restrictive, and also support scheduled events:
https://gist.github.com/DanielRuskin1/cdddfa951bdb68fc09cea59cdd909935

One question that I had with your version: I'm not 100% familiar with Cloudformation, but I noticed that it has the ability to create IAM roles. Could giving Serverless access to create roles (via Cloudformation), and add them to Lambda functions (via the iam:PassRole permission), result in a priv escalation?

@c-knowles

This comment has been minimized.

c-knowles commented May 31, 2016

Maybe I missed something, there is an IAM role creation in serverless projects by default but not in the policy above. Do you mean the iam:PassRole part? If so, that part does not create a role but allows the IAM user/group associated this policy to give other processes the roles matching the resource expression. The resource I set is "arn:aws:iam::*:role/${project}*" which matches the role created by default by serverless inside s-resources-cf.json. In other words, this deployment IAM policy allows the deployment process to grant the Lambda runtime the appropriate project specific IAM Role specified as IamRoleLambda in a default serverless v0.5.5 project. I think that's restrictive enough assuming no name clashes with other projects in your account.

Privilege escalation really depends on what is granted to that IamRoleLambda role which serverless needs to pass to Lambda to function correctly. By default the role just has the permissions related to creation of logs although I believe it's not as restrictive as it could be by default (it does not really restrict the ARN that much). Some people add extra permissions in there such as access to DynamoDB/S3.

If you need extra security around production, then in turn this depends on where you are putting keys for stack creation/deploy and who has access to those.

Some more info on PassRole here and here if you want more familiarity with it, these pages are not specific to Lambda but I believe the IAM capabilities are the same.

@DanielRuskin1

This comment has been minimized.

DanielRuskin1 commented May 31, 2016

Ah I just meant these two policy statements:

 {
    "Effect": "Allow",
    "Action": [
      "cloudformation:CreateStack",
      "cloudformation:UpdateStack",
      "cloudformation:DeleteStack"
    ],
    "Resource": "arn:aws:cloudformation:*:*:stack/${project}*"
}
{
    "Effect": "Allow",
    "Action": [
      "iam:PassRole"
    ],
    "Resource": "arn:aws:iam::*:role/${project}*"
}

It seems like CloudFormation has the ability to create IAM roles. Does that mean Serverless can create an IAM role via CloudFormation, then pass it to a Lambda? If so - does that result in a potential priv escalation (create role with expanded permissions -> pass it to a lambda with some malicious code -> access to expanded permissions)?

@c-knowles

This comment has been minimized.

c-knowles commented May 31, 2016

Ah sure, got you. In my case that vector is less attackable as the automated pipeline does not have access to our production system at all. If you want to add that, I would take a look at ResourceTypes as those allow restrictions on the types of resources that can be created via the CF calls, in this case we'd be adding a "not IAM" to the cloudformation actions. I may look into adding that to ours as it's a good vector to tighten regardless

@DanielRuskin1

This comment has been minimized.

DanielRuskin1 commented May 31, 2016

Cool, makes sense. I ended up just removing the CF permission completely for my deployment, I'll probably just make those updates manually.

@flomotlik

This comment has been minimized.

Contributor

flomotlik commented Aug 19, 2016

Duplicate of #1439 so I'm closing here but we'll keep the discussion going there.

@flomotlik flomotlik closed this Aug 19, 2016

@berry2012

This comment has been minimized.

berry2012 commented Jan 26, 2018

create a serverless.json policy with the code below and attach to user or role that runs serverless project in your environment.

    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateUploadBucket",
                "cloudformation:CancelUpdateStack",
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack",
                "cloudformation:UpdateStack",
                "cloudformation:UpdateTerminationProtection",
                "cloudformation:CreateChangeSet",
                "cloudformation:Describe*",
                "cloudformation:ContinueUpdateRollback",
                "cloudformation:EstimateTemplateCost",
                "cloudformation:PreviewStackUpdate",
                "cloudformation:List*",
                "cloudformation:ExecuteChangeSet",
                "cloudformation:ValidateTemplate",
                "cloudformation:Get*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "lambda:AddPermission",
                "lambda:InvokeFunction",
                "lambda:DeleteFunction",
                "lambda:PublishVersion",
                "lambda:List*",
                "lambda:CreateFunction",
                "lambda:Get*",
                "lambda:RemovePermission",
                "lambda:CreateAlias",
                "lambda:Update*",
                "apigateway:GET"
            ],
            "Resource": [
                "arn:aws:lambda:*:*:function:${project}*",
                "arn:aws:apigateway:*::/restapis"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "apigateway:*",
            "Resource": "arn:aws:apigateway:*::/restapis/GATEWAY_ID/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:Describe*",
                "ec2:createTags",
                "ec2:CreateInternetGateway",
                "ec2:DeleteInternetGateway",
                "ec2:CreateVpc",
                "ec2:DeleteVpc",
                "ec2:ModifyVpcAttribute",
                "iam:CreateRole",
                "iam:DeleteRole",
                "ec2:DeleteInternetGateway",
                "dynamodb:CreateTable",
                "dynamodb:DescribeTable",
                "dynamodb:DeleteTable",
                "kinesis:CreateStream",
                "kinesis:DescribeStream",
                "kinesis:DeleteStream",
                "ec2:CreateNetworkAcl",
                "ec2:DeleteNetworkAcl",
                "ec2:CreateRouteTable",
                "ec2:DeleteRouteTable",
                "ec2:AttachInternetGateway",
                "ec2:DetachInternetGateway",
                "ec2:CreateSubnet",
                "ec2:DeleteSubnet",
                "ec2:CreateSecurityGroup",
                "ec2:DeleteSecurityGroup",
                "ec2:CreateNetworkAclEntry",
                "ec2:DeleteNetworkAclEntry",
                "iam:PutRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:GetRole",
                "states:CreateStateMachine",
                "states:DeleteStateMachine"
            ],
            "Resource": "*"
        }
    ]
}```
@tommedema

This comment has been minimized.

tommedema commented Aug 10, 2018

@berry2012 this gives me:

An error occurred: ServerlessDeploymentBucket - API: s3:CreateBucket Access Denied.
@erikerikson

This comment has been minimized.

Member

erikerikson commented Aug 10, 2018

@tommedema - most users in a more restrictive environment pre-create their bucket and specify it using the deploymentBucket parameter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment