Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New options to configure roles and VPC #11779

Merged
merged 6 commits into from Apr 23, 2019
Merged

New options to configure roles and VPC #11779

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4dd8fcb
add more granular options for permissions
kvch Apr 12, 2019
12703e6
fix invalid account id in cloudwatch unit test
kvch Apr 12, 2019
c7a12fe
add changelog entry
kvch Apr 12, 2019
fe79fc4
add source of pattern
kvch Apr 12, 2019
160ee53
log if custom role is used && minor refactoring
kvch Apr 12, 2019
14991ec
lowercase policy name to stay consistent with the docs
kvch Apr 12, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
24 changes: 24 additions & 0 deletions x-pack/functionbeat/_meta/beat.reference.yml
View file
Open in desktop
Expand Up @@ -34,6 +34,14 @@ functionbeat.provider.aws.functions:
# There is a hard limit of 3008MiB for each function. Default is 128MiB.
#memory_size: 128MiB

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

Expand Down Expand Up @@ -71,6 +79,14 @@ functionbeat.provider.aws.functions:
# There is a hard limit of 3008MiB for each function. Default is 128MiB.
#memory_size: 128MiB

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

Expand Down Expand Up @@ -113,6 +129,14 @@ functionbeat.provider.aws.functions:
# There is a hard limit of 3008MiB for each function. Default is 128MiB.
#memory_size: 128MiB

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

Expand Down
24 changes: 24 additions & 0 deletions x-pack/functionbeat/_meta/beat.yml
View file
Open in desktop
Expand Up @@ -38,6 +38,14 @@ functionbeat.provider.aws.functions:
# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Optional fields that you can specify to add additional information to the
# output. Fields can be scalar values, arrays, dictionaries, or any nested
# combination of these.
Expand Down Expand Up @@ -75,6 +83,14 @@ functionbeat.provider.aws.functions:
# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Optional fields that you can specify to add additional information to the
# output. Fields can be scalar values, arrays, dictionaries, or any nested
# combination of these.
Expand Down Expand Up @@ -117,6 +133,14 @@ functionbeat.provider.aws.functions:
# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Optional fields that you can specify to add additional information to the
# output. Fields can be scalar values, arrays, dictionaries, or any nested
# combination of these.
Expand Down
24 changes: 24 additions & 0 deletions x-pack/functionbeat/functionbeat.reference.yml
View file
Open in desktop
Expand Up @@ -34,6 +34,14 @@ functionbeat.provider.aws.functions:
# There is a hard limit of 3008MiB for each function. Default is 128MiB.
#memory_size: 128MiB

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

Expand Down Expand Up @@ -71,6 +79,14 @@ functionbeat.provider.aws.functions:
# There is a hard limit of 3008MiB for each function. Default is 128MiB.
#memory_size: 128MiB

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

Expand Down Expand Up @@ -113,6 +129,14 @@ functionbeat.provider.aws.functions:
# There is a hard limit of 3008MiB for each function. Default is 128MiB.
#memory_size: 128MiB

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

Expand Down
24 changes: 24 additions & 0 deletions x-pack/functionbeat/functionbeat.yml
View file
Open in desktop
Expand Up @@ -38,6 +38,14 @@ functionbeat.provider.aws.functions:
# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Optional fields that you can specify to add additional information to the
# output. Fields can be scalar values, arrays, dictionaries, or any nested
# combination of these.
Expand Down Expand Up @@ -75,6 +83,14 @@ functionbeat.provider.aws.functions:
# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Optional fields that you can specify to add additional information to the
# output. Fields can be scalar values, arrays, dictionaries, or any nested
# combination of these.
Expand Down Expand Up @@ -117,6 +133,14 @@ functionbeat.provider.aws.functions:
# Dead letter queue configuration, this must be set to an ARN pointing to a SQS queue.
#dead_letter_config.target_arn:

# Execution role of the function.
#role: arn:aws:iam::123456789012:role/MyFunction

# Connect to private resources in an Amazon VPC.
#virtual_private_cloud:
# security_group_ids: []
# subnet_ids: []

# Optional fields that you can specify to add additional information to the
# output. Fields can be scalar values, arrays, dictionaries, or any nested
# combination of these.
Expand Down
60 changes: 39 additions & 21 deletions x-pack/functionbeat/provider/aws/cli_manager.go
View file
Open in desktop
Expand Up @@ -106,29 +106,37 @@ func (c *CLIManager) template(function installer, name, codeLoc string) *cloudfo
// Merge any specific policies from the service.
policies = append(policies, function.Policies()...)

// Create the roles for the lambda.
template := cloudformation.NewTemplate()
// doc: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html
template.Resources[prefix("")+"IAMRoleLambdaExecution"] = &cloudformation.AWSIAMRole{
AssumeRolePolicyDocument: map[string]interface{}{
"Statement": []interface{}{
map[string]interface{}{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": map[string]interface{}{
"Service": cloudformation.Join("", []string{
"lambda.",
cloudformation.Ref("AWS::URLSuffix"),
}),

role := lambdaConfig.Role
dependsOn := make([]string, 0)
if lambdaConfig.Role == "" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should log that we are using a custom role, I presume that we will get a few questions concerning the policies required if they use their role. So I think we will need to create a followup issue doc issue to describe them. Can you coordinate with @dedemorton for that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created issue with the required policies: #11787

// Create the roles for the lambda.
// doc: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html
template.Resources[prefix("")+"IAMRoleLambdaExecution"] = &cloudformation.AWSIAMRole{
AssumeRolePolicyDocument: map[string]interface{}{
"Statement": []interface{}{
map[string]interface{}{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": map[string]interface{}{
"Service": cloudformation.Join("", []string{
"lambda.",
cloudformation.Ref("AWS::URLSuffix"),
}),
},
},
},
},
},
Path: "/",
RoleName: "functionbeat-lambda-" + name,
// Allow the lambda to write log to cloudwatch logs.
// doc: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-policy.html
Policies: policies,
Path: "/",
RoleName: "functionbeat-lambda-" + name,
// Allow the lambda to write log to cloudwatch logs.
// doc: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-policy.html
Policies: policies,
}

role = cloudformation.GetAtt(prefix("")+"IAMRoleLambdaExecution", "Arn")
dependsOn = []string{prefix("") + "IAMRoleLambdaExecution"}
}

// Configure the Dead letter, any failed events will be send to the configured amazon resource name.
Expand All @@ -139,6 +147,15 @@ func (c *CLIManager) template(function installer, name, codeLoc string) *cloudfo
}
}

// Configure VPC
var vcpConf *cloudformation.AWSLambdaFunction_VpcConfig
if lambdaConfig.VPCConfig != nil && len(lambdaConfig.VPCConfig.SecurityGroupIDs) != 0 && len(lambdaConfig.VPCConfig.SubnetIDs) != 0 {
vcpConf = &cloudformation.AWSLambdaFunction_VpcConfig{
SecurityGroupIds: lambdaConfig.VPCConfig.SecurityGroupIDs,
SubnetIds: lambdaConfig.VPCConfig.SubnetIDs,
}
}

// Create the lambda
// Doc: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-function.html
template.Resources[prefix("")] = &AWSLambdaFunction{
Expand All @@ -156,15 +173,16 @@ func (c *CLIManager) template(function installer, name, codeLoc string) *cloudfo
},
},
DeadLetterConfig: dlc,
VpcConfig: vcpConf,
FunctionName: name,
Role: cloudformation.GetAtt(prefix("")+"IAMRoleLambdaExecution", "Arn"),
Role: role,
Runtime: runtime,
Handler: handlerName,
MemorySize: lambdaConfig.MemorySize.Megabytes(),
ReservedConcurrentExecutions: lambdaConfig.Concurrency,
Timeout: int(lambdaConfig.Timeout.Seconds()),
},
DependsOn: []string{prefix("") + "IAMRoleLambdaExecution"},
DependsOn: dependsOn,
}

// Create the log group for the specific function lambda.
Expand Down
2 changes: 1 addition & 1 deletion x-pack/functionbeat/provider/aws/cloudwatch_logs_test.go
View file
Open in desktop
Expand Up @@ -49,7 +49,7 @@ func TestCloudwatchLogs(t *testing.T) {
cfg := common.MustNewConfigFrom(map[string]interface{}{
"name": "foobar",
"description": "my long description",
"role": "arn:aws:iam::00000000:role/functionbeat",
"role": "arn:aws:iam::000000000000:role/functionbeat",
"triggers": []map[string]interface{}{
map[string]interface{}{
"log_group_name": "foo",
Expand Down
27 changes: 22 additions & 5 deletions x-pack/functionbeat/provider/aws/config.go
View file
Open in desktop
Expand Up @@ -6,6 +6,7 @@ package aws

import (
"fmt"
"regexp"
"time"
"unicode"

Expand All @@ -23,18 +24,25 @@ type Config struct {
const maxMegabytes = 3008

// DefaultLambdaConfig confguration for AWS lambda function.
var DefaultLambdaConfig = &lambdaConfig{
MemorySize: 128 * 1024 * 1024,
Timeout: time.Second * 3,
Concurrency: 5,
}
var (
DefaultLambdaConfig = &lambdaConfig{
MemorySize: 128 * 1024 * 1024,
Timeout: time.Second * 3,
Concurrency: 5,
}

arnRolePattern = "arn:(aws[a-zA-Z-]*)?:iam::\\d{12}:role/?[a-zA-Z_0-9+=,.@\\-_/]+"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using the arn reference in a few places in the code, we might want to create a custom type for it and have a single place to validate it.

As a note, I've decided not to add a validation for arn, since I was a bit worried that we would get it wrong. Can you add a link concerning the rules for an ARN.

roleRE = regexp.MustCompile(arnRolePattern)
)

type lambdaConfig struct {
Concurrency int `config:"concurrency" validate:"min=0,max=1000"`
DeadLetterConfig *deadLetterConfig `config:"dead_letter_config"`
Description string `config:"description"`
MemorySize MemSizeFactor64 `config:"memory_size"`
Timeout time.Duration `config:"timeout" validate:"nonzero,positive"`
Role string `config:"role"`
VPCConfig *vpcConfig `config:"virtual_private_cloud"`
}

func (c *lambdaConfig) Validate() error {
Expand All @@ -46,13 +54,22 @@ func (c *lambdaConfig) Validate() error {
return fmt.Errorf("'memory_size' must be lower than %d", maxMegabytes)
}

if c.Role != "" && !roleRE.MatchString(c.Role) {
return fmt.Errorf("invalid role: '%s', name must match pattern %s", c.Role, arnRolePattern)
}

return nil
}

type deadLetterConfig struct {
TargetArn string `config:"target_arn"`
}

type vpcConfig struct {
SecurityGroupIDs []string `config:"security_group_ids" validate:"required"`
SubnetIDs []string `config:"subnet_ids" validate:"required"`
}

// MemSizeFactor64 implements a human understandable format for bytes but also make sure that all
// values used are a factory of 64.
type MemSizeFactor64 int
Expand Down