S3 Server-Side Encryption #784

austinbyers · 2018-07-06T21:15:19Z

to: @jacknagz
cc: @airbnb/streamalert-maintainers
size: medium
part of #758

Background

S3 Server-Side Encryption adds a layer of protection to data in S3 buckets. If encrypted with KMS, additional permissions to use the KMS key are required for access (in addition to access to S3).

Changes

Create a new KMS key for server-side encryption of most StreamAlert S3 buckets: athena results, streamalert.data, streamalert.secrets, streamalert.terraform.state, streamalerts
- This means data in streamalert.secrets is actually encrypted twice (client-side and server-side). We will eventually move these into Secrets Manager and get rid of this S3 bucket completely
Create a new KMS key for each configured CloudTrail
The data/alerts firehoses are configured to use the KMS key corresponding to their target S3 buckets
After @ryandeivert upgraded moto, some of the Lambda mocking has taken longer or started throwing exceptions. For these, I just used basic mocks in place of moto.

Testing

Deploy from scratch from this branch with cloudtrail and a firehose enabled. Alerts landed in S3, encrypted, and were searchable from Athena. The firehose data also landed in the data bucket (encrypted).
Deploy from master, then apply these changes:

Terraform will perform the following actions:

  + aws_kms_alias.server_side_encryption
      id:                                                                                                        <computed>
      arn:                                                                                                       <computed>
      name:                                                                                                      "alias/encryptallthings_server-side-encryption"
      target_key_arn:                                                                                            <computed>
      target_key_id:                                                                                             "${aws_kms_key.server_side_encryption.key_id}"

  + aws_kms_key.server_side_encryption
      id:                                                                                                        <computed>
      arn:                                                                                                       <computed>
      description:                                                                                               "StreamAlert S3 Server-Side Encryption"
      enable_key_rotation:                                                                                       "true"
      is_enabled:                                                                                                "true"
      key_id:                                                                                                    <computed>
      key_usage:                                                                                                 <computed>
      policy:                                                                                                    "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Action\": \"kms:*\", \"Principal\": {\"AWS\": \"arn:aws:iam::ACCOUNTID:root\"}, \"Resource\": \"*\", \"Effect\": \"Allow\", \"Sid\": \"Enable IAM User Permissions\"}, {\"Resource\": \"*\", \"Effect\": \"Allow\", \"Sid\": \"Allow principals in the account to use the key\", \"Action\": [\"kms:Decrypt\", \"kms:GenerateDataKey*\", \"kms:Encrypt\"], \"Condition\": {\"StringEquals\": {\"kms:CallerAccount\": \"ACCOUNTID\"}}, \"Principal\": \"*\"}]}"

  ~ aws_s3_bucket.logging_bucket
      server_side_encryption_configuration.#:                                                                    "0" => "1"
      server_side_encryption_configuration.0.rule.#:                                                             "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.#:                   "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.sse_algorithm:     "" => "AES256"

  ~ aws_s3_bucket.stream_alert_secrets
      server_side_encryption_configuration.#:                                                                    "0" => "1"
      server_side_encryption_configuration.0.rule.#:                                                             "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.#:                   "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.kms_master_key_id: "" => "${aws_kms_key.server_side_encryption.key_id}"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.sse_algorithm:     "" => "aws:kms"

  ~ aws_s3_bucket.streamalerts
      server_side_encryption_configuration.#:                                                                    "0" => "1"
      server_side_encryption_configuration.0.rule.#:                                                             "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.#:                   "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.kms_master_key_id: "" => "${aws_kms_key.server_side_encryption.key_id}"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.sse_algorithm:     "" => "aws:kms"

  ~ aws_s3_bucket.terraform_remote_state
      server_side_encryption_configuration.#:                                                                    "0" => "1"
      server_side_encryption_configuration.0.rule.#:                                                             "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.#:                   "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.kms_master_key_id: "" => "${aws_kms_key.server_side_encryption.key_id}"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.sse_algorithm:     "" => "aws:kms"

 <= module.alert_processor_iam.data.aws_iam_policy_document.output_secrets
      id:                                                                                                        <computed>
      json:                                                                                                      <computed>
      statement.#:                                                                                               "2"
      statement.0.actions.#:                                                                                     "2"
      statement.0.actions.1493550787:                                                                            "kms:DescribeKey"
      statement.0.actions.3733000735:                                                                            "kms:Decrypt"
      statement.0.effect:                                                                                        "Allow"
      statement.0.resources.#:                                                                                   <computed>
      statement.1.actions.#:                                                                                     "1"
      statement.1.actions.2071725391:                                                                            "s3:GetObject"
      statement.1.effect:                                                                                        "Allow"
      statement.1.resources.#:                                                                                   "1"
      statement.1.resources.1019851817:                                                                          "arn:aws:s3:::encryptallthings.streamalert.secrets/*"

  ~ module.alert_processor_iam.aws_iam_role_policy.output_secrets
      policy:                                                                                                    "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"kms:DescribeKey\",\n        \"kms:Decrypt\"\n      ],\n      \"Resource\": \"arn:aws:kms:us-east-1:121671312944:key/UUID\"\n    },\n    {\n      \"Sid\": \"\",\n      \"Effect\": \"Allow\",\n      \"Action\": \"s3:GetObject\",\n      \"Resource\": \"arn:aws:s3:::encryptallthings.streamalert.secrets/*\"\n    }\n  ]\n}" => "${data.aws_iam_policy_document.output_secrets.json}"

  ~ module.cloudtrail_prod.aws_cloudtrail.streamalert_no_cloudwatch
      kms_key_id:                                                                                                "" => "${aws_kms_key.cloudtrail_encryption.arn}"

  + module.cloudtrail_prod.aws_kms_alias.cloudtrail_encryption
      id:                                                                                                        <computed>
      arn:                                                                                                       <computed>
      name:                                                                                                      "alias/encryptallthings-prod-streamalert-cloudtrail"
      target_key_arn:                                                                                            <computed>
      target_key_id:                                                                                             "${aws_kms_key.cloudtrail_encryption.key_id}"

  + module.cloudtrail_prod.aws_kms_key.cloudtrail_encryption
      id:                                                                                                        <computed>
      arn:                                                                                                       <computed>
      description:                                                                                               "Encrypt Cloudtrail logs for encryptallthings.prod.streamalert.cloudtrail"
      enable_key_rotation:                                                                                       "true"
      is_enabled:                                                                                                "true"
      key_id:                                                                                                    <computed>
      key_usage:                                                                                                 <computed>
      policy:                                                                                                    "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"Enable IAM User Permissions\",\n      \"Effect\": \"Allow\",\n      \"Action\": \"kms:*\",\n      \"Resource\": \"*\",\n      \"Principal\": {\n        \"AWS\": \"arn:aws:iam::ACCOUNTID:root\"\n      }\n    },\n    {\n      \"Sid\": \"Allow CloudTrail to encrypt logs\",\n      \"Effect\": \"Allow\",\n      \"Action\": \"kms:GenerateDataKey*\",\n      \"Resource\": \"*\",\n      \"Principal\": {\n        \"Service\": \"cloudtrail.amazonaws.com\"\n      },\n      \"Condition\": {\n        \"StringLike\": {\n          \"kms:EncryptionContext:aws:cloudtrail:arn\": \"arn:aws:cloudtrail:*:ACCOUNTID:trail/*\"\n        }\n      }\n    },\n    {\n      \"Sid\": \"Allow CloudTrail to describe key\",\n      \"Effect\": \"Allow\",\n      \"Action\": \"kms:DescribeKey\",\n      \"Resource\": \"*\",\n      \"Principal\": {\n        \"Service\": \"cloudtrail.amazonaws.com\"\n      }\n    },\n    {\n      \"Sid\": \"Allow principals in the account to decrypt log files\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"kms:ReEncryptFrom\",\n        \"kms:Decrypt\"\n      ],\n      \"Resource\": \"*\",\n      \"Principal\": \"*\",\n      \"Condition\": {\n        \"StringEquals\": {\n          \"kms:CallerAccount\": \"ACCOUNTID\"\n        },\n        \"StringLike\": {\n          \"kms:EncryptionContext:aws:cloudtrail:arn\": \"arn:aws:cloudtrail:*:ACCOUNTID:trail/*\"\n        }\n      }\n    }\n  ]\n}"

  ~ module.cloudtrail_prod.aws_s3_bucket.cloudtrail_bucket
      server_side_encryption_configuration.#:                                                                    "0" => "1"
      server_side_encryption_configuration.0.rule.#:                                                             "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.#:                   "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.kms_master_key_id: "" => "${aws_kms_key.cloudtrail_encryption.key_id}"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.sse_algorithm:     "" => "aws:kms"

  ~ module.kinesis_firehose_binaryalert.aws_kinesis_firehose_delivery_stream.stream_alert_data
      s3_configuration.0.kms_key_arn:                                                                            "" => "${var.kms_key_arn}"

 <= module.kinesis_firehose_setup.data.aws_iam_policy_document.firehose_s3
      id:                                                                                                        <computed>
      json:                                                                                                      <computed>
      statement.#:                                                                                               "2"
      statement.0.actions.#:                                                                                     "6"
      statement.0.actions.2071725391:                                                                            "s3:GetObject"
      statement.0.actions.315547055:                                                                             "s3:PutObject"
      statement.0.actions.329996733:                                                                             "s3:AbortMultipartUpload"
      statement.0.actions.3948445924:                                                                            "s3:GetBucketLocation"
      statement.0.actions.486976917:                                                                             "s3:ListBucket"
      statement.0.actions.548422:                                                                                "s3:ListBucketMultipartUploads"
      statement.0.effect:                                                                                        "Allow"
      statement.0.resources.#:                                                                                   "2"
      statement.0.resources.1001773326:                                                                          "arn:aws:s3:::encryptallthings.streamalert.data/*"
      statement.0.resources.1026980626:                                                                          "arn:aws:s3:::encryptallthings.streamalert.data"
      statement.1.actions.#:                                                                                     "3"
      statement.1.actions.271337083:                                                                             "kms:Encrypt"
      statement.1.actions.3733000735:                                                                            "kms:Decrypt"
      statement.1.actions.886236925:                                                                             "kms:GenerateDataKey*"
      statement.1.effect:                                                                                        "Allow"
      statement.1.resources.#:                                                                                   <computed>

  ~ module.kinesis_firehose_setup.aws_iam_role_policy.stream_alert_firehose_s3
      policy:                                                                                                    "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"s3:PutObject\",\n        \"s3:ListBucketMultipartUploads\",\n        \"s3:ListBucket\",\n        \"s3:GetObject\",\n        \"s3:GetBucketLocation\",\n        \"s3:AbortMultipartUpload\"\n      ],\n      \"Resource\": [\n        \"arn:aws:s3:::encryptallthings.streamalert.data/*\",\n        \"arn:aws:s3:::encryptallthings.streamalert.data\"\n      ]\n    }\n  ]\n}" => "${data.aws_iam_policy_document.firehose_s3.json}"

  ~ module.kinesis_firehose_setup.aws_s3_bucket.stream_alert_data
      server_side_encryption_configuration.#:                                                                    "0" => "1"
      server_side_encryption_configuration.0.rule.#:                                                             "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.#:                   "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.kms_master_key_id: "" => "${var.kms_key_id}"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.sse_algorithm:     "" => "aws:kms"

  ~ module.stream_alert_athena.aws_s3_bucket.athena_results_bucket
      server_side_encryption_configuration.#:                                                                    "0" => "1"
      server_side_encryption_configuration.0.rule.#:                                                             "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.#:                   "0" => "1"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.kms_master_key_id: "" => "${var.kms_key_id}"
      server_side_encryption_configuration.0.rule.0.apply_server_side_encryption_by_default.0.sse_algorithm:     "" => "aws:kms"

  ~ module.globals.module.alerts_firehose.aws_kinesis_firehose_delivery_stream.stream_alerts
      s3_configuration.0.kms_key_arn:                                                                            "" => "${var.kms_key_arn}"


Plan: 4 to add, 12 to change, 0 to destroy.

Athena queries worked the same (with no observable performance impact) with none, some, or all objects KMS encrypted.

jacknagz · 2018-07-06T21:29:32Z

terraform/modules/tf_stream_alert_cloudtrail/main.tf

@@ -1,3 +1,87 @@
+// KMS key for encrypting CloudTrail logs


Is there a reference in AWS docs for the policies below? If so could you add it? 🙇

Sadly not (that I'm aware of). I got the policy by manually creating a CloudTrail, encrypting it from the console, and seeing what policy AWS automatically attached to the key. I'll add a comment to that effect

jacknagz · 2018-07-06T21:30:07Z

terraform/modules/tf_stream_alert_cloudtrail/main.tf

+
+data "aws_iam_policy_document" "cloudtrail_encryption" {
+  statement {
+    sid = "Enable IAM User Permissions"


Isn't this policy to allow root to manage the key?

Sort of - it's a required policy that enables any IAM user in the account (with the appropriate permissions) to manage the key - https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html#key-policy-default-allow-root-enable-iam

So if this permission is granted to "root", I as an IAM user in the account am allowed to modify the key, provided that I have kms:* permissions on my IAM user

jacknagz

Two comments, LGTM otherwise!

coveralls · 2018-07-06T21:44:43Z

Coverage remained the same at 97.385% when pulling 16924fe on austin-s3-encryption into ea5ebe3 on master.

Austin Byers added 5 commits July 6, 2018 01:27

Encrypt CloudTrail

fd3768b

Use key for generated buckets (alerts, tfstate, logging, secrets)

19181f3

Encrypt firehoses and Athena bucket

3d8828d

Grant alert processor permissions to use SSE key

1f3945c

Fix unit tests and pylint

32d7e67

austinbyers added this to the 2.0.0 milestone Jul 6, 2018

austinbyers requested a review from jacknagz July 6, 2018 21:15

jacknagz reviewed Jul 6, 2018

View reviewed changes

jacknagz approved these changes Jul 6, 2018

View reviewed changes

Add comment about KMS policy

16924fe

austinbyers merged commit dab2658 into master Jul 6, 2018

austinbyers deleted the austin-s3-encryption branch July 6, 2018 21:58

ryandeivert added tech debt athena and removed athena tech debt labels Jul 9, 2018

ryandeivert modified the milestones: 2.0.0, pro Jul 9, 2018

ryandeivert added the improvement label Jul 9, 2018

ryandeivert modified the milestones: pro, 2.0.0 Jul 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 Server-Side Encryption #784

S3 Server-Side Encryption #784

austinbyers commented Jul 6, 2018 •

edited

jacknagz Jul 6, 2018

austinbyers Jul 6, 2018

jacknagz Jul 6, 2018

austinbyers Jul 6, 2018 •

edited

jacknagz left a comment

coveralls commented Jul 6, 2018 •

edited

S3 Server-Side Encryption #784

S3 Server-Side Encryption #784

Conversation

austinbyers commented Jul 6, 2018 • edited

Background

Changes

Testing

jacknagz Jul 6, 2018

Choose a reason for hiding this comment

austinbyers Jul 6, 2018

Choose a reason for hiding this comment

jacknagz Jul 6, 2018

Choose a reason for hiding this comment

austinbyers Jul 6, 2018 • edited

Choose a reason for hiding this comment

jacknagz left a comment

Choose a reason for hiding this comment

coveralls commented Jul 6, 2018 • edited

austinbyers commented Jul 6, 2018 •

edited

austinbyers Jul 6, 2018 •

edited

coveralls commented Jul 6, 2018 •

edited