Skip to content

Commit

Permalink
Merge pull request #5 from dgtized/list-permissions
Browse files Browse the repository at this point in the history
Calculate IAM Policy required for cloning environment
  • Loading branch information
dgtized committed Jan 23, 2021
2 parents 35065ef + e7638fa commit c204bff
Show file tree
Hide file tree
Showing 16 changed files with 372 additions and 51 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: 2
jobs:
build:
docker:
- image: circleci/clojure:openjdk-11-tools-deps-1.10.1.483
- image: circleci/clojure:openjdk-11-tools-deps-1.10.1.754

working_directory: ~/repo

Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Changelog

## Unreleased

### Added

## [0.5.0]

- Added `--iam-policy` option for generating a IAM policy for a user or role to clone a replica with a minimal set of permissions.
- Updated dependencies

2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
BSD 3-Clause License

Copyright (c) 2019, Charles L.G. Comstock
Copyright (c) 2019-2021, Charles L.G. Comstock
All rights reserved.

Redistribution and use in source and binary forms, with or without
Expand Down
72 changes: 71 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,9 @@ Hopefully in the future this can be parsed directly from the `AWS_CONFIG` file.
clj -m stack-mitosis.cli \
--source mitosis-production --target mitosis-staging \
--restart "./restart-service.sh"
--credentials resources/role.edn
[--credentials resources/role.edn]
[--plan]
[--iam-policy]

## Flight Plan

Expand Down Expand Up @@ -112,7 +113,76 @@ Flight plan:

Note that for many cases, even if the clone process is interrupted, the flight plan will show steps it will try to execute again, and steps it will skip because it has detected that the instance has already been created or modified to the right attribute values. In other words, it tries to pickup where it left-off if there is a failure.

## IAM Policy Generation

Stack-mitosis can also generate an IAM policy for an automated user to update a particular environment. The policy uses the database names from the planned changeset to calculate these minimal permissions. While they have been elided from the example below, the ARNs are locked to the specific account & region used.

```
$ clj -m stack-mitosis.cli --source mitosis-prod --target mitosis-demo --iam-policy
{"Version":"2012-10-17",
"Statement":
[{"Effect":"Allow",
"Action":["rds:DescribeDBInstances", "rds:ListTagsForResource"],
"Resource":["arn:aws:rds:*:*:db:*"]},
{"Effect":"Allow",
"Action":["rds:CreateDBInstanceReadReplica"],
"Resource":
["arn:aws:rds:*:*:og:*", "arn:aws:rds:*:*:pg:*",
"arn:aws:rds:*:*:subgrp:*",
"arn:aws:rds:*:*:db:mitosis-prod",
"arn:aws:rds:*:*:db:temp-mitosis-demo",
"arn:aws:rds:*:*:db:temp-mitosis-demo-replica"]},
{"Effect":"Allow",
"Action":["rds:AddTagsToResource"],
"Resource":
["arn:aws:rds:*:*:db:temp-mitosis-demo",
"arn:aws:rds:*:*:db:temp-mitosis-demo-replica"]},
{"Effect":"Allow",
"Action":["rds:PromoteReadReplica"],
"Resource":
["arn:aws:rds:*:*:db:temp-mitosis-demo"]},
{"Effect":"Allow",
"Action":["rds:ModifyDBInstance"],
"Resource":
["arn:aws:rds:*:*:og:*", "arn:aws:rds:*:*:pg:*",
"arn:aws:rds:*:*:secgrp:*", "arn:aws:rds:*:*:subgrp:*",
"arn:aws:rds:*:*:db:temp-mitosis-demo",
"arn:aws:rds:*:*:db:temp-mitosis-demo-replica",
"arn:aws:rds:*:*:db:mitosis-demo-replica",
"arn:aws:rds:*:*:db:old-mitosis-demo-replica",
"arn:aws:rds:*:*:db:mitosis-demo",
"arn:aws:rds:*:*:db:old-mitosis-demo"]},
{"Effect":"Allow",
"Action":["rds:RebootDBInstance"],
"Resource":
["arn:aws:rds:*:*:db:temp-mitosis-demo",
"arn:aws:rds:*:*:db:temp-mitosis-demo-replica",
"arn:aws:rds:*:*:db:mitosis-demo-replica",
"arn:aws:rds:*:*:db:mitosis-demo"]},
{"Effect":"Allow",
"Action":["rds:DeleteDBInstance"],
"Resource":
["arn:aws:rds:*:*:db:old-mitosis-demo-replica",
"arn:aws:rds:*:*:db:old-mitosis-demo"]}]}
```

This ensures that a continuous integration or cronjob server like Jenkins can clone production to demo environments on a weekly basis restricted to the minimal permissions necessary. If a user needs to run stack-mitosis for multiple environments (demo, staging, random developer test environment), then a policy can be attached for each environment.

# Testing

bin/kaocha # basic unit tests
bin/kaocha --plugin cloverage # with coverage output

# Frequently Asked Questions

## Why not use Cloudformation/Terraform

Cloudformation and Terraform are wonderful tools focused on declarative architecture transformation from one steady state to another. Stack-mitosis is focused on safely cloning the contents of a database in one environment to another without changing from one steady state to another. As example, for an environment with production and demo environments, they both exist in the correct configuration before running stack-mitosis, and then after running stack-mitosis the configuration remains the same but the demo environment has a fresh copy of the data from production.

I suspect this could also be accomplished using one of these declarative infrastructure tools by transitioning through multiple intervening states, but have not found any examples of anyone doing that.

# License

Copyright © 2019-2021 Charles L.G. Comstock

Distributed under the BSD-3 Clause License (see LICENSE file)
2 changes: 1 addition & 1 deletion bin/ci
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ bin/kaocha --plugin cloverage \
--plugin kaocha.plugin/junit-xml \
--junit-xml-file test-results/kaocha/results.xml

clojure -Aclj-kondo --lint src
clojure -Aclj-kondo -M --lint src
2 changes: 1 addition & 1 deletion bin/kaocha
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
#!/usr/bin/env bash

clojure -A:kaocha -m kaocha.runner --config-file test/tests.edn "$@"
clojure -Mkaocha -m kaocha.runner --config-file test/tests.edn "$@"
32 changes: 14 additions & 18 deletions deps.edn
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
{:deps
{org.clojure/clojure {:mvn/version "1.10.1"}
org.clojure/core.async {:mvn/version "1.2.603"}
com.cognitect.aws/api {:mvn/version "0.8.456"}
com.cognitect.aws/endpoints {:mvn/version "1.1.11.783"}
org.clojure/core.async {:mvn/version "1.3.610"}
com.cognitect.aws/api {:mvn/version "0.8.498"}
com.cognitect.aws/endpoints {:mvn/version "1.1.11.934"}

com.cognitect.aws/rds {:mvn/version "796.2.662.0"}
com.cognitect.aws/rds {:mvn/version "810.2.817.0"}

;; for STS refresh
com.cognitect.aws/iam {:mvn/version "796.2.654.0"}
com.cognitect.aws/sts {:mvn/version "798.2.678.0"}
com.cognitect.aws/iam {:mvn/version "801.2.704.0"}
com.cognitect.aws/sts {:mvn/version "809.2.784.0"}

;; logging
org.clojure/tools.logging {:mvn/version "1.1.0"}
Expand All @@ -23,21 +23,17 @@
}
:paths ["src" "resources"]
:aliases
{;; clj -Aoutdated
:outdated {:extra-deps {olical/depot {:mvn/version "RELEASE"}}
:main-opts ["-m" "depot.outdated.main"]}

;; clj -A:kaocha -m kaocha.runner --config-file test/tests.edn
{;; clj -Mkaocha -m kaocha.runner --config-file test/tests.edn
:kaocha {:extra-paths ["test"]
:extra-deps {lambdaisland/kaocha {:mvn/version "1.0.632"}
lambdaisland/kaocha-junit-xml {:mvn/version "0.0-70"}
lambdaisland/kaocha-cloverage {:mvn/version "1.0-45"}}}
:extra-deps {lambdaisland/kaocha {:mvn/version "1.0.732"}
lambdaisland/kaocha-junit-xml {:mvn/version "0.0.76"}
lambdaisland/kaocha-cloverage {:mvn/version "1.0.75"}}}

;; clj -Aclj-kondo --lint src
;; clj -Mclj-kondo --lint src
:clj-kondo
{:extra-deps {clj-kondo {:mvn/version "RELEASE"}}
{:extra-deps {clj-kondo/clj-kondo {:mvn/version "RELEASE"}}
:main-opts ["-m" "clj-kondo.main"]}

;; clj -Acoverage
:coverage {:extra-deps {cloverage {:mvn/version "RELEASE"}}
;; clj -Mcoverage
:coverage {:extra-deps {cloverage/cloverage {:mvn/version "RELEASE"}}
:main-opts ["-m" "cloverage.coverage" "-p" "src"]}}}
18 changes: 13 additions & 5 deletions src/stack_mitosis/cli.clj
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
(ns stack-mitosis.cli
(:require [clojure.tools.cli :as cli]
(:require [clojure.data.json :as json]
[clojure.string :as str]
[clojure.tools.cli :as cli]
[clojure.tools.logging :as log]
[stack-mitosis.interpreter :as interpreter]
[stack-mitosis.planner :as plan]
[stack-mitosis.policy :as policy]
[stack-mitosis.request :as r]
[clojure.string :as str]
[stack-mitosis.sudo :as sudo]
[clojure.tools.logging :as log]))
[stack-mitosis.sudo :as sudo]))

;; TODO: add max-timeout for actions
;; TODO: show attempt info like skipped steps in flight plan?
Expand All @@ -16,7 +18,8 @@
["-t" "--target DST" "Root identifier of database tree to copy over"]
[nil "--restart CMD" "Blocking script to restart application."]
["-c" "--credentials FILENAME" "Credentials file in edn for iam assume-role"]
["-p" "--plan", "Display expected flightplan for operation."]
["-p" "--plan" "Display expected flightplan for operation."]
["-i" "--iam-policy" "Generate IAM policy for planned actions."]
["-h" "--help"]])

(defn parse-args [args]
Expand Down Expand Up @@ -55,6 +58,9 @@
(cond (:plan options)
(do (println (flight-plan (interpreter/check-plan instances plan)))
true)
(:iam-policy options)
(do (json/pprint (policy/from-plan instances plan))
true)
:else
(let [last-action (interpreter/evaluate-plan rds plan)]
(not (contains? last-action :ErrorResponse))))))))
Expand All @@ -72,4 +78,6 @@
"--plan" "--restart" "'./service-restart.sh'"]))
(process (parse-args ["--source" "mitosis-prod" "--target" "mitosis-demo"
"--plan" "--credentials" "resources/role.edn"]))
(process (parse-args ["--source" "mitosis-prod" "--target" "mitosis-demo"
"--iam-policy"]))
(process (parse-args ["--source" "mitosis-prod" "--target" "mitosis-demo"])))
11 changes: 8 additions & 3 deletions src/stack_mitosis/example_environment.clj
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,13 @@
(:require [stack-mitosis.helpers :as helpers]
[stack-mitosis.operations :as op]
[stack-mitosis.planner :as plan]
[stack-mitosis.policy :as policy]
[stack-mitosis.predict :as predict]))

(def template
{:DBInstanceClass "db.t3.micro"
{:DBInstanceClass "db.t2.micro"
:Engine "postgres" ;;"mysql"
:StorageType "gp2"
:StorageType "gp2" ;; ie ssd storage
:AllocatedStorage 5
:PubliclyAccessible false
:MasterUsername "root"})
Expand Down Expand Up @@ -53,4 +54,8 @@
(concat (plan/delete-tree state "mitosis-demo")
(plan/delete-tree state "mitosis-prod"))))


(comment
;; Add fake arn to create template?
(policy/generate [] (create template))
;; Fix this to actual return a functioning delete policy?
(policy/generate (predict/state [] (create template)) (destroy)))
5 changes: 3 additions & 2 deletions src/stack_mitosis/interpreter.clj
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,9 @@

(defn databases
[rds]
{:post [(seq %)]}
(:DBInstances (invoke-logged! rds {:op :DescribeDBInstances})))
;; exclude nil, but fresh account might return empty list
{:post [(sequential? %)]}
(:DBInstances (invoke-logged! rds (op/describe))))

;; TODO: verify that "old-" database copies do not exist before running
(defn verify-databases-exist
Expand Down
8 changes: 5 additions & 3 deletions src/stack_mitosis/operations.clj
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,11 @@
{:op :PromoteReadReplica
:request {:DBInstanceIdentifier id}})

(defn describe [id]
{:op :DescribeDBInstances
:request {:DBInstanceIdentifier id}})
(defn describe
([] {:op :DescribeDBInstances})
([id]
{:op :DescribeDBInstances
:request {:DBInstanceIdentifier id}}))

(defn tags [db-arn]
{:op :ListTagsForResource
Expand Down
106 changes: 106 additions & 0 deletions src/stack_mitosis/policy.clj
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
(ns stack-mitosis.policy
(:require [stack-mitosis.request :as r]
[stack-mitosis.lookup :as lookup]
[stack-mitosis.predict :as predict]
[clojure.string :as str]))

(defn make-arn
[db-id & {:keys [account-id region type]
:or {account-id "*" region "*" type "db"}}]
(str/join ":" ["arn:aws:rds" region account-id type db-id]))

(defmulti permissions
"Calculate permissions required for a given operation.
(permissions [instances action]) => [{:op _ :arn _} ...]"
(fn [_ action] (get action :op)))

(defmethod permissions :shell-command
[_ _]
[])

(defmethod permissions :CreateDBInstanceReadReplica
[instances action]
;; For create replica, use the ARN from the source database
(let [source-id (r/source-id action)
db-id (r/db-id action)
source-arn (:DBInstanceArn (lookup/by-id instances source-id))
target-arn (:DBInstanceArn (lookup/by-id (predict/predict instances action) db-id))]
[{:op (:op action)
;; TODO: can these permissions be more specific instead of wildcard?
;; a) re-use the base ARN (ie region:account-id) from source
;; b) generate named ARNs for each subtype used in source?
:arn (into [(make-arn "*" :type "og")
(make-arn "*" :type "pg")
(make-arn "*" :type "subgrp")]
[source-arn target-arn])}
{:op :AddTagsToResource :arn target-arn}]))

(defmethod permissions :ModifyDBInstance
[instances action]
(let [db-id (r/db-id action)
arn (:DBInstanceArn (lookup/by-id instances db-id))]
(keep identity
[{:op (:op action)
;; TODO: see above about tightening permissions
:arn [(make-arn "*" :type "og")
(make-arn "*" :type "pg")
(make-arn "*" :type "secgrp")
(make-arn "*" :type "subgrp")
arn]}
;; If renaming we need the new arn too
(when-let [new-arn
(:DBInstanceArn (lookup/by-id (predict/predict instances action)
(r/new-id action)))]
{:op (:op action)
:arn new-arn})
;; RebootDBInstance is necessary for :ApplyImmediately true
{:op :RebootDBInstance
:arn arn}])))

(defmethod permissions :default
[instances action]
(let [db-id (r/db-id action)]
(if-let [instance (lookup/by-id instances db-id)]
[{:op (:op action)
:arn (:DBInstanceArn instance)}]
;; TODO handle ResourceName for ListTagsForResource
[{:op (:op action)}])))

;; TODO possibly generate optional statement identifier?
;; TODO simplify action/resource to singular if only one value?
(defn allow [actions resources]
{:Effect "Allow"
:Action (mapv (partial str "rds") actions)
:Resource resources})

(defn create-example []
;; TODO handle og, pg, subgrp, secgrp permissions?
(allow [:CreateDBInstance :AddTagsToResource :DeleteDBInstance]
[(make-arn "mitosis-*")]))

(defn globals []
(allow [:DescribeDBInstances :ListTagsForResource]
[(make-arn "*")]))

;; TODO breakup permissions per operation type with better granularity
;; ie Delete should only have permissions on old-, not temp- or current staging.
;; TODO tighten restrictions on DB OptionGroup (og), DB ParameterGroup (pg) and DB Subnet Group (subgrp)
;; these were listed as warnings in the policy editor so leaving wildcard for now
(defn generate [instances operations]
(let [all-permissions
(mapcat permissions
(reductions predict/predict instances operations)
operations)]
(for [[op ops] (group-by :op all-permissions)]
(allow [op] (distinct (flatten (map :arn ops)))))))

(defn policy [statements]
{:Version "2012-10-17" :Statement statements})

(defn from-plan [instances operations]
(policy (concat [(globals)] (generate instances operations))))

(comment
(require '(clojure.data.json :as json))
(json/pprint (policy [(globals) (create-example)])))
Loading

0 comments on commit c204bff

Please sign in to comment.