Skip to content

full terraform plan implementation #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 16 commits into from
Jul 11, 2023
Merged

full terraform plan implementation #8

merged 16 commits into from
Jul 11, 2023

Conversation

@DavidS-ovm
Copy link
Contributor

@DavidS-ovm DavidS-ovm commented Jul 11, 2023

Not my neatest code, but it works!

@DavidS-ovm
Embed git version into tracing info
23cfffa
@DavidS-ovm
Add aws-source symlink/checkout to support generating tf transforms f…
19ac06c 
…rom it
@DavidS-ovm
First prototype of tfplan parsing and evaluating
82cd9d5 
All input data still manually assembled for testing purposes

* add `go generate` script `extractmaps.go` as a placeholder for transforming the docs-data from aws-source into TfMapData. The data in extractmaps is manually copied over from aws-source for now.
* add example resource from our dogfood environment plan to be extracted from `terraform show -json`
* map the example resource into an `sdp.Query` and send that off as changing item to the rest of the processing

Working run:
```
vscode ➜ /workspace/ovm-cli (main) $ go run main.go change-from-tfplan --frontend https://frontend-knkxto8fa.preview.overmind-demo.com/
ERRO[0000] Error reading config file                     err="Config File \"config\" Not Found in \"[]\""
INFO[0000] set log level from config                     fields.level=trace
INFO[0000] otlptracehttp client configured itself: &{traces {api.honeycomb.io false <nil> map[x-honeycomb-team:Quu6uCifruYihvi1Lzj3MC] 0 10000000000 /v1/traces <nil>} {{api.honeycomb.io false <nil> map[x-honeycomb-team:Quu6uCifruYihvi1Lzj3MC] 0 10000000000 /v1/traces <nil>} {true 5000000000 30000000000 60000000000} 0  [] <nil>} 0xc9f440 0xc0005540f0 0xc0000483c0 {0 {0 0}}}
DEBU[0000] Connecting to overmind API: https://api.df.overmind-demo.com/api/gateway
[auth redacted]
INFO[0017] created a new change                          change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 url="https://api.df.overmind-demo.com/api/gateway"
INFO[0017] resolving items from terraform plan           change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 url="https://api.df.overmind-demo.com/api/gateway"
DEBU[0017] query status update                           change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 query=99d81090-03dd-4f6d-8783-d3360c68a446 status=STARTED url="https://api.df.overmind-demo.com/api/gateway"
DEBU[0017] query status update                           change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 query=0588a10a-9cc1-41d0-abb5-02767172578f status=STARTED url="https://api.df.overmind-demo.com/api/gateway"
INFO[0017] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=944651592624.eu-west-2.elbv2-load-balancer.ingress url="https://api.df.overmind-demo.com/api/gateway"
INFO[0018] still waiting for responders                  change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 post_processing_complete=false queriesSent=true responders=2 summary="working:1 complete:1 responders:2" url="https://api.df.overmind-demo.com/api/gateway"
DEBU[0018] query status update                           change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 query=99d81090-03dd-4f6d-8783-d3360c68a446 status=FINISHED url="https://api.df.overmind-demo.com/api/gateway"
INFO[0019] still waiting for responders                  change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 post_processing_complete=false queriesSent=true responders=2 summary="working:1 complete:1 responders:2" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0019] still waiting for responders                  change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 post_processing_complete=false queriesSent=true responders=2 summary="working:1 complete:1 responders:2" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.aws-service-role/AWSSSOServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.job-function/ViewOnlyAccess url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.ReadOnlyAccess url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.AdministratorAccess url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.aws-service-role/AWSConfigServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.aws-service-role/AWSOrganizationsServiceTrustPolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.aws-service-role/AmazonSSMServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.aws-service-role/AWSTrustedAdvisorServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.aws-service-role/AWSSupportServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.service-role/AWSLambdaBasicExecutionRole url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.aws-service-role/CloudTrailServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.AWSOrganizationsFullAccess url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.PowerUserAccess url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=aws.iam-policy.service-role/AWS_ConfigRole url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.aws-service-role/AWSSSOServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.job-function/ViewOnlyAccess url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.AdministratorAccess url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.ReadOnlyAccess url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.aws-service-role/AWSOrganizationsServiceTrustPolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.AWSOrganizationsFullAccess url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.aws-service-role/AWSSupportServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.aws-service-role/AmazonSSMServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.aws-service-role/AWSTrustedAdvisorServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.service-role/AWSLambdaBasicExecutionRole url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.aws-service-role/CloudTrailServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.aws-service-role/AWSConfigServiceRolePolicy url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.PowerUserAccess url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] new item                                      change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 item=337346983342.iam-policy.service-role/AWS_ConfigRole url="https://api.df.overmind-demo.com/api/gateway"
DEBU[0023] query status update                           change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 query=0588a10a-9cc1-41d0-abb5-02767172578f status=FINISHED url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] all responders and queries done               allDone=true change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 post_processing_complete=true queriesSent=true responders=2 summary="complete:2 responders:2" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0023] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0024] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:1" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0025] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:10 numEdges:11" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0025] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:12 numEdges:13" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0026] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:14 numEdges:13" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0027] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:16 numEdges:13" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0028] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:17 numEdges:13" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0029] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:19 numEdges:13" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0030] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:33 numEdges:26" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0030] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:39 numEdges:35" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0030] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:44 numEdges:43" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0031] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:49 numEdges:47" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0031] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DISCOVERING numItems:51 numEdges:48" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0032] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_FINDING_APPS numItems:52 numEdges:55" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0032] status update                                 change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 fields.msg="state:STATE_DONE numItems:52 numEdges:55" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0032] change ready                                  change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 change-url="https://frontend-knkxto8fa.preview.overmind-demo.com//changes/b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0033] Websocket closing                             change=b0209f6c-5b46-4ce5-8fae-b9d4d3c612a2 code=StatusNormalClosure reason= url="https://api.df.overmind-demo.com/api/gateway"
TRAC[0033] tracing has shut down
vscode ➜ /workspace/ovm-cli (main) $
```
@DavidS-ovm
Generate aws-source mappings
8a1822a 
Since some sources can have overlapping queries, this adds another
layer of array into the data structure.
@DavidS-ovm
Mapping tf resources to SDP queries
e65ff58
@DavidS-ovm
Avoid infinite loop when there's no item mapped
4fc5c94
@DavidS-ovm
Add debug logging for which items are getting mapped
36646c3
@DavidS-ovm
Remove test flag
b75079b
@DavidS-ovm DavidS-ovm merged commit 0811ad6 into main Jul 11, 2023
@DavidS-ovm DavidS-ovm deleted the tfplan-parse branch July 11, 2023 15:04
dylanratcliffe
dylanratcliffe reviewed Jul 11, 2023
View reviewed changes
Copy link
Member

@dylanratcliffe dylanratcliffe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something that might be worth considering would be just constructing these manually i.e. call CreateBookmark and put the queries in there, without running them. Then call CreateChange, then CalculateBlastRadius. That way the CLI wouldn't need to deal with a websocket.

@DavidS-ovm
Copy link
Contributor Author

DavidS-ovm commented Jul 12, 2023

Something that might be worth considering would be just constructing these manually i.e. call CreateBookmark and put the queries in there, without running them. Then call CreateChange, then CalculateBlastRadius. That way the CLI wouldn't need to deal with a websocket.

That would be nice. To get there requires fleshing out GetAffectedBookmarks to work with queries that are not only GET queries with no recursion.

@dylanratcliffe dylanratcliffe mentioned this pull request Jul 12, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dylanratcliffe dylanratcliffe dylanratcliffe left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@DavidS-ovm @dylanratcliffe