Skip to content

Prepare change-from-tfplan for terraform mapping #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Jul 10, 2023
Merged

Prepare change-from-tfplan for terraform mapping #7

merged 8 commits into from
Jul 10, 2023

Conversation

@DavidS-ovm
Copy link
Contributor

@DavidS-ovm DavidS-ovm commented Jul 10, 2023 •
edited
Loading

This still doesn't do any actual tfplan mapping, but now has all the structure in place to work with the data from the plan and the aws-source's docs-data.

Example run:

vscode ➜ /workspace/ovm-cli (main) $ go run main.go change-from-tfplan --frontend https://frontend-p6zwiposu.preview.overmind-demo.com --title test-from-cli
ERRO[0000] Error reading config file                     err="Config File \"config\" Not Found in \"[]\""
INFO[0000] set log level from config                     fields.level=trace
INFO[0000] otlptracehttp client configured itself: &{traces {api.honeycomb.io false <nil> map[x-honeycomb-team:abc] 0 10000000000 /v1/traces <nil>} {{api.honeycomb.io false <nil> map[x-honeycomb-team:abc] 0 10000000000 /v1/traces <nil>} {true 5000000000 30000000000 60000000000} 0  [] <nil>} 0xc9f000 0xc0005780c0 0xc0000ba360 {0 {0 0}}} 
DEBU[0000] Connecting to overmind API: https://api.df.overmind-demo.com/api/gateway 
INFO[0000] created a new change                          change=976effc0-13cc-4ba6-babb-ff3b9589f205 url="https://api.df.overmind-demo.com/api/gateway"
INFO[0000] resolving items from terraform plan           change=976effc0-13cc-4ba6-babb-ff3b9589f205 url="https://api.df.overmind-demo.com/api/gateway"
DEBU[0000] query status update                           change=976effc0-13cc-4ba6-babb-ff3b9589f205 query=260189a4-155a-4f11-a5cf-178957fc0b87 status=STARTED url="https://api.df.overmind-demo.com/api/gateway"
INFO[0000] new item                                      change=976effc0-13cc-4ba6-babb-ff3b9589f205 item=944651592624.eu-west-2.elbv2-load-balancer.ingress url="https://api.df.overmind-demo.com/api/gateway"
INFO[0001] still waiting for responders                  change=976effc0-13cc-4ba6-babb-ff3b9589f205 post_processing_complete=false queriesSent=true responders=1 summary="complete:1  responders:1" url="https://api.df.overmind-demo.com/api/gateway"
DEBU[0001] query status update                           change=976effc0-13cc-4ba6-babb-ff3b9589f205 query=260189a4-155a-4f11-a5cf-178957fc0b87 status=FINISHED url="https://api.df.overmind-demo.com/api/gateway"
INFO[0001] all responders and queries done               allDone=true change=976effc0-13cc-4ba6-babb-ff3b9589f205 post_processing_complete=true queriesSent=true responders=1 summary="complete:1  responders:1" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0001] status update                                 change=976effc0-13cc-4ba6-babb-ff3b9589f205 fields.msg="state:STATE_DISCOVERING" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0002] status update                                 change=976effc0-13cc-4ba6-babb-ff3b9589f205 fields.msg="state:STATE_DISCOVERING  numItems:14  numEdges:13" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0002] status update                                 change=976effc0-13cc-4ba6-babb-ff3b9589f205 fields.msg="state:STATE_DISCOVERING  numItems:21  numEdges:29" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0002] status update                                 change=976effc0-13cc-4ba6-babb-ff3b9589f205 fields.msg="state:STATE_DISCOVERING  numItems:22  numEdges:29" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0003] status update                                 change=976effc0-13cc-4ba6-babb-ff3b9589f205 fields.msg="state:STATE_DISCOVERING  numItems:27  numEdges:34" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0004] status update                                 change=976effc0-13cc-4ba6-babb-ff3b9589f205 fields.msg="state:STATE_FINDING_APPS  numItems:27  numEdges:35" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0004] status update                                 change=976effc0-13cc-4ba6-babb-ff3b9589f205 fields.msg="state:STATE_DONE  numItems:27  numEdges:35" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0006] change ready                                  change=976effc0-13cc-4ba6-babb-ff3b9589f205 change-url="https://frontend-p6zwiposu.preview.overmind-demo.com/changes/976effc0-13cc-4ba6-babb-ff3b9589f205" url="https://api.df.overmind-demo.com/api/gateway"
INFO[0006] Websocket closing                             change=976effc0-13cc-4ba6-babb-ff3b9589f205 code=StatusNormalClosure reason= url="https://api.df.overmind-demo.com/api/gateway"
TRAC[0006] tracing has shut down                        
vscode ➜ /workspace/ovm-cli (main) $

@DavidS-ovm
Cleanups
1ca0320
@DavidS-ovm
Adjust message size to current client limits
9dd39a7 
Since we have messages that regularly break the 32k barrier (see [BACKEND-Z](https://overmindtech.sentry.io/issues/4295740369)) and we already have the api-server gateway client configured to accept messages of up to this size (see SetReadLimit in api-server).
@DavidS-ovm
Wrap up ensureToken migration
d441d1f
@DavidS-ovm
Prepare change-from-tfplan for terraform mapping
d1bc353
dylanratcliffe
dylanratcliffe requested changes Jul 10, 2023
View reviewed changes
Copy link
Member

@dylanratcliffe dylanratcliffe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless I'm mistaken this will re-introduce a previously squashed bug. Also the use of GOTOs sent me on a rollercoaster of emotions from Windows batch file PTSD all the way back to "actually these do make sense"

dylanratcliffe
dylanratcliffe approved these changes Jul 10, 2023
View reviewed changes
Copy link
Member

@dylanratcliffe dylanratcliffe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of potential simplifications but looks good

cmd/changefromtfplan.go
// fall through from all "final" query states, check if there's still queries in progress;
// only break from the loop if all queries have already been sent
// TODO: see above, still needs DefaultStartTimeout implemented to account for slow sources
allDone := allDone(ctx, activeQueries, lf)
Copy link
Member

@dylanratcliffe dylanratcliffe Jul 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is necessary but no hard in having it I guess. If you were to get rid of this it would make sense to also get rid if the whole *sdp.GatewayResponse_QueryStatus branch too

Copy link
Contributor Author

@DavidS-ovm DavidS-ovm Jul 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another notch for having to write a proper client library as the responder status does not provide detailed information on what is and is not done. I'll leave it as is for now as a note-to-self that this part of SDP also needs to be considered as part of the client work.

cmd/request.go
// fall through from all "final" query states, check if there's still queries in progress;
// only break from the loop if all queries have already been sent
// TODO: see above, still needs DefaultStartTimeout implemented to account for slow sources
allDone := allDone(ctx, activeQueries, lf)
Copy link
Member

@dylanratcliffe dylanratcliffe Jul 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once again I think this could be completely removed if you want, no pressure though

@DavidS-ovm
Use status.Done()
6e02337 
Co-authored-by: Dylan <dylanratcliffe@outlook.com>
@DavidS-ovm DavidS-ovm merged commit 359424b into main Jul 10, 2023
@DavidS-ovm DavidS-ovm deleted the tfplan-prep branch July 10, 2023 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dylanratcliffe dylanratcliffe dylanratcliffe approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@DavidS-ovm @dylanratcliffe