Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(terraform): reduce parsing time for large TF plan files #3757

Merged
merged 3 commits into from
Oct 31, 2022

Conversation

gruebel
Copy link
Contributor

@gruebel gruebel commented Oct 28, 2022

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Description

  • Added a json.laods() fallback for TF plan files, which are bigger than 50mb (the created ticket says 100mb, but currently it takes ~40sec to parse a 100mb file, compared to ~2-3sec via json.loads)
  • Added a couple of logs for better troubleshooting
  • Interestingly we also have a custom JSON parser, but it performs worse compared to the YAML loader 馃し
  • Added some small performance improvements in the TF edge creation, which makes it ~20% faster for TF plan files and ~10% for HCL files 馃殌

Fixes #3754

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my feature, policy, or fix is effective and works
  • New and existing tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

Copy link
Contributor

@nimrodkor nimrodkor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice!

checkov/cloudformation/parser/cfn_yaml.py Outdated Show resolved Hide resolved
checkov/cloudformation/parser/cfn_yaml.py Outdated Show resolved Hide resolved
@gruebel gruebel merged commit 0b73830 into bridgecrewio:master Oct 31, 2022
@gruebel gruebel deleted the add-json-fallback branch October 31, 2022 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Terraform plan files with a size of over 100mb
3 participants