Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pretty-printed JSON representation of a state increases file size by ~20-25% #35174

Open
alexott opened this issue May 17, 2024 · 0 comments · May be fixed by #35175
Open

Pretty-printed JSON representation of a state increases file size by ~20-25% #35174

alexott opened this issue May 17, 2024 · 0 comments · May be fixed by #35175
Labels
enhancement new new issue not yet triaged

Comments

@alexott
Copy link
Contributor

alexott commented May 17, 2024 •
edited

Terraform Version

all versions

Use Cases

We have a few use cases when the state size is very big, and we were looking into optimizing performance. What I noticed is that by default, we're using pretty-printed JSON when storing the state, and the resulting file size is much bigger than if we use compact representation, and this make read/write of state slower, not taking into account increased storage and egress costs when using cloud backends.

For example, here is comparison of the state with ~70k objects - original state (mkdirs.tfstate) is converted into compact representation by running jq -c . on it.

84096 -rw-rw-r--@ 1 alex  staff  42996911 Apr 27 10:53 mkdirs-compact.tfstate
114112 -rw-r--r--@ 1 alex  staff  57680266 Apr 27 10:50 mkdirs.tfstate

Attempted Solutions

I have PR for it...

Proposal

We should be able to store the state in compact JSON format. This could be done either way:

  1. Use compact representation by default
  2. Make it configurable with some environment variable

References

No response
@alexott alexott added enhancement new new issue not yet triaged labels May 17, 2024
alexott added a commit to alexott/terraform that referenced this issue May 17, 2024
@alexott
Allow to write state using compact representation
3cb7fc1 
This PR allows to write JSON using compact (non-indented) representation if the
`TF_USE_COMPACT_STATE_FORMAT` environment variable is defined.  This significantly
improves performance when working with huge states.

For example (on ~450k objects in state):

* state size is decreased from 510.1MB to 386.6MB
* Time to write to S3 decreased from 112 seconds to 94 seconds
* Time to read from S3 decreased from 87 seconds -> 55 seconds

Fixes hashicorp#35174

Signed-off-by: Alex Ott <alexott@gmail.com>
@alexott alexott linked a pull request May 17, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement new new issue not yet triaged
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Allow to write state using compact representation alexott/terraform
1 participant
@alexott