Write state using compact JSON representation #1647
Merged
+35
−1,693
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR allows to write JSON using compact (non-indented) representation if the `TF_USE_COMPACT_STATE_FORMAT` environment variable is defined. This significantly improves performance when working with huge states. For example (on ~450k objects in state): * state size is decreased from 510.1MB to 386.6MB * Time to write to S3 decreased from 112 seconds to 94 seconds * Time to read from S3 decreased from 87 seconds -> 55 seconds Signed-off-by: Alex Ott <alexott@gmail.com>
|
Reminder for the PR assignee: If this is a user-visible change, please update the changelog as part of the PR. |
janosdebugs
reviewed
May 13, 2024
Signed-off-by: Alex Ott <alexott@gmail.com>
alexott
changed the title
cam72cam
previously approved these changes
May 21, 2024
janosdebugs
previously approved these changes
May 21, 2024
|
@alexott looks like some test failures to fix before this can be merged |
|
Tests are fixed |
|
I had a small heart attack when looking at the LoC totals in the diff, before remembering that it was the whole goal of this PR 😆 |
cam72cam
previously approved these changes
May 21, 2024
ollevche
reviewed
May 21, 2024
oh, yes - missed this one Co-authored-by: Oleksandr Levchenkov <ollevche@gmail.com> Signed-off-by: Alex Ott <alexott@gmail.com>
ollevche
approved these changes
May 21, 2024
cam72cam
approved these changes
May 21, 2024
cam72cam
added a commit
that referenced
this pull request
May 22, 2024
Signed-off-by: Christian Mesh <christianmesh1@gmail.com>
3 tasks
cam72cam
added a commit
that referenced
this pull request
May 22, 2024
bunniseng
pushed a commit
to bunniseng/opentofu
that referenced
this pull request
May 28, 2024
Signed-off-by: Alex Ott <alexott@gmail.com> Co-authored-by: Oleksandr Levchenkov <ollevche@gmail.com>
bunniseng
pushed a commit
to bunniseng/opentofu
that referenced
this pull request
May 28, 2024
Signed-off-by: Christian Mesh <christianmesh1@gmail.com>
|
@cam72cam Some of the testdata in |
|
@eduzgun, sure create an issue. I can't reproduce that myself, the tests are passing on main. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes the writing of the state using compact JSON (non-indented) representation. This significantly improves performance when working with huge states.
For example (on ~450k objects in the state):
Resolves #1593
Target Release
1.8.0
Checklist
Go checklist
<-- If your PR contains Go code, please make sure you check off all items on this list: -->
Website/documentation checklist