Serialize memory improvement #729
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Minimal time-to-serialize change (4s vs 3s, likely a rounding error), but a huge memory improvement by not keeping a serialized copy of everything in memory. Time with change (one step, direct to disk, about 4 seconds): ``` .... Fri Dec 08 18:04:07 PST 2017: Convert configurations to vendor-independent format: 1086/1086 .... Fri Dec 08 18:04:13 PST 2017: Serializing 'org.batfish.datamodel.Configuration' instances to disk: 1086/1086 .... Fri Dec 08 18:04:17 PST 2017: TERMINATEDNORMALLY ``` Time without change (two steps, serialize then pack/write; about 3 seconds).: ``` .... Fri Dec 08 18:15:28 PST 2017: Convert configurations to vendor-independent format: 1086/1086 .... Fri Dec 08 18:15:34 PST 2017: Serializing 'org.batfish.datamodel.Configuration' instances: 1086/1086 .... Fri Dec 08 18:15:37 PST 2017: Packing and writing 'org.batfish.datamodel.Configuration' instances to disk: 1086/1086 .... Fri Dec 08 18:15:37 PST 2017: TERMINATEDNORMALLY ```
Don't stop in-memory first
|
Reviewed 1 of 1 files at r1. Comments from Reviewable |
The JsonFactory override does not use our custom printer on all paths.
|
Reviewed 1 of 1 files at r2. Comments from Reviewable |
dhalperi
added a commit
that referenced
this pull request
Dec 12, 2017
* Batfish: serialize directly to disk Minimal time-to-serialize change (4s vs 3s, likely a rounding error), but a huge memory improvement by not keeping a serialized copy of everything in memory. Time with change (one step, direct to disk, about 4 seconds): ``` .... Fri Dec 08 18:04:07 PST 2017: Convert configurations to vendor-independent format: 1086/1086 .... Fri Dec 08 18:04:13 PST 2017: Serializing 'org.batfish.datamodel.Configuration' instances to disk: 1086/1086 .... Fri Dec 08 18:04:17 PST 2017: TERMINATEDNORMALLY ``` Time without change (two steps, serialize then pack/write; about 3 seconds).: ``` .... Fri Dec 08 18:15:28 PST 2017: Convert configurations to vendor-independent format: 1086/1086 .... Fri Dec 08 18:15:34 PST 2017: Serializing 'org.batfish.datamodel.Configuration' instances: 1086/1086 .... Fri Dec 08 18:15:37 PST 2017: Packing and writing 'org.batfish.datamodel.Configuration' instances to disk: 1086/1086 .... Fri Dec 08 18:15:37 PST 2017: TERMINATEDNORMALLY ``` * Write JSON directly to output files Don't stop in-memory first * BatfishObjectMapper: override PrettyPrinter properly The JsonFactory override does not use our custom printer on all paths.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Instead of stopping in-memory, serialize directly to disk. On a small-scale testbed (~1100 configs), these two changes let me reduce memory requirements to under 500MiB from a couple of gigs.
This change is