Fix #5: Support for very large revisions and very large files. #9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… grained (down from SvnRevision to SvnNode). * Move TerminatingValidator out of test source into main source, as it might be useful.
… having the Consumer do that. * SvnDumpFileParser will not add nodes to an SvnRevision, as those should be consumed separately. * Update SvnDumpWriterImpl and SvnDumpInMemory to work with this new way.
…ad of getting the SvnDump as a whole.
* Previously, this was implemented as a separate class, ConsumerChain. * Because of the granularity changes to SvnDumpFileParser to parse SvnNodes separately from their SvnRevision, many SvnDumpConsumers can no longer properly operate. * Instead of teaching the ConsumerChain how to represent and handle SvnNode additions/deletions that may come from an SvnDumpMutator, I'm teaching SvnDumpConsumers how to chain to each other. * This removes the need for a separate ConsumerChain class. * This also places the burder of continuing the consumer chain on each consumer, which is a downside of this approach.
…rForEmptyRevisions to properly chain.
added 24 commits
October 2, 2015 11:31
…ded in SvnDumpFileParserTest.consume
…ependently from other nodes in the revision. * This means we don't have to hold all the nodes in memory at the same time, just one node. * This should fix the "SvnRevision >30GB" requirement.
* We're going to split up the file content of SvnNodes into smaller pieces, so that we don't have to store the entire file content in memory.
* Add ending calls to SvnDumpConsumer: endRevision(...), endNode(...), and endChunks(). Makes SvnDumpWriterImpl code much cleaner and clearer.
…fixes #5. * You can change the chunk size by calling setFileContentChunkSize(long) on the SvnDumpFileParser.
cstroe
pushed a commit
that referenced
this pull request
Oct 5, 2015
Fix #5: Support for very large revisions and very large files.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
File content for files committed to an SVN repository is now processed in chunks. This allows processing of any size files that may have been committed to an SVN repository.
To achieve this, the
SvnDumpConsumermechanism was made more granular, all the way to the file content level (consume(SvnNode), consume(FileContentChunk)). More consumption and ending methods were added toSvnDumpConsumerin order to support this.The ending methods (
endRevision,endNode,endChunks) were added because it allows the implementation code ofSvnDumpConsumers to be much cleaner without much complexity added. It also gives a lot of flexibility over the parsing process.Added an
SvnFileDumpParserDoppelgangerin order to support parsingSvnDumpobjects in memory. Possibly should be removed in the future and the functionality merged intoSvnDumpFileParser.