KV BulkWrite #223
Merged
KV BulkWrite #223
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… durning load. This should streamline the loader, so that the index code just writes, with not gets to increment counters.
…ifferent when using badger driver Fixing lint
It turned out the aggrigation processor was dependent on a bug in the KVIndex code, and when we fixed that, it broke the aggigator Fixing lint Fixing lint
Fixing lint Adding level testing back into travis Removing rocks from docs
kellrott
reviewed
Dec 6, 2019
|
|
||
| // StreamBatch a stream of inputs and loads them into the graph | ||
| // This function assumes incoming stream is GraphElemnts from a single graph | ||
| func StreamBatch(stream <-chan *gripql.GraphElement, batchSize int, graph string, vertexAdd func([]*gripql.Vertex) error, edgeAdd func([]*gripql.Edge) error) error { |
There was a problem hiding this comment.
It might make more sense to use key/value serialized value size to determine batches, rather then use a batchSize. But that can probably left for another PR.
kellrott
reviewed
Dec 6, 2019
| @@ -34,3 +37,30 @@ def test_bulkload(O): | |||
| (res["count"], 6)) | |||
|
|
|||
| return errors | |||
|
|
|||
|
|
|||
| def test_bulkload_validate(O): | |||
There was a problem hiding this comment.
Would it be worth adding in a unit test that actually tests the batching? IE, lots of vertices or elements with large amounts of attached data?
|
👍 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replaces #211
Currently unable to load BMEG due to OOM error in badger that occurs while loading expression data.