Skip to content

Commit

Permalink
graph: batch quads automatically if caller uses single AddQuad on writer
Browse files Browse the repository at this point in the history
  • Loading branch information
dennwc committed May 6, 2017
1 parent decd378 commit a420e62
Showing 1 changed file with 18 additions and 2 deletions.
20 changes: 18 additions & 2 deletions graph/quadwriter.go
Expand Up @@ -187,18 +187,34 @@ func NewWriter(qs QuadWriter) BatchWriter {

type batchWriter struct {
qs QuadWriter
buf []quad.Quad
}

func (w *batchWriter) flushBuffer(force bool) error {
if !force && len(w.buf) < quad.DefaultBatch {
return nil
}
_, err := w.WriteQuads(w.buf)
w.buf = w.buf[:0]
return err
}

func (w *batchWriter) WriteQuad(q quad.Quad) error {
return w.qs.AddQuad(q)
if err := w.flushBuffer(false); err != nil {
return err
}
w.buf = append(w.buf, q)
return nil
}
func (w *batchWriter) WriteQuads(quads []quad.Quad) (int, error) {
if err := w.qs.AddQuadSet(quads); err != nil {
return 0, err
}
return len(quads), nil
}
func (w *batchWriter) Close() error { return nil }
func (w *batchWriter) Close() error {
return w.flushBuffer(true)
}

// NewRemover creates a quad writer for a given QuadStore which removes quads instead of adding them.
func NewRemover(qs QuadWriter) BatchWriter {
Expand Down

2 comments on commit a420e62

@edfungus
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dennwc This commit seems to have broken writing structs directly into the graph. I haven't looked too much into it but recently my tests were failing. Essentially, my graph would be populated using commits before this one and my graph would be empty on this commit and current master (a50a6ef)

A sample of how I am writing structs to the graph is here (https://github.com/edfungus/conduction/blob/master/storage/storage.go#L76)

I can also make an issue too if you would like

@dennwc
Copy link
Member Author

@dennwc dennwc commented on a420e62 May 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@edfungus The reason the storage is empty is because Writer does internal buffering now and needs to be closed to write it to the storage. It's similar to bufio.Writer in this regard. Creating a new writer has a very small overhead, so you can make a new one each time you want to write a chunk of data.

But we should document it better, I guess, or expose a Flush method.

Please sign in to comment.