on-disk size always 0

[missinglink]

heya, I'm trying out badger as an alternative to leveldb, when I ported my code over I find that there is no data being synced to disk:

total 252K
drwxrwxr-x   2 peter peter 4.0K Jun 28 14:28 .
drwxrwxrwt 710 root  root  244K Jun 28 14:28 ..
-rw-rw-r--   1 peter peter    0 Jun 28 14:28 000000.vlog
-rw-rw-r--   1 peter peter    0 Jun 28 14:28 clog

note that the log.Println("wrote", item.ID) line get's called for each item and no error is produced for log.Println("write error", err).

I have tested the encoding/decoding logic and it works fine with leveldb, so don't worry about that

any help would be appreciated :)

// open database connection
conn := &badger.Connection{}
conn.Open(leveldbPath)
defer conn.Close()

package badger

import "github.com/dgraph-io/badger/badger"

// Connection - Connection
type Connection struct {
	DB *badger.KV
}

// Open - open connection and set up
func (c *Connection) Open(path string) {
	opt := badger.DefaultOptions
	opt.Dir = path
	opt.ValueDir = opt.Dir
	db, err := badger.NewKV(&opt)
	if err != nil {
		panic(err)
	}
	c.DB = db
}

// Close - close connection and clean up
func (c *Connection) Close() {
	c.DB.Close()
}

package badger

import (
	"encoding/binary"
	"log"
	"math"

	"github.com/dgraph-io/badger/badger"
	"github.com/missinglink/gosmparse"
)

// WriteCoord - encode and write lat/lon pair to db
func (c *Connection) WriteCoord(item gosmparse.Node) error {

	// encode id
	key := make([]byte, 8)
	binary.BigEndian.PutUint64(key, uint64(item.ID))

	// encode lat
	lat := make([]byte, 8)
	binary.BigEndian.PutUint64(lat, math.Float64bits(item.Lat))

	// encode lon
	lon := make([]byte, 8)
	binary.BigEndian.PutUint64(lon, math.Float64bits(item.Lon))

	// value
	value := append(lat, lon...)

	// write to db
	err := c.DB.Set(key, value)
	log.Println("wrote", item.ID)
	if err != nil {
		log.Println("write error", err)
		return err
	}

	return nil
}

// ReadCoord - read lat/lon pair from db and decode
func (c *Connection) ReadCoord(id int64) (*gosmparse.Node, error) {

	// encode id
	key := make([]byte, 8)
	binary.BigEndian.PutUint64(key, uint64(id))

	// read from db
	var item badger.KVItem
	err := c.DB.Get(key, &item)
	if err != nil {
		return nil, err
	}

	data := item.Value()

	// decode item
	return &gosmparse.Node{
		ID:  id,
		Lat: math.Float64frombits(binary.BigEndian.Uint64(data[:8])),
		Lon: math.Float64frombits(binary.BigEndian.Uint64(data[8:])),
	}, nil
}

$ go version
go version go1.8.3 linux/amd64

$ uname -a
Linux peterpro 4.6.0-040600-generic #201606100558 SMP Fri Jun 10 10:01:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/issue
Ubuntu 16.04.2 LTS \n \l

$ cd ${GOPATH}/src/github.com/dgraph-io/badger/

$ git log -1
commit 8809f1c3328a7f17a5bf89a1a14f4ee11124e4c6
Author: Jason Chu <1lann@users.noreply.github.com>
Date:   Wed Jun 28 15:02:09 2017 +0800

[missinglink]

I tried isolating the issue further by reducing my script to this:

package main

import (
	"encoding/binary"
	"fmt"
	"log"

	"github.com/dgraph-io/badger/badger"
)

func main() {

	// connection
	opt := badger.DefaultOptions
	opt.Dir = "/tmp/leveldb"
	opt.ValueDir = opt.Dir
	db, err := badger.NewKV(&opt)
	if err != nil {
		panic(err)
	}
	defer db.Close()

	// writes
	for i := 0; i < 10000; i++ {

		bytes := make([]byte, 8)
		binary.BigEndian.PutUint64(bytes, uint64(i))

		// write to db
		err = db.Set(bytes, bytes)
		if err != nil {
			log.Println("write error", err)
		}
	}

	// fetch one key
	keyOne := make([]byte, 8)
	binary.BigEndian.PutUint64(keyOne, uint64(1))

	// read from db
	var item badger.KVItem
	err = db.Get(keyOne, &item)
	if err != nil {
		log.Println("read error", err)
	}

	// debug
	data := item.Value()
	fmt.Println("val", data)
}

it seems like there are two issues:

• it is really slow, taking 11s to insert 10000 records, surely this can't be right?
• it doesn't flush the data to disk until the call to Close(), so it looks like it's not flushing when it actually is.

I think the issue for my original script it I got tired of waiting and cancelled the job, using leveldb it takes about 1.5s, I waited about 15s before giving it the 'ol cntl+c

[missinglink]

is it possible that I'm getting poor performance because I'm using an ext4 filesystem?

[manishrjain]

You should batch your writes. And, do them concurrently to achieve the best performance from Badger. Serial one by one writes are slow. Also, by default the writes are not sync. Which is why you don't see much on disk until close. Sent from Nexus 6P

…

On Jun 28, 2017 11:14 PM, "Peter Johnson" ***@***.***> wrote: is it possible that I'm getting poor performance because I'm using an ext4 filesystem? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#81 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABsyNIqJiyqtw5fBV_1yfdcwsSpHLH4Pks5sIlHEgaJpZM4OH5wN> .

[manishrjain]

Let me know if there's something more I can help you with.

[missinglink]

hey @manishrjain thanks for your reply. I can re-jig my code to use batched writes instead of single writes, but the issue remains that comparing apples v. apples with leveldb, badger is much slower?

are you suggesting that the batch mode of badger is much faster than the batch mode of leveldb?

[manishrjain]

I haven't tried running any benchmarks against leveldb. Also, serial writes in a single goroutine isn't something that we have optimized for. Most practical applications which care about performance would do batched writes, concurrently.