mtbl.Merger returns corrupted values with mtbl.Get

[hdm]

The mtbl.Merger interface returns corrupted values when Get() is called. In the example below, the MTBL contains keys that are domains in reverse order, with the values being a set of related DNS records. This should repro with any MTBL and seems specific to mtbl.Merger (mtbl.Reader works as expected).

We need also need to set GODEBUG=cgocheck=0 to avoid the issue in #4.

$ GODEBUG=cgocheck=0 go run get_merge.go moc.elpmaxe 20161210_dnsrecords_all-names.mtbl
Reader: "moc.elpmaxe" "[[\"aaaa\",\"2606:2800:220:1:248:1893:25c8:1946\"]]"
Merger: "moc.elpmaxe" "\x00s\xb4\x01\x00\x00\x00\x00x\x9b\b2 \u007f\x00\x00800:220:1:248:1893:25c8:1946\"]]"

The source for get_merge.go

package main

import "fmt"
import "os"

import "github.com/edmonds/golang-mtbl"

func mergeFunc(key []byte, val0 []byte, val1 []byte) (mergedVal []byte) {
	return []byte(string(val0) + " + " + string(val1))
}

func main() {
	if len(os.Args) != 3 {
		fmt.Fprintf(os.Stderr, "Usage: %s <KEY> <MTBL FILE>\n", os.Args[0])
		os.Exit(1)
	}

	key := []byte(os.Args[1])
	fname := os.Args[2]


	r, e := mtbl.ReaderInit(fname, &mtbl.ReaderOptions{VerifyChecksums: true})
	defer r.Destroy()
	if e != nil {
		fmt.Fprintf(os.Stderr, "Error: %s\n", e)
		os.Exit(1)
	}



	rval, rok := mtbl.Get(r, key)
	if rok {
		fmt.Printf("Reader: %q %q\n", key, rval)
	}

	m := mtbl.MergerInit(&mtbl.MergerOptions{Merge: mergeFunc})
	m.AddSource(r)
	defer m.Destroy()

	mval, mok := mtbl.Get(m, key)
	if mok {
		fmt.Printf("Merger: %q %q\n", key, mval)
	}

}

[hdm]

This seems to apply to mtbl.Reader() as well, but happens much less frequently. The only workaround is avoiding Get() entirely and moving to Prefix() with manual filtering.

[edmonds]

Hi, HD:

I never really finished golang-mtbl. I started working on it years ago when I first started learning Go, and I'm not really working on much Go code or projects that use mtbl these days.

Feel free to fork this repo if you'd like to improve it, or maybe ping my former colleagues @vixie @merkletrie @bapril to see if they'd like to bring it under https://github.com/farsightsec. I don't think they use golang-mtbl, though.

[hdm]

Thanks Robert! I am still getting used to cgo, but will take a crack at these issues in a fork in the near future.