Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Added some major enhancements #19

Closed
wants to merge 4 commits into from

1 participant

sougou
sougou

I wanted to use BSON for a high performance rpc server in go. If you like my changes so far, I can do some more iterations, and we can even try to get this added to the main go rpc package.

sougou added some commits
sougou sougou Revamp of Unmarshal and alternate implementation for Marshal (Marshal2).
These changes are needed so this package can be used for high performance rpc servers.

- Support for non-struct types for Unmarshal: maps, slices, simple types, and interface{}
- Support for byte array encoding and decoding as packed binary
- An alternate implementation of Marshal that avoids unnecessary object creation and mem copies.

Details:
- I used the panic mechanism to consolidate errors. I can change it to making functions return os.Error if you dislike this scheme.
- Extended structBuilder to support deferred updates for interface objects
- Added isSimple member to structBuilder: go rpc supports simple types. But BSON does not support simple types, I had to hack this
  such that Unmarshal will decode into a simple object as long as there is only one value in the document.
- Added three constructors for structBuilder, to avoid proliferation of PtrValue handling, and TopValueBuilder performs necessary
  structBuilder initialization for a top level object to handle simple types, interfaces, etc.
- Any time an interface{} is encountered, a default object type is created based on the type of the incoming stream.
- In the case of Binary and OID, I directly point the slice into the slice of the input stream. This is an optimization that will avoid
  unnecessary copies, but the input stream will not be garbage collected until the decoded objects go out of scope. I can change
  this to always copy if this will be an issue.
- Created _Binary conforming the BSON interface to support binary encoding
- Improved Parse() function to use bytes.Buffer.Next(). Avoids copying.
- Added a few more tests to bson_test.go. Sorry, it doesn't cover everythin, but it has enough to show that things are generally stable.
- Marshal2() directly decodes the objects into a buffer without any additional allocations or copying, and returns the final byte array.
  I implemented it as a separate function because I didn't know why you built a sytem where you first decode objects into this intermediate
  BSON representation. If this representation is necessary, then Marshal2 will have to be on its own. Otherwise, it can replace most of
  what's in bson.go. It has duplicated code from there. If you like Marshal2, and still want to keep the BSON representation, we can change
  bson.go to reuse functions in Marshal2 to avoid duplication

I know these changes are somewhat radical, and I'm willing to make all necessary cosmetic adjustments if you think they are going in
the right direction.
2995b04
sougou sougou I'm a git noob. Second commit attempt.
Revamp of Unmarshal and alternate implementation for Marshal (Marshal2).
These changes are needed so this package can be used for high performance rpc servers.

- Support for non-struct types for Unmarshal: maps, slices, simple types, and interface{}
- Support for byte array encoding and decoding as packed binary
- An alternate implementation of Marshal that avoids unnecessary object creation and mem copies.

Details:
- I used the panic mechanism to consolidate errors. I can change it to making functions return os.Error if you dislike this scheme.
- Extended structBuilder to support deferred updates for interface objects
- Added isSimple member to structBuilder: go rpc supports simple types. But BSON does not support simple types, I had to hack this such that Unmarshal will decode into a simple object as long as there is only one value in the document.
- Added three constructors for structBuilder, to avoid proliferation of PtrValue handling, and TopValueBuilder performs necessary structBuilder initialization for a top level object to handle simple types, interfaces, etc.
- Any time an interface{} is encountered, a default object type is created based on the type of the incoming stream.
- In the case of Binary and OID, I directly point the slice into the slice of the input stream. This is an optimization that will avoid unnecessary copies, but the input stream will not be garbage collected until the decoded objects go out of scope. I can change this to always copy if this will be an issue.
- Created _Binary conforming the BSON interface to support binary encoding
- Improved Parse() function to use bytes.Buffer.Next(). Avoids copying.
- Added a few more tests to bson_test.go. Sorry, it doesn't cover everythin, but it has enough to show that things are generally stable.
- Marshal2() directly decodes the objects into a buffer without any additional allocations or copying, and returns the final byte array. I implemented it as a separate function because I didn't know why you built a sytem where you first decode objects into this intermediate BSON representation. If this representation is necessary, then Marshal2 will have to be on its own. Otherwise, it can replace most of what's in bson.go. It has duplicated code from there. If you like Marshal2, and still want to keep the BSON representation, we can change bson.go to reuse functions in Marshal2 to avoid duplication

I know these changes are somewhat radical, and I'm willing to make all necessary cosmetic adjustments if you think they are going in the right direction.
dbd0b4e
sougou sougou Fixed typos in bson_test.go da42f6d
sougou sougou - Added streaming counterparts for Marshal & Unmarshal
- Changed Marshal2 to use reflection for simple types because the regular type checking mechanism does not handle named types
6ba506c
sougou sougou closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 7, 2011
  1. sougou

    Revamp of Unmarshal and alternate implementation for Marshal (Marshal2).

    sougou authored
    These changes are needed so this package can be used for high performance rpc servers.
    
    - Support for non-struct types for Unmarshal: maps, slices, simple types, and interface{}
    - Support for byte array encoding and decoding as packed binary
    - An alternate implementation of Marshal that avoids unnecessary object creation and mem copies.
    
    Details:
    - I used the panic mechanism to consolidate errors. I can change it to making functions return os.Error if you dislike this scheme.
    - Extended structBuilder to support deferred updates for interface objects
    - Added isSimple member to structBuilder: go rpc supports simple types. But BSON does not support simple types, I had to hack this
      such that Unmarshal will decode into a simple object as long as there is only one value in the document.
    - Added three constructors for structBuilder, to avoid proliferation of PtrValue handling, and TopValueBuilder performs necessary
      structBuilder initialization for a top level object to handle simple types, interfaces, etc.
    - Any time an interface{} is encountered, a default object type is created based on the type of the incoming stream.
    - In the case of Binary and OID, I directly point the slice into the slice of the input stream. This is an optimization that will avoid
      unnecessary copies, but the input stream will not be garbage collected until the decoded objects go out of scope. I can change
      this to always copy if this will be an issue.
    - Created _Binary conforming the BSON interface to support binary encoding
    - Improved Parse() function to use bytes.Buffer.Next(). Avoids copying.
    - Added a few more tests to bson_test.go. Sorry, it doesn't cover everythin, but it has enough to show that things are generally stable.
    - Marshal2() directly decodes the objects into a buffer without any additional allocations or copying, and returns the final byte array.
      I implemented it as a separate function because I didn't know why you built a sytem where you first decode objects into this intermediate
      BSON representation. If this representation is necessary, then Marshal2 will have to be on its own. Otherwise, it can replace most of
      what's in bson.go. It has duplicated code from there. If you like Marshal2, and still want to keep the BSON representation, we can change
      bson.go to reuse functions in Marshal2 to avoid duplication
    
    I know these changes are somewhat radical, and I'm willing to make all necessary cosmetic adjustments if you think they are going in
    the right direction.
  2. sougou

    I'm a git noob. Second commit attempt.

    sougou authored
    Revamp of Unmarshal and alternate implementation for Marshal (Marshal2).
    These changes are needed so this package can be used for high performance rpc servers.
    
    - Support for non-struct types for Unmarshal: maps, slices, simple types, and interface{}
    - Support for byte array encoding and decoding as packed binary
    - An alternate implementation of Marshal that avoids unnecessary object creation and mem copies.
    
    Details:
    - I used the panic mechanism to consolidate errors. I can change it to making functions return os.Error if you dislike this scheme.
    - Extended structBuilder to support deferred updates for interface objects
    - Added isSimple member to structBuilder: go rpc supports simple types. But BSON does not support simple types, I had to hack this such that Unmarshal will decode into a simple object as long as there is only one value in the document.
    - Added three constructors for structBuilder, to avoid proliferation of PtrValue handling, and TopValueBuilder performs necessary structBuilder initialization for a top level object to handle simple types, interfaces, etc.
    - Any time an interface{} is encountered, a default object type is created based on the type of the incoming stream.
    - In the case of Binary and OID, I directly point the slice into the slice of the input stream. This is an optimization that will avoid unnecessary copies, but the input stream will not be garbage collected until the decoded objects go out of scope. I can change this to always copy if this will be an issue.
    - Created _Binary conforming the BSON interface to support binary encoding
    - Improved Parse() function to use bytes.Buffer.Next(). Avoids copying.
    - Added a few more tests to bson_test.go. Sorry, it doesn't cover everythin, but it has enough to show that things are generally stable.
    - Marshal2() directly decodes the objects into a buffer without any additional allocations or copying, and returns the final byte array. I implemented it as a separate function because I didn't know why you built a sytem where you first decode objects into this intermediate BSON representation. If this representation is necessary, then Marshal2 will have to be on its own. Otherwise, it can replace most of what's in bson.go. It has duplicated code from there. If you like Marshal2, and still want to keep the BSON representation, we can change bson.go to reuse functions in Marshal2 to avoid duplication
    
    I know these changes are somewhat radical, and I'm willing to make all necessary cosmetic adjustments if you think they are going in the right direction.
Commits on Feb 8, 2011
  1. sougou

    Fixed typos in bson_test.go

    sougou authored
Commits on Feb 10, 2011
  1. sougou

    - Added streaming counterparts for Marshal & Unmarshal

    sougou authored
    - Changed Marshal2 to use reflection for simple types because the regular type checking mechanism does not handle named types
This page is out of date. Refresh to see the latest.
1  mongo/Makefile
View
@@ -11,6 +11,7 @@ GOFILES=\
message.go\
bson.go\
bson-struct.go\
+ marshal2.go\
include $(GOROOT)/src/Make.pkg
356 mongo/bson-struct.go
View
@@ -8,6 +8,7 @@
package mongo
import (
+ "io"
"reflect"
"strings"
"fmt"
@@ -18,160 +19,242 @@ import (
"strconv"
)
+type bsonError struct {
+ msg string
+}
+
+func NewBsonError(format string, args ...interface{}) bsonError {
+ return bsonError{fmt.Sprintf(format, args...)}
+}
+
+func (self bsonError) String() string {
+ return self.msg
+}
+
+// Maps & interface values will not give you a reference to their underlying object.
+// You can only update them through their Set methods.
type structBuilder struct {
val reflect.Value
- // if map_ != nil, write val to map_[key] on each change
+ // isSimple == bool: It's the top level structBuilder object and it has a simple value
+ isSimple bool
+
+ // if map_ != nil, write val to map_[key] when val is finalized, performed by Flush()
map_ *reflect.MapValue
key reflect.Value
-}
-var nobuilder *structBuilder
+ // if interface_ != nil, write val to interface_ when val is finalized. Performed by Flush()
+ interface_ *reflect.InterfaceValue
+}
-func isfloat(v reflect.Value) bool {
- switch v.(type) {
- case *reflect.FloatValue:
- return true
+func NewStructBuilder(val reflect.Value) *structBuilder {
+ // Dereference pointers here so we don't have to handle this case everywhere else
+ if v, ok := val.(*reflect.PtrValue); ok {
+ if v.IsNil() {
+ v.PointTo(reflect.MakeZero(v.Type().(*reflect.PtrType).Elem()))
+ }
+ val = v.Elem()
}
- return false
+ return &structBuilder{val: val}
}
-func setfloat(v reflect.Value, f float64) {
- switch v := v.(type) {
- case *reflect.FloatValue:
- v.Set(f)
+func MapValueBuilder(val reflect.Value, map_ *reflect.MapValue, key reflect.Value) *structBuilder {
+ if v, ok := val.(*reflect.PtrValue); ok {
+ if v.IsNil() {
+ v.PointTo(reflect.MakeZero(v.Type().(*reflect.PtrType).Elem()))
+ }
+ map_.SetElem(key, v)
+ val = v.Elem()
+ return &structBuilder{val: val}
}
+ return &structBuilder{val: val, map_: map_, key: key}
}
-func setint(v reflect.Value, i int64) {
- switch v := v.(type) {
- case *reflect.IntValue:
- v.Set(i)
- case *reflect.UintValue:
- v.Set(uint64(i))
- }
+// Returns a valid unmarshalable structBuilder or an error
+func TopLevelBuilder(val interface{}) (sb *structBuilder, err os.Error) {
+ ival := reflect.NewValue(val)
+ v, ok := ival.(*reflect.PtrValue)
+ if !ok {
+ return nil, os.NewError(fmt.Sprintf("expecting pointer value, received %v", ival.Type()))
+ }
+ // We'll allow one level of indirection
+ switch actual := v.Elem().(type) {
+ case *reflect.FloatValue, *reflect.StringValue, *reflect.BoolValue,
+ *reflect.IntValue, *reflect.SliceValue, *reflect.ArrayValue:
+ sb := NewStructBuilder(actual)
+ sb.isSimple = true // Prepare to receive a simple value
+ return sb, nil
+ case *reflect.MapValue, *reflect.StructValue, *reflect.InterfaceValue:
+ sb := NewStructBuilder(actual)
+ sb.Object() // Allocate memory if necessary
+ return sb, nil
+ }
+ return nil, os.NewError(fmt.Sprintf("unrecognized type %v", ival.Type()))
}
-// If updating self.val is not enough to update the original,
-// copy a changed self.val out to the original.
+// Flush handles the final update for map & interface objects.
func (self *structBuilder) Flush() {
- if self == nil {
- return
+ if self.interface_ != nil {
+ self.interface_.Set(self.val)
}
if self.map_ != nil {
- self.map_.SetElem(self.key, self.val)
+ if self.interface_ != nil {
+ self.map_.SetElem(self.key, self.interface_)
+ } else {
+ self.map_.SetElem(self.key, self.val)
+ }
}
}
+// Defer update if it's an interface, handled by Flush().
+func (self *structBuilder) DeferSet(val reflect.Value) {
+ self.interface_ = self.val.(*reflect.InterfaceValue)
+ self.val = val
+}
+
func (self *structBuilder) Int64(i int64) {
- if self == nil {
- return
- }
- v := self.val
- if isfloat(v) {
- setfloat(v, float64(i))
- } else {
- setint(v, i)
+ switch v := self.val.(type) {
+ case *reflect.IntValue:
+ v.Set(i)
+ case *reflect.UintValue:
+ v.Set(uint64(i))
+ case *reflect.FloatValue:
+ v.Set(float64(i))
+ case *reflect.InterfaceValue:
+ self.DeferSet(reflect.NewValue(i))
+ default:
+ panic(NewBsonError("unable to convert int64 %v to %s", i, self.val.Type()))
}
}
func (self *structBuilder) Date(t *time.Time) {
- if self == nil {
- return
- }
- if v, ok := self.val.(*reflect.PtrValue); ok {
- v.PointTo(reflect.Indirect(reflect.NewValue(t)))
+ switch v := self.val.(type) {
+ case *reflect.StructValue:
+ v.SetValue(reflect.NewValue(*t))
+ case *reflect.InterfaceValue:
+ self.DeferSet(reflect.NewValue(*t))
+ default:
+ panic(NewBsonError("unable to convert time %v to %s", *t, self.val.Type()))
}
}
func (self *structBuilder) Int32(i int32) {
- if self == nil {
- return
- }
- v := self.val
- if isfloat(v) {
- setfloat(v, float64(i))
- } else {
- setint(v, int64(i))
+ switch v := self.val.(type) {
+ case *reflect.IntValue:
+ v.Set(int64(i))
+ case *reflect.UintValue:
+ v.Set(uint64(uint32(i)))
+ case *reflect.FloatValue:
+ v.Set(float64(i))
+ case *reflect.InterfaceValue:
+ self.DeferSet(reflect.NewValue(i))
+ default:
+ panic(NewBsonError("unable to convert int32 %v to %s", i, self.val.Type()))
}
}
func (self *structBuilder) Float64(f float64) {
- if self == nil {
- return
- }
- v := self.val
- if isfloat(v) {
- setfloat(v, f)
- } else {
- setint(v, int64(f))
+ switch v := self.val.(type) {
+ case *reflect.IntValue:
+ v.Set(int64(f))
+ case *reflect.UintValue:
+ v.Set(uint64(f))
+ case *reflect.FloatValue:
+ v.Set(f)
+ case *reflect.InterfaceValue:
+ self.DeferSet(reflect.NewValue(f))
+ default:
+ panic(NewBsonError("unable to convert float64 %v to %s", f, self.val.Type()))
}
}
func (self *structBuilder) Null() {}
func (self *structBuilder) String(s string) {
- if self == nil {
- return
- }
- if v, ok := self.val.(*reflect.StringValue); ok {
+ switch v := self.val.(type) {
+ case *reflect.StringValue:
v.Set(s)
+ case *reflect.InterfaceValue:
+ self.DeferSet(reflect.NewValue(s))
+ default:
+ panic(NewBsonError("unable to convert string %v to %s", s, self.val.Type()))
}
}
func (self *structBuilder) Regex(regex, options string) {
- // Ignore options for now...
- if self == nil {
- return
- }
- if v, ok := self.val.(*reflect.StringValue); ok {
- v.Set(regex)
- }
+ // No special treatment
+ self.String(regex)
}
func (self *structBuilder) Bool(tf bool) {
- if self == nil {
- return
- }
- if v, ok := self.val.(*reflect.BoolValue); ok {
+ switch v := self.val.(type) {
+ case *reflect.BoolValue:
v.Set(tf)
+ case *reflect.InterfaceValue:
+ self.DeferSet(reflect.NewValue(tf))
+ default:
+ panic(NewBsonError("unable to convert bool %v to %s", tf, self.val.Type()))
}
}
func (self *structBuilder) OID(oid []byte) {
- if self == nil {
- return
- }
- if v, ok := self.val.(*reflect.SliceValue); ok {
- if v.Cap() < 12 {
- nv := reflect.MakeSlice(v.Type().(*reflect.SliceType), 12, 12)
- v.Set(nv)
- }
- for i := 0; i < 12; i++ {
- v.Elem(i).(*reflect.UintValue).Set(uint64(oid[i]))
- }
- }
+ self.Binary(oid)
}
func (self *structBuilder) Array() {
- if self == nil {
- return
- }
- if v, ok := self.val.(*reflect.SliceValue); ok {
+ switch v := self.val.(type) {
+ case *reflect.ArrayValue:
+ // no op
+ case *reflect.SliceValue:
if v.IsNil() {
v.Set(reflect.MakeSlice(v.Type().(*reflect.SliceType), 0, 8))
}
+ case *reflect.InterfaceValue:
+ self.DeferSet(reflect.NewValue(make([]interface{}, 0, 8)))
+ default:
+ panic(NewBsonError("unable to convert array to %s", self.val.Type()))
+ }
+}
+
+func (self *structBuilder) Binary(bindata []byte) {
+ switch v := self.val.(type) {
+ case *reflect.ArrayValue:
+ if v.Cap() < len(bindata) {
+ panic(NewBsonError("insufficient space in array. Have: %v, Need: %v", v.Cap(), len(bindata)))
+ }
+ for i := 0; i < len(bindata); i++ {
+ v.Elem(i).(*reflect.UintValue).Set(uint64(bindata[i]))
+ }
+ case *reflect.SliceValue:
+ if v.IsNil() {
+ // Just point it to the bindata object
+ v.SetValue(reflect.NewValue(bindata))
+ return
+ }
+ if v.Cap() < len(bindata) {
+ nv := reflect.MakeSlice(v.Type().(*reflect.SliceType), len(bindata), len(bindata))
+ v.Set(nv)
+ }
+ for i := 0; i < len(bindata); i++ {
+ v.Elem(i).(*reflect.UintValue).Set(uint64(bindata[i]))
+ }
+ case *reflect.InterfaceValue:
+ self.DeferSet(reflect.NewValue(bindata))
+ default:
+ panic(NewBsonError("unable to convert oid %v to %s", bindata, self.val.Type()))
}
}
func (self *structBuilder) Elem(i int) Builder {
- if self == nil || i < 0 {
- return nobuilder
+ if i < 0 {
+ panic(NewBsonError("negative index %v for array element", i))
}
switch v := self.val.(type) {
case *reflect.ArrayValue:
if i < v.Len() {
- return &structBuilder{val: v.Elem(i)}
+ return NewStructBuilder(v.Elem(i))
+ } else {
+ panic(NewBsonError("array index %v out of bounds", i))
}
case *reflect.SliceValue:
if i > v.Cap() {
@@ -190,41 +273,38 @@ func (self *structBuilder) Elem(i int) Builder {
v.SetLen(i + 1)
}
if i < v.Len() {
- return &structBuilder{val: v.Elem(i)}
+ return NewStructBuilder(v.Elem(i))
+ } else {
+ panic(NewBsonError("internal error, realloc failed?"))
}
}
- return nobuilder
+ panic(NewBsonError("unexpected type %s, expecting slice or array", self.val.Type()))
}
func (self *structBuilder) Object() {
- if self == nil {
- return
- }
- if v, ok := self.val.(*reflect.PtrValue); ok && v.IsNil() {
+ switch v := self.val.(type) {
+ case *reflect.MapValue:
if v.IsNil() {
- v.PointTo(reflect.MakeZero(v.Type().(*reflect.PtrType).Elem()))
- self.Flush()
+ v.Set(reflect.MakeMap(v.Type().(*reflect.MapType)))
}
- self.map_ = nil
- self.val = v.Elem()
- }
- if v, ok := self.val.(*reflect.MapValue); ok && v.IsNil() {
- v.Set(reflect.MakeMap(v.Type().(*reflect.MapType)))
+ case *reflect.StructValue:
+ // no op
+ case *reflect.InterfaceValue:
+ self.DeferSet(reflect.NewValue(make(map[string]interface{})))
+ default:
+ panic(NewBsonError("unexpected type %s, expecting composite type", self.val.Type()))
}
}
func (self *structBuilder) Key(k string) Builder {
- if self == nil {
- return nobuilder
- }
- switch v := reflect.Indirect(self.val).(type) {
+ switch v := self.val.(type) {
case *reflect.StructValue:
t := v.Type().(*reflect.StructType)
// Case-insensitive field lookup.
k = strings.ToLower(k)
for i := 0; i < t.NumField(); i++ {
if strings.ToLower(t.Field(i).Name) == k {
- return &structBuilder{val: v.Field(i)}
+ return NewStructBuilder(v.Field(i))
}
}
case *reflect.MapValue:
@@ -238,37 +318,63 @@ func (self *structBuilder) Key(k string) Builder {
v.SetElem(key, reflect.MakeZero(t.Elem()))
elem = v.Elem(key)
}
- return &structBuilder{val: elem, map_: v, key: key}
- case *reflect.SliceValue:
+ return MapValueBuilder(elem, v, key)
+ case *reflect.SliceValue, *reflect.ArrayValue:
+ if self.isSimple {
+ self.isSimple = false
+ return self
+ }
index, err := strconv.Atoi(k)
if err != nil {
- return nobuilder
- }
- if index < v.Len() {
- return &structBuilder{val: v.Elem(index)}
- }
- if index < v.Cap() {
- v.SetLen(index + 1)
- return &structBuilder{val: v.Elem(index)}
+ panic(bsonError{err.String()})
}
- newCap := v.Cap() * 2
- if index >= newCap {
- newCap = index*2 + 1
+ return self.Elem(index)
+ case *reflect.FloatValue, *reflect.StringValue, *reflect.BoolValue, *reflect.IntValue:
+ // Special case. We're unmarshaling into a simple type.
+ if self.isSimple {
+ self.isSimple = false
+ return self
}
- temp := reflect.MakeSlice(v.Type().(*reflect.SliceType), index+1, newCap)
- reflect.Copy(temp, v)
- v.Set(temp)
- return &structBuilder{val: v.Elem(index)}
}
- return nobuilder
+ panic(NewBsonError("%s not supported as a BSON document", self.val.Type()))
}
func Unmarshal(b []byte, val interface{}) (err os.Error) {
- sb := &structBuilder{val: reflect.NewValue(val)}
+ sb, terr := TopLevelBuilder(val)
+ if terr != nil {
+ return terr
+ }
err = Parse(bytes.NewBuffer(b[4:len(b)]), sb)
+ sb.Flush()
return
}
+func UnmarshalFromStream(reader io.Reader, val interface{}) (err os.Error) {
+ lenbuf := make([]byte, 4)
+ var n int
+ n, err = reader.Read(lenbuf)
+ if err != nil {
+ return err
+ }
+ if n != 4 {
+ return io.ErrUnexpectedEOF
+ }
+ length := pack.Uint32(lenbuf)
+ buf := make([]byte, length)
+ pack.PutUint32(buf, length)
+ n, err = reader.Read(buf[4:])
+ if err != nil {
+ if err == os.EOF {
+ return io.ErrUnexpectedEOF
+ }
+ return err
+ }
+ if n != int(length-4) {
+ return io.ErrUnexpectedEOF
+ }
+ return Unmarshal(buf, val)
+}
+
func Marshal(val interface{}) (BSON, os.Error) {
if val == nil {
return Null, nil
@@ -289,6 +395,8 @@ func Marshal(val interface{}) (BSON, os.Error) {
return &_Long{int64(v), _Null{}}, nil
case *time.Time:
return &_Date{v, _Null{}}, nil
+ case []byte:
+ return &_Binary{v, _Null{}}, nil
}
var value reflect.Value
56 mongo/bson.go
View
@@ -9,8 +9,6 @@ package mongo
import (
"os"
- "io"
- "io/ioutil"
"fmt"
"math"
"time"
@@ -185,6 +183,23 @@ func (self *_Array) Bytes() []byte {
return append(w32, buf.Bytes()...)
}
+type _Binary struct {
+ value []byte
+ _Null
+}
+
+func (self *_Binary) Kind() int { return BinaryKind }
+func (self *_Binary) Bytes() []byte {
+ w32 := make([]byte, _WORD32)
+ pack.PutUint32(w32, uint32(len(self.value)))
+
+ buf := bytes.NewBuffer(w32)
+ buf.WriteByte(0)
+ buf.Write(self.value)
+
+ return buf.Bytes()
+}
+
type _OID struct {
value []byte
_Null
@@ -331,6 +346,7 @@ type Builder interface {
Null()
Object()
Array()
+ Binary(bytes []byte)
// Create sub-Builders
Key(s string) Builder
@@ -377,6 +393,7 @@ func (self *_BSONBuilder) Float64(f float64) { self.Put(&_Number{f, _Null{}}) }
func (self *_BSONBuilder) String(s string) { self.Put(&_String{s, _Null{}}) }
func (self *_BSONBuilder) Object() { self.Put(&_Object{make(map[string]BSON), _Null{}}) }
func (self *_BSONBuilder) Array() { self.Put(&_Array{new(vector.Vector), _Null{}}) }
+func (self *_BSONBuilder) Binary(b []byte) { self.Put(&_Binary{b, _Null{}}) }
func (self *_BSONBuilder) Bool(b bool) { self.Put(&_Boolean{b, _Null{}}) }
func (self *_BSONBuilder) Date(t *time.Time) { self.Put(&_Date{t, _Null{}}) }
func (self *_BSONBuilder) Null() { self.Put(Null) }
@@ -437,6 +454,12 @@ func readCString(buf *bytes.Buffer) string {
}
func Parse(buf *bytes.Buffer, builder Builder) (err os.Error) {
+ defer func() {
+ if x := recover(); x != nil {
+ err = x.(bsonError)
+ }
+ }()
+
kind, _ := buf.ReadByte()
err = nil
@@ -452,27 +475,28 @@ func Parse(buf *bytes.Buffer, builder Builder) (err os.Error) {
switch kind {
case NumberKind:
- lr := io.LimitReader(buf, 8)
- bits, _ := ioutil.ReadAll(lr)
- ui64 := pack.Uint64(bits)
+ ui64 := pack.Uint64(buf.Next(8))
fl64 := math.Float64frombits(ui64)
b2.Float64(fl64)
case StringKind:
- bits, _ := ioutil.ReadAll(io.LimitReader(buf, 4))
- l := pack.Uint32(bits)
- s, _ := ioutil.ReadAll(io.LimitReader(buf, int64(l-1)))
+ l := int(pack.Uint32(buf.Next(4)))
+ s := buf.Next(l - 1)
buf.ReadByte()
b2.String(string(s))
case ObjectKind:
b2.Object()
- ioutil.ReadAll(io.LimitReader(buf, 4))
+ buf.Next(4)
err = Parse(buf, b2)
case ArrayKind:
b2.Array()
- ioutil.ReadAll(io.LimitReader(buf, 4))
+ buf.Next(4)
err = Parse(buf, b2)
+ case BinaryKind:
+ l := int(pack.Uint32(buf.Next(4)))
+ buf.Next(1) // Skip the subtype, we don't care
+ b2.Binary(buf.Next(l))
case OIDKind:
- oid, _ := ioutil.ReadAll(io.LimitReader(buf, 12))
+ oid := buf.Next(12)
b2.OID(oid)
case BooleanKind:
b, _ := buf.ReadByte()
@@ -482,24 +506,22 @@ func Parse(buf *bytes.Buffer, builder Builder) (err os.Error) {
b2.Bool(false)
}
case DateKind:
- bits, _ := ioutil.ReadAll(io.LimitReader(buf, 8))
- ui64 := pack.Uint64(bits)
+ ui64 := pack.Uint64(buf.Next(8))
b2.Date(time.SecondsToUTC(int64(ui64) / 1000))
case RegexKind:
regex := readCString(buf)
options := readCString(buf)
b2.Regex(regex, options)
case IntKind:
- bits, _ := ioutil.ReadAll(io.LimitReader(buf, 4))
- ui32 := pack.Uint32(bits)
+ ui32 := pack.Uint32(buf.Next(4))
b2.Int32(int32(ui32))
case LongKind:
- bits, _ := ioutil.ReadAll(io.LimitReader(buf, 8))
- ui64 := pack.Uint64(bits)
+ ui64 := pack.Uint64(buf.Next(8))
b2.Int64(int64(ui64))
default:
err = os.NewError(fmt.Sprintf("don't know how to handle kind %v yet", kind))
}
+ b2.Flush()
kind, _ = buf.ReadByte()
}
110 mongo/bson_test.go
View
@@ -1,4 +1,3 @@
-// Copyright 2009-2011 The gomongo Authors. All rights reserved.
// Use of this source code is governed by the 3-clause BSD License
// that can be found in the LICENSE file.
@@ -23,11 +22,11 @@ type OtherStruct struct {
}
type ExampleStruct struct {
- First int32
- Second float64
- Third string
- Fourth EmptyStruct
- Fifth OtherStruct
+ First int32
+ Second float64
+ Third string
+ Fourth EmptyStruct
+ Fifth OtherStruct
}
type ExampleWithId struct {
@@ -100,3 +99,102 @@ func TestMarshal(t *testing.T) {
Unmarshal(bs2.Bytes(), es2)
assertTrue(es2.Date.Seconds() == d.Seconds(), "date unmarshal", t)
}
+
+func TestVariety(t *testing.T) {
+ in := map[string]string{"val": "test"}
+ encoded := VerifyMarshal2(t, in)
+ expected := []byte("\x13\x00\x00\x00\x02val\x00\x05\x00\x00\x00test\x00\x00")
+ compare(t, encoded, expected)
+
+ out := make(map[string]interface{})
+ err := Unmarshal(encoded, &out)
+ if in["val"] != out["val"] {
+ t.Errorf("unmarshal doesn't match input: %v\n%v\n%v\n", err, in, out)
+ }
+
+ var out1 string
+ err = Unmarshal(encoded, &out1)
+ if out1 != "test" {
+ t.Errorf("unmarshal doesn't match input: %v\n%v\n%v\n", err, in, out1)
+ }
+
+ var out2 interface{}
+ err = Unmarshal(encoded, &out2)
+ if out2.(map[string]interface{})["val"].(string) != "test" {
+ t.Errorf("unmarshal doesn't match input: %v\n%v\n%v\n", err, in, out1)
+ }
+
+ type mystruct struct {
+ Val string
+ }
+ var out3 mystruct
+ err = Unmarshal(encoded, &out3)
+ if out3.Val != "test" {
+ t.Errorf("unmarshal doesn't match input: %v\n%v\n%v\n", err, in, out1)
+ }
+}
+
+func TestBinary(t *testing.T) {
+ in := map[string][]byte{"val": []byte("test")}
+ encoded := VerifyMarshal2(t, in)
+ expected := []byte("\x13\x00\x00\x00\x05val\x00\x04\x00\x00\x00\x00test\x00")
+ compare(t, encoded, expected)
+
+ out := make(map[string]interface{})
+ err := Unmarshal(encoded, &out)
+ if string(out["val"].([]byte)) != "test" {
+ t.Errorf("unmarshal doesn't match input: %v\n%v\n%v\n", err, in, out)
+ }
+
+ var out1 []byte
+ err = Unmarshal(encoded, &out1)
+ if string(out1) != "test" {
+ t.Errorf("unmarshal doesn't match input: %v\n%v\n%v\n", err, in, out1)
+ }
+}
+
+func TestInt(t *testing.T) {
+ in := map[string]int{"val": 20}
+ encoded := VerifyMarshal2(t, in)
+ expected := []byte("\x12\x00\x00\x00\x12val\x00\x14\x00\x00\x00\x00\x00\x00\x00\x00")
+ compare(t, encoded, expected)
+
+ out := make(map[string]interface{})
+ err := Unmarshal(encoded, &out)
+ if out["val"].(int64) != 20 {
+ t.Errorf("unmarshal doesn't match input: %v\n%v\n%v\n", err, in, out)
+ }
+
+ var out1 int
+ err = Unmarshal(encoded, &out1)
+ if out1 != 20 {
+ t.Errorf("unmarshal doesn't match input: %v\n%v\n%vn", err, in, out1)
+ }
+}
+
+func VerifyMarshal2(t *testing.T, val interface{}) []byte {
+ bs, err := Marshal(val)
+ if err != nil {
+ t.Errorf("marshal error: %s\n", err)
+ }
+ expected := bs.Bytes()
+ encoded, err2 := Marshal2(val)
+ if err2 != nil {
+ t.Errorf("marshal2 error: %s\n", err2)
+ }
+ compare(t, encoded, expected)
+ return expected
+}
+
+func compare(t *testing.T, encoded []byte, expected []byte) {
+ if len(encoded) != len(expected) {
+ t.Errorf("encoding mismatch:\n%#v\n%#v\n", string(encoded), string(expected))
+ } else {
+ for i, _ := range encoded {
+ if encoded[i] != expected[i] {
+ t.Errorf("encoding mismatch:\n%#v\n%#v\n", string(encoded), string(expected))
+ break
+ }
+ }
+ }
+}
224 mongo/marshal2.go
View
@@ -0,0 +1,224 @@
+// Copyright 2009 The Go Authors. All rights reserved.
+// Copyright 2009,2010, The 'gomongo' Authors. All rights reserved.
+// Use of this source code is governed by the 3-clause BSD License
+// that can be found in the LICENSE and LICENSE.GO files.
+
+package mongo
+
+import (
+ "io"
+ "os"
+ "math"
+ "reflect"
+ "strings"
+ "time"
+ "bytes"
+ "strconv"
+)
+
+type SimpleContainer struct {
+ Val interface{}
+}
+
+// LenWriter records the current write postion on the buffer
+// and can later be used to recor the number of bytes written
+// in conformance to BSON spec
+type LenWriter struct {
+ buf *bytes.Buffer
+ len_offset int
+}
+
+func NewLenWriter(buf *bytes.Buffer) *LenWriter {
+ len_offset := len(buf.Bytes())
+ w32 := make([]byte, _WORD32)
+ buf.Write(w32)
+ return &LenWriter{buf, len_offset}
+}
+
+func (self *LenWriter) RecordLen() {
+ buf := self.buf.Bytes()
+ final_len := len(buf)
+ w32 := buf[self.len_offset : self.len_offset+_WORD32]
+ pack.PutUint32(w32, uint32(final_len-self.len_offset))
+}
+
+func MarshalToStream(writer io.Writer, val interface{}) (err os.Error) {
+ var encoded []byte
+ encoded, err = Marshal2(val)
+ if err != nil {
+ return err
+ }
+ _, err = writer.Write(encoded)
+ return err
+}
+
+func Marshal2(val interface{}) (encoded []byte, err os.Error) {
+ defer func() {
+ if x := recover(); x != nil {
+ err = x.(bsonError)
+ }
+ }()
+
+ if val == nil {
+ return nil, os.NewError("Cannot marshal empty object")
+ }
+
+ // Dereference pointer types
+ switch v := reflect.NewValue(val).(type) {
+ case *reflect.PtrValue:
+ val = v.Elem().Interface()
+ }
+
+ buf := bytes.NewBuffer(make([]byte, 0, 32))
+ switch fv := reflect.NewValue(val).(type) {
+ case *reflect.FloatValue, *reflect.StringValue, *reflect.BoolValue,
+ *reflect.IntValue, *reflect.UintValue, *reflect.SliceValue, *reflect.ArrayValue:
+ // Wrap simple types in a container
+ val = SimpleContainer{fv.Interface()}
+ EncodeStruct(buf, reflect.NewValue(val).(*reflect.StructValue))
+ case *reflect.StructValue:
+ EncodeStruct(buf, fv)
+ case *reflect.MapValue:
+ EncodeMap(buf, fv)
+ default:
+ panic(NewBsonError("Unexpected type %v\n", fv.Type()))
+ }
+ return buf.Bytes(), err
+}
+
+func EncodeField(buf *bytes.Buffer, key string, val interface{}) {
+ // MongoDB uses '_id' as the primary key, but this
+ // name is private in Go. Use 'Id_' for this purpose
+ // instead.
+ if key == "id_" {
+ key = "_id"
+ }
+ switch v := val.(type) {
+ case time.Time:
+ EncodePrefix(buf, '\x11', key)
+ EncodeTime(buf, v)
+ case []byte:
+ EncodePrefix(buf, '\x05', key)
+ EncodeBinary(buf, v)
+ default:
+ goto CompositeType
+ }
+ return
+
+CompositeType:
+ switch fv := reflect.NewValue(val).(type) {
+ case *reflect.FloatValue:
+ EncodePrefix(buf, '\x01', key)
+ EncodeFloat64(buf, fv.Get())
+ case *reflect.StringValue:
+ EncodePrefix(buf, '\x02', key)
+ EncodeString(buf, fv.Get())
+ case *reflect.BoolValue:
+ EncodePrefix(buf, '\x08', key)
+ EncodeBool(buf, fv.Get())
+ case *reflect.IntValue:
+ EncodePrefix(buf, '\x12', key)
+ EncodeUint64(buf, uint64(fv.Get()))
+ case *reflect.UintValue:
+ EncodePrefix(buf, '\x12', key)
+ EncodeUint64(buf, fv.Get())
+ case *reflect.StructValue:
+ EncodePrefix(buf, '\x03', key)
+ EncodeStruct(buf, fv)
+ case *reflect.MapValue:
+ EncodePrefix(buf, '\x03', key)
+ EncodeMap(buf, fv)
+ case *reflect.SliceValue:
+ EncodePrefix(buf, '\x04', key)
+ EncodeSlice(buf, fv)
+ case *reflect.PtrValue:
+ EncodeField(buf, key, fv.Elem().Interface())
+ default:
+ panic(NewBsonError("don't know how to marshal %v\n", reflect.NewValue(val).Type()))
+ }
+}
+
+func EncodePrefix(buf *bytes.Buffer, etype byte, key string) {
+ buf.WriteByte(etype)
+ buf.WriteString(key)
+ buf.WriteByte(0)
+}
+
+func EncodeFloat64(buf *bytes.Buffer, val float64) {
+ bits := math.Float64bits(val)
+ w64 := make([]byte, _WORD64)
+ pack.PutUint64(w64, bits)
+ buf.Write(w64)
+}
+
+func EncodeString(buf *bytes.Buffer, val string) {
+ w32 := make([]byte, _WORD32)
+ pack.PutUint32(w32, uint32(len(val)+1))
+ buf.Write(w32)
+ buf.WriteString(val)
+ buf.WriteByte(0)
+}
+
+func EncodeBool(buf *bytes.Buffer, val bool) {
+ if val {
+ buf.WriteByte(1)
+ } else {
+ buf.WriteByte(0)
+ }
+}
+
+func EncodeUint64(buf *bytes.Buffer, val uint64) {
+ w64 := make([]byte, _WORD64)
+ pack.PutUint64(w64, val)
+ buf.Write(w64)
+}
+
+func EncodeTime(buf *bytes.Buffer, val time.Time) {
+ w64 := make([]byte, _WORD64)
+ mtime := val.Seconds() * 1000
+ pack.PutUint64(w64, uint64(mtime))
+ buf.Write(w64)
+}
+
+func EncodeBinary(buf *bytes.Buffer, val []byte) {
+ w32 := make([]byte, _WORD32)
+ pack.PutUint32(w32, uint32(len(val)))
+ buf.Write(w32)
+ buf.WriteByte(0)
+ buf.Write(val)
+}
+
+func EncodeStruct(buf *bytes.Buffer, val *reflect.StructValue) {
+ lenWriter := NewLenWriter(buf)
+ t := val.Type().(*reflect.StructType)
+ for i := 0; i < t.NumField(); i++ {
+ key := strings.ToLower(t.Field(i).Name)
+ EncodeField(buf, key, val.Field(i).Interface())
+ }
+ buf.WriteByte(0)
+ lenWriter.RecordLen()
+}
+
+func EncodeMap(buf *bytes.Buffer, val *reflect.MapValue) {
+ lenWriter := NewLenWriter(buf)
+ mt := val.Type().(*reflect.MapType)
+ if mt.Key() != reflect.Typeof("") {
+ panic(NewBsonError("can't marshall maps with non-string key types"))
+ }
+ keys := val.Keys()
+ for _, k := range keys {
+ key := k.(*reflect.StringValue).Get()
+ EncodeField(buf, key, val.Elem(k).Interface())
+ }
+ buf.WriteByte(0)
+ lenWriter.RecordLen()
+}
+
+func EncodeSlice(buf *bytes.Buffer, val *reflect.SliceValue) {
+ lenWriter := NewLenWriter(buf)
+ for i := 0; i < val.Len(); i++ {
+ EncodeField(buf, strconv.Itoa(i), val.Elem(i).Interface())
+ }
+ buf.WriteByte(0)
+ lenWriter.RecordLen()
+}
Something went wrong with that request. Please try again.