Switch branches/tags
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
..
Failed to load latest commit information.
codecgen codecgen: fix decStruct to handle all variations (pointers, fields, s… Dec 31, 2017
0doc.go codec: update that mid-stack linling is expected in go 1.11 developme… Feb 22, 2018
README.md codec: support mapping non-interface type to an interface Dec 25, 2017
binc.go codec: use decReader and encWriter interfaces (not the struct decRead… Jul 29, 2018
build.sh codec: build.sh: support flag for inlining diagnostics Jul 29, 2018
cbor.go codec: use decReader and encWriter interfaces (not the struct decRead… Jul 29, 2018
cbor_test.go codec: make tests more robust Jun 27, 2018
codec_test.go codec: rpc tests should listen on ipv4 localhost only. Jun 28, 2018
decode.go codec: use decReader and encWriter interfaces (not the struct decRead… Jul 29, 2018
encode.go codec: use decReader and encWriter interfaces (not the struct decRead… Jul 29, 2018
fast-path.generated.go codec: clean up, improved symbol handling and numeric overflow support Dec 12, 2017
fast-path.go.tmpl codec: clean up, improved symbol handling and numeric overflow support Dec 12, 2017
fast-path.not.go codec: re-factor and streamline code for publishing standards, consis… Dec 24, 2017
gen-dec-array.go.tmpl codec: streamline chan encoding/decoding Feb 23, 2018
gen-dec-map.go.tmpl codec: streamline library: refactor sendContainerState and clean up h… Oct 18, 2017
gen-enc-chan.go.tmpl codec: streamline chan encoding/decoding Feb 23, 2018
gen-helper.generated.go codec: if json and the struct field name (or encode name) is ascii an… Jul 28, 2018
gen-helper.go.tmpl codec: if json and the struct field name (or encode name) is ascii an… Jul 28, 2018
gen.generated.go codec: streamline chan encoding/decoding Feb 23, 2018
gen.go codec: if json and the struct field name (or encode name) is ascii an… Jul 28, 2018
goversion_arrayof_gte_go15.go codec: update copyright to say 2012-2018 Dec 10, 2017
goversion_arrayof_lt_go15.go codec: update copyright to say 2012-2018 Dec 10, 2017
goversion_makemap_gte_go19.go codec: update copyright to say 2012-2018 Dec 10, 2017
goversion_makemap_lt_go19.go codec: update copyright to say 2012-2018 Dec 10, 2017
goversion_unexportedembeddedptr_gte_go110.go codec: update copyright to say 2012-2018 Dec 10, 2017
goversion_unexportedembeddedptr_lt_go110.go codec: update copyright to say 2012-2018 Dec 10, 2017
goversion_unsupported_lt_go14.go codec: update copyright to say 2012-2018 Dec 10, 2017
goversion_vendor_eq_go15.go codec: update copyright to say 2012-2018 Dec 10, 2017
goversion_vendor_eq_go16.go codec: update copyright to say 2012-2018 Dec 10, 2017
goversion_vendor_gte_go17.go codec: update copyright to say 2012-2018 Dec 10, 2017
goversion_vendor_lt_go15.go codec: update copyright to say 2012-2018 Dec 10, 2017
helper.go codec: codecFner.reset should reset all typeInfos (in case *TypeInfos… Aug 10, 2018
helper_internal.go codec: refactor codecgen, optimize isEmptyValue and cache more static… Dec 30, 2017
helper_not_unsafe.go codec: remove double-indirection and fix race in Copy-On-Write during… Jan 12, 2018
helper_test.go codec: faster unsafe variants of reflect.Value.SetXXX and update msgp… Oct 30, 2017
helper_unsafe.go codec: represent correctly, that a ptr is nil if it "points" to a nil… Apr 6, 2018
json.go codec: use decReader and encWriter interfaces (not the struct decRead… Jul 29, 2018
mammoth-test.go.tmpl codec: fix suggestions raised by golint -min_confidence 0.9, misspell… Nov 12, 2017
mammoth2-test.go.tmpl codec: update comment in generated code per accepted proposal: https:… Nov 9, 2017
mammoth2_codecgen_generated_test.go codec: if json and the struct field name (or encode name) is ascii an… Jul 28, 2018
mammoth2_generated_test.go codec: update generated files (per recent changes to the generators) Nov 13, 2017
mammoth_generated_test.go codec: update generated files (per recent changes to the generators) Nov 13, 2017
msgpack.go codec: msgpack: support option to encode positive numbers as unsigned… Aug 13, 2018
py_test.go codec: update copyright to say 2012-2018 Dec 10, 2017
rpc.go codec: rpc: remove comments Feb 21, 2018
shared_test.go codec: update copyright to say 2012-2018 Dec 10, 2017
simple.go codec: use decReader and encWriter interfaces (not the struct decRead… Jul 29, 2018
test-cbor-goldens.json codec: some overflow and 0-length container fixes, and cbor tests. Nov 2, 2014
test.py codec: test.py: use 127.0.0.1 always for localhost Aug 26, 2017
values_flex_test.go codecgen: handle struct fields of type *time.Time correctly Apr 7, 2018
values_test.go codec: refactor struct field loading and data structures, for perform… Dec 26, 2017
xml.go codec: re-factor and streamline code for publishing standards, consis… Dec 24, 2017
z_all_test.go codec: add omitempty tests Apr 6, 2018

README.md

Codec

High Performance, Feature-Rich Idiomatic Go codec/encoding library for binc, msgpack, cbor, json.

Supported Serialization formats are:

To install:

go get github.com/ugorji/go/codec

This package will carefully use 'unsafe' for performance reasons in specific places. You can build without unsafe use by passing the safe or appengine tag i.e. 'go install -tags=safe ...'. Note that unsafe is only supported for the last 3 go sdk versions e.g. current go release is go 1.9, so we support unsafe use only from go 1.7+ . This is because supporting unsafe requires knowledge of implementation details.

Online documentation: http://godoc.org/github.com/ugorji/go/codec
Detailed Usage/How-to Primer: http://ugorji.net/blog/go-codec-primer

The idiomatic Go support is as seen in other encoding packages in the standard library (ie json, xml, gob, etc).

Rich Feature Set includes:

  • Simple but extremely powerful and feature-rich API
  • Support for go1.4 and above, while selectively using newer APIs for later releases
  • Excellent code coverage ( > 90% )
  • Very High Performance. Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
  • Careful selected use of 'unsafe' for targeted performance gains. 100% mode exists where 'unsafe' is not used at all.
  • Lock-free (sans mutex) concurrency for scaling to 100's of cores
  • Coerce types where appropriate e.g. decode an int in the stream into a float, decode numbers from formatted strings, etc
  • Corner Cases: Overflows, nil maps/slices, nil values in streams are handled correctly
  • Standard field renaming via tags
  • Support for omitting empty fields during an encoding
  • Encoding from any value and decoding into pointer to any value (struct, slice, map, primitives, pointers, interface{}, etc)
  • Extensions to support efficient encoding/decoding of any named types
  • Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
  • Support IsZero() bool to determine if a value is a zero value. Analogous to time.Time.IsZero() bool.
  • Decoding without a schema (into a interface{}). Includes Options to configure what specific map or slice type to use when decoding an encoded list or map into a nil interface{}
  • Mapping a non-interface type to an interface, so we can decode appropriately into any interface type with a correctly configured non-interface value.
  • Encode a struct as an array, and decode struct from an array in the data stream
  • Option to encode struct keys as numbers (instead of strings) (to support structured streams with fields encoded as numeric codes)
  • Comprehensive support for anonymous fields
  • Fast (no-reflection) encoding/decoding of common maps and slices
  • Code-generation for faster performance.
  • Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
  • Support indefinite-length formats to enable true streaming (for formats which support it e.g. json, cbor)
  • Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes. This mostly applies to maps, where iteration order is non-deterministic.
  • NIL in data stream decoded as zero value
  • Never silently skip data when decoding. User decides whether to return an error or silently skip data when keys or indexes in the data stream do not map to fields in the struct.
  • Encode/Decode from/to chan types (for iterative streaming support)
  • Drop-in replacement for encoding/json. json: key in struct tag supported.
  • Provides a RPC Server and Client Codec for net/rpc communication protocol.
  • Handle unique idiosyncrasies of codecs e.g.

Extension Support

Users can register a function to handle the encoding or decoding of their custom types.

There are no restrictions on what the custom type can be. Some examples:

type BisSet   []int
type BitSet64 uint64
type UUID     string
type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
type GifImage struct { ... }

As an illustration, MyStructWithUnexportedFields would normally be encoded as an empty map because it has no exported fields, while UUID would be encoded as a string. However, with extension support, you can encode any of these however you like.

Custom Encoding and Decoding

This package maintains symmetry in the encoding and decoding halfs. We determine how to encode or decode by walking this decision tree

  • is type a codec.Selfer?
  • is there an extension registered for the type?
  • is format binary, and is type a encoding.BinaryMarshaler and BinaryUnmarshaler?
  • is format specifically json, and is type a encoding/json.Marshaler and Unmarshaler?
  • is format text-based, and type an encoding.TextMarshaler?
  • else we use a pair of functions based on the "kind" of the type e.g. map, slice, int64, etc

This symmetry is important to reduce chances of issues happening because the encoding and decoding sides are out of sync e.g. decoded via very specific encoding.TextUnmarshaler but encoded via kind-specific generalized mode.

Consequently, if a type only defines one-half of the symmetry (e.g. it implements UnmarshalJSON() but not MarshalJSON() ), then that type doesn't satisfy the check and we will continue walking down the decision tree.

RPC

RPC Client and Server Codecs are implemented, so the codecs can be used with the standard net/rpc package.

Usage

Typical usage model:

// create and configure Handle
var (
  bh codec.BincHandle
  mh codec.MsgpackHandle
  ch codec.CborHandle
)

mh.MapType = reflect.TypeOf(map[string]interface{}(nil))

// configure extensions
// e.g. for msgpack, define functions and enable Time support for tag 1
// mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)

// create and use decoder/encoder
var (
  r io.Reader
  w io.Writer
  b []byte
  h = &bh // or mh to use msgpack
)

dec = codec.NewDecoder(r, h)
dec = codec.NewDecoderBytes(b, h)
err = dec.Decode(&v)

enc = codec.NewEncoder(w, h)
enc = codec.NewEncoderBytes(&b, h)
err = enc.Encode(v)

//RPC Server
go func() {
    for {
        conn, err := listener.Accept()
        rpcCodec := codec.GoRpc.ServerCodec(conn, h)
        //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
        rpc.ServeCodec(rpcCodec)
    }
}()

//RPC Communication (client side)
conn, err = net.Dial("tcp", "localhost:5555")
rpcCodec := codec.GoRpc.ClientCodec(conn, h)
//OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
client := rpc.NewClientWithCodec(rpcCodec)

Running Tests

To run tests, use the following:

go test

To run the full suite of tests, use the following:

go test -tags alltests -run Suite

You can run the tag 'safe' to run tests or build in safe mode. e.g.

go test -tags safe -run Json
go test -tags "alltests safe" -run Suite

Running Benchmarks

Please see http://github.com/ugorji/go-codec-bench .

Caveats

Struct fields matching the following are ignored during encoding and decoding

  • struct tag value set to -
  • func, complex numbers, unsafe pointers
  • unexported and not embedded
  • unexported and embedded and not struct kind
  • unexported and embedded pointers (from go1.10)

Every other field in a struct will be encoded/decoded.

Embedded fields are encoded as if they exist in the top-level struct, with some caveats. See Encode documentation.