Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/json: document and limit buffering #1955

gopherbot opened this issue Jun 14, 2011 · 12 comments

encoding/json: document and limit buffering #1955

gopherbot opened this issue Jun 14, 2011 · 12 comments


Copy link


What steps will reproduce the problem?
1. Compile & run the attached

What is the expected output?
"Everything is A-OK"

What do you see instead?
"Didn't get proper exdata, got ''"

Which compiler are you using (5g, 6g, 8g, gccgo)?

Which operating system are you using?

Which revision are you using?  (hg identify)

Please provide any additional information below.

I believe the cause is that the decoder creates (and grows) a buffer without confirming
that the total amount is needed.  This approach is probably far more efficient than a
correct 'byte.Read && process || byte.Unread' approach, but it would be nice if
there were an exported 
json.DecodeElement(io.Reader, interface{})(os.Error) that didn't "over-feed".


  1. json_bug.go (465 bytes)
Copy link

peterGo commented Jun 14, 2011

Comment 1:

func (*Decoder) Decode
Decode reads the next JSON-encoded value from its input and stores it in the value
pointed to by v.
func (*Buffer) Read
Read reads the next len(p) bytes from the buffer or until the buffer is drained.
When json.Decode reads from bytes.Buffer (buf.Read), the bytes in the buffer are
package main
import (
func main() {
    buf := bytes.NewBufferString(`{"k":"v"}exdata`)
    m := map[string]string{}
    log.Println("buf.Len:", buf.Len())
    err := json.NewDecoder(buf).Decode(&m)
    if err != nil {
        log.Fatalf("Couldn't even decode...")
    log.Println("buf.Len:", buf.Len())
    exbuff := make([]byte, 80)
    n, err := buf.Read(exbuff)
    if string(exbuff[0:n]) != "exdata" {
        log.Fatalf("Didn't get proper exdata, got '%s'", exbuff[0:n])
    log.Printf("Everything was A-OK")
2011/06/13 23:45:02 buf.Len: 15
2011/06/13 23:45:02 buf.Len: 0
2011/06/13 23:45:02 Didn't get proper exdata, got ''

Copy link

Comment 2 by

I totally agree that is what's happening;  The bug (as I view it) is that the json
element should be decoded only to the end of the object ('}');  At that point, I should
be able to resume the read and read 'exdata' (non-JSON data) if so desired.
It's not the standard JSON stream -- but it doesn't seem unreasonable to read a JSON
element and not 'extra' non-JSON data.
Or did I misunderstand your comment?

Copy link

rsc commented Jun 14, 2011

Comment 3:

We should fix this.
Note that in this specific case JSON can do without reading ahead
but in general it cannot.  If the JSON value is 123, it has to
consume the byte that follows in order to find out that the
number is over.

Status changed to Accepted.

Copy link

Comment 4:

http.ReadRequest takes a *bufio.Reader.  JSON could also?  Then you could Peek at it.
Perhaps add json.NewBufferedDecoder() and keep json.NewDecoder() as a wrapper.
Or a fun technique I used recently:  you can have json.Decoder.Decode return not just an
os.Error but instead a (remaining io.Reader, err os.Error) where remaining is:
   remaining = io.MultiReader(bytes.NewBuffer(slop), originalReader)

Copy link

rsc commented Oct 6, 2011

Comment 5:

I think the fix is to do what flate and gob do: define that
if the input has a ReadByte then more buffering won't be added.

Status changed to HelpWanted.

Copy link

rsc commented Dec 9, 2011

Comment 7:

Labels changed: added priority-later.

Copy link

rsc commented Dec 12, 2011

Comment 8:

Labels changed: added priority-go1.

Copy link

robpike commented Jan 13, 2012

Comment 9:

Owner changed to

Copy link

Comment 11 by monnand:

I submitted a CL to solve this issue. Ready to be reviewed:

Copy link

rsc commented Feb 8, 2012

Comment 12:

This issue was closed by revision 49110ea.

Status changed to Fixed.

Copy link

Comment 13:

Proposed addition for this:

Copy link

Comment 14:

This issue was updated by revision 91e99c1.

R=golang-dev, adg, rsc

@rsc rsc added this to the Go1 milestone Apr 10, 2015
@rsc rsc removed the priority-go1 label Apr 10, 2015
@golang golang locked and limited conversation to collaborators Jun 24, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
None yet

No branches or pull requests

5 participants