close(::GenericIOBuffer) has a bug and is too aggressive in some cases

[kwdye]

I encountered what appears to be a small bug in close(::GenericIOBuffer). Here is the code from v1.5.3

@noinline function close(io::Base.GenericIOBuffer{T}) where T
    io.readable = false
    io.writable = false
    io.seekable = false
    io.size = 0
    io.maxsize = 0
    io.ptr = 1
    io.mark = -1
    if io.writable
        resize!(io.data, 0)
    end
    nothing
end

The first issue is just that the "if io.writable" condition can never be true because writable is set to false earlier in the function. I don't understand why we should even want to resize the data to zero in any case though since you can optionally provide this data yourself in the IOBuffer constructor. I think most users would find their data array being resized to zero to be an unhelpful surprise.

The second issue which is more important is that close() is very aggressive about resetting the size and ptr fields, making the data effectively inaccessible. This causes difficulties when using GenericIOBuffer as a stream that's been wrapped by another stream ie:

data = zeros(UInt8,80)
iob = IOBuffer(data)
seek(iob,0)
compressor = TranscodingStream(ZstdCompressor(), iob)

The issue is that compressor won't necessarily have written all of its output to the IOBuffer until it has been closed, and closing it calls close on the IOBuffer, rendering the data inaccessible (besides manually inspecting the data field). There is no way to tell how many bytes were actually written to data, unless data was extended in size because it was too small.

I'm not sure if there would be any bad effects to making close(::GenericIOBuffer) less aggressive, perhaps only setting writable to false and otherwise doing nothing. It seems like this is a case where the behavior you expect from a buffer and the behavior you expect from a stream diverge.

[vtjnash]

Perhaps you're looking for a Base.BufferStream?

[kwdye]

Yes thanks BufferStream looks like what I'd want here. Is there a reason it isn't exported?

[StefanKarpinski]

In my experience Base.BufferStream is quite buggy. You may want to use https://github.com/staticfloat/SimpleBufferStream.jl instead. Perhaps @staticfloat can elaborate on why Base.BufferStream didn't do what he needed (I can't recall).

[staticfloat]

I don't remember what the exact issue was; it may have been as simple as that Base.BufferStream doesn't inherit from the right abstract interface for usage with HTTP. In any case, SimpleBufferStream does work pretty well for what it is (it's not fully-featured, e.g. there's no mark()/unmark() interface).