Decide on Bytes

[pchlupacek]

Current implementation of Bytes is somehow second class citizen in the library.

We shall probably extend it or decide on alternative approach that will not require allocation of arrays on every read request.

I think the contract shall be either :

• read-only wrapper around the Array[Bytes]. Ref to Array will be held separatelly from the wrapper closed over Process that actually reads the data from input source.

trait Bytes {
  // writes to os
  def writeTo(os:OutputStrem):Int

  // return read only nio.ByteBuffer
  def toBuffer : ByteBuffer 

//efficient append
 def append(other:Bytes) : Bytes

 def ++(other:bytes) : Bytes

// will copy content to new array
 def toArray : Array[Byte]
}

//somewhere in process
var array = new Array.ofDim[Byte](1024)

val is: InputStream = ???


///in step 

Bytes.apply(is.read(array), array) : Bytes

//on other end of Process 
bytes ++ otherBytes
//or
(bytes:Bytes).writeTo(output:OutputStream)

• State machine, that will not expose internal array to public (but will keep it internally) and will be either in Full or Empty state. In Full state it will allow only operations that write content to some input to be performed, in Empty it will allow only single read from some output to be performed

Both scenarios will have some effective append and seek operations on them inspired for example by akka.ByteString : http://doc.akka.io/api/akka/2.0/akka/util/ByteString.html

What are your thoughts? We need efficient byte streaming for scalaz-stream-mongo, and other scalaz-stream goodies....

Thanks,

P.

[pchiusano]

I think I have a nice solution for this, described the basic idea here, but need to do some fiddling with it to settle on an API. What I'd like to do is actually just subclass Seq with a special 'copy-on-write' version of Seq using the technique described there. Then we have a Seq[Byte] which is a copy-on-write Seq backed by java.nio.ByteBuffer (and we can have similar ones for ints, floats, etc).

So io.chunkR will return a copy-on-write Seq[Byte]. Downstream can recover this exact Seq[Byte] using this combinator:

// 'natural' buffering of this process
trait Process[F[_],+O] { 
  def available: Process[F,Seq[O]] = this match {
    case Halt(e) => Halt(e)
    case Emit(h,t) => emit(h) ++ t.available
    ...
}

Downstream can then pattern match on the Seq returned from available to get access to the actual fancy nio backed buffer, which can have all sorts of special purpose functions for doing things 'unboxed'.

[pchlupacek]

Hmm, seems nice, so essentially we just wrap the Seq over buffer :-)

Cool

P.

On Oct 11, 2013, at 6:59 PM, pchiusano notifications@github.com wrote:

I think I have a nice solution for this, described the basic idea here, but need to do some fiddling with it to settle on an API. What I'd like to do is actually just subclass Seq with a special 'copy-on-write' version of Seq using the technique described there. Then we have a Seq[Byte] which is a copy-on-write Seq backed by java.nio.ByteBuffer (and we can have similar ones for ints, floats, etc).

So io.chunkR will return a copy-on-write Seq[Byte]. Downstream can recover this exact Seq[Byte] using this combinator:

// 'natural' buffering of this process
trait Process[F[_],+O] {
def available: Process[F,Seq[O]] = this match {
case Halt(e) => Halt(e)
case Emit(h,t) => emit(h) ++ t.available
...
}
Downstream can then pattern match on the Seq returned from available to get access to the actual fancy nio backed buffer, which can have all sorts of special purpose functions for doing things 'unboxed'.

—
Reply to this email directly or view it on GitHub.

[pchlupacek]

@pchiusano I just though a bit about this and not sure if I have fully understood the idea. I mean how you want to check whether someone is holding the weak reference? Whent the get == null then this got gced and you cannot really recover this object somehow. The only solution I found is via using phantom reference, but still it heavily relies on gc to check whether it is just phantom reachable or not.
This overall concept to rely on gc and its setting seems to be a quite hacky to me? I mean you will really hard to be able to reason about program behaviour....
Or maybe I am not getting the idea right?