Need help - Ruby FFI and Go - Pointers and []Byte

[TomFreudenberg]

I am sorry to post such kind as question as an issue but hopefully some may take assist.

We do not know how to write a go (Golang) package that we can exchange binary data between Ruby and Go functions as arguments and results.

Thanks to any help
Tom

The question is (partly) copied from StackExchange:

http://stackoverflow.com/questions/41204483/ruby-ffi-and-go-golang-how-to-get-and-set-a-binary-data-block-or-byte-array

sample.go:

package main
{...}
//export VarEncode
func VarEncode() unsafe.Pointer {
  var buffer = new(bytes.Buffer)
  {...}
  // How to return the data to Ruby
  return buffer.Bytes() // THIS DOES NOT WORK
}

//export VarDecode
func VarDecode(bufFromRuby unsafe.Pointer, length C.int) bool {
  // How to access the data from Ruby
  var buffer = bytes.NewBuffer(C.GoBytes(bufFromRuby, length))
  {...}
}

{...}

sample.rb:

require 'ffi'

module Sample
  sample_so = File.expand_path('./sample.so', File.dirname(__FILE__))

  extend FFI::Library
  ffi_lib sample_so

  attach_function :VarEncode, [], :pointer
  attach_function :VarDecode, [:pointer, :int], :bool
end

# get the object as binary dump from go
dumpedStructFromGo = Sample.VarEncode

# How to access the buffer ???
puts "Size of dumped object: #{dumpedStructFromGo.Length}"
puts dumpedStructFromGo.inspect

# Write buffer to tempfile
WriteToFile dumpedStructFromGo ????

# Re-Read buffer from tempfile
newBuf = ReadFromFile ????

# Re-Instantiate an object in Ruby
res = Sample.VarDecode(newBuf, newBuf.Length)
puts "Instance successfully restored: #{res}"

[tduehr]

I didn't know this would work with Go. This doesn't seem to be a particularly good path as you're disabling some of the protections provided by Go.

I need to know more about your use case before I can provide much help.

[TomFreudenberg]

@tduehr Thanks for your response.

We have developed some kind of workflow / state machine engine in ruby. In case that we won't re-develop all in Go again we are looking forward to just replace some intensive tasks in Go.

Most of the time this means, we are running tasks from ruby in different queues handled by different ruby processes and want using "speedy" functions from Go.

Let's say something like incoming request

Rabbit Message Queues:
Queue A: processed by scripta.rb
Queue B: processed by scriptb.rb

A new message is posted to Queue A and then the first scripta process creates something like a CONFIG struct via Go function

After that the binary dump of that CONFIG struct should be stored as messagebody to Queue B

When scriptb processes the message, the messagebody should be instantiated again in another Go function to work with just instantiating the passed object.

Hopefully you may understand what we are targetting

[TomFreudenberg]

Just to ensure - we already run this but need to convert for and back the Go object binary memory dump into a MIME encoded string.

This consumes time and RAM.

[tduehr]

There is no way for FFI to know how large the memory allocated to a pointer on its own. If this is a data structure of some kind, you can define a subclass of FFI::Struct with the details and instantiate it with the pointer returned from VarEncode.

[TomFreudenberg]

From VarEncode there is no structure but byte[]. After VarEncode in Go we know the size of the data block and could copy that into a fix length reserved Ruby memory block. Would this be possible but how to copy?

[tduehr]

That partially depends on they nature of the data in the array. If it's a C string, then the pointer is fine and you can just call `read_string` off the pointer. If there's another function you can call to obtain the length, it can be used as the argument to `read_string` to get a Ruby String containing the info. Otherwise, you're best option is to modify the Go to accept `int*` and `byte**` to be set as part of the call.

[TomFreudenberg]

Is is possible for you to give me a short code sample about byte**. I am struggling with other than Strings.

[tduehr]

I don't know how to pass by reference in Go.

[TomFreudenberg]

Hm ... that is just *byte[]

But how is that defined in Ruby?

[tduehr]

The argument to attach_function is still :pointer.

ptr = MemoryPointer.new :pointer
lptr = MemoryPointer.new :long
Sample.VerEncode ptr, lptr
ptr.read_pointer.readh_string(lptr.read_long)

[Jared-Prime]

for VarEncode, can you return the result as a string? return C.CString(buffer.String()) // type will be *C.char in the exported function, then set the ffi return type to :string

do the complementary work when passing from ruby: the ffi input type would be :string and the go coercion would be C.GoString(bufFromRuby) // type will be string