Build a library usable for embedding & FFI

[byteit101]

Using a library via FFI is usually done via a shared library load. When I was attempting to embed HashLink via an FFI, I had a need for a shared library version of hl so that I could use the functions for loading and initializing .hl bytecode. This PR adds make targets for such a library, plus exposing a few more methods necessary when looking up and calling objects as I've seen referenced in the community and issues here. Additionally, there is a thread-helper method I found necessary when calling from not just a single spot in the C stack. Currently, I make it build as libhl-jit.so, but I'm not married to the name. I originally was thinking libhl-embed.so.

Things not in this PR, but could be helpful for embedding via an FFI:

It may be helpful to also document how to integrate with the GC, that took me a while to figure out.

An entrypoint helper. It's a bit more verbose in non-C languages via FFI

Method lookup/calling helpers. This surprised me at the lack of helpers. Right now my FFI code looks like this for driving looking up methods:

def lookup_function(name, type, instance)
	hash = field_hash(name.to_s)
	lookup = case type.kind
		when Hl::HOBJ
			Hl.obj_resolve_field(type.details, hash)
		when Hl::HVIRTUAL
			tv = Hl::TypeVirtual.new(type.details) # type cast, not instantiation
			Hl.lookup_find(tv.lookup, tv.nfields, hash)
		else
			raise "TODO: implement lookup for #{type.kind}"
	end
	raise NameError.new("Method doesn't exist #{name}") if lookup.null?
	fl = Hl::FieldLookup.new(lookup) # type cast, not instantiation
	return Hl.dyn_getp(instance, fl.hashed_name, Hl.dyn)
end

I didn't implement these as I wanted to discuss them first, after a minimum basic solution (the PR right now).

[ncannasse]

hi !

I think you need a bit more than hl_enter_thread_stack in order to correctly init the VM.
You need to call both hl_global_init() and hl_register_thread()

As for the helpers, you're right the VM was designed to be used by compiler generated code, not directly with FFI, so the helpers we have are mostly focused on wrapping C primitives for VM, not for the user to manage the VM objects directly.

[byteit101]

Yes, sorry I didn't provide enough context. I was assuming the other bits were already called. Here is my current usage of enter_thread_stack and the global init. Note the enter_thread_stack is called per-call (call_raw), while the read init/register is just once

def initialize(bytecode)
	Hl.global_init
	Hl.sys_init(nil, 0, nil) # TODO: args?  #hl_sys_init((void**)argv,argc,file);
	# the GC stops at the stack pointer. Don't scan any ruby code
	Hl.register_thread(nil)
	
	...
end

def dispose
	Hl.module_free(@m)
	alloc_field = FFI::Pointer.new(:pointer, @code.pointer.address + @code.offset_of(:alloc))
	Hl.free alloc_field #(@code.alloc)
	Hl.unregister_thread
	Hl.global_free
end

def call_raw(closure, args)
	Hl.profile_setup(-1) # TODO: profile setup?
	# avoid the ruby&ffi stack, we must keep this updated for each call
	Hl::enter_thread_stack(0)

	ret = if args == []
		Hl.dyn_call_safe(closure,nil,0,@isExc)
	else
		# TODO: HL_MAX_ARGS!
		HashLink.memory_pointer(:pointer, 10) do |cargs|
			cargs.write_array_of_pointer(args)
			Hl.dyn_call_safe(closure,cargs,args.length,@isExc)
		end
	end
	Hl.profile_end
	if @isExc.get_int8(0) != 0
		a = Hl.exception_stack
		hlstr = Hl.to_string(ret).read_wstring
		ex = ::HashLink::Exception.new("Uncaught HashLink exception: #{hlstr}")
		ex.set_backtrace(Hl::Varray.new(a).asize.times.map do |i|
			_read_array(a, i).read_wstring
		end)
		raise ex
	end
	return ret
end			

I'm still working on cleaning it up, but will be publishing the full FFI source shortly

[byteit101]

Here is how I'm using this, with full versions of the methods pasted above as examples: https://github.com/byteit101/hashlink-embed/blob/8288463a76bd520920f10b1eec69f98ca496e2aa/lib/hashlink-embed/hashlink.rb#L118