New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Basic C API for the runtime #17048

Open
DemiMarie opened this Issue Sep 10, 2016 · 17 comments

Comments

Projects
None yet
7 participants
@DemiMarie
Copy link

DemiMarie commented Sep 10, 2016

Purpose

This is a proposal for a basic C API for the runtime. It allows for non-Go main programs and for using specially written Go libraries from any language that can call C functions, with a few additional features beyond what are currently available by buildmode=c-shared. This will be a new buildmode.

Background

Go can call almost any C function via CGo. However, the ability to have a non-Go main program is much more limited. Go does have buildmode=c-shared, but it is limited by #11100 and also does not allow for useful features like passing GOMAXPROCS other than as an environment variable.

Goals

  • Allow for writing programs in a mixture of Go and another language, such that Go is not necessarily the language in which the program's entry point is written.
  • Allow for writing a a library partly or completely in Go for use by another language.
  • Provide handles that can passed to C and which can refer to any Go object.

Non-Goals

  • Allowing arbitrary Go functions to be called from C
  • Allowing C code to manipulate arbitrary Go data structures.

The API

Ground rules.

The entire API is provided in a single C header file, goapi.h.

Unless otherwise specified, any function can be called from multiple threads simultaneously. However, a single pointer in the API cannot be passed to API functions in 2 or more threads simultaneously, unless both functions take pointers to const data.

The API itself

int go_runtime_init(char * const* errormsg, ssize_t *already_initialized, const char * const *options, const void * const* arguments);

Initializes the Go runtime. Must be called before calling any other Go function. Thread-safe and may be called multiple times.

options is a pointer to an array of NUL-terminated strings. Each string corresponds to a matching void* in arguments that corresponds to a matching pointer, which points to a flag that can be used to configure the runtime. Currently, the only string in options that is meaningful is GOMAXPROCS, which replaces the GOMAXPROCS environment variable: the corresponding void* must point to a size_t*. All other values are reserved and must not be used.

If the defaults are OK, both arguments and options may be set to NULL. They are ignored if the runtime is already initialized.

Returns 0 on success, a negative number on failure.

  • On success, *errormsg is set to NULL and *already_initialized is set to the number of times the runtime has been initialized (including this one). It is safe to set errormsg and/or already_initialized to NULL, in which case they are not accessed.
  • On failure, *errormsg points to a NUL-terminated, human-readable error message, and *already_initialized holds a negative number. Again, neither are written to if NULL.
int go_runtime_shutdown(void);

This function shuts down the runtime. It must be called once for each call to go_runtime_init. Only the last such call has any effect.

Once the number of calls to go_runtime_shutdown equals the number of calls to go_runtime_init, this function terminates all active goroutines and resets any signal handlers.

The following sentence may not be initially implemented:
Afterwards, it is safe to dlclose the shared library containing the runtime, or to restart the runtime with go_runtime_init.

typedef struct Go_Handle *Go_Handle;

A handle to an arbitrary Go object. Analogous to a Haskell stable pointer. It stays valid even across garbage collections.

A Go_Handle is not guaranteed to point to valid memory. Dereferencing it invokes undefined behavior.

void go_runtime_delete_handle(Go_Handle handle_to_be_freed);

Deletes the handle passed as argument, rendering it invalid and freeing all underlying resources. After this function is called, the object the handle pointed to may be garbage collected if there are no more references to it.

Go_Handle go_runtime_duplicate_handle(Go_Handle handle_to_be_duplicated);

Duplicates the handle passed as argument. The returned handle points to the same object as the original handle, but must be deallocated separately.

uint8_t go_runtime_are_identical_handles(Go_Handle handle1, Go_Handle handle2);

Tests if the 2 handles passed are identical. Returns true if they point to the same object in memory. Otherwise, returns false.

The Go side

These are additional functions exposed from the runtime package.

type CHandle

The type of C handles to Go data. When passed via CGo, becomes a Go_Handle on the C side.

May be accessed by multiple goroutines simultaneously.

func NewHandle(object interface {}) CHandle

Creates a handle to a Go object that can safely be passed to C.

func DuplicateHandle(handle CHandle) CHandle

Duplicates the handle.

func TestIfIdenticalHandles(handle1 CHandle, handle2 CHandle) bool

Tests if the 2 handles passed point to the same object; that is, if modifications of the object pointed to by one will affect the object pointed to by the other.

func DeallocateHandle(handle CHandle)

Deallocates the handle, rendering it invalid.

func DereferenceHandle(handle CHandle) interface {}

Dereferences the handle, returning the contained object. If the handle is invalid, invokes undefined behavior.

@minux

This comment has been minimized.

Copy link
Member

minux commented Sep 10, 2016

@DemiMarie

This comment has been minimized.

Copy link

DemiMarie commented Sep 10, 2016

There are two advantages.

The less important one is that go_runtime_shutdown is called explicitly, rather than implicitly on dlclose. This might be helpful.

The bigger one is the handle feature. This makes the task of turning arbitrary Go data into a void* used by C libraries much easier.

@crawshaw

This comment has been minimized.

Copy link
Contributor

crawshaw commented Sep 10, 2016

If you put aside #11100 for the moment (making go_runtime_shutdown work would also make -buildmode=c-shared work, it's the same problem), I believe you can build this interface today on top of c-shared.

You can declare CHandle to be an int, allocate them by incrementing a global counter, and keep a map[interface{}]CHandle and map[CHandle]interface{} in a management package. This is roughly how the inside of the gobind tool in the mobile repository works.

@minux

This comment has been minimized.

Copy link
Member

minux commented Sep 10, 2016

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Sep 10, 2016

I believe the handle code can be written using an ordinary Go package.

You can write a non-Go program that calls Go code today using either buildmode=c-shared or buildmode=c-archive.

I don't understand how to implement go_runtime_shutdown. That's the hard part.

@quentinmit quentinmit added this to the Proposal milestone Sep 12, 2016

@quentinmit quentinmit added the Proposal label Sep 12, 2016

@DemiMarie

This comment has been minimized.

Copy link

DemiMarie commented Oct 11, 2016

@minux This is not intended to be like JNI at all. There is no analog of the JNI functions for accessing handles. In fact, the direct inspiration is Haskell's excellent Foreign Function Interface, which includes stable pointers (which refer to an arbitrary Haskell object, and which can then be passed back into Haskell). The purpose of a handle is not so that C code can manipulate Go objects. What it is is a way for Go code to wrap a reference to some Go value, such that a later callback from C into Go can use that value.

This can be implemented in the runtime more efficiently than in pure Go. For one, using a deleted handle is use-after free.

@minux

This comment has been minimized.

Copy link
Member

minux commented Oct 11, 2016

@RLH

This comment has been minimized.

Copy link
Contributor

RLH commented Oct 11, 2016

W.R.T. the level of indirection / pointer handle implementation the next
step seems is a pure Go library implementation. This could then be used to
generate real world use cases that in turn can identify if there are indeed
performance bottlenecks the runtime/compiler need to look at. Expanding
the runtime's API surface needs to be motivated by real world use cases
that aren't apparent from this discussion.

On Tue, Oct 11, 2016 at 4:12 AM, Minux Ma notifications@github.com wrote:

I don't think it requires any special facilities from the runtime
to be performant.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#17048 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7Wn0IMFf9H_xZvFxBExEj4xapV1LJwks5qy0T2gaJpZM4J5mFi
.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Nov 7, 2016

Like @ianlancetaylor said, the key insight for cgo was that people writing these kinds of bindings want to write Go code, not C code, so we made it so that wrappers are written in Go. We really don't want to write wrappers in C. C code can call custom cgo-exported Go code, of course.

Is there some specific functionality that this approach (write Go code called from C) doesn't work for?

@DemiMarie

This comment has been minimized.

Copy link

DemiMarie commented Nov 9, 2016

@rsc You are correct that writing stubs in C would not be a good thing. That is why .NET's PInvoke is
much better than the JVM's JNI – marshaling is done in the higher level language, where it belongs.

I can think of two cases for this API. Neither involves writing stubs in C:

  • The C library exposes a callback-based API, and the user would like to pass a closure handle as the void*
    parameter to the callback. This requires a way for Go code to go from a void* to a Go object.
    This could be done with maps in user code at an uncertain (but probably low) performance penalty. As
    such, I am not too worried about this use case.

  • The Go program is a plugin that is repeatedly loaded and unloaded, often by a proprietary software
    package. This is what motivated this PR to OCaml and cannot
    be done without runtime support.

    I am much more motivated by this latter use case. The host application may (quite reasonably) expect the
    plugin to clean up after itself – Java does this, for example. Furthermore, the host application is often
    proprietary and/or written by a third party, so changing the behavior of the host may not be an option.

Since the latter use case is much more important to me, I am willing to restrict my proposal to it.

@minux

This comment has been minimized.

Copy link
Member

minux commented Nov 9, 2016

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Nov 9, 2016

@DemiMarie I have no objection in principle to support for shutting down the Go runtime, but I have no idea how to implement it. As @minux says, the Go runtime doesn't even have a way to shut down a stopped goroutine. I think that moving forward on this proposal would require a design that would make that possible. The design would have to not require major changes to the current runtime, because we do not want to rewrite everything to support a minor use case. Unless and until such a design is made, I would have to recommend that we decline this proposal.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Nov 9, 2016

@DemiMarie Let's assume the "handle" half of the suggestion is off the table, since, as you agree, it can be done on the Go side.

Then what's left is the go_runtime_shutdown. I think it would be plausible - provided it is not an invasive change - to add an exported C function like "bool go_runtime_shutdown(void)" that simply returns false (and does not do any shutdown) if there are any goroutines running. Then it's up to the author of the plugin to arrange that the goroutines are all gone before calling shutdown. If not, the shutdown doesn't shut anything down. As Minux and Ian said, this is the best one could really hope for.

I don't understand the need for go_runtime_init. The runtime already initializes itself, as far as I understand. It especially doesn't make sense to me to call it and not know whether the runtime is initialized or not. Again, the caller really needs to understand the state for any of this to work. It can't be stabbing in the dark.

@minux

This comment has been minimized.

Copy link
Member

minux commented Nov 9, 2016

@DemiMarie

This comment has been minimized.

Copy link

DemiMarie commented Nov 10, 2016

@minux The problem with a Go side shutdown function is that you can't run any (user) Go code after the runtime has shutdown – the runtime Go needs isn't there anymore. It would work only if the function never returned.

Expecting the user code to shut down all goroutines is okay.

@minux

This comment has been minimized.

Copy link
Member

minux commented Nov 10, 2016

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Nov 14, 2016

I wrote:

Then what's left is the go_runtime_shutdown. I think it would be plausible - provided it is not an invasive change - to add an exported C function like "bool go_runtime_shutdown(void)" that simply returns false (and does not do any shutdown) if there are any goroutines running. Then it's up to the author of the plugin to arrange that the goroutines are all gone before calling shutdown. If not, the shutdown doesn't shut anything down. As Minux and Ian said, this is the best one could really hope for.

I think this proposal is essentially on hold until someone does the work to find out whether this is an invasive change or not. It may be necessary to have sysmon shut itself down, but hopefully there isn't more to do.

@rsc rsc added the Proposal-Hold label Nov 14, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment