plugin: Add support for closing plugins

[zacps]

Adding support for closing plugins would give a simple way to do hot code reloading in long running applications like web servers.

This would add another method to src/plugin/plugin.go and a new file src/plugin/dl_close.go that would handle closing the dlhandle and destroying the go symbol map.
It would also add the handle as an un-exported unsafe pointer to the go plugin struct.

The main issue is how to signal that a plugin has been closed without breaking API compatibility. Currently the plugin has a channel called loaded which is closed when the plugin is loaded. The simplest solution is to add another channel which is closed when the plugin is closed, however this doesn't seem very 'neat' to me.

[davecheney]

I don't see how this could be implemented safely. Other Go code could have references to functions and symbols loaded from the plugin-would unloading the plugin make those dangling references, or delay the unloading of a plugin until all the references have been reclaimed? Neither sound workable.

[zacps]

Would it be possible to make close fail until the reference count of all symbols in the plugin reached 1, so that the plugin itself was the only structure that held a reference?

Alternatively you could have a different function which checked whether references to the symbols existed and if there were none close the dlhandle, leaving the destruction of the plugin object and any other references up to the program. While this is far from ideal I think with adequate caution it could be a workable solution.

[davecheney]

Would it be possible to make close fail until the reference count of all symbols in the plugin reached 1, so that the plugin itself was the only structure that held a reference?

This is spinny and racy.

Alternatively you could have a different function which checked whether references to the symbols existed and if there were none close the dlhandle, leaving the destruction of the plugin object and any other references up to the program. While this is far from ideal I think with adequate caution it could be a workable solution.

This sounds like a finaliser, with the associated problems of finaliser non determinism.

[zacps]

Could you be a bit more specific about the second point?

[davecheney]

Your proposal is to add a way to free resources associated with a plugin. If this is delegated to a finaliser it becomes non deterministic when this freeing occurs as finalisers are not guaranteed to run promptly, or indeed, at all.

[zacps]

I'm not suggesting that it's run automatically by the GC, or in a defer (unfortunately).

I'm suggesting that if the needs to be unloaded that the developer will keep track of references to it's symbols then when they no longer need them destroy the references then explicitly call close.

In that situation they will either forget to call close, leaving the handle open, close successfully or attempt to close when there are still references which will fail. It could possibly fail with a panic rather than a plain error as attempting the close a resource that still has a reference is a pretty big bug.

I don't see what issues this approach has beyond reliance on the developer to not make mistakes. It does introduce a need to keep track of references which is far from perfect and not something I'd like to introduce to go. However I think that it's worth having this for those that want to use it and understand the additional complexity it introduces.

[ianlancetaylor]

What if the plugin started a goroutine?

Go as a language and a runtime environment tends to avoid providing facilities that are easy to get wrong when getting them wrong causes the program to crash or behave unpredictably. I don't see how to make this at all reliable, so to me it doesn't seem to be a good fit for Go.

[zacps]

goroutines would have to be handled in the same way, requiring the developer to close them.

I can understand not wanting to introduce functionality that's easy to misuse. I just think that it's better to have the ability to do it if you know what you're doing. I guess it does go against the philosophy of go though.

[davecheney]

How can you close a goroutine? The runtime doesn't give you any way to get a handle or an identifier for a goroutine?

…

On Tue, 23 May 2017, 09:54 Zac Pullar-Strecker ***@***.***> wrote: goroutines would have to be handled in the same way, requiring the developer to close them. I can understand not wanting to introduce functionality that's easy to misuse I just think that it's better to have the ability to do it if you know what you're doing. I guess it does go against the philosophy of go though. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#20461 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA7TSJ5h7YYZU1PZDXlrNZfjSaaifks5r8iBGgaJpZM4Ni-0b> .

[zacps]

I took a quick look at runtime but couldn't find where exactly the goroutine is created. Are goroutines not closed when they return?

[davecheney]

Yup, a goroutine dies/quits/goes away when the function that was passed to the go statement returns. So that leaves the problem of ensuring every goroutine that may have access to a symbol from a plugin must exit before the plugin can be unloaded. This means either finalisers, which won't give you a timely response, or manual bookkeeping, which as Ian said is unreliable.

…

On Tue, May 23, 2017 at 12:16 PM, Zac Pullar-Strecker < ***@***.***> wrote: I took a quick look at runtime but couldn't find where exactly the goroutine is created. Are goroutines not closed when they return? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#20461 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA-7gIyQn1lrDQJPC5XHyF1Oe4u_Mks5r8kGDgaJpZM4Ni-0b> .

[zacps]

Ok, that's what I thought.

I personally think that manual bookkeeping wouldn't be too bad. If you're writing something designed to be loaded/unloaded you're probably going to make sure you keep track of references and goroutines as is.

I'm trying to think whether there's a better way to do this. I guess it's not something other languages have had problems with.

[zacps]

I have another idea for how this could be solved, it's not simple though and I'm not sure whether it would make sense.

Basically I think it would be possible to implement a compile-time check that ensured there was no possibility for a dangling reference when a program was closed.

The compiler could check to see if close was called on any plugin object in the program. If it was we'd find all of the lookup calls associated with that plugin and determine whether there were any paths that could leave a live reference when or after closed was called.

I have no idea how you'd implement this or whether it'd be fast enough to be viable. Just an idea.

[DemiMarie]

What about having the GC close any plugins that are no longer referenced?

That's what the JVM does. And forcing a full GC whenever Go code must be unloaded is not hard.