Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Go 2: Formal Destructors #38057

Open
markdingo opened this issue Mar 25, 2020 · 9 comments
Open

proposal: Go 2: Formal Destructors #38057

markdingo opened this issue Mar 25, 2020 · 9 comments

Comments

@markdingo
Copy link

@markdingo markdingo commented Mar 25, 2020

I'm sure this has been discussed many times before - perhaps more around the desire for
constructors - but I couldn't find any obvious proposals with Go2/LanguageChange tags so
if this is a dupe, my apologies.

I don't have any proposed syntax or tightly defined semantics, this is more about what
people think of the idea.

Proposal

Simply put, the proposal is to be able to attach a formal Destructor (in Java-speak, a
Finalizer) to a type. This should be something that the type can explicitly define without
participation by users of that type.

The motivation is to be able to garbage collect resources invisible to the go GC and not
have to rely on packages users to "do the right thing". Such resources include
process-based resources such as file descriptors and sockets as well as external resources
such as disk files. Any non-go resource really.

Rationale

While go isn't particularly designed to directly manipulate underlying OS resources the
way one might do in lower-level languages, there are still many occasions when a go
program does directly create OS resources which it wants to actively destroy.

Examples within go include os.File, http.Body and the net.*Conn family. Essentially any
type which offers a Close() call.

Examples outside of go include file system objects such as cache files, named pipes and
.lock files used to signal other applications via the file system.

More obscure examples include OS resources created by the likes of dup(), Dup2() and
Flock() out of the https://godoc.org/golang.org/x/sys/unix package.

Any package which creates these sort of resources usually wants to offer a way to destroy
them at the end of their life-cycle. This is what Destructors traditionally do in other
languages.

Why not Close() as the defacto Destructor?

The obvious counter to introducing a formal Destructor is that go already has an idiomatic
Destructor in the form of Close(). So what's wrong with continuing with this approach?

Lots of things frankly. First off, it's a bit antithetical as go programmers are not
well-conditioned to worrying about resource cleanup due to the GC nature of go and
corresponding lack of formal constructors. In other words, we've spent the last decade
training go programmers to code like this:

   import "packageX"

   func do() {
     x := packageX.Stuff{}
     x.Do()
   }

which means they aren't particularly on the lookout for formal Constructors/Destructors
yet alone informal Destructors like Close() in packages they import. Nor are they
conditioned to adding Close() in any package they create.

One could even argue that the absence of formal Destructors encourages ephemeral system
resource usage within packages. Is that always a good thing? I doubt it.

Secondly, it's not intuitive which packages need a Close() and which don't. I was
surprised to find that net.Resolver doesn't have a Close() whereas the popular
https://github.com/miekg/dns Client exposes a Conn which does require a Close(). Clearly
the net DNS resolver doesn't cache UDP sockets whereas the miekg/dns resolver
can. Intuitive much?

Perhaps an even better example of counter-intuitive uses of Close() is the http package. I
don't know how many times I've tried to remind myself of http.Body.Close() because in the
back of my mind I know there is some weird Close() when one would never expect it. It
doesn't help that there is a Server.Close(), a Body.Close(), but no Client.Close(),
Request.Close() or Response.Close(). To an outside observer this non-symmetric, random
smattering of Close() calls must be quite confusing. To me Body.Close() is just too easy
to overlook since I rarely code http clients.

Third, if a package initially ships without Close() but a later version requires one -
perhaps because the author has determined resource caching offers substantial performance
improvements - they can't do so transparently without risking resource leaks. Sure they
could introduce a pseudo-constructor, such as:

      resolver := package.NewWithClose()

but this means that no existing importers automatically benefit from the package
enhancements. They all have to change application code first.

This can be a cascading problem of course: If package A imports package B which
subsequently introduces B.NewWithClose() and B.Close(), then A is forced to introduce
an A.NewWithClose() and A.Close() so applications can reach down to the
B.Close(). That seems pretty ugly to me.

The only way to avoid this retrofit problem is for package designers to have perfect
foresight to know if they or any other contributor will ever require a Close() at any time
in the future of their package or its imports. Reductio ad absurdum leads us to the
conclusion that every package type should offer a Close() even if they are mostly noops.

Summary

I don't think the whole "Close() as a proxy for explicit Destructors" is working very well
as idioms go and it likely stifles package evolution as one cannot transparently retrofit
Close() semantics.

Clearly with the presence of Close() in many packages and interfaces, there is demand for
Destructors, so why not formalise Destructors and eliminate our imperfect Close() idiom?

As a bonus, if we have formal Destructors, we can simplify the interface surface of many
packages. Just looking at the "io" package: we could get rid of Closer, ReadCloser,
WriteCloser and ReadWriteCloser.

Complications

I admit that I don't know the complexities of implementing formal Destructors into the go
runtime yet alone determining the exact semantics. E.g., when should the Destructor be
called? When the instances go out of scope? When the GC cleans them up? Or do we invent a
special Destructor goroutine that reads a channel of types ready for destruction?
Dunno. Obviously if it's extremely difficult then that has some sway on the idea.

PS. Inexperienced with the proper submission protocol for proposals.

@gopherbot gopherbot added this to the Proposal milestone Mar 25, 2020
@gopherbot gopherbot added the Proposal label Mar 25, 2020
@robpike
Copy link
Contributor

@robpike robpike commented Mar 25, 2020

Have you seen runtime.SetFinalizer? Things such as os.File already use that mechanism.

@markdingo
Copy link
Author

@markdingo markdingo commented Mar 25, 2020

I confess that I have noticed SetFinalizer before but have never really looked at it in any depth. But I guess I'm confused as to why packages still offer Close() if SetFinalizer solves their problem? Is it purely historical and no future package should ever offer Close()? If so, I'm a happy camper.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 25, 2020

runtime.SetFinalizer is generally not appropriate for collecting non-memory resources: the delay between a Go value becoming unreachable and its finalizer being run can be arbitrarily long, and the garbage collector paces its rate of collection (and thus finalization) based only on the process's memory footprint — not on the quantities of other resources in use.

If you want to be careful about avoiding resource leaks, I would recommend using the finalizer to verify that Close was called, rather than using it as a backstop to free the actual resources. For example, see the usage in cmd/go/internal/lockedfile:

// Although the operating system will drop locks for open files when the go
// command exits, we want to hold locks for as little time as possible, and we
// especially don't want to leave a file locked after we're done with it. Our
// Close method is what releases the locks, so use a finalizer to report
// missing Close calls on a best-effort basis.
runtime.SetFinalizer(f, func(f *File) {
panic(fmt.Sprintf("lockedfile.File %s became unreachable without a call to Close", f.Name()))
})

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 25, 2020

For language-change proposals, please also answer the questions in the language change template.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Mar 25, 2020

I'm sorry, I don't understand what the actual proposal is. Are you just suggesting an idea? Do you have a specific language change in mind?

@markdingo
Copy link
Author

@markdingo markdingo commented Mar 25, 2020

I'm sorry, I don't understand what the actual proposal is. Are you just suggesting an idea? Do you have a specific language change in mind?

I was trying to avoid proposing a specific language change as I was more interested in whether the idea of being able to define a destructor was of more general interest beyond my own needs, or not.

But yes, a concrete proposal might make my ramblings clearer.

One implementation might be a SetFinalizer variant which offers more certainty about when the finalizer function gets called. Specifically that the finalizer function gets called as soon as the object becomes unreachable, not at some indeterminate time in the future when the GC happens to notice. More like the semantics of a C++ smart pointer than a Java Finalizer.

One goal is to replicate the typical Open;defer Close() idiom in terms of the timing of releasing resources without objects needing to introduce Close() functions and have programmers remember to call them.

Things get a lot trickier when object references are passed around and copied. But semantically this variant finalizer call should occur as soon as all references disappear.

From my limited internals knowledge of go, this seems like it might unfortunately be hard to implement, but I really don't have much of a clue on that front. And frankly it doesn't matter if hardly anyone thinks it's a good idea in the first instance.

But, having objects be able to reliably release resources on their own volition as soon as they can no longer be reached by the application is the over-arching goal.

I am guessing but if we could wave a magic wand and replace all existing Close() calls with SetFinalizer equivalents, that there would be plenty of applications which would blow out resource usage in some way. So go is currently stuck with these bodged up, hand-crafted destructors called Close(). Are we ok with that?

@markdingo
Copy link
Author

@markdingo markdingo commented Mar 25, 2020

If you want to be careful about avoiding resource leaks, I would recommend using the finalizer to verify that Close was called

That is an excellent tip, thanks.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Mar 25, 2020

It's very difficult for a garbage collected language to have destructors that take effect as soon as the object becomes unreachable, because there is nothing that tracks whether objects are reachable. The best that a garbage collected language can normally do is run a destructor when the garbage collector discovers that the object is unreachable; that is often called a finalizer, and Go already has finalizers.

Destructors work in languages like C++ because C++ doesn't have garbage collection. Therefore, every object has a lifespan that is precisely controlled by the program, either because the object is a local variable or because it lives until the program explicitly calls delete.

@markdingo
Copy link
Author

@markdingo markdingo commented Mar 25, 2020

It's very difficult for a garbage collected language to have destructors

Yup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.