terraform creates sockets named /tmp/tf-plugin12345678 and never deletes them

[joekhoobyar]

No description provided.

[josephholsten]

On mac you'll find these files in $TMPDIR, which usually lives somewhere in /var/folders.

I can confirm I have tons of these files.

[catsby]

Those are temporary files created when loading the plugins:

• https://github.com/hashicorp/terraform/blob/master/plugin/server.go#L122

We're just using Go's built in TempFile util:

TempFile uses the default directory for temporary files (see os.TempDir)

• https://golang.org/pkg/io/ioutil/#TempFile

Those files are per-user temporary files and caches; I've found a fairly detailed writeup about them here:

• http://www.magnusviri.com/OS_X_Admin/what-is-var-folders.html

This is the result of Terraform using temporary file space for scratch. I have 1,200 some files in there (only ~700 are tf-plugin)... but all have been modified in the last 3 days (the author above mentions docs that indicate files are cleared out after 3 days if not used).

Do you have any of these files that have not been modified in the past 3 days?

[josephholsten]

So long as those files don't contain credentials or sensitive information, I'm comfortable trusting the operating system's cleanup.

[thomascellerier]

It will only be cleaned up on reboot, just so you know...

[wleese]

Surely this isn't acceptable? I now have 64840 directories after a few weeks - I cringe to think of how bad this will be after a few years of uptime.

I'm comfortable trusting the operating system's cleanup.

Which will hopefully never happen .. at least if you value high uptime.

Update: a colleague of mine just mentioned that the filesystem inodes are depleted for our tmp volumes within 12 days due to this.

[josephholsten]

@wleese: are you seeing files older than 3 days? Cleanup occurs more frequently than mere restarts on systems.

Have you confirmed that the fs does runs out of inodes in practice? On what platform is this occurring?

[wleese]

RHEL6, indeed just 3 days old. Regardless of if we run out in the end, the fact that a tool puts so much pressure on a resource that we need to worry about it indicates there's something to be fixed. This would be more forgivable is terraform had some kind of incredibly storage heavy state to manage, but yeah..

[josephholsten]

I'm sorry this is breaking things for you. Do you need any assistance writing a patch to fix this?

[wleese]

I'd expect this to be sufficient:

diff --git a/plugin/server.go b/plugin/server.go
index 3daa8a3..a1427b9 100644
--- a/plugin/server.go
+++ b/plugin/server.go
@@ -86,6 +86,7 @@ func Serve(opts *ServeOpts) {

        // Serve
        server.Accept(listener)
+       defer os.Remove(listener.Addr().String())
 }

Sadly it seems that anything deferred is not run - likely due to the way plugins are handled.

[wleese]

Will this ticket be re-opened by the way?

[bflad]

We're seeing this in our CI environment. Hundreds of /tmp/tf-plugin* files over 30 days old. I'd +1 reopening this as a bug.

[jleclanche]

Same. This is a problem, and I suspect it might cause some issues with the kernel limits if there's a lot of terraform instances being run very fast. You can't rely on reboots here.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.