Variable Scope and Garbage Collection

Or, more colorfully: "Where the #$^%#$@$ did my timer go?"

In Lua, memory management is handled through a process known as garbage collection. The way garbage collection (sometimes just shortened to GC) works is that at certain intervals, while running Lua code, the Lua interpreter decides to check if there are any user objects (of the types userdata, table, function, or thread (coroutines) -- string and number are handled differently) that are no longer being referenced -- if so, it clears them freeing up the memory used. Usually this happens without our notice and allows us to ignore things like freeing up variables we no longer need or releasing memory that we're no longer using -- it happens automatically for us.

In Hammerspoon, most modules use either a userdata or a table to provide an interface to macOS system objects so we can do interesting things like execute a function at scheduled intervals (hs.timer), manipulate other applications or windows (hs.application and hs.window), watch for file changes (hs.pathwatcher), etc.

Here I discuss the two most common mistakes that can cause these to seem to randomly stop working and a variety of ways to correct them.

In all cases where code is presented and you want to try it out for youself in the Hammerspoon Console, you must copy the entire code block and paste it into the console as one entry before hitting Enter -- entering these as separate lines into the console won't work (the reason for this is described in the section about code blocks and scopes later on in this document).

Mistake 1: Failing to capture the userdata:

As an example, consider the following code:

hs.timer.doEvery(60, function()
    print("Another minute has gone by...")
end)

If you enter this example into the Hammerspoon Console, you will see something similar to hs.timer: running (0x16ecb14d8) printed to the console. This is because the hs.timer.doEvery function returns a userdata object, but we didn't capture it -- so, the console does what it does with every command which returns a value that isn't captured: it prints a string representation of it. (This is why you can type arithmetical expressions directly into the console, e.g. 1 + 1, and get back the answer)

But because we didn't capture the userdata, it is not actually referenced anywhere, so when the Lua garbage collector gets around to discovering it, the timer object is collected, disabled, and the memory is released.

The simplest fix for this mistake is to capture the userdata in a global variable:

myTimer = hs.timer.doEvery(60, function()
    print("Another minute has gone by...")
end)

This time if you type it into the console, nothing is printed after the command is entered -- because the return value is now captured in a global variable.

The global variable holds a reference to the userdata, so garbage collection will leave it alone.

Because it's captured in the myTimer variable, you can stop it whenever you want with myTimer:stop() and restart it again with myTimer:start().

Note that issuing myTimer:stop() only stops the timer -- it does not clear the reference or release the memory. This is because stop (like start) is a method for an hs.timer object -- it performs an action on or for the object.

To fully clear the timer and release the memory when we are done with it, all you need to do is assign something else to myTimer since this is the only reference to the object. myTimer = nil would be sufficient; now when garbage collection notices it, the timer will be automatically stoped and cleared from memory.

Notice that I said "when garbage collection notices it"... garbage collection can only occur while the Lua interpreter is executing Lua code, so in a seeming paradox, the busier your Hammerspoon setup is, the faster it will likely be automatically cleared, though it may still take a while (and if the timer has a particularly short repeat interval, it may even invoke the callback a few more times before being collected). For this reason, it's usually safer to stop the timer (or the equivalent for other module objects) yourself before clearing it. The following would be a better way to finish with the timer:

myTimer:stop()
myTimer = nil -- we could assign anything we wanted to it, even a brand new timer if we want to

The down side to this approach is that we are using a global variable for every object (timer, image, pathwatcher, etc.) that we want to make sure remains until we explicitely clear it. Global variables are stored in the common global name space, which is just a fancy way of saying that it can be accessed from any Lua code running in Hammerspoon -- the console, another file loaded via require, hs.loadSpoon, dofile, loadfile, or even a string or function passed to load. Global variables are visible to anything running in the same Lua instance.

In a simple configuration, especially if only one or two people are contributing to its maintenance, this might be fine, as long as you're careful. In a more complex setup with a lot of separate files being loaded at various times or with multiple contributors, you can quickly forget what global variables you've defined in what file... if a second file also stores a timer in the myTimer global variable, the first one is clobbered if it doesn't have any other references as soon as the second one is setup.

Additional ways to save a reference to the object

To minimize the likliehood of accidently overwriting a global variable used by another file, there are some common approaches that people use:

Create a special "unique" global table for each separate file that you may load and store objects you need to maintain a reference to as members of the given files unique table.
Use require or hs.loadSpoon to load all of your files and take advantage of the Lua built-in package.loaded global table.
Use local variables.

Number 3 has its own common pitfall, so discussion of it will wait until later.

Approach 1

For this example, I'd like to credit jeffmikels@gmail.com (https://groups.google.com/g/hammerspoon/c/ROEqi2cUvZQ/m/fL-egFu-AAAJ).

Let's assume you've defined all of your hotkeys in a file name hotkeys.lua in your ~/.hammerspoon directory and that it is loaded by your ~/.hammerspoon/init.lua file (how is unimportant in this case). hotkeys.lua might look something like this:

hk = {}
hk.var1 = 'this will never go out of scope'
hk.var2 = 'this will also not go out'
hk.ctrlA = hs.hotkey.bind({'ctrl'}, 'a', 'CTRL+A', ...)

Now instead of having to remember all of the names for possible global variables defined in hotkeys.lua, you only have to remember one -- hk.

Pros:

Significantly reduces the number of global variables you have to remember not to define in multiple files
Allows easy investigation and possibly modification from wtihin the Hammerspoon Console (e.g. hs.inspect(hk)) if something isn't working quite right, rather than having to edit a file and reload multiple times.

Cons:

We still have to remember which global variables are created by other files, but as mentioned above, the number has been significantly reduced.

Approach 2

Every file which is loaded via require can optionally return a value which will be stored in package.loaded. Most commonly this is a table, and in the case of modules, it is the table which contains the top level functions for the module. As an example:

hs.inspect(package.loaded["hs.timer"])

will return:

{
  __refTable = 9,
  absoluteTime = <function 1>,
...cut for brevity...
  weeks = <function 19>
}

We can take advantage of this with our own files if we define them following a simple pattern and then load them with require (dofile or loadfile won't work).

Continuing with the hotkey example, let's modify it so it looks like this:

local hk = {} -- more on local later

hk.var1 = 'this will never go out of scope'
hk.var2 = 'this will also not go out'
hk.ctrlA = hs.hotkey.bind({'ctrl'}, 'a', 'CTRL+A', ...)

return hk -- here we return the table with all of its contents

Now, if we load this with require("hotkeys"), a reference to the table is automatically saved for us as package.loaded["hotkey"].

Pros:

no additional global variables were created -- we're just taking advantage of one that already always exists in Lua.

Cons:

a little more difficult, though not by much, to access for investigation or modification from the console (i.e. hs.inspect(package.loaded["hotkeys"])) instead of hs.inspect(hk).

It should be noted that hs.loadSpoon uses require internally to load spoons, so if you follow the conventions and examples described at https://github.com/Hammerspoon/hammerspoon/blob/master/SPOONS.md when writing your own Spoons, objects stored in the spoon obj table will be protected by this method.

Now on to local variables...

Mistake 2: Failing to maintain a reference to a local variable

In the discussion of the first common mistake, we fixed it by assinging the timer to a global variable. What if we don't want to use global variables at all?

What it means for a variable to be `local`

A local variable has a more limited scope and effectively doesn't exist outside of that scope. You can think of a variable's scope as the code block in which it is defined. Basic examples of code blocks are:

The text you type into the Hamemrspoon Console's input field -- each time you enter commands and hit Enter, the whole input field is treated as a single code block.
The string, or the concatenation of strings produced by the generator function, passed to load
Each individual file loaded via require, hs.loadSpoon, dofile, or loadfile.
Any block of code that ends with end (or for the if construct, ends with else). Some examples (illustrative, not meant to be exhaustive):
- for i, v in pairs(tbl) do .. this is a code block .. end
- if true then .. this is a code block .. else .. this is a different one .. end
- while i ~= 10 do .. this is a code block .. end
- function myFunction() .. this is a code block .. end

As a first attempt, we might try the following:

local myTimer = hs.timer.doEvery(60, function()
    print("One more minute has passed")
end)

If you paste this into the Hammerspoon Console, or if it were in a file that you loaded via require, the same thing would happen as when we didn't assign the hs.timer object to a global variable -- it will get collected at some point and stop working.

In this case, however, it's not because the object isn't being captured -- it is; the problem is that where the object is being captured is going out of scope (either as soon as the Console executes the code, or as soon as execution of the Lua code within the loaded file has completed).

Some background first

To understand how to fix this without using global variables, we need to understand a little bit more about how Lua handles things internally.

References

First, let me be a little more specific about what it means to "reference" a value in Lua. A reference for a userdata, table, function, or thread (strings and numbers are handled differently and don't concern us here) can be thought of as:

the assignment of a specific instance of an object to a variable
storing a specific instance of an object in a table, either as a value or a key.

a = hs.timer.doEvery(1, function() print("beep") end)
b = { a }
c = { [a] = b }

The timer userdata has a reference count of 3 -- the variable a, as a value in the table assigned to b, and as a key in the table assigned to c.

The table assigned to b has a reference count of 2 -- the variable b and as a value in c.

The table assigned to c has a reference count of 1 -- the variable c.

If we set a = nil, the timer won't be collected because it is still a value in b and a key in c. These additional references are still members of tables that have their own independant reference counts, so the timer is protected as long as they aren't collectable. However, if we also set b = nil and c = nil, then the timer can finally be collected because it will no longer have a reference.

The Lua Registry

The Lua registry is an internal table used by modules which include compiled code (i.e. non-Lua code) to store references for objects that the compiled code needs to save. Unless you are planning to write your own modules for Hammerspoon, the only things you really need to know about it are:

that it exists
and that modules save things like tables or callback functions in it so that they don't get collected prematurely. As an example:

local myCallback = function()
    print("I'm a callback function")
end
myTimer = hs.timer.doEvery(5, myCallback)

In this varient, we define the callback function for the timer in the myCallback local variable and then specify it as the second argument to hs.timer.doEvery. Internally hs.timer.doEvery stores a reference to this function in the Lua registry, so even though myCallback will be cleared as soon as this code block is executed, the function won't be collected and cleared from memory because it is also referenced by the Lua registry.

Some additional notes about `local`

When defining local variables, there are a couple of other details that are worth noting:

Local variables defined within a block will "hide" any variable of the same name which is defined in an outer scope or the global scope after it is defined:

j = 10
for i = 1, 5, 1 do
    local j = j + i
    print(i, j)
end
print("outside", j)

results in:

First, note that j + i refers to the global j in its expression (because j hasn't been defined local until after the evaluation of j + i). Second, note that even though the loop is repeated, each time the end line is reached, the local j goes out of scope, so as i increases, j + i uses the global j again.

A code block can refer to any local variable defined before it, if it's still in scope and if it's defined within a code block that contains the inner one:

-- a silly example, I know, but it illustrates the point
local i = 0
while i ~= 10 do
    if i == 5 then
        local halfway = true
    end
    if i == 7 then
        print(i, halfway, later)
    end
    local later = "a string"
    i = i + 1
end

results in:

7	nil	nil

Note that the block containing print(i, halfway, later) could acces i, but halfway and later were nil (undefined)

halfway because even though 7 > 5 -- the scope where halfway was defined as true had ended and thus so had the variable's definition
later because even though it's within a containing scope of the code block with the print line, it's defined after it.

Fixing Mistake 2 (finally)

With these notes in mind, we can now fix the second mistake using what Lua calls an up-value, also known as a closure in other languages:

local myTimer -- because we're going to use this variable name within the assignment, we have to make the declaration and the assignment two separate lines
myTimer = hs.timer.doEvery(60, function()
    print("One more minute has passed")
    local a = myTimer -- makes myTimer an up-value
end)

In this example, we separate the defining of myTimer as local and its assignment into two separate lines. This is necessary because we need to use myTimer from within the function that hs.timer.doEvery will be invoking. If you forget to do this, then the local a = myTimer line will be assigning the global variable named myTimer (which will probably be nil) to a.

Also remember that hs.timer.doEvery stores the function it's going to call in the Lua registry. So we're referring to myTimer from within a function that is protected from being collected -- even when this code block ends and the local myTimer variable should be cleared, a reference to it remains protected -- in Lua terms, myTimer has become an up-value because it's defined at least one level "up from" the function code block where it is being used, which means that when the function is stored in the registry table, it also has to maintain a reference to myTimer.

Of course, this creates a timer that cannot be stopped short of quitting Hammerspoon, a more useful example might be something like:

local count = 0
local myTimer -- declare before assignment
myTimer = hs.timer.doEvery(60, function()
    print("One more minute has passed")
    count = count + 1 -- makes count an upvalue
    if count == 5 then
        myTimer:stop() -- makes myTimer an up-value
        myTimer = nil -- this releases the up-value reference to the timer object
    end
end)

In this example, we use a second up-value, count which tracks the number of times the timer has called its callback function, though you could use any logic here you wanted. After 5 times, it stops and clears the timer. When garbage collection clears the timer object from memory, it will remove the function reference from the Lua registry.

Final Notes

In summary, the most common reason that timers and watchers may randomly seem to stop working comes down to garbage collection. And the best solution is what works for your code -- depending upon what you're trying to do, one method described above may be simpler or more convienent to use than another.

Use what makes the most sense to you at the time -- there is not a wrong answer if it works; just tradeoffs as each approach has its strengths and its weaknesses. The Lua portions of the Hammerspoon modules themselves use all of these approaches in different places, depending upon the specific need and intent.

Home | Tutorials | Sample Configurations

What's New?

0.9.79

Modules

hs.canvas Examples
hs.axuielement

Developer

Coroutines

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Variable Scope and Garbage Collection

Mistake 1: Failing to capture the userdata:

Additional ways to save a reference to the object

Approach 1

Approach 2

Mistake 2: Failing to maintain a reference to a local variable

What it means for a variable to be `local`

Some background first

References

The Lua Registry

Some additional notes about `local`

Fixing Mistake 2 (finally)

Final Notes

Table of Contents

What's New?

Modules

Developer

Clone this wiki locally

Variable Scope and Garbage Collection

Mistake 1: Failing to capture the userdata:

Additional ways to save a reference to the object

Approach 1

Approach 2

Mistake 2: Failing to maintain a reference to a local variable

What it means for a variable to be local

Some background first

References

The Lua Registry

Some additional notes about local

Fixing Mistake 2 (finally)

Final Notes

Table of Contents

What's New?

Modules

Developer

Clone this wiki locally

What it means for a variable to be `local`

Some additional notes about `local`