Recycling Powered‐Off Servers

By default, recycling of powered-off servers that have completed executing a job is turned on.

Recycling allows for minimizing costs by allowing multiple runners to be brought up on the same server instance as Hetzner Cloud, which bills servers in 1-hour increments. Therefore, it is inefficient to delete a server if it only executed a job that runs for a few minutes. Instead, after completing a job, the server is powered off and if it can be recycled, it is rebuilt from scratch by reinstalling the image thus providing a clean environment for the next job.

Powered-off servers are marked as recyclable by changing their name to github-hetzner-runner-recycle-{uid}.

Recyclable servers are deleted when they reach their end of life period which is defined by the --end-of-life option, and by default is set to 50 minutes. The end of life is calculated on an hourly basis and must be greater than 0 and less than 60.

For example, with the default value of the --end-of-life option set to the 50 minutes, if the server is running for 2 hours and 50 minutes, then it will be considered to have reached its end of life and is deleted because it has only 10 minutes or less of useful life left in the current hour period. However, if the server is running for 2 hours and 30 minutes, then it could potentially has 30 minutes of life left, and it will be kept around to be available for recycling.

Sometimes a job might need a server that does not match any recyclable servers, if the maximum number of runners has been reached, then by default, one of the recyclable servers will be picked to be deleted to make room for a new server. By default, the recyclable server that is deleted is picked based on the server's price per hour and its remaining useful life. The server with the lowest unused budget is deleted.

The unused budget is defined as follows:

unused budget:	server_life = 60 - server_age.minutes price_per_minute = price_per_hour / 60 unused_budget = server_life * price_per_minute
✋ Note:	You can also use the --delete-random option to randomly pick a recyclable server to be deleted. Deleting servers at random is a legacy feature.

A recyclable server is recycled for a new job if it matches the following:

• The server type matches exactly what the job requires, or the default type
• The server location matches exactly if a job requests a runner in a specific location or the default location is specified
• The server has matching SSH keys

✋ Note:

Matching server type exactly means that even if a bigger, more expensive server type could be potentially recycled if it is not used, even though a job that actually requires that expensive server might not be queued before the server's end of life. This is intensional, as we can't predict when a job that actually requires the more expensive server type could be queued. If the program would allow recycling of higher server types than actually requested by a job, then we could run into cases when a job that requires a smaller and less expensive server runs on a bigger and more expensive server instead. In this case, a job that actually requires a bigger server would force a new, expensive server to be created and thus causing more expensive servers to be created than are actually necessary.

If needed, you can turn recycling off using the --recycle {on,off} option.

github-hetzner-runners --recycle off