Add instance uptime value

[stgraber]

It would be pretty useful to know how long an instance has been running.

This should be done with a millisecond value which for containers can be calculated from when the monitor process was first started, whereas for VMs, we can ask the agent for the uptime and fallback to when the qemu process was started.

We'll want the field added to InstanceState and then exposed in incus info and as a column in incus list.

[BajuMcBites]

Hello, my friend @NotPranav0, and I are students at UT Austin and after taking a virtualization class, we wanted to contribute to this project. Could we be assigned this issue? Thanks.

[stgraber]

Sure, I assigned it to you!

If @NotPranav0 comments in this issue I can also assign them to it.

[NotPranav0]

Comment

[BajuMcBites]

Hello, I'm looking to start working on this issue,

• The field being added to IncusState would be something along the lines of InstanceUptime and would track the total time the instance had been up.

• The main question I have is whether we are tracking the total time an instance has been up or just the time since the last start (does the timer reset upon an instance stop)?

In the first case, would this approach work:

• We would update InstanceUptime whenever the instance was stopped by using the time the monitor process was started and the current time (assuming new monitor process is created upon instance start call, if not we’d have to store the new start time on every instance start call)
• When returning this information in incus info or incus list we would return the value of the InstanceUptime + the time elapsed since the last start to get the total uptime.

In the second case:

• Would we need the instanceUptime variable at all because we could just calculate it whenever it is needed.

Finally, I was wondering where I could find the structs/data for the monitor processes, VM agents, and qemu processes. Thanks.

[stgraber]

• The field being added to IncusState would be something along the lines of InstanceUptime and would track the total time the instance had been up.

We can probably make that StartedAt time.Time json:"started_at" yaml:"started_at"`

That makes it a bit more consistent with other timestamps type stuff we have.
Unix users are usually used to a timestamp in seconds since boot, but a time.Time is cleaner at the API level, so let's go with that.

• The main question I have is whether we are tracking the total time an instance has been up or just the time since the last start (does the timer reset upon an instance stop)?

Time since last start. We should be able to just read the creation time of the container PID and use that as the start time so we don't need to record anything.

So you're looking at roughly this set of commits:

• api: instance_started_at (doc/api-extensions.md, internal/version/api.go)
• shared/api: Add StartedAt to InstanceState (shared/api/instance_state.go)
• doc/rest-api.yaml: Refresh swagger YAML (result of make update-api)
• incusd/instance/lxc: Implement StartedAt (internal/server/instance/driver/driver_lxc.go)
• incusd/instance/qemu: Implement StartedAt (internal/server/instance/driver/driver_qemu.go)

As mentioned, for the initial implementation we can just rely on the instance pid's creation time which should be good enough in most cases. For containers that ends up being the [lxc monitor] process, for VMs, that's the qemu process.

[BajuMcBites]

Hello, I managed to get the instance start time added to the state api, I had a few questions though.

My implementation (same for qemu and lxc so probably should make a shared method, however wasn't too sure where to put it):

// Gets the time the instance was started at
func (d *qemu) StartedAt() (time.Time, error) {
	const START_TIME_INDEX = 21
	clctck, err := sysconf.Sysconf(sysconf.SC_CLK_TCK)
	if err != nil {
		return time.Time{}, err
	}

	pid := d.InitPID()

	processInfo, err := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid))
	if err != nil {
		return time.Time{}, err
	}

	fields := strings.Fields(string(processInfo))

	bootTimeCmd, err := exec.Command("who", "-b").Output()
	if err != nil {
		return time.Time{}, err 
	}

	bootTimeString := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(string(bootTimeCmd)), "system boot"))

	timeZone, err := exec.Command("date", "+%Z").Output()
	if err != nil {
		return time.Time{}, err 
	}

    bootTimeString = bootTimeString + strings.TrimSpace(string(timeZone))	  
	bootTime, err := time.Parse(`2006-01-02 15:04MST`, bootTimeString)
	if err != nil {
		return time.Time{}, err 
	}

	startTime, err := strconv.ParseInt(fields[START_TIME_INDEX], 10, 64)
	if err != nil {
		return time.Time{}, err 
	}
	startTime = startTime / clctck
	
	return time.Unix(bootTime.Unix() + startTime, 0), nil
}

• Is how I get the boot time and time zone acceptable? I use exec.Command().Output(), which might not be the best approach, however I couldn't find another easy way to get those values.
• to get the time after boot that the process started in seconds, I had to get the clock tick speed, and I couldn't find a way to do this with using this external package ("github.com/tklauser/go-sysconf"). Is that ok?

[stgraber]

stgraber@dakara:~$ cat example.go 
package main

import (
	"fmt"
	"os"
	"syscall"
	"time"
)

func main() {
	file, err := os.Stat(fmt.Sprintf("/proc/%s", os.Args[1]))
	if err != nil {
		os.Exit(1)
	}

	linuxInfo := file.Sys().(*syscall.Stat_t)
	createdAt := time.Unix(linuxInfo.Ctim.Sec, linuxInfo.Ctim.Nsec)

	fmt.Printf("%s\n", createdAt)
}
stgraber@dakara:~$ ./example 1
2024-03-10 14:54:11.971999991 -0400 EDT
stgraber@dakara:~$ 

[stgraber]

That should be easier, just looking at the creation time on the /proc entry for the PID.

[BajuMcBites]

That was a lot easier, thanks. I made those changes and drafted PR #829.

[BajuMcBites]

I realize I haven't exposed those values in incus list and incus info, do these call the instance state endpoint and then I could simply retrieve the info in those calls and print it? Where does this happen?

[stgraber]

I've just pushed that logic and a few fixes to the branch now.