Instance: Fix containers not always starting up after host reboot

[tomponline]

Ignore liblxc.ErrNotRunning error when stopping.

If the container refuses to stop, then check if the error is ErrNotRunning, and if so ignore it, because sometimes if an earlier shutdown request was sent, but timed out, the actual guest shutdown can still be proceeding and the container may have reached a stop state by now and is in the process of running the onStop hook to cleanup host side devices. If we returned here with ErrNotRunning then this would be incorrect as the onStop hook could still be running and we aren't fully cleaned up yet, which can cause issues with state reporting after Stop has returned.

[MusicDin]

Cluster tests are failing

[tomponline]

Thanks @MusicDin looks like an issue when the cluster DB is offline and we try and shutdown the last LXD member:

time="2024-07-04T08:43:07Z" level=info msg="Stopping instances" stopPriority=0
time="2024-07-04T08:43:07Z" level=info msg="Shutting down instance" action=shutdown created="0001-01-01 00:00:00 +0000 UTC" ephemeral=false instance=foo instanceType=container project=default timeout=30s used="0001-01-01 00:00:00 +0000 UTC"
time="2024-07-04T08:43:09Z" level=warning msg="Failed loading instance from database, trying backup file" err="Failed to begin transaction: sql: database is closed" instance=foo project=default
time="2024-07-04T08:43:09Z" level=error msg="The stopns hook failed to load" err="Failed to begin transaction: sql: database is closed"
time="2024-07-04T08:43:09Z" level=warning msg="Failed loading instance from database, trying backup file" err="Failed to begin transaction: sql: database is closed" instance=foo project=default
time="2024-07-04T08:43:09Z" level=error msg="The stop hook failed to load" err="Failed to begin transaction: sql: database is closed"
time="2024-07-04T08:43:37Z" level=warning msg="Failed shutting down instance, forcefully stopping" err="Failed shutting down instance, status is \"Running\": context deadline exceeded" instance=foo project=default
time="2024-07-04T08:43:37Z" level=info msg="Stopping instance" action=stop created="0001-01-01 00:00:00 +0000 UTC" ephemeral=false instance=foo instanceType=container project=default stateful=false used="0001-01-01 00:00:00 +0000 UTC"

[tomponline]

That fixed it!

[MusicDin]

Thanks! This looks good, especially having less DB queries.

Just curious whether it occur that the database contains different expanded config compared to what is in backup file? Though, I cannot think of any scenario

[simondeziel]

Nice, thank you!

[tomponline]

Just curious whether it occur that the database contains different expanded config compared to what is in backup file? Though, I cannot think of any scenario

Its possible, but unlikely, as the backup file is written from the DB at each start up or config change.