Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

devicestate: keep log from install-mode on installed system #9545

Merged
merged 8 commits into from
Feb 2, 2021

Conversation

mvo5
Copy link
Contributor

@mvo5 mvo5 commented Oct 26, 2020

This commit ensures that we capture the full log of the ephemeral
install mode on the writable disk when installing the system.

This is useful for debugging if install mode worked as expected
or to get logs about why things are different than they should
have been.

This is an initial PR - if this approach looks reasonable I will add
the missing unit tests (spread is there already).

We recently got asked to add this for QA purposes.

@mvo5 mvo5 added the UC20 label Oct 26, 2020
Copy link
Collaborator

@zyga zyga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love where this is going, left some ideas.

// is no systemd-journal-remote on core{,18,20}
//
// XXX: or only log if persistant journal is enabled?
logPath := filepath.Join(rootdir, "var/log/install-mode.log")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small suggestion to store this on the FAT partition appropriate for the device. We could also zip-compress it with a bunch of other files, like seed.yaml. This would be excellent for easy (windows) diagnostic of failed startup.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's an interessting idea. I wonder if there is any risk in storing this on a unencrypted partition, i.e. if it could contain sensitive information. This is why I opted for ubuntu-data but maybe this is more a development helper?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the log could contain sensitive data so I think it should go on ubuntu-data

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how long does this take? do we know how big those logs are for example for our default amd64 install?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The log is 214K on my nested-google amd64 install with debug. I have no exact timing data from the install bug running the equivalent command journalctl > file takes: real 0m0.049s.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that for now storing it on ubuntu-data is the safest approach.

This commit ensures that we capture the full log of the
ephemeral install mode on the writable disk when installing
the system.

This is useful for debugging if install mode worked as expected
or to get logs about why things are different than they should
have been.
Copy link
Member

@anonymouse64 anonymouse64 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this, I think it's good, I don't think we need to care right now whether persistent journal is enabled or not, this will be helpful even for folks trying uc20 on i.e. a stock rpi image where we can ask them for install mode logs from just putting their SD card into their main machine and looking at the specified file (assuming that install mode finishes up to that point).

I would really like to see the logs get integrated into the journald properly, is it really just /etc/machine-id we need to copy over from install mode to run mode for that to work? If so I think we could copy the ephemeral one generated at install time to run mode at the same place in devicestate

overlord/devicestate/handlers_install.go Outdated Show resolved Hide resolved
@pedronis pedronis self-requested a review November 19, 2020 16:48
Copy link
Collaborator

@pedronis pedronis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

questions

// is no systemd-journal-remote on core{,18,20}
//
// XXX: or only log if persistant journal is enabled?
logPath := filepath.Join(rootdir, "var/log/install-mode.log")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how long does this take? do we know how big those logs are for example for our default amd64 install?

@anonymouse64 anonymouse64 self-requested a review December 4, 2020 20:53
@mvo5 mvo5 marked this pull request as ready for review January 7, 2021 21:15
@mvo5 mvo5 added the Run nested The PR also runs tests inluded in nested suite label Jan 7, 2021
}
defer f.Close()

cmd := exec.Command("journalctl")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there should always be only the log from current but, but maybe as extra safety we should call journalctl -b 0.

defer f.Close()

cmd := exec.Command("journalctl")
cmd.Stdout = f
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just thinking, would this work?

logPath := filepath.Join(rootdir, "var/log/install-mode.log.gz")
...
f, err := os.Create(logPath)
...
gw := gzip.NewWriter(f)
defer gw.Close()
cmd.Stdout = gw
if err := cmd.Run(); err != nil {
    return err
}
return gw.Flush()

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reference, the log gets compressed quite vell:

216 -rw-r--r-- 1 root root 220057 Jan 12 13:37 install-mode.log
 36 -rw-r--r-- 1 root root  35052 Jan 12 13:37 install-mode.log.gz

Signed-off-by: Maciej Borzecki <maciej.zenon.borzecki@canonical.com>
Signed-off-by: Maciej Borzecki <maciej.zenon.borzecki@canonical.com>
Copy link
Member

@anonymouse64 anonymouse64 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, this is certainly better than no logs at all

@mvo5
Copy link
Contributor Author

mvo5 commented Jan 13, 2021

Thanks @bboozzoo for adding the gzip in there.

Copy link
Collaborator

@pedronis pedronis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks reasonable to me

@bboozzoo bboozzoo merged commit 8afbd38 into snapcore:master Feb 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Run nested The PR also runs tests inluded in nested suite
Projects
None yet
5 participants