logvm(s)

[marmarek]

Reported by joanna on 25 Apr 2014 16:41 UTC
A VM to collect logs from other VMs via qrexec service. Allow tools to easy copy logs to other VMs, as well as do log processing (logs are untrusted, but the logvm should be considered not-sensitive).

Potentially allow more than one logvm for paranoid configurations, cofigurable via qrexec policy.

Migrated-From: https://wiki.qubes-os.org/ticket/830

[Rudd-O]

This is EXCELLENT.

Plus, with journald log replication, it should be trivial to do.

Care must be taken that the files received by the logvm do not compromise the logvm upon using journalctl to read those logs.

[Rudd-O]

This is EXCELLENT.

Plus, with journald log replication, it should be trivial to do.

Care must be taken that the files received by the logvm do not compromise the logvm upon using journalctl to read those logs.

[andrewdavidwong]

@Rudd-O: Would you be interested in taking this on?

[andrewdavidwong]

@Rudd-O: Would you be interested in taking this on?

[Rudd-O]

I wish I could, but I am extremely busy.

Is this intended to send/receive arbitrary logs, or specific structured logs like systemd ones? systemd has a good protocol for incremental send and receive of logs, and it is quite possible that it can be adapted to this use case with a minimal shim.

What's the idea w.r.t. who initiates the copy? Is it the logvm that extracts logs from other machines? Or is it the VM producing the log entries which sends it to the logvm?

[Rudd-O]

I wish I could, but I am extremely busy.

Is this intended to send/receive arbitrary logs, or specific structured logs like systemd ones? systemd has a good protocol for incremental send and receive of logs, and it is quite possible that it can be adapted to this use case with a minimal shim.

What's the idea w.r.t. who initiates the copy? Is it the logvm that extracts logs from other machines? Or is it the VM producing the log entries which sends it to the logvm?

[marmarek]

Is this intended to send/receive arbitrary logs, or specific structured logs like systemd ones? systemd has a good protocol for incremental send and receive of logs, and it is quite possible that it can be adapted to this use case with a minimal shim.

I think it's fair assumption to have systemd (or generally syslog-like) logs. But this shouldn't use any complex protocol, to not expose too much code for attacks. It looks like a simple "one log message per line" should be enough.

What's the idea w.r.t. who initiates the copy? Is it the logvm that extracts logs from other machines? Or is it the VM producing the log entries which sends it to the logvm?

Logically VM producing the logs should send them to logvm. After all it know when there is anything for sending, so no need for polling. Maybe even send logs to logvm instead of writing them to /var/log (or /var/log/journal).

[marmarek]

Is this intended to send/receive arbitrary logs, or specific structured logs like systemd ones? systemd has a good protocol for incremental send and receive of logs, and it is quite possible that it can be adapted to this use case with a minimal shim.

I think it's fair assumption to have systemd (or generally syslog-like) logs. But this shouldn't use any complex protocol, to not expose too much code for attacks. It looks like a simple "one log message per line" should be enough.

What's the idea w.r.t. who initiates the copy? Is it the logvm that extracts logs from other machines? Or is it the VM producing the log entries which sends it to the logvm?

Logically VM producing the logs should send them to logvm. After all it know when there is anything for sending, so no need for polling. Maybe even send logs to logvm instead of writing them to /var/log (or /var/log/journal).

[Rudd-O]

On 06/12/2016 06:09 PM, Marek Marczykowski-Górecki wrote:

Is this intended to send/receive arbitrary logs, or specific
structured logs like systemd ones? systemd has a good protocol for
incremental send and receive of logs, and it is quite possible
that it can be adapted to this use case with a minimal shim.

I think it's fair assumption to have systemd (or generally
syslog-like) logs. But this shouldn't use any complex protocol, to not
expose too much code for attacks. It looks like a simple "one log
message per line" should be enough.

You would lose a LOT of useful and certified metadata if you only
ingested "one message per line" syslog-style logs. Perhaps the format
used by systemd-journald log forwarding is not suitable for what you
want to do, or perhaps it is.

Look into how it works.

What's the idea w.r.t. who initiates the copy? Is it the logvm
that extracts logs from other machines? Or is it the VM producing
the log entries which sends it to the logvm?

Logically VM producing the logs should send them to logvm. After all
it know when there is anything for sending, so no need for polling.
Maybe even send logs to logvm /instead of/ writing them to |/var/log|
(or |/var/log/journal|).

Send instead of write sounds like a sensible thing.

It sounds like what you want is a systemd-journald log forwarder on each
client VM, continually sending log data over a vchan connection to the
log VM. That should not be very difficult to do.

Rudd-O
http://rudd-o.com/

[Rudd-O]

On 06/12/2016 06:09 PM, Marek Marczykowski-Górecki wrote:

Is this intended to send/receive arbitrary logs, or specific
structured logs like systemd ones? systemd has a good protocol for
incremental send and receive of logs, and it is quite possible
that it can be adapted to this use case with a minimal shim.

I think it's fair assumption to have systemd (or generally
syslog-like) logs. But this shouldn't use any complex protocol, to not
expose too much code for attacks. It looks like a simple "one log
message per line" should be enough.

You would lose a LOT of useful and certified metadata if you only
ingested "one message per line" syslog-style logs. Perhaps the format
used by systemd-journald log forwarding is not suitable for what you
want to do, or perhaps it is.

Look into how it works.

What's the idea w.r.t. who initiates the copy? Is it the logvm
that extracts logs from other machines? Or is it the VM producing
the log entries which sends it to the logvm?

Logically VM producing the logs should send them to logvm. After all
it know when there is anything for sending, so no need for polling.
Maybe even send logs to logvm /instead of/ writing them to |/var/log|
(or |/var/log/journal|).

Send instead of write sounds like a sensible thing.

It sounds like what you want is a systemd-journald log forwarder on each
client VM, continually sending log data over a vchan connection to the
log VM. That should not be very difficult to do.

Rudd-O
http://rudd-o.com/

[marmarek]

It sounds like what you want is a systemd-journald log forwarder on each client VM, continually sending log data over a vchan connection to the log VM.

Yes, something like this. But the priority is to not allow such logvm being compromised with some mis-formatted message. If that requirement means dropping some metadata, so be it.

[marmarek]

It sounds like what you want is a systemd-journald log forwarder on each client VM, continually sending log data over a vchan connection to the log VM.

Yes, something like this. But the priority is to not allow such logvm being compromised with some mis-formatted message. If that requirement means dropping some metadata, so be it.

[Rudd-O]

On 06/12/2016 06:31 PM, Marek Marczykowski-Górecki wrote:

It sounds like what you want is a systemd-journald log forwarder
on each client VM, continually sending log data over a vchan
connection to the log VM.

Yes, something like this. But the priority is to not allow such logvm
being compromised with some mis-formatted message. If that requirement
means dropping some metadata, so be it.

This is why the receiving code in the systemd log forwarder needs to be
reviewed. I expect it to be generally better than other stuff, if only
because it is meant to be exposed as a service on a network. If, for
some reason, the code does not meet your security standards, then the
two sides of the vchan connection can do serialization and deserialization.

If you reuse the log forwarder system, you also get forward secure
sealing of logs from VMs. That means a VM cannot tamper with old log
entries (without it becoming evident) no matter what it tries to do.

Rudd-O
http://rudd-o.com/

[Rudd-O]

On 06/12/2016 06:31 PM, Marek Marczykowski-Górecki wrote:

It sounds like what you want is a systemd-journald log forwarder
on each client VM, continually sending log data over a vchan
connection to the log VM.

Yes, something like this. But the priority is to not allow such logvm
being compromised with some mis-formatted message. If that requirement
means dropping some metadata, so be it.

This is why the receiving code in the systemd log forwarder needs to be
reviewed. I expect it to be generally better than other stuff, if only
because it is meant to be exposed as a service on a network. If, for
some reason, the code does not meet your security standards, then the
two sides of the vchan connection can do serialization and deserialization.

If you reuse the log forwarder system, you also get forward secure
sealing of logs from VMs. That means a VM cannot tamper with old log
entries (without it becoming evident) no matter what it tries to do.

Rudd-O
http://rudd-o.com/

[donob4n]

So one option would be redirect journald using a version of systemd-journal-remote with vchan connections but it could be security problem due complex protocol and potentially a VM could compromise it.
As benefit logVM would have a powerful tool for monitoring, filter, search....

Another option I see, doing a very simple logVM wich just receives messages and save thems, is using an unikernel for it. It could dump the received lines in xen console so you can easily see all logs events in realtime and save them in files (separed by VM, dates, ...), if you need more detailed analysis of logs you could then bypass them (maybe with readonly access) to guiVM, dom0 or another trusted VM.

[donob4n]

So one option would be redirect journald using a version of systemd-journal-remote with vchan connections but it could be security problem due complex protocol and potentially a VM could compromise it.
As benefit logVM would have a powerful tool for monitoring, filter, search....

Another option I see, doing a very simple logVM wich just receives messages and save thems, is using an unikernel for it. It could dump the received lines in xen console so you can easily see all logs events in realtime and save them in files (separed by VM, dates, ...), if you need more detailed analysis of logs you could then bypass them (maybe with readonly access) to guiVM, dom0 or another trusted VM.

[Rudd-O]

I like the unikernel idea to save RAM in the VM that receives logs, but then querying the log would be tricky. I'd very much like to aggregate logs from several Qubes servers as well.

[Rudd-O]

I like the unikernel idea to save RAM in the VM that receives logs, but then querying the log would be tricky. I'd very much like to aggregate logs from several Qubes servers as well.

[donob4n]

Yes the idea is querying the log with another trusted VM which direct access to log files, it will not be as powerful as journald but the unikernel logic would be remain very minimal.

By Qubes servers do you mean like AppVM's?

[donob4n]

Yes the idea is querying the log with another trusted VM which direct access to log files, it will not be as powerful as journald but the unikernel logic would be remain very minimal.

By Qubes servers do you mean like AppVM's?

[donob4n]

Another option could be using the unikernel for saving raw json output from journald without parsing it. Maybe with a very basic parsing logic so it could dump something readable in the console. Then, if you want a detailed log analysys, you create a offline disposableVM which loads the desired logs from the read-only data (or a copy of it) in logVM.

This way the only vulnerable part is the disposableVM, if later appears some bug in journald you could load the unmodified original logs with a patched version. If some VM tried to exploit this bug you could detect it later.

[donob4n]

Another option could be using the unikernel for saving raw json output from journald without parsing it. Maybe with a very basic parsing logic so it could dump something readable in the console. Then, if you want a detailed log analysys, you create a offline disposableVM which loads the desired logs from the read-only data (or a copy of it) in logVM.

This way the only vulnerable part is the disposableVM, if later appears some bug in journald you could load the unmodified original logs with a patched version. If some VM tried to exploit this bug you could detect it later.

[marmarek]

See also related discussion from last year here

[marmarek]

See also related discussion from last year here

[donob4n]

After reading it, I assume that journald support should be optional.

For an initial proof of concept what do you think about using 'journalctl -f' output and the @HW42 'qubes.AppendLog' service?

[donob4n]

After reading it, I assume that journald support should be optional.

For an initial proof of concept what do you think about using 'journalctl -f' output and the @HW42 'qubes.AppendLog' service?