New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

meminfo writer service: improve consistency #3207

Closed
rootkovska opened this Issue Oct 24, 2017 · 19 comments

Comments

@rootkovska
Member

rootkovska commented Oct 24, 2017

Currently there is a somehow inconsistent algorithm used to determine whether to start the meminfo-writer service (Dynamic Memory Balancing) for the VM or not. Typically it is started whenever the meminfo-writer service key is present, except if the VM gets PCI devices assigned, when it is not started.

Rather than having this weird logic in the code starting the service, move the logic away to the code which creates/disabled the service key creation. Thanks to this it will be much easier for both (advanced) users as well as other software (e.g. GUI manager) to find out if memory balancing is enabled for the VM or not.

Currently (4.0-rc2), Qubes Manager (i.e. the Settings window) displays wrong information for all device-hosting VMs, which is a direct result of the problem described here.

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Oct 24, 2017

Member

Technically, HVM with PCI device cannot be started with maxmem > memory. But, when started, balloon driver can be used to take memory out of that VM. And thanks to IOMMU, memory fragmentation is not an issue anymore here (which was initial reason why we disable meminfo-writer in PCI-having VMs). So, we can support VMs with PCI devices and dynamic memory management. But such a VM needs to have all its memory during its early startup (until balloon driver is loaded).
Actually, that last point may not be needed if we use memory hotplug, I've tested it some time ago and it just work. Require one option to be enabled in VM kernel (CONFIG_XEN_BALLOON_MEMORY_HOTPLUG). And adjustments to qmemman.

This will in practice allow to use some actual software in a VM with PCI device, not only be a proxy (for network, USB etc).

So, maybe the right thing to do is not move this logic into one place, but remove it entirely? And if someone want to disable dynamic memory management of a VMs, he/she can do it anyway. And salt can do it for sys-net/sys-usb if we think its appropriate.

Member

marmarek commented Oct 24, 2017

Technically, HVM with PCI device cannot be started with maxmem > memory. But, when started, balloon driver can be used to take memory out of that VM. And thanks to IOMMU, memory fragmentation is not an issue anymore here (which was initial reason why we disable meminfo-writer in PCI-having VMs). So, we can support VMs with PCI devices and dynamic memory management. But such a VM needs to have all its memory during its early startup (until balloon driver is loaded).
Actually, that last point may not be needed if we use memory hotplug, I've tested it some time ago and it just work. Require one option to be enabled in VM kernel (CONFIG_XEN_BALLOON_MEMORY_HOTPLUG). And adjustments to qmemman.

This will in practice allow to use some actual software in a VM with PCI device, not only be a proxy (for network, USB etc).

So, maybe the right thing to do is not move this logic into one place, but remove it entirely? And if someone want to disable dynamic memory management of a VMs, he/she can do it anyway. And salt can do it for sys-net/sys-usb if we think its appropriate.

@rootkovska

This comment has been minimized.

Show comment
Hide comment
@rootkovska

rootkovska Oct 24, 2017

Member

Let's not be so fast about removing the option to disable mem balancing: we still offer an (easy) option to run VMs as PV and in the future might offer even more options.

Member

rootkovska commented Oct 24, 2017

Let's not be so fast about removing the option to disable mem balancing: we still offer an (easy) option to run VMs as PV and in the future might offer even more options.

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Oct 24, 2017

Member

It's not about removing option to disable mem balancing. It's about doing it (or not) automatically depending on various factors (PCI device having here). The later, as you noticed, currently is very non intuitive currently.

Member

marmarek commented Oct 24, 2017

It's not about removing option to disable mem balancing. It's about doing it (or not) automatically depending on various factors (PCI device having here). The later, as you noticed, currently is very non intuitive currently.

@rootkovska

This comment has been minimized.

Show comment
Hide comment
@rootkovska

rootkovska Oct 24, 2017

Member

Ok, so you're proposing to have no-meminfo-writer service key, instead, and if not present, then always start the service, right?

Member

rootkovska commented Oct 24, 2017

Ok, so you're proposing to have no-meminfo-writer service key, instead, and if not present, then always start the service, right?

@marmarek

This comment has been minimized.

Show comment
Hide comment
@marmarek

marmarek Oct 24, 2017

Member

Something like this - keep service.meminfo-writer name and always start the service, unless this feature is present and set to false.

Member

marmarek commented Oct 24, 2017

Something like this - keep service.meminfo-writer name and always start the service, unless this feature is present and set to false.

marmarek added a commit to marmarek/qubes-core-agent-linux that referenced this issue Nov 20, 2017

Dumb down meminfo-writer enabling logic
Let dom0 decide whether it should be enabled or not, regardless of PCI
devices or any other factor.

Fixes QubesOS/qubes-issues#3207

marmarek added a commit to marmarek/qubes-mgmt-salt-dom0-virtual-machines that referenced this issue Nov 21, 2017

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Nov 21, 2017

Automated announcement from builder-github

The package qubes-mgmt-salt-dom0-virtual-machines-4.0.7-1.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

Automated announcement from builder-github

The package qubes-mgmt-salt-dom0-virtual-machines-4.0.7-1.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Nov 21, 2017

Automated announcement from builder-github

The package qubes-core-agent_4.0.13-1+deb8u1 has been pushed to the r4.0 testing repository for the Debian jessie template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing jessie-testing, then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

Automated announcement from builder-github

The package qubes-core-agent_4.0.13-1+deb8u1 has been pushed to the r4.0 testing repository for the Debian jessie template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing jessie-testing, then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@qubesos-bot qubesos-bot referenced this issue in QubesOS/updates-status Nov 21, 2017

Closed

core-agent-linux v4.0.13 (r4.0) #308

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Nov 21, 2017

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 testing repository for the CentOS centos7 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 testing repository for the CentOS centos7 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Nov 21, 2017

Automated announcement from builder-github

The package qubes-core-agent_4.0.13-1+deb9u1 has been pushed to the r4.0 testing repository for the Debian stretch template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing, then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

Automated announcement from builder-github

The package qubes-core-agent_4.0.13-1+deb9u1 has been pushed to the r4.0 testing repository for the Debian stretch template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing, then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Nov 21, 2017

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc24 has been pushed to the r4.0 testing repository for the Fedora fc24 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc24 has been pushed to the r4.0 testing repository for the Fedora fc24 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Nov 21, 2017

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc25 has been pushed to the r4.0 testing repository for the Fedora fc25 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc25 has been pushed to the r4.0 testing repository for the Fedora fc25 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Nov 21, 2017

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc26 has been pushed to the r4.0 testing repository for the Fedora fc26 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc26 has been pushed to the r4.0 testing repository for the Fedora fc26 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

marmarek added a commit to marmarek/qubes-core-admin that referenced this issue Nov 30, 2017

qubesvm: make initial qmemman request consistent with libvirt config
If HVM have PCI device, it can't use PoD, so need 'maxmem' memory to be
started. Request that much from qmemman.
Note that is is somehow independent of enabling or not dynamic memory
management for the VM (`service.meminfo-writer` feature). Even if VM
initially had assigned maxmem memory, it can be later ballooned down.

QubesOS/qubes-issues#3207

marmarek added a commit to marmarek/qubes-core-admin that referenced this issue Dec 5, 2017

qubesvm: make initial qmemman request consistent with libvirt config
If HVM have PCI device, it can't use PoD, so need 'maxmem' memory to be
started. Request that much from qmemman.
Note that is is somehow independent of enabling or not dynamic memory
management for the VM (`service.meminfo-writer` feature). Even if VM
initially had assigned maxmem memory, it can be later ballooned down.

QubesOS/qubes-issues#3207

@marmarek marmarek referenced this issue in QubesOS/qubes-core-admin Dec 5, 2017

Merged

Various fixes for tests and the code #171

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Dec 11, 2017

Automated announcement from builder-github

The package qubes-core-agent_4.0.13-1+deb8u1 has been pushed to the r4.0 stable repository for the Debian jessie template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

Automated announcement from builder-github

The package qubes-core-agent_4.0.13-1+deb8u1 has been pushed to the r4.0 stable repository for the Debian jessie template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Dec 11, 2017

Automated announcement from builder-github

The package qubes-core-agent_4.0.13-1+deb9u1 has been pushed to the r4.0 stable repository for the Debian stretch template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

Automated announcement from builder-github

The package qubes-core-agent_4.0.13-1+deb9u1 has been pushed to the r4.0 stable repository for the Debian stretch template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Dec 11, 2017

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 stable repository for the Fedora centos7 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 stable repository for the Fedora centos7 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Dec 11, 2017

Automated announcement from builder-github

The package qubes-mgmt-salt-dom0-virtual-machines-4.0.7-1.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

Automated announcement from builder-github

The package qubes-mgmt-salt-dom0-virtual-machines-4.0.7-1.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Dec 12, 2017

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc24 has been pushed to the r4.0 stable repository for the Fedora fc24 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc24 has been pushed to the r4.0 stable repository for the Fedora fc24 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Dec 12, 2017

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc25 has been pushed to the r4.0 stable repository for the Fedora fc25 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc25 has been pushed to the r4.0 stable repository for the Fedora fc25 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@qubesos-bot

This comment has been minimized.

Show comment
Hide comment
@qubesos-bot

qubesos-bot Dec 12, 2017

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc26 has been pushed to the r4.0 stable repository for the Fedora fc26 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-4.0.13-1.fc26 has been pushed to the r4.0 stable repository for the Fedora fc26 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@qubesos-bot qubesos-bot referenced this issue in QubesOS/updates-status Dec 22, 2017

Closed

core-admin v4.0.15 (r4.0) #327

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment