fresh Debian VM with big private storage fails to start

[thejh]

Qubes OS version:

R4.0

Affected component(s):

probably core-agent-linux?

---

Steps to reproduce the behavior:

1. Set up a clean Qubes 4.0 installation, as described in the update instructions.
2. Create a new qube, with template debian-9, with the box "launch settings after creation" ticked.
3. Set the private storage max size to 110000MiB.
4. Save the settings.
5. Attempt to start the newly created VM.

Expected behavior:

The new VM should start, going into Transient state first, then into Running state.

Actual behavior:

The new VM starts and goes into Transient state, uses a lot of CPU power for some time, then turns off again. The logfile shows that the VM spends around a minute in the stage "A start job is running for initiali...unt /rw and /home".

General notes:

I think the problem might be that the VM attempts to initialize the whole disk or so while some other system component is applying a timeout to VM startup? Creating VMs with smaller storage sizes (like 3GiB) seems to work fine, including Debian VMs.
I ended up working around this by first creating a VM with normal private storage size, then resizing the private storage at runtime.
(The reason I'm creating such a gigantic VM is that I need a place to which I can download my Qubes backup from R3.)
My disk speed (when dd'ing from /dev/xvdb to /dev/null in a VM) is around 200-300MB/s.

---

Related issues:

[marmarek]

On first VM startup there is need for mkfs. And to be absolutely sure that no data is overridden, the script compares the whole device to /dev/zero (actually as a trigger for mkfs). This takes time on large device...
Enlarging private volume after first VM startup avoids this problem, as you've already noticed.
Any better idea for reliable method of checking if device is empty?

[marmarek]

On first VM startup there is need for mkfs. And to be absolutely sure that no data is overridden, the script compares the whole device to /dev/zero (actually as a trigger for mkfs). This takes time on large device...
Enlarging private volume after first VM startup avoids this problem, as you've already noticed.
Any better idea for reliable method of checking if device is empty?

[thejh]

From dom0, I think it should be doable by checking whether the allocated size of the disk image file / LVM logical volume is zero.

For LVM logical volumes, I think the DM_TABLE_STATUS ioctl can be used on the control device to determine the allocated size; in the kernel, it goes via retrieve_status, which does an indirect call to a function like thin_status, which should give you a string with the number of used sectors and the index of the highest used sector AFAIU. The LVM userspace library also has some code for interacting with that; this code is used by the userspace tool lvdisplay for calculating the "Mapped size" line.

For disk image files, AFAIU you could use lseek(fd, 0, SEEK_DATA) to determine whether any part of the file is allocated.

[thejh]

From dom0, I think it should be doable by checking whether the allocated size of the disk image file / LVM logical volume is zero.

For LVM logical volumes, I think the DM_TABLE_STATUS ioctl can be used on the control device to determine the allocated size; in the kernel, it goes via retrieve_status, which does an indirect call to a function like thin_status, which should give you a string with the number of used sectors and the index of the highest used sector AFAIU. The LVM userspace library also has some code for interacting with that; this code is used by the userspace tool lvdisplay for calculating the "Mapped size" line.

For disk image files, AFAIU you could use lseek(fd, 0, SEEK_DATA) to determine whether any part of the file is allocated.

[marmarek]

But not that easy (I think?) from VM...

[marmarek]

But not that easy (I think?) from VM...

[marmarek]

Another idea is relaxing the check, to compare for example only first 1GB or such. This takes only few seconds. Not as reliable as full device check, but IMO still much better than heuristics for known filesystems employed by tools like blkid.

[marmarek]

Another idea is relaxing the check, to compare for example only first 1GB or such. This takes only few seconds. Not as reliable as full device check, but IMO still much better than heuristics for known filesystems employed by tools like blkid.

[qubesos-bot]

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 testing repository for the CentOS centos7 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 testing repository for the CentOS centos7 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.0.32-1+deb9u1 has been pushed to the r4.0 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.0.32-1+deb9u1 has been pushed to the r4.0 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-agent-linux (including package python2-dnf-plugins-qubes-hooks-4.0.32-1.fc26) has been pushed to the r4.0 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-agent-linux (including package python2-dnf-plugins-qubes-hooks-4.0.32-1.fc26) has been pushed to the r4.0 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-agent-linux (including package python2-dnf-plugins-qubes-hooks-3.2.31-1.fc26) has been pushed to the r3.2 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.2-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-agent-linux (including package python2-dnf-plugins-qubes-hooks-3.2.31-1.fc26) has been pushed to the r3.2 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.2-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_3.2.31-1+deb9u1 has been pushed to the r3.2 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_3.2.31-1+deb9u1 has been pushed to the r3.2 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-agent-linux (including package python2-dnf-plugins-qubes-hooks-4.0.33-1.fc26) has been pushed to the r4.0 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component core-agent-linux (including package python2-dnf-plugins-qubes-hooks-4.0.33-1.fc26) has been pushed to the r4.0 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 stable repository for the Fedora centos7 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 stable repository for the Fedora centos7 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.0.33-1+deb9u1 has been pushed to the r4.0 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-agent_4.0.33-1+deb9u1 has been pushed to the r4.0 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update