Consider: Distribution of Disk/Appvm/Template usage (Qubes-diskspace-Widget)

[Aekez]

Qubes OS version:

Qubes 4.0

Affected component(s):

• Dom0
  • Qubes-DiskSpace-Widget

---

Steps to reproduce the behavior:

From a users perspective, missing a quick and easy means to gauge the systems distribution of disk space.

• This is a much bigger problem for some users or scenarios, compared to others.
  • For example some may have plenty of disk space and runer run out, while others have little disk space and easily run out of disk space.
  • Heavy user of a lot of files and/or programs that easily eat up drive space, compared to light-users who overall only have some small files.
  • When an individual is looking for places to delete old files that are no longer needed, i.e. when running out of space and need to find more free space, or if a user does "spring cleaning" on general free-disk-space maintenance. Having no quick and easy proper insight of disk space distribution makes the task bigger.
  • It is harder to regularly keep system clean from unneeded / forgotten about data. Regular maintenance means less overall work later, i.e. like many other things in life becomes a bigger work-burden if waiting for it to build-up before doing anything about it.
  • Unexpected situations are maybe not easily realized, i.e. the free disk space trimming of AppVM's, Templates, and dom0 isn't working, and the user does not realize this. Having an easy daily overview increases the chances that a user catches issues (like lack of trimming) before they build-up or start cause problems.
  • and other scenarios that I didn't think of.

Expected behavior:

From a users perspective, a quick and easy means to gauge the systems distribution of disk space.

• Below picture is a comparative example, it does not nessicarily have to be as complex as this, or even anywhere near this example. But it gives a good idea of what type of information-direction might be desirable.

• Furthermore, maybe the code of the open source disk usage analyser can be borrowed? I.e. to show Templates / AppVM disk space use.

• The old Qube Manager also has a layout/code that maybe can be re-used.

Actual behavior:

Only total disk-size/disk-used is shown, but it gives next to none information on how it is distributed, and provides otherwise no different easy means, yet alone any means, to attain this information. A user needs to do various things, like i.e. running qvm-ls --disk in dom0, install separate dom0 disk analytic tools manually on their own, or manually go through files or VM's to figure out where space is used.

General notes:

• If any third party tools can be useful for this, then currently no third party tools is being officially recommended. Users are left alone to try figure out what to do here.
  • In addition many (all?) third party tools won't currently work either due to not accurately showing disk-usage, making it harder for users to find their own solutions to the problem.
• Not everyone can read the output of qvm-ls --disk, i.e. it may be trivial for some, but people are different, some have shown to have difficulties reading/understanding i.e. this particular command output.
• Even if a user is an advanced user, it still takes up extra time to do a search in terminal or similar methods, and in general takes more time and focus to maintain a systems management of disk-space.
• Some users might just give up, or spend a lot of time to try figure it out, which can potentially be avoided with more qubes-disk-space-analytic solutions.

Suggestion for the widget solution

Suggestion is a shot-term suggestion, or maybe even sticking with it? Since it might not be so bad because the Disk Usage Analyzer tool is pretty neat (above picture), and saves some work on the tool itself.

• dom0: sudo qubes-dom0-update baobab

  • Disk Usage Analyzer (baobab old name).
  • https://en.wikipedia.org/wiki/Baobab_(software)
  • License | GNU General Public License (source from wiki)
  • Their Git's
    • https://github.com/GNOME/baobab
    • https://gitlab.gnome.org/GNOME/baobab/tree/master/

• Dom0 Shortcuts
  sudo baobab /
sudo baobab /var/lib/qubes/appvms
sudo baobab /var/lib/qubes/vm-templates

• Suggestion to include tool in dom0 as default and use above commands in the Qubes-DiskSpace-widget as clickable links.

• Current disk space values doesn't show correctly in baobab.

  • Is it feasible and desirable to fix this for this disk tool? or maybe other solutions is better instead.

---

Related issues:

Putting this reference here, in case this suggestion is being picked up, and because its postulated to _ maybe _ be useful if the same code is being worked on when fixing the widgets printed widget-space number accuracies #4039

[airelemental]

For GUI, there's already a "disk size" column in the qubes manager.

(Which I think should be called "disk usage" or "real size" or something. "Disk size" could be misread as the "size of the disk", aka total space allocated, as opposed to "on-disk size").

[airelemental]

For GUI, there's already a "disk size" column in the qubes manager.

(Which I think should be called "disk usage" or "real size" or something. "Disk size" could be misread as the "size of the disk", aka total space allocated, as opposed to "on-disk size").

[marmarta]

Currently, the idea is that for most users, the disk space widget is mostly to be used as an alert - when disk space runs out, they will be alerted to that. They can also see at a glance if disk space situation is bad. For more complex management there is - as @airelemental said - the size on disk column in Qube Manager (I think there's a pull request somewhere to rename it to 'size' or something like that, actually). I'm not sure if making the disk size widget more complicated is warranted - I'd guess that daily checking VM sizes is not a common use case.

[marmarta]

Currently, the idea is that for most users, the disk space widget is mostly to be used as an alert - when disk space runs out, they will be alerted to that. They can also see at a glance if disk space situation is bad. For more complex management there is - as @airelemental said - the size on disk column in Qube Manager (I think there's a pull request somewhere to rename it to 'size' or something like that, actually). I'm not sure if making the disk size widget more complicated is warranted - I'd guess that daily checking VM sizes is not a common use case.

[Aekez]

I see, I did indeed overlook the Qube Manager (I thought it was affected by the disk space calculation issue as well). I barely ever use it anymore since I'm among the early Qubes 4 adopters, so I got pretty used to live without it before it was brought back, and it appears I naively thought the disk issues was happening in it too (like most/(all?) other third party disk space tools currently not working in Qubes 4).

Some problems with the Qube Manager in regard as a disk space viewer though, it might be worth bringing up before considering whether to close this or not, as I realize it might be better to make a new issue more directly focusing on specific issue, rather than one this broad in coverage in multiple directions, and leaving out the elements that doesn't matter (i.e. should support for a third party disk-space tools be a separate issue or not. User issues with the Qube Manager itself, the question how users might interact with disk-space-viewing on a Qubes system, and so on).

In regard to the Qube Manger itself as a disk space viewer

• Qube Manager takes a while to load even on the faster machines, and the loading animation feels a bit laggy (getting stuck), it feels a little bit old fashioned too, but it might just be my opinion that's off. All in all, it doesn't feel "nice" when starting the Qube Manager, it doesn't quite feel like it did back in Qubes 3.2. which was more snappy, although I realize it's due to Qubes 4 changes and new limitations, and not the Qube Managers fault.
• It's hard getting back into using Qube Manager (at least from an anecdotal single users perspective), since it had a rough start in Qubes 4, with a lot of early bugs (understandably though), but also hard to forget the almost hostility towards Qube Manager users back when it was officially announced and brought back, and the understandably tricky bugs it had due to Qubes 4 being so different. I highly respect your work marmarta, don't get me wrong, I'm fully aware it won't happen overnight and that Qubes 4 was already made before your work started on adapting the Qube Manager, so it's unfair to be critical of the Qube Manager in that way. Nevertheless, I thought it might be a good idea to bring up as feedback seems hard to come by.
• The many bugs that naturally comes with the Qube Manager (naturally due to the differences with Qubes 4 of course), but it still unavoidably makes it feel less reliable, though that impression might change after some time again as the Qube Manager keeps improving again now. But from an end-users perspective, the question easily comes up (What works in the Qube Manager? and what does not work?). Since disk space can easily be wrong, it's hard to be sure disk space work correctly in the Qube Manager, since all other disk space tools also fail in Qubes 4, especially third-party aps. It's tricky for end-users to know when these problems have been resolved or not, as disk-space bugs is not a problem easily seen in plain sight for normal users.
• The end-users perspective again, the question comes up, is new work going to be going into the Qube Manager in the future? or is it purely work on bringing it back to a state like it was in Qubes 3.2. and nothing new? i.e. will there never be improvements for the Qube Manager? I think it was Joanna who said there probably won't be, but it was never clear if it won't be for Qubes 4.0., or not at all in the future in general. Is the Qube Manager (and therein the disk space tools), going to see new improvements in the future?

Qubes end-users might be left with too many questions in present times
The biggest question though, can we rely on the numbers printed from the Qube Manger's disk space viewer in its current state? and if not, how can be communicate with the end-users who do not read on GitHub to get rid of all the uncertainty that swarms in general disk-space bugs and the Qube Manager bugs, especially during early days of Qubes 4.0.?

In response to the daily check of system disk space question, from a single end-users feedback, I have about 24 templates, and about 45-50 AppVM's, on a limited 128GB disk, although I very recently upgraded it to 256GB (and my amount of templates has also increased somewhat since my 128GB-256GB move, so an area where I as a user also might need to change instead to fit-in, and make fewer specialized templates). But the problem still lingers if users continuously run out of space, and need to keep track where to find new free space. Having limited space, especially if one has one copy of Win7 installed (never-mind two or more copies of Win7, although I don't personally use Win7 much, I still keep it around if I need it (wasting further scarce disk space)), and some other HVM's, data, and so on, it becomes a very big deal to continuously look for wasted disk space, and continuously cleaning it up so it doesn't hook up, before running out of disk space (which also corrupts system files, i.e. like some of the XFCE4 system files). But I suppose it's hard to tell how many people actually have this issue, and it isn't personally an issue for me anymore since I upgraded to 256GB instead to overcome the disk space issue. But even 256GB is outgrowing me at this rate... already starting to think about a 512GB disk, or maybe changing my habits and use two Qubes systems and move data/AppVM's between them to make space on laptop.

Final thoughts
Thanks for the hard work and effort btw, while I may seem critical of the Qube Manager, I do actually appreciate the work you put into it, and I do see it as important to keep it around for some type of users and groups too, even if I don't personally use it much right now.

Just my 2 cents, feel free to close if it's not currently an issue that can be looked further into, or in case if you feel it should be separated into smaller issues. Hopefully these thoughts can be useful, even though it's only from a single user (and probably biased in some parts to some degree too).

[Aekez]

I see, I did indeed overlook the Qube Manager (I thought it was affected by the disk space calculation issue as well). I barely ever use it anymore since I'm among the early Qubes 4 adopters, so I got pretty used to live without it before it was brought back, and it appears I naively thought the disk issues was happening in it too (like most/(all?) other third party disk space tools currently not working in Qubes 4).

Some problems with the Qube Manager in regard as a disk space viewer though, it might be worth bringing up before considering whether to close this or not, as I realize it might be better to make a new issue more directly focusing on specific issue, rather than one this broad in coverage in multiple directions, and leaving out the elements that doesn't matter (i.e. should support for a third party disk-space tools be a separate issue or not. User issues with the Qube Manager itself, the question how users might interact with disk-space-viewing on a Qubes system, and so on).

In regard to the Qube Manger itself as a disk space viewer

• Qube Manager takes a while to load even on the faster machines, and the loading animation feels a bit laggy (getting stuck), it feels a little bit old fashioned too, but it might just be my opinion that's off. All in all, it doesn't feel "nice" when starting the Qube Manager, it doesn't quite feel like it did back in Qubes 3.2. which was more snappy, although I realize it's due to Qubes 4 changes and new limitations, and not the Qube Managers fault.
• It's hard getting back into using Qube Manager (at least from an anecdotal single users perspective), since it had a rough start in Qubes 4, with a lot of early bugs (understandably though), but also hard to forget the almost hostility towards Qube Manager users back when it was officially announced and brought back, and the understandably tricky bugs it had due to Qubes 4 being so different. I highly respect your work marmarta, don't get me wrong, I'm fully aware it won't happen overnight and that Qubes 4 was already made before your work started on adapting the Qube Manager, so it's unfair to be critical of the Qube Manager in that way. Nevertheless, I thought it might be a good idea to bring up as feedback seems hard to come by.
• The many bugs that naturally comes with the Qube Manager (naturally due to the differences with Qubes 4 of course), but it still unavoidably makes it feel less reliable, though that impression might change after some time again as the Qube Manager keeps improving again now. But from an end-users perspective, the question easily comes up (What works in the Qube Manager? and what does not work?). Since disk space can easily be wrong, it's hard to be sure disk space work correctly in the Qube Manager, since all other disk space tools also fail in Qubes 4, especially third-party aps. It's tricky for end-users to know when these problems have been resolved or not, as disk-space bugs is not a problem easily seen in plain sight for normal users.
• The end-users perspective again, the question comes up, is new work going to be going into the Qube Manager in the future? or is it purely work on bringing it back to a state like it was in Qubes 3.2. and nothing new? i.e. will there never be improvements for the Qube Manager? I think it was Joanna who said there probably won't be, but it was never clear if it won't be for Qubes 4.0., or not at all in the future in general. Is the Qube Manager (and therein the disk space tools), going to see new improvements in the future?

Qubes end-users might be left with too many questions in present times
The biggest question though, can we rely on the numbers printed from the Qube Manger's disk space viewer in its current state? and if not, how can be communicate with the end-users who do not read on GitHub to get rid of all the uncertainty that swarms in general disk-space bugs and the Qube Manager bugs, especially during early days of Qubes 4.0.?

In response to the daily check of system disk space question, from a single end-users feedback, I have about 24 templates, and about 45-50 AppVM's, on a limited 128GB disk, although I very recently upgraded it to 256GB (and my amount of templates has also increased somewhat since my 128GB-256GB move, so an area where I as a user also might need to change instead to fit-in, and make fewer specialized templates). But the problem still lingers if users continuously run out of space, and need to keep track where to find new free space. Having limited space, especially if one has one copy of Win7 installed (never-mind two or more copies of Win7, although I don't personally use Win7 much, I still keep it around if I need it (wasting further scarce disk space)), and some other HVM's, data, and so on, it becomes a very big deal to continuously look for wasted disk space, and continuously cleaning it up so it doesn't hook up, before running out of disk space (which also corrupts system files, i.e. like some of the XFCE4 system files). But I suppose it's hard to tell how many people actually have this issue, and it isn't personally an issue for me anymore since I upgraded to 256GB instead to overcome the disk space issue. But even 256GB is outgrowing me at this rate... already starting to think about a 512GB disk, or maybe changing my habits and use two Qubes systems and move data/AppVM's between them to make space on laptop.

Final thoughts
Thanks for the hard work and effort btw, while I may seem critical of the Qube Manager, I do actually appreciate the work you put into it, and I do see it as important to keep it around for some type of users and groups too, even if I don't personally use it much right now.

Just my 2 cents, feel free to close if it's not currently an issue that can be looked further into, or in case if you feel it should be separated into smaller issues. Hopefully these thoughts can be useful, even though it's only from a single user (and probably biased in some parts to some degree too).

[donob4n]

@Aekez

Qube Manager takes a while to load even on the faster machines, and the loading animation feels a bit laggy (getting stuck), it feels a little bit old fashioned too, but it might just be my opinion that's off. All in all, it doesn't feel "nice" when starting the Qube Manager, it doesn't quite feel like it did back in Qubes 3.2. which was more snappy, although I realize it's due to Qubes 4 changes and new limitations, and not the Qube Managers fault.

Are you using XFCE? I'm using KDE since some months ago and I think that Qube Manager appearance looks a lot better than on XFCE. Last time I tested XFCE I saw some minor things that should be fixed like refresh icons being bad drawn.

Respect the 'Loading Qube Manager', in the older version the window appeared near instantly but you have to wait like the same time until it was full filled and usable, so I think that the loading window (which should be improved) is a better solution.

There is a way for improve the 'laggy' effect and the getting stuck, calling more frequently qt_app.processEvents(). The problem is after all the table is filled, it enables sorting and I don't think there is a simple/reliable way for handling events while doing the sort process. So for fixing the final stuck time (pretty noticeable with few VM's compared to you) I think the whole table filling process should be modified in some way it works with 'Sorting' enabled.

I would like to work on it but there are some more important bugs in current version which I will try to fix first.

The end-users perspective again, the question comes up, is new work going to be going into the Qube Manager in the future? or is it purely work on bringing it back to a state like it was in Qubes 3.2. and nothing new? i.e. will there never be improvements for the Qube Manager? I think it was Joanna who said there probably won't be, but it was never clear if it won't be for Qubes 4.0., or not at all in the future in general. Is the Qube Manager (and therein the disk space tools), going to see new improvements in the future?

I'm not sure about the official plans of Qubes devs, but I would like to at least recover RAM and CPU view on Qube Manager. Since it's already receiving dbus events about them, it should be easy to do. As a future improvement I would like to add LVM snapshots management, configure number of snapshots, rolling back, delete,.. and related to it a better disk usage report. But this is pretty far at this time and I don't know if there is some official plans about some GUI for LVM or qvm-volume.

[donob4n]

@Aekez

Qube Manager takes a while to load even on the faster machines, and the loading animation feels a bit laggy (getting stuck), it feels a little bit old fashioned too, but it might just be my opinion that's off. All in all, it doesn't feel "nice" when starting the Qube Manager, it doesn't quite feel like it did back in Qubes 3.2. which was more snappy, although I realize it's due to Qubes 4 changes and new limitations, and not the Qube Managers fault.

Are you using XFCE? I'm using KDE since some months ago and I think that Qube Manager appearance looks a lot better than on XFCE. Last time I tested XFCE I saw some minor things that should be fixed like refresh icons being bad drawn.

Respect the 'Loading Qube Manager', in the older version the window appeared near instantly but you have to wait like the same time until it was full filled and usable, so I think that the loading window (which should be improved) is a better solution.

There is a way for improve the 'laggy' effect and the getting stuck, calling more frequently qt_app.processEvents(). The problem is after all the table is filled, it enables sorting and I don't think there is a simple/reliable way for handling events while doing the sort process. So for fixing the final stuck time (pretty noticeable with few VM's compared to you) I think the whole table filling process should be modified in some way it works with 'Sorting' enabled.

I would like to work on it but there are some more important bugs in current version which I will try to fix first.

The end-users perspective again, the question comes up, is new work going to be going into the Qube Manager in the future? or is it purely work on bringing it back to a state like it was in Qubes 3.2. and nothing new? i.e. will there never be improvements for the Qube Manager? I think it was Joanna who said there probably won't be, but it was never clear if it won't be for Qubes 4.0., or not at all in the future in general. Is the Qube Manager (and therein the disk space tools), going to see new improvements in the future?

I'm not sure about the official plans of Qubes devs, but I would like to at least recover RAM and CPU view on Qube Manager. Since it's already receiving dbus events about them, it should be easy to do. As a future improvement I would like to add LVM snapshots management, configure number of snapshots, rolling back, delete,.. and related to it a better disk usage report. But this is pretty far at this time and I don't know if there is some official plans about some GUI for LVM or qvm-volume.