Feature request: test the free space in /opt/alien

[DrYak]

Use case:

• for users of microG (like me), enabling the older mapsv1 API used by some older apps requires patching system.img
• as far as I've heard using the original Google Play Services requires patching system.img

So if a user keeps both a backup of system.img and the patched one, the /opt partition doesn't have enough room left for an updated system.img and the upgrade fails.

Checking that the partition usage is <60% should do the trick

[Olf0]

Sorry, I currently fail to comprehend the issue:

• All devices supported by Jolla (except for the Jolla 1 phone) use LVM to to create two volumes: One for / and one for /home. IIRC the directory /opt resides on the root volume.
  What do you intend to address with the term "/opt partition"?
• Why are the extant checks for space on the root volume (which include an /opt directory there) not sufficient?

[Olf0]

Actually I have more questions:

• for users of microG (like me), enabling the older mapsv1 API used by some older apps requires patching system.img

The way I understood your guides at TJC and FSO, especially your latest posting, you advise against patching system.img, because deploying and using microG works fine without it (and with a GUI-only setup process) on all officially supported devices since the Xperia XA2 series (i.e., all Xperias except the Xperia X), correct?

• as far as I've heard using the original Google Play Services requires patching system.img

• Source of this hearsay?
• On which devices and SailfishOS releases?
• Do you think that is really relevant (because SailfishOS is rather used to evade Google's software)?

So if a user keeps both a backup of system.img and the patched one, the /opt partition doesn't have enough room left for an updated system.img and the upgrade fails.

IMO backups shall not reside on system partitions, right?

I am critically questioning this on a technical level (which I fail to fully understand, yet) as well as I am trying to assess how relevant this is.
I will weigh the latter against how much effort I have to put into this and how intrusive this is (to the extant sfos-upgrade script).

• Why are the extant checks for space on the root volume (which include an /opt directory there) not sufficient?

Can you please provide a pull request or at least some shell code (in this thread here) to provide an example what you are thinking of.

[Olf0]

@DrYak: Ping.

[Olf0]

@DrYak, I will close this after Christmas due to the lack of (any) response, if this is still the case then.

[DrYak]

Sorry for the glavial speed of answers: extremely busy with the data analysis with the current pandemic.

IIRC the directory /opt resides on the root volume.

That is not waht I am observing on my Xperia XA2 Ultra:

> ssh xperia 
Last login: Wed Dec 22 14:10:50 2021 from 192.168.2.3
,---
| Sailfish OS 4.3.0.12 (Suomenlinna)
'---
[nemo@Sailfish ~]$ mount
/dev/mapper/sailfish-root on / type ext4 (rw,noatime,data=ordered)
/dev/mmcblk0p68 on /odm type ext4 (ro,relatime,data=ordered)
/dev/mmcblk0p71 on /opt type ext4 (rw,relatime,data=ordered)
/dev/mmcblk0p42 on /firmware type vfat (ro,relatime,uid=1000,gid=1000,fmask=0337,dmask=0227,codepage=437,iocharset=iso8859-1,shortname=lower,errors=remount-ro)
/dev/mmcblk0p40 on /bt_firmware type vfat (ro,relatime,uid=1002,gid=3002,fmask=0337,dmask=0227,codepage=437,iocharset=iso8859-1,shortname=lower,errors=remount-ro)
/dev/mmcblk0p44 on /dsp type ext4 (rw,nosuid,nodev,relatime,nodelalloc,errors=panic,data=ordered)
/dev/mmcblk0p2 on /persist type ext4 (rw,nosuid,nodev,noatime,nodelalloc,errors=panic,data=ordered)
/dev/mmcblk1p1 on /run/media/nemo/Ulysse31 type btrfs (rw,relatime,thread_pool=8,ssd,noacl,space_cache,subvolid=257,subvol=/data)
/dev/mapper/sailfish-home on /home type ext4 (rw,noatime,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user)
 :

Android smartphones tend to have a giant pile of various partition, and even after switching to Sailfish OS that's still the case on Sony Xperia:

[nemo@Sailfish ~]$ lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0               7:0    0 334.7M  0 loop 
mmcblk0           179:0    0  29.1G  0 disk 
|-mmcblk0p1       179:1    0     2M  0 part 
|-mmcblk0p2       179:2    0    32M  0 part /persist
|-mmcblk0p3       179:3    0     1K  0 part 
|-mmcblk0p4       179:4    0     2M  0 part 
|-mmcblk0p5       179:5    0     2M  0 part 
|-mmcblk0p6       179:6    0     2M  0 part 
|-mmcblk0p7       179:7    0    16M  0 part 
|-mmcblk0p8       179:8    0    64K  0 part 
|-mmcblk0p9       179:9    0    64K  0 part 
|-mmcblk0p10      179:10   0   3.5M  0 part 
|-mmcblk0p11      179:11   0   3.5M  0 part 
|-mmcblk0p12      179:12   0     4M  0 part 
|-mmcblk0p13      179:13   0     4M  0 part 
|-mmcblk0p14      179:14   0   512K  0 part 
|-mmcblk0p15      179:15   0   512K  0 part 
|-mmcblk0p16      179:16   0   512K  0 part 
|-mmcblk0p17      179:17   0   512K  0 part 
|-mmcblk0p18      179:18   0   512K  0 part 
|-mmcblk0p19      179:19   0   512K  0 part 
|-mmcblk0p20      179:20   0     1M  0 part 
|-mmcblk0p21      179:21   0     1M  0 part 
|-mmcblk0p22      179:22   0     1M  0 part 
|-mmcblk0p23      179:23   0     1M  0 part 
|-mmcblk0p24      179:24   0   256K  0 part 
|-mmcblk0p25      179:25   0     1M  0 part 
|-mmcblk0p26      179:26   0     1M  0 part 
|-mmcblk0p27      179:27   0     1M  0 part 
|-mmcblk0p28      179:28   0     1M  0 part 
|-mmcblk0p29      179:29   0     1M  0 part 
|-mmcblk0p30      179:30   0     1M  0 part 
|-mmcblk0p31      179:31   0   128K  0 part 
|-mmcblk0p32      259:0    0    64M  0 part 
|-mmcblk0p33      259:1    0    64M  0 part 
|-mmcblk0p34      259:2    0   256K  0 part 
|-mmcblk0p35      259:3    0   256K  0 part 
|-mmcblk0p36      259:4    0   256K  0 part 
|-mmcblk0p37      259:5    0   256K  0 part 
|-mmcblk0p38      259:6    0    64M  0 part 
|-mmcblk0p39      259:7    0    64M  0 part 
|-mmcblk0p40      259:8    0     1M  0 part /bt_firmware
|-mmcblk0p41      259:9    0     1M  0 part 
|-mmcblk0p42      259:10   0   110M  0 part /firmware
|-mmcblk0p43      259:11   0   110M  0 part 
|-mmcblk0p44      259:12   0    16M  0 part /dsp
|-mmcblk0p45      259:13   0    16M  0 part 
|-mmcblk0p46      259:14   0     4M  0 part 
|-mmcblk0p47      259:15   0     4M  0 part 
|-mmcblk0p48      259:16   0    32M  0 part 
|-mmcblk0p49      259:17   0    32M  0 part 
|-mmcblk0p50      259:18   0     1M  0 part 
|-mmcblk0p51      259:19   0     4K  0 part 
|-mmcblk0p52      259:20   0   256K  0 part 
|-mmcblk0p53      259:21   0   256K  0 part 
|-mmcblk0p54      259:22   0     4K  0 part 
|-mmcblk0p55      259:23   0  32.7M  0 part 
|-mmcblk0p56      259:24   0     4K  0 part 
|-mmcblk0p57      259:25   0     1M  0 part 
|-mmcblk0p58      259:26   0     8M  0 part 
|-mmcblk0p59      259:27   0     1M  0 part 
|-mmcblk0p60      259:28   0    16K  0 part 
|-mmcblk0p61      259:29   0     8K  0 part 
|-mmcblk0p62      259:30   0   512K  0 part 
|-mmcblk0p63      259:31   0     2M  0 part 
|-mmcblk0p64      259:32   0     1M  0 part 
|-mmcblk0p65      259:33   0    64M  0 part 
|-mmcblk0p66      259:34   0    64M  0 part 
|-mmcblk0p67      259:35   0   512K  0 part 
|-mmcblk0p68      259:36   0   450M  0 part /odm
|-mmcblk0p69      259:37   0   450M  0 part 
|-mmcblk0p70      259:38   0   850M  0 part 
|-mmcblk0p71      259:39   0   850M  0 part /opt
|-mmcblk0p72      259:40   0     1M  0 part 
|-mmcblk0p73      259:41   0    16M  0 part 
|-mmcblk0p74      259:42   0    32M  0 part 
|-mmcblk0p75      259:43   0    20M  0 part 
|-mmcblk0p76      259:44   0    20G  0 part 
| |-sailfish-root 252:0    0   2.5G  0 lvm  /
| `-sailfish-home 252:1    0  17.6G  0 lvm  /home
|-mmcblk0p77      259:45   0    20M  0 part 
|-mmcblk0p78      259:46   0   2.8G  0 part 
`-mmcblk0p79      259:47   0   2.8G  0 part 

Here you can clearly see that the /opt is in its own 850 partition. (That's where the vendor's android installation would have went on a normal Android installation):

[nemo@Sailfish ~]$ devel-su blkid /dev/-mmcblk0p7[01]
Password: 
/dev/mmcblk0p70: LABEL="vendor" UUID="8d792c1b-3680-44ad-b07a-c6e73c3c0bde" TYPE="ext4" PARTLABEL="vendor_a" PARTUUID="af51f270-1648-4538-87bd-c3e7e707cd28"
/dev/mmcblk0p71: LABEL="vendor" UUID="8d792c1b-3680-44ad-b07a-c6e73c3c0bde" TYPE="ext4" PARTLABEL="vendor_b" PARTUUID="88ba9aad-b155-4e71-b665-a01ce5d4ddb3"

What do you intend to address with the term "/opt partition"?
Why are the extant checks for space on the root volume (which include an /opt directory there) not sufficient?

If you have both a backup of the original system.img by Jolla and own patch file, the remaining space left isn't enough to unpack and update aliendalvik-system-10.0.0.58.2.1-1.5.1.jolla.armv7hl :

[nemo@Sailfish ~]$ ls -l /opt/alien/
total 690164
-rw-r--r-- 1 root root      4048 Oct  7 23:21 build.prop
lrwxrwxrwx 1 root root        17 Nov 14 12:23 system.img -> system.img.mapsv1
-rw-r--r-- 1 root root 350945280 Nov 14 12:21 system.img.mapsv1
-rw-r--r-- 1 root root 355770368 Oct  7 23:21 system.img.orig
[nemo@Sailfish ~]$ df /opt/
Filesystem      1K-blocks   Used Available Use% Mounted on
/dev/mmcblk0p71    840320 691052    124644  85% /opt

you advise against patching system.img, because deploying and using microG works fine without it (and with a GUI-only setup process) on all officially supported devices since the Xperia XA2 series (i.e., all Xperias except the Xperia X), correct?

Yes, indeed. Patching system.img is only required for corner cases (mapsv1 API). Still, some people do it (me), and get regularily bitten in the ass by forgetting to free that space before starting the upgrade. The official Sailfish OS doesn't check for space in /opt , it too only checks for space in /.

Source of this hearsay?
On which devices and SailfishOS releases?

Last time I checked, getting the real-deal Google Play Service require having files in the Android's /system directory.
On Sailfish OS devices whose aliendalvik supports AOSP 8 and later, this directory is inside the system.img image.

Alos, the Google Play Service users where the first to not the "boken SELinux XATTR" bug that poses problems when decompressing and recompressing the image (and which is why my script instead uses some loopback and bind mounting instead of decompressing). I guess that these means they also still need to patch it.

Do you think that is really relevant (because SailfishOS is rather used to evade Google's software)?

Most of the users try indeed to stay as much within non-Google software as possible.
Still some people are confronted with Apps that they absolutely need to run.
(e.g.: in my case, my friends keep insisting on using WhatsApp, and Meta keeps insisting on banning users that don't use the official app).
Some users' "that one app that the must run" might refuse to run on anything but the official Google Pay Service API. That was the case of some banking app and other such critical apps.
(Though the recent signing of the AOSP image by Jolla has aleviated some of these problems).

IMO backups shall not reside on system partitions, right?

Depends on the use case. In my case I like having the option to switch back to the original file with a simple symlink.
(Though I haven't needed that).
Also for people not keeping the backup, there should be a risk that the first Sailfish upgrade decides to keep the old file as system.img.rpmsave because the content was changed, at which point you HAVE a risk of having no enough free space.

I am critically questioning this on a technical level (which I fail to fully understand, yet) as well as I am trying to assess how relevant this is.

My reasoning goes:

• you currently check the free space on any partition that is affected by an upgrade: /, /home, etc.
• /opt is another partition that is affected by upgrade where the the space might be affected.

But you're right that it might be too much of a corner case, for you to handle in your limited volunteered time.

I might write a pull request with the upgrade at some time in the future (Next SFOS upgrade ?)

Can you please provide a pull request or at least some shell code (in this thread here) to provide an example what you are thinking of.

For the general idea, I hope you got it with the example above.
I'll try writing a PR with the actual shell code when time permits (around the next SFOS upgrade, probably).

[Olf0]

For the general idea, I hope you got it with the example above.

Yes, thank you very much. This was very helpful to comprehend your issue.

I'll try writing a PR with the actual shell code when time permits (around the next SFOS upgrade, probably).

Never mind: As you pointed out, that is easy.

• Check if /opt is on a separate partition.
• If so, warn when below a certain free space threshold.

(Or did you consider a different approach?)

What I do need help with, is to assess, what exactly to check for.
Currently I am considering to check for at least 400 MBytes free space, because the size of a system image is ≥ 350 MBytes.
Opinions, other ideas?

P.S.: Have pleasant Christmas holidays, and please mind (you sure know that longer and better than I do): This pandemic will not go away anytime soon, so after 1,5 years and more to come there is no reason to overly stress oneself. IMO there is nothing to be achieved quickly. 😉

[Olf0]

I created commit 4e44e4c, but I am not convinced that this is the right thing to do!

• Your non-technical arguments are trying to counter the "Jolla users want a Google-free system" assumption. While this is definitely true for a few, IMO they are misguided and shall use an Android device instead, even more so people who insist in using software from Facebook (now "Meta"), e.g., Whatsapp, the Facebook app, the Twitter app etc.! Still Jolla is eliminating hurdles for banking apps etc. step-by-step, and with microG and Jolla's system signing most apps should work.
  More importantly all this is no reason to keep a backup of the system.img file in /opt, that may just be a reason to patch it (see next point).

• Your technical point was:

  Also for people not keeping the backup, there should be a risk that the first Sailfish upgrade decides to keep the old file as system.img.rpmsave because the content was changed, at which point you HAVE a risk of having no enough free space.

  But that is not true, please take a look at the output of rpm -ql aliendalvik-system, which lists /opt/alien/system.img as the only file in that RPM, and rpm -qc aliendalvik-system, which shows that no file in that RPM is defined as a "config file". Thus Jolla does simply overwrites the extant system.img in /opt/alien when updating it, regardless if its content was altered.

  Consequently when upgrading SailfishOS there is no need for more than a few 10 MBytes free space on an /opt partition the way Jolla has designed this.
  Only if one insists in keeping backups of the system.img file in /opt, space there may become scarce.

Thus, despite having developed a "solution" in the branch Olf0-check-opt, I still lack to see the problem it solves.
As the system.img file does not contain any privacy relevant data (regardless if patched or not), one can backup the original and the patched version anywhere, e.g. on (unencrypted) SD-card or "in the cloud". But most appropriate places (somewhere under /home, on an encrypted SD-card or per scp on one's PC) even provide confidentiality and tamper resistance.
To use an arbitrarily named symbolic link (e.g., system.img.patched) as a reminder which points to a patched system.img is fine (vice versa your scheme), but to keep a backup of this huge file in /opt is structurally wrong IMO: There should only be one system.img file in /opt/alien/, which should be named system.img. Both, the original and the patched system.img files shall be backed up at one of the aforementioned appropriate places.

If you still want to pursue this, please provide a scheme which ensures that regular users (i.e., those without multiple system.img files in /opt/alien/) are not spammed with senseless warnings when using sfos-upgrade (because for them a few 10 MBytes free space on an /opt partition is sufficient for upgrading SailfishOS successfully), preferably as a Merge Request against the Olf0-check-opt branch.

[Olf0]

@DrYak: Ping.

[Olf0]

Closing due to lack of feedback.
Will reopen this issue, if these points are addressed convincingly, specifically the last paragraph.

[DrYak]

Consequently when upgrading SailfishOS there is no need for more than a few 10 MBytes free space on an /opt partition the way Jolla has designed this.

That's not my own experience.
AFAIK (but I am not certain, I haven't check the code) RPM doesn't overwrite files in-place (perhaps in order to be able to abort installation?) so even with way more than 10MiB (close to 128MiB in my case) the aliendalvik-system package fails to install.

Also, for the sake of the finer details:

As the system.img file does not contain any privacy relevant data (regardless if patched or not), one can backup the original and the patched version anywhere, e.g. on (unencrypted) SD-card or "in the cloud". But most appropriate places (somewhere under /home, on an encrypted SD-card or per scp on one's PC) even provide confidentiality and tamper resistance.

My logic of keeping the backup on the same partition is for the ease switching while on the move, to not depend on any external device (so no cloud, no PC), while also avoiding the extremely limited space on /home (as currently aliendalvik will always use the space in /home for its storage, it can't offload to an SD card like a real Android device does).

SD card is indeed a viable alternative as you point out (though at the cost of some transfer speed) - at some point I even tested my patching script to check if it bahaves correctly when run from an SD card on the phone (though it's not my own typical use pattern, so I haven't checked recently if it still works).

Your non-technical arguments are trying to counter the "Jolla users want a Google-free system" assumption.

It's an assumption. While it is certainly true for you (I suppose you wouldn't touch anything like this even with a 10 foot pole) and me (I try as much as possible to reduce this kind of closed source crap, though keeping balanced with some practical needs), that definitely not the case for all users of Sailfish, see the various questions that pop-up from people wanting to run apps that rely on Google Play Service.

even more so people who insist in using software from Facebook (now "Meta"), e.g., Whatsapp, the Facebook app, the Twitter app etc.!

(Now I am completely veering of the initial point)

it's a bit more complex because the choice of the platform isn't 100% yours, but also depends on your circle of friends. And then the "network effect" is strong, with those who are completely into running closed smartphones and apps from psychopath companies "pulling in" the rest of their friends.
At some point you risk losing contact with friends, specially if you're not that good at keeping contact, and said friends are too lazy to keep in touch with whom they consider "that weird friends who insist on not using a 'normal phone' like everybody else".
I've shut myself off from friends as an accidental consequence of not using Facebook anymore.

On the other hand newer networks like Signal and Twitter (that aren't on a crusade to ban unauthorized 3rd party application) are helping a lot.

Still Jolla is eliminating hurdles for banking apps etc. step-by-step, and with microG and Jolla's system signing most apps should work.

Yup, in my opinion, this is the best path for the future, an alternative with true freedom, rather than the "going for a system that tries not be Google, but then putting back the Google in, because you want the apps".
And I do appreciate the progress (my banking apps have stopped complaining about running on a rooted phone sign the signing!)

(End of tangeant about freedom, let's go back to the main discussion).

More importantly all this is no reason to keep a backup of the system.img file in /opt, that may just be a reason to patch it (see next point).

In the end, I think that my request is too much of a corner case that won't represent the typical user. I am not sure how many of them are really affected.
Given the lack of other people jumping into this discussion, it seems to me that there isn't a big overlap between "people who want a patched system.img with google" and "people who run sfos-upgrade".
The later being probably more technically minded and using solutions (microG) which don't normally require patching (except of extreme corner cases - apps require mapsv1 are a dying breed).
While the former are not really digging into the details (they just want the Google Play Store to work for their favorite game or app to work, while not appreciating the irony of what they're trying to achieve). The complains from users who don't understand while they are still running the older alien-dalvik after upgrading sailfish that pop on forums (to me this rings like a aliendalvik-system package failing to install) could support that those users just patch their stuff and don't give much thoughs into that and don't seem to look for more advanced upgrading solution.

So yeah, I think you're right at closing the issue.
Probably isn't worth the effort in maintaining, and the alienation that the false alerts will bring just for very few corner cases.
(And I can try to remember checking my space in /opt)

[Olf0]

So yeah, I think you're right at closing the issue.
Probably isn't worth the effort in maintaining, and the alienation that the false alerts will bring just for very few corner cases.

Thank you for your understanding.
Specifically the "false alerts" for many, which may help very few, are my main concern.