Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iiab-expand-rootfs for more HW/OS's w/ safety checks, per raspi-config's do_expand_rootfs() #3137

Merged
merged 11 commits into from
Mar 15, 2022
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
13 changes: 6 additions & 7 deletions roles/1-prep/README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,15 @@ username and group, to log into Admin Console
* Ubermix (distro) needs /etc/tmpfiles.d/iiab.conf to create essential
/var/log subdirs on each boot
* *_Hardware actions:_*
** Install package cloud-guest-utils for growpart
** Install link:templates/iiab-expand-rootfs[iiab-expand-rootfs] that acts on flag `/.expand-rootfs` at boot
** link:tasks/raspberry_pi.yml[*_raspberry_pi.yml_*]:
*** RTC (real-time clock): install udev rule, configure, enable
*** *_Install packages related to:_*
**** growpart
**** swapfile
*** *_Install packages:_*
**** dphys-swapfile (for swapfile)
**** fake-hwclock (as RTC is often missing or dead!)
**** Wi-Fi
*** Increase swap file size
*** https://github.com/iiab/iiab/blob/master/roles/1-prep/templates/iiab-rpi-max-rootfs.sh[rootfs
auto-resizing]
**** iw, rfkill, wireless-tools (for internal Wi-Fi)
*** Increase swap file size (to `pi_swap_file_size`)
** NUC 6 Wi-Fi firmware

Recap: Similar to 0-init, 2-common, 3-base-server, 4 server-options and
Expand Down
25 changes: 23 additions & 2 deletions roles/1-prep/tasks/hardware.yml
Original file line number Diff line number Diff line change
@@ -1,17 +1,38 @@
- name: Install package 'cloud-guest-utils' so growpart is available for expanding rootfs partition during boot
package:
name: cloud-guest-utils # 2022-03-15: For RasPiOS especially. Ubuntu has still pre-installed, for use with cloud-init.
state: present

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are using raspi-config on RaspOS why install universally unless the use of raspi-config is going to go away?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Installing cloud-guest-utils is indeed currently overkill on RasPiOS.

/usr/bin/growpart is only 22 kB however, and the total package size is also tiny:

root@box:~# apt show cloud-guest-utils | grep Size

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Installed-Size: 62.5 kB
Download-Size: 23.7 kB
root@box:~# dpkg-query -L cloud-guest-utils
/.
/usr
/usr/bin
/usr/bin/ec2metadata
/usr/bin/growpart
/usr/bin/vcs-run
/usr/share
/usr/share/doc
/usr/share/doc/cloud-guest-utils
/usr/share/doc/cloud-guest-utils/NEWS.Debian.gz
/usr/share/doc/cloud-guest-utils/changelog.Debian.gz
/usr/share/doc/cloud-guest-utils/changelog.gz
/usr/share/doc/cloud-guest-utils/copyright
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/growpart.1.gz

Currently I'm inclined to continue installing it for all OS's, in case such infra utilities later help others.

But if others feel otherwise (e.g. if growpart proves to be unmaintained and/or causing problems?) then we could later try to remove it completely?

- name: "Install from templates: /usr/sbin/iiab-expand-rootfs, /etc/systemd/system/iiab-expand-rootfs.service"
template:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
mode: "{{ item.mode }}"
with_items:
- { src: 'iiab-expand-rootfs', dest: '/usr/sbin/', mode: '0755' }
- { src: 'iiab-expand-rootfs.service', dest: '/etc/systemd/system/', mode: '0644' }

- name: Enable /etc/systemd/system/iiab-expand-rootfs.service
systemd:
name: iiab-expand-rootfs
enabled: yes


## DISCOVER PLATFORMS ######
# Put conditional actions for hardware platforms here

- include_tasks: raspberry_pi.yml
when: first_run and rpi_model != "none"


- name: Check if the identifier for Intel's NUC6 built-in WiFi is present
shell: "lsusb | grep 8087:0a2b | wc | awk '{print $1}'"
register: usb_NUC6
ignore_errors: True

- name: Download {{ iiab_download_url }}/iwlwifi-8000C-13.ucode to /lib/firmware for built-in WiFi on NUC6 # iiab_download_url is http://download.iiab.io/packages
- name: Download {{ iiab_download_url }}/iwlwifi-8000C-13.ucode to /lib/firmware for built-in WiFi on NUC6
get_url:
url: "{{ iiab_download_url }}/iwlwifi-8000C-13.ucode"
url: "{{ iiab_download_url }}/iwlwifi-8000C-13.ucode" # http://download.iiab.io/packages
dest: /lib/firmware
timeout: "{{ download_timeout }}"
when: usb_NUC6.stdout|int > 0
37 changes: 12 additions & 25 deletions roles/1-prep/tasks/raspberry_pi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,21 +39,25 @@
meta: noop

# 2021-07-27 explanation from @jvonau: The 3 BELOW (iw, rfkill, wireless-tools)
# are provided by RaspiOS. Ubuntu|Debian on the other hand are hit or miss:
# are provided by RasPiOS. Ubuntu|Debian on the other hand are hit or miss:
# desktops might have some/all 3 preinstalled, while servers tend not to have
# these present at all, but are needed to be installed if you want to take full
# advantage of WiFi on Ubuntu and friends -- but it's only enforced on RPi
# hardware where we know in advance of the likelihood of WiFi being present.

- name: 'Install packages: cloud-guest-utils, dphys-swapfile, fake-hwclock, iw, rfkill, wireless-tools'
# 2022-03-15: This section is effectively now useless on RasPiOS (but apparently
# serves a purpose on Ubuntu on RPi, where it installs dphys-swapfile and
# fake-hwclock). Still: @jvonau's above explanation of other OS's / other HW
# suggest this code should possibly move in future, to helps others too?

- name: 'Install packages: dphys-swapfile, fake-hwclock, iw, rfkill, wireless-tools'
package:
name:
- cloud-guest-utils # Contains 'growpart' for resizing a partition during boot, which is normally done with the aid of cloud-init
- dphys-swapfile # 2021-07-27: RaspiOS installs this regardless -- autogenerate and use a swap file
- fake-hwclock # 2021-07-27: RaspiOS installs this regardless -- save/restore system clock on machines without working RTC hardware
- iw # 2021-07-27: RaspiOS installs this regardless -- configure Linux wireless devices -- hard dependence for ap0 creation, SEE https://github.com/iiab/iiab/blob/master/roles/network/templates/hostapd/iiab-clone-wifi.service.j2
- rfkill # 2021-07-27: RaspiOS installs this regardless -- enable & disable wireless devices
- wireless-tools # 2021-07-27: RaspiOS installs this regardless -- manipulate Linux Wireless Extensions
- dphys-swapfile # 2021-03-15: For Ubuntu on RPi? RasPiOS installs this regardless -- autogenerate and use a swap file
- fake-hwclock # 2021-03-15: For Ubuntu on RPi? RasPiOS installs this regardless -- save/restore system clock on machines without working RTC hardware
- iw # 2021-07-27: RasPiOS installs this regardless -- configure Linux wireless devices -- hard dependence for ap0 creation, SEE https://github.com/iiab/iiab/blob/master/roles/network/templates/hostapd/iiab-clone-wifi.service.j2
- rfkill # 2021-07-27: RasPiOS installs this regardless -- enable & disable wireless devices
- wireless-tools # 2021-07-27: RasPiOS installs this regardless -- manipulate Linux Wireless Extensions
state: present


Expand All @@ -68,20 +72,3 @@
systemd: # Had been...a rare/legacy service that was NOT systemd
name: dphys-swapfile
state: restarted


- name: Install RPi rootfs resizing (/usr/sbin/iiab-rpi-max-rootfs.sh) and its systemd service (/etc/systemd/system/iiab-rpi-root-resize.service), from templates (root:root by default)
template:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
#owner: root
#group: root
mode: "{{ item.mode }}"
with_items:
- { src: 'iiab-rpi-max-rootfs.sh', dest: '/usr/sbin/', mode: '0755' }
- { src: 'iiab-rpi-root-resize.service', dest: '/etc/systemd/system/', mode: '0644' }

- name: Enable RPi rootfs resizing (systemd service iiab-rpi-root-resize.service)
systemd:
name: iiab-rpi-root-resize
enabled: yes
57 changes: 57 additions & 0 deletions roles/1-prep/templates/iiab-expand-rootfs
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/bin/bash -x

# Expand rootfs partition to its maximum size, if /.expand-rootfs exists.
# Used by /etc/systemd/system/iiab-expand-rootfs.service on IIAB boot.

# Should work with most Linux OS's, regardless whether Raspberry Pi SD cards
# or external USB boot disks, etc. Verifies that rootfs is the last partition.

if [ -f /.expand-rootfs ] || [ -f /.resize-rootfs ]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why there would be a need to introduce another flag to act upon, all the tools in iiab-factory know nothing about the new 'expand-rootfs' flag.

Copy link
Member Author

@holta holta Mar 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've implemented consistent /.expand-rootfs naming across the board in both repos, per raspi-config norms:

(Legacy flag name /.resize-rootfs will still work during the interim.)

echo "$0: Expanding rootfs partition"

if [ -x /usr/bin/raspi-config ]; then
# 2022-02-17: Works in many more situations, e.g. with USB disks (not
# just microSD cards). IF ONLY THIS ALSO WORKED ON Ubuntu/Mint/etc !

# Uses do_expand_rootfs() from:
# https://github.com/RPi-Distro/raspi-config/blob/master/raspi-config
raspi-config --expand-rootfs
else
# 2022-03-15: Borrows from above raspi-config URL's do_expand_rootfs()
ROOT_PART="$(findmnt / -o SOURCE -n)" # e.g. /dev/sda2 or /dev/mmcblk0p2
ROOT_DEV="/dev/$(lsblk -no pkname "$ROOT_PART")" # e.g. /dev/sda or /dev/mmcblk0

ROOT_PART_NUM="$(echo "$ROOT_PART" | grep -o "[[:digit:]]*$")" # e.g. 2
# SLOW (~10 seconds) but it works!
LAST_PART_NUM=$(parted "$ROOT_DEV" -ms unit s p | tail -n 1 | cut -f 1 -d:)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should really add 'parted' as a dependency, same as cloud-guest-utils

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: 85eb059


if [ $ROOT_PART_NUM -ne $LAST_PART_NUM ]; then
echo "ERROR: $ROOT_PART partition ($ROOT_PART_NUM) is not the last partition ($LAST_PART_NUM). Don't know how to expand."
return 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would exit 1 be better here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

I borrowed return 0 from https://github.com/RPi-Distro/raspi-config/blob/684fbe183a53f60cccee21c341d5754357cf1288/raspi-config#L170 but exit 1 is a useful fix and improvement I'll add now.

fi

# Expand partition
growpart $ROOT_DEV $ROOT_PART_NUM
resize2fs $ROOT_PART

# # ASSUMES SD CARD STYLE PARTITION NAME LIKE <device>p<partition number>
# # e.g. /dev/mmcblk0p2 mounts at / (typical RasPiOS microSD)
# # BUT /dev/sda2 mounts at /media/usb1 (RasPiOS USB boot disk...
# # ...WON'T WORK BELOW; recap @ PR #3121)

# # Calculate root partition
# root_part=`lsblk -aP -o NAME,MOUNTPOINT | grep 'MOUNTPOINT="/"' | awk -F\" '{ print $2 }'` # e.g. mmcblk0p2
# root_dev=${root_part:0:-2} # e.g. mmcblk0
# # bash substring expansion: "negative offset [below, but not above]
# # must be separated from the colon by at least one space to avoid
# # being confused with the ‘:-’ expansion"
# # https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
# root_part_no=${root_part: -1} # e.g. 2

# # Resize partition
# growpart /dev/$root_dev $root_part_no
# resize2fs /dev/$root_part
fi

rm -f /.expand-rootfs /.resize-rootfs
fi
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Description=Root Filesystem Auto-Resizer
[Service]
Environment=TERM=linux
Type=oneshot
ExecStart=/usr/sbin/iiab-rpi-max-rootfs.sh
ExecStart=/usr/sbin/iiab-expand-rootfs
StandardError=syslog
RemainAfterExit=no

Expand Down
41 changes: 0 additions & 41 deletions roles/1-prep/templates/iiab-rpi-max-rootfs.sh

This file was deleted.