[Regression] all documentation links and explanatory text vanished from MOTD

[ogra1]

With a recent core18 upgrade all explanatory text and the various documentation links that are shown in the MOTD messages on login have vanished on all my UbuntuCore18 installs:

it looks like this now:

$ ssh 192.168.2.85
Welcome to Ubuntu Core 18 (GNU/Linux 5.3.0-1033-raspi2 armv7l)
Last login: Fri Sep 18 12:25:11 2020 from 192.168.2.48
ogra@localhost:~$

While it is supposed to give some hints and intro text about the snap ecosystem as well as help and support URLs like:

Welcome to Ubuntu Core 18 (GNU/Linux 5.3.0-1027-raspi2 armv7l)
 * Ubuntu Core:     https://www.ubuntu.com/core
 * Community:       https://forum.snapcraft.io
 * Snaps:           https://snapcraft.io

This Ubuntu Core 18 machine is a tiny, transactional edition of Ubuntu,
designed for appliances, firmware and fixed-function VMs.

If all the software you care about is available as snaps, you are in
the right place. If not, you will be more comfortable with classic
deb-based Ubuntu Server or Desktop, where you can mix snaps with
traditional debs. It's a brave new world here in Ubuntu Core!

Please see 'snap --help' for app installation and updates.
Last login: Mon Jun 29 12:04:15 2020 from 192.168.2.48

[anonymouse64]

Are you able to revert the core18 snap to confirm that the problem goes away with an old version of core18 ? Also what is snap list core18?

[ogra1]

yes, rolling back gets me the messages back:

ogra@acheron:~$ ssh 192.168.2.85
Welcome to Ubuntu Core 18 (GNU/Linux 5.3.0-1033-raspi2 armv7l)
Last login: Tue Sep 22 10:24:40 2020 from 192.168.2.48
ogra@localhost:~$ snap changes
error: no changes found
ogra@localhost:~$ snap list core18
Name    Version   Rev   Tracking       Publisher   Notes
core18  20200724  1889  latest/stable  canonical✓  base
ogra@localhost:~$ snap revert core18
Ensure prerequisites for "core18" are available                                                              /
Broadcast message from root@localhost (Tue 2020-09-22 10:26:23 UTC):

reboot scheduled to update the system

ogra@localhost:~$ sudo reboot
Connection to 192.168.2.85 closed by remote host.
Connection to 192.168.2.85 closed.
ogra@acheron:~$ ssh 192.168.2.85
Welcome to Ubuntu Core 18 (GNU/Linux 5.3.0-1033-raspi2 armv7l)
 * Ubuntu Core:     https://www.ubuntu.com/core
 * Community:       https://forum.snapcraft.io
 * Snaps:           https://snapcraft.io

This Ubuntu Core 18 machine is a tiny, transactional edition of Ubuntu,
designed for appliances, firmware and fixed-function VMs.

If all the software you care about is available as snaps, you are in
the right place. If not, you will be more comfortable with classic
deb-based Ubuntu Server or Desktop, where you can mix snaps with
traditional debs. It's a brave new world here in Ubuntu Core!

Please see 'snap --help' for app installation and updates.
Last login: Tue Sep 22 10:25:31 2020 from 192.168.2.48
ogra@localhost:~$ snap list core18
Name    Version   Rev   Tracking       Publisher   Notes
core18  20200707  1882  latest/stable  canonical✓  base
ogra@localhost:~$ 

[ogra1]

there is also https://bugs.launchpad.net/bugs/1888575

[panlinux]

Did you guys revert? I just deployed ubuntu core 18, and I have r1885 in latest/stable, not 1889:

ahasenack@localhost:~$ snap list
Name       Version         Rev   Tracking       Publisher   Notes
core18     20200724        1885  latest/stable  canonical✓  base
pc         18-2            104   18/stable      canonical✓  gadget
pc-kernel  4.15.0-118.119  602   18/stable      canonical✓  kernel
snapd      2.46.1          9279  latest/stable  canonical✓  snapd
ahasenack@localhost:~$ snap info core18
name:      core18
summary:   Runtime environment based on Ubuntu 18.04
publisher: Canonical✓
store-url: https://snapcraft.io/core18
license:   unset
description: |
  The base snap based on the Ubuntu 18.04 release.
type:         base
snap-id:      CSO04Jhav2yK0uz97cr0ipQRyqg0qQL6
tracking:     latest/stable
refresh-date: today at 12:57 UTC
channels:
  latest/stable:    20200724 2020-08-06 (1885) 58MB -
  latest/candidate: 20200907 2020-09-08 (1927) 58MB -
  latest/beta:      20200907 2020-09-07 (1927) 58MB -
  latest/edge:      20200907 2020-09-07 (1927) 58MB -
installed:          20200724            (1885)   0B base

[ogra1]

the revisions are per-arch (or rather "per-binary-snap") ... i'm on armhf here (which typically builds slower than x86 so it usually gets a higher revision number)...

ogra@localhost:~$ snap info core18|grep stable
tracking:     latest/stable
  latest/stable:    20200724 2020-08-06 (1889) 47MB -

[panlinux]

Is there another way to identify the build? Because I do see the motd with the amd64 1885 snap, but I can't tell if it has the change that is affecting you.

[ogra1]

the version string should be unique "20200724" ... (a second build on that same day should become "20200724.1")

[panlinux]

It's the same then:
core18 20200724 1885 latest/stable canonical✓ base

I used the amd64 image with kvm. Can you troubleshoot furhter what's going on in /etc/update-motd.d and /etc/motd?

ahasenack@localhost:~$ ll /etc/update-motd.d/
total 7
drwxr-xr-x  2 root root   52 Jul 24 12:47 ./
drwxr-xr-x 42 root root 1459 Jul 24 12:47 ../
-rwxr-xr-x  1 root root 1220 Apr  9  2018 00-header*
-rwxr-xr-x  1 root root 4646 Sep 27  2019 50-motd-news*

ahasenack@localhost:~$ ll /etc/motd 
lrwxrwxrwx 1 root root 13 Jul 24 12:46 /etc/motd -> writable/motd

ahasenack@localhost:~$ ll /etc/writable/motd 
-rw-r--r-- 1 root root 591 Jul 24 12:46 /etc/writable/motd

ahasenack@localhost:~$ cat /etc/writable/motd 
 * Ubuntu Core:     https://www.ubuntu.com/core
 * Community:       https://forum.snapcraft.io
 * Snaps:           https://snapcraft.io

This Ubuntu Core 18 machine is a tiny, transactional edition of Ubuntu,
designed for appliances, firmware and fixed-function VMs.

If all the software you care about is available as snaps, you are in
the right place. If not, you will be more comfortable with classic
deb-based Ubuntu Server or Desktop, where you can mix snaps with
traditional debs. It's a brave new world here in Ubuntu Core!

Please see 'snap --help' for app installation and updates.

[ogra1]

what is supposed to put /etc/writable/motd in place on existing systems ?

ogra@acheron:~$ ssh 192.168.2.85
Welcome to Ubuntu Core 18 (GNU/Linux 5.3.0-1033-raspi2 armv7l)
Last login: Tue Sep 22 13:56:18 2020 from 192.168.2.48
ogra@localhost:~$ ls -l /etc/writable/motd
ls: cannot access '/etc/writable/motd': No such file or directory
ogra@localhost:~$ 

note that i see the file on a freshly written SD card that uses 20200724, but not on any of the long running systems that got upgraded

[panlinux]

No idea, and if the image was produced on the date 20200724, then it doesn't have the base-files update from https://bugs.launchpad.net/ubuntu/+source/base-files/+bug/1888575.

I'm not super familiar with ubuntu core, I don't know how and when updates from ubuntu trickle down into core. I know there is no dpkg in there, but /usr/share/doc/base-files/ also doesn't have a changelog file, so I can't tell which version was used exactly. This needs help from the team producing the images perhaps?

[anonymouse64]

We meet again our good friend, writable-paths(5), where /etc/writable is listed thusly in core18:

/etc/writable                           auto                    persistent  transition        none

with the third column "persistent" being the action taken, and here's the quote from the manpage about persistent:

          persistent
                 Writes to the mount point will be persisted to the writable partition.

This means that new files from the base snap / root filesystem do not get copied to the writable partition, for that we must look at synced:

          synced Any  file  appearing in the root filesystem will also be copied over to writable
                 storage. However file removals are still not synced and files existing  in  both
                 read-only and writeable storage will not be updated.

however synced is terrible for other reasons and we have dropped support for synced entirely from core20, though it still exists for core18 obviously due to compatibility reasons. The "right but awful" fix here would be to switch to using synced instead of persistent here, but it's totally unclear to me how writable-paths(5) works when the action is changed from persistent to synced. I presume it probably will not explode, but it may have subtleties that are not well understood and could be buggy, perhaps deleting all existing files there.

This does prove the case that we need to be very careful about adding files to the base snap, but I think we need to be even more careful about adding files to the base snaps which add files to persistent directories from writable-paths(5) because files added there do not get copied into existing writable partitions and thus leads to bugs such as this one.

[ogra1]

we have switched from persistent to synced several times in the past without issues (and it was designed to be robust in that sense)...
that said ... writable-paths has indeed not been massively maintained or used (beyond UC) since 2016 ...

[anonymouse64]

If that's the case, then sure perhaps the fix for this particular bug is to make /etc/writable synced for core18 now but AIUI talking to @mvo5 we do not want more synced things uc, and we specifically don't want any synced dirs in uc20 at all, at least with the current implementation of writable-paths in shell (we are looking at eventually re-writing it in Go to use from the initramfs on uc20).