Support Ubuntu 22.04 Jammy Jellyfish

[JoshData]

This is a draft PR combining #1752, #2008, and some additional changes to prepare for switching our support to Ubuntu 22.04, which is the next long-term support release coming in April.

[myfirstnameispaul]

Should I submit 22.04 PRs to the new branch?

[JoshData]

Go for it. :)

Right now, Nextcloud doesn't appear to be installing correctly and it seems the PHP apcu apt package is built incorrectly. It has the module for PHP 8.1 instead of 8.0. I'm not sure if these are related, but probably. (https://bugs.launchpad.net/ubuntu/+source/php-apcu/+bug/1957261)

[myfirstnameispaul]

I haven't tried installing yet, but I did discover php-apcu-bc APCu Backwards Compatibility Module. Can that work?

[JoshData]

I think that's something different.

[Synchro]

That is indeed different. apc is an older extension that was replaced by apcu, however, not all apc functions are supported in apcu, and the apcu-bc extension adds the missing pieces. This is only really relevant for legacy apps running on newer PHP versions, as apcu has been around for a long time now, and some of what the old extension did provide is now handled by zend opcache, which is newer/better.

[myfirstnameispaul]

I wonder if they will fix this. After all, the bug is so far only affecting two people...

[myfirstnameispaul]

What about installing from pecl/pear (I never know the difference) until it's fixed?

[myfirstnameispaul]

How should we post issues or questions on this release? For example, I just ran apt upgrade and am greeted with a dialog box I don't recall seeing before:

 ┌─────────────────────────────┤ Configuring opendmarc ├──────────────────────────────┐
 │                                                                                    │ 
 │ The opendmarc package must have a database installed and configured before it can  │ 
 │ be used. This can be optionally handled with dbconfig-common.                      │ 
 │                                                                                    │ 
 │ If you are an advanced database administrator and know that you want to perform    │ 
 │ this configuration manually, or if your database has already been installed and    │ 
 │ configured, you should refuse this option. Details on what needs to be done        │ 
 │ should most likely be provided in /usr/share/doc/opendmarc.                        │ 
 │                                                                                    │ 
 │ Otherwise, you should probably choose this option.                                 │ 
 │                                                                                    │ 
 │ Configure database for opendmarc with dbconfig-common?                             │ 
 │                                                                                    │ 
 │                       <Yes>                          <No>                          │ 
 │                                                                                    │ 
 └────────────────────────────────────────────────────────────────────────────────────┘ 
                                                                                        

This is after a fresh install, no edits, yet, and not a restore.

Maybe we should have a v60 milestone?

[JoshData]

Yes, please post issues with this PR here.

I have no idea what that opendmarc configuration dialog is about, so I don't have any response to that now, but we'll have to figure it out eventually. Installing apuc from pecl/pear is a good idea and a fine workaround so we can keep developing this PR, but with luck Ubuntu will fix that before April.

[3lliryc]

Good evening,
is the quota in mailinbox will be integrate for ubuntu 22.04 version?

Best,

[downtownallday]

FWIW

Everyone has probably realized by now that jammy will ship with PHP 8.1, not 8.0 so the apcu issue should go away. However Nextcloud does not currently support 8.1:

nextcloud/server#29287

Roundcube also appears to lack support for 8.1 because the test matrix for the 1.5 branch only includes php up to version 8.0, but master includes 8.1. No specific bug or issue that I could find, though. I did not check the Roundcube plugins.

[myfirstnameispaul]

For Roundcube it looks like in the 1.6-beta milestone the PHP 8.1 tasks are complete with one still open in 1.6-rc milestones.

A quick skimming of the plugins and I didn't see any open issues or PRs mentioning 8.1.

[myfirstnameispaul]

Has anyone seen this before in Ubuntu? This is after running apt-get -qqy upgrade:

Package configuration


  ┌────┤ Daemons using outdated libraries ├─────┐
  │                                             │ 
  │                                             │ 
  │ Which services should be restarted?         │ 
  │                                             │ 
  │    [ ] dbus.service                         │ 
  │    [*] dovecot.service                      │ 
  │    [*] multipathd.service                   │ 
  │    [*] named.service                        │ 
  │    [ ] networkd-dispatcher.service          │ 
  │    [*] nginx.service                        │ 
  │    [*] nsd.service                          │ 
  │    [*] ntp.service                          │ 
  │    [*] opendkim.service                     │ 
  │    [*] packagekit.service                   │ 
  │    [*] php8.0-fpm.service                   │ 
  │    [*] polkit.service                       │ 
  │    [*] postfix@-.service                    │ 
  │    [*] rsyslog.service                      │ 
  │    [*] ssh.service                          │ 
  │    [ ] systemd-logind.service               │ 
  │    [*] udisks2.service                      │ 
  │    [ ] unattended-upgrades.service          │ 
  │    [*] upower.service                       │ 
  │    [ ] user@1000.service                    │ 
  │                                             │ 
  │                                             │ 
  │          <Ok>              <Cancel>         │ 
  │                                             │ 
  └─────────────────────────────────────────────┘ 

Maybe need to document something for new users.

[downtownallday]

man needrestart

[JoshData]

Everyone has probably realized by now that jammy will ship with PHP 8.1, not 8.0 so the apcu issue should go away. However Nextcloud does not currently support 8.1:

Uh oh. It's actually much worse for us than that. Since Nextcloud only (officially) supports upgrades from consecutive versions, we need there to be two consecutive versions that support PHP 7.2 (which we have on Ubuntu 18.04 now) and PHP 8.1. Nextcloud 20 is the last version to support 7.2. (We have that version on Mail-in-a-Box now.) So unless Nextcloud backports PHP 8.1 support to Nextcloud 21 (unlikely, since they're up to 23) or backports PHP 7.2 support to any of the newer versions (even less likely), we can't perform the upgrade. So this is very bad.

[JoshData]

I'll wait a bit to see if Ubuntu changes course and goes back to PHP 8.0, but if they proceed with 8.1 what we'll have to do is release a new version for Ubuntu 18.04 that adds a PHP PPA and install 7.3, 7.4 or 8.0. (Even 7.3 would be enough to move to the current latest version of Nextcloud and presumably would be lower risk than installing 8.0, so it would probably be 7.3.)

[myfirstnameispaul]

Sury several years back Sury started making all of the packages so that even the main repo won't interfere with each other (e.g., php7.2-fpm.sock, php7.3-fpm.sock, etc.) If php is invoked from command line, that has to be configured using update-alternatives --set php /usr/bin/phpx.x.

[downtownallday]

IMHO, it seems unlikely php 8.0 would be included with jammy because it would leave just 6 or 7 months of active php support assuming an April release.

https://www.php.net/supported-versions.php

[raccettura]

It might be worth removing haveged in this PR.

Doing some research on it's relevance today it seems like the consensus even from the developer is newer kernels don't need it:
jirka-h/haveged#57

Given 22.04 will ship with kernel 5.15, I think this dependency can be removed and streamline things ever so slightly.

[JoshData]

It might be worth removing haveged in this PR.

Nice catch. I'd happily accept a PR to this branch removing it. (If no one does it, I will.)

[0xFelix]

Here you go #2090 :)

[downtownallday]

FWIW, I ran php static analyzers phan and phpcompatibility against the three roundcube plugins (html_notifier, rcmcarddav, and persistent_login). Phan only does backward compatibility checks, so not that useful and unfortunately phpcompatibility does not yet support php 8.1. Anyway, no compatibility issues were found at least up to php 8.0 for those plugins.

[Synchro]

Exakat supports compatibility checks for 8.1. It's very thorough!

[downtownallday]

I don't see any compatibility checks for php 8.1, or at least it's not listed in the rulesets section of exakat doctor:

exakat : 
    executable           : /usr/src/exakat/exakat-2.3.0.phar
    version              : 2.3.0
    build                : 1265
    exakat.ini           : ./config/exakat.ini,
                           test/config.ini
    graphdb              : gsneo4j
    reports              : 
    rulesets             : CompatibilityPHP53,
                           CompatibilityPHP54,
                           CompatibilityPHP55,
                           CompatibilityPHP56,
                           CompatibilityPHP70,
                           CompatibilityPHP71,
                           CompatibilityPHP72,
                           CompatibilityPHP73,
                           CompatibilityPHP74,
                           CompatibilityPHP80,
                           Analyze,
                           Preferences,
                           Appinfo,
                           Appcontent,
                           Dead code,
                           ClassReview,
                           Performances,
                           Security,
                           Custom,
                           Inventory,
                           Stats,
                           php-cs-fixable,
                           Rector
    extra rulesets       : 
    ignored rules        : 
    tokenslimit          : 100 000 000
    stubs                : 

PHP : 
    binary               : 7.4.11
    memory_limit         : -1
    short_open_tags      : Off
    ext/curl             : Yes
    ext/hash             : Yes
    ext/phar             : Yes
    ext/sqlite3          : Yes
    ext/tokenizer        : Yes
    ext/mbstring         : Yes
    ext/json             : Yes
    ext/xmlwriter        : Yes
    pcre.jit             : On (Must be off on PHP 7.3 and OSX)

java : 
    installed            : Yes
    type                 : OpenJDK Runtime Environment (build 11.0.8+10-post-Debian-1deb10u1)
    version              : openjdk
    $JAVA_HOME           : <none>
    $JAVA_OPTIONS        : 

tinkergraph : 
    configured           : No tinkergraph configured in config/exakat.ini.

tinkergraphv3 : 
    configured           : No tinkergraph configured in config/exakat.ini.

gsneo4j : 
    installed            : Yes (folder : /usr/src/exakat/tinkergraph)
    host                 : 127.0.0.1
    port                 : 8182
    gremlin version      : 3.4.8
    neo4j version        : 3.2.3

gsneo4jv3 : 
    configured           : No gsneo4jv3_folder configured in config/exakat.ini.

nogremlin : 
    installed            : Always

project : 
    name                 : html5_notifier
    url                  : projects/html5_notifier
    phpversion           : 7.4
    reports              : ""
    rulesets             : CompatibilityPHP53, CompatibilityPHP54, CompatibilityPHP55, CompatibilityPHP56, CompatibilityPHP70, CompatibilityPHP71, CompatibilityPHP72, CompatibilityPHP73, CompatibilityPHP74, CompatibilityPHP80, Analyze, Preferences, Appinfo, Appcontent, Dead code, ClassReview, Performances, Security, Custom, Inventory, Stats, php-cs-fixable, Rector
    included dirs        : 
    ignored dirs         : /assets, /cache, /css, /data, /doc, /docker, /docs, /example, /examples, /images, /js, /lang, /spec, /sql, /test, /tests, /tmp, /version, /var, /vendor, /vendor, /vendor
    ignored rules        : 
    file extensions      : php, php3, inc, tpl, phtml, tmpl, phps, ctp, module

folders : 
    projects folder      : Yes

php74 : 
    configured           : Yes (/usr/local/bin/php)
    actual version       : 7.4.11
    auditing             : with this version

[Synchro]

That's odd, as it definitely exists! Docs for it start here: https://exakat.readthedocs.io/en/latest/Reference/Rulesets.html#ruleset-compatibilityphp81

I note that the output you posted is from an old version 2.3.0; it's currently at 2.3.6.

[downtownallday]

Further complication - Roundcube 1.6 will no longer support PHP 7.2.

See:

roundcube/roundcubemail#7976
https://github.com/roundcube/roundcubemail/blob/master/CHANGELOG.md

[downtownallday]

Several times, the status page reports "something went wrong sorry".

Perhaps the crash when loading the status page is causing gunicorn to timeout.

[downtownallday]

Or, 30 seconds isn't long enough to load the status page and gunicorn is killing it.

[stvhay]

The .well-known redirects for CardDAV and CalDAV point to the wrong URL. It should just be "dav" Same problem in the main branch--so maybe we wait, but it is a pretty simple fix:

rewrite ^/.well-known/carddav /cloud/remote.php/dav/ redirect;
rewrite ^/.well-known/caldav /cloud/remote.php/dav/ redirect;

The ZPush redirects could use some love too, but isn't that broken anyways? At any rate, I just commited a fix for both in the nextcloud24 PR so we can wait if we don't think its urgent.

[downtownallday]

I am having some success getting the daemon to hang during status checks, which causes gunicorn to terminate the worker process (and for me I end up with 3 process after the gunicorn restarts the worker, one of which must be killed with prejudice).

This seems to only happen using the web interface, not from the command line.

In reading the multiprocessing pool docs, it says

[multiprocessing.pool](https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool) objects have internal resources that need to be properly managed (like any other resource) by using the pool as a context manager or by calling [close()](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.close) and [terminate()](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.terminate) manually. Failure to do this can lead to the process hanging on finalization.

If I change daemon.py's /system/status POST handler, and add pool.close() and pool.join() after run_checks(), it seems to avoid the hang.

[stvhay]

If I change daemon.py's /system/status POST handler, and add pool.close() and pool.join() after run_checks(), it seems to avoid the hang.

That's interesting, because it looks like the code is using the pool as a context manager:

	with multiprocessing.pool.Pool(processes=5) as pool:
		run_checks(False, env, output, pool)

[downtownallday]

yes, I noticed that too, and tried a bunch of other stuff thinking the cleanup was automatic, but nothing worked. the docs say close/terminate must be called manually, so I guess that holds.

[stvhay]

Hmm. Found this article https://superfastpython.com/multiprocessing-pool-context-manager/ and it would suggest you are right.

Therefore, if we wish to wait for all issued tasks to complete before closing the pool, we must explicitly add a call to close() and join().

I've patched this on my server as well and things seem to be running better with the change and I don't see any obvious problems.

[downtownallday]

Feel free to add the commit to one of your PRs, or I can add a new PR later today.

[stvhay]

I kept it as a separate PR because I think @JoshData wants to hold off on the other PRs for now (#2162 and #2159)

[downtownallday]

great

[downtownallday]

There's another timeout that causes a gunicorn worker to get killed which doesn't appear to be documented or configurable. That is, when I pause my VM for an hour, then continue (eg. closing my laptop and opening it a while later), a new gunicorn worker is created (and old one is killed). It's probably not a use case we care about, just documenting it as it's different than the old behavior.

[downtownallday]

the docs say close/terminate must be called manually, so I guess that holds

multiprocessing.Pool's exit() function, which is automatically called by python's 'with' builtin when the scope is closed, calls terminate() (see https://github.com/python/cpython/blob/3.10/Lib/multiprocessing/pool.py) . That should prevent a hang during garbage collection according to the docs. However, the process clearly hangs in practice, and I don't think we want to terminate either so close()/join() still makes sense and seems to work.

[stvhay]

The postgrey changes will be great for this upcoming migration!

[jvolkenant]

The postgrey changes will be great for this upcoming migration!

I don't believe this will help for this migration (it will for future ones and reinstalls). This branch (what will be v0.60?) should only be run is after you have reinstalled to 22.04 and copied your /home/userdata/. You could backup your /var/lib/postgrey/ before you migrate and stop postgrey and copy/chown/chmod the files afterward. (I did this from 14.04 ->18.04)

[stvhay]

Good point. I wasn't thinking too deeply about it and forgot the change would have to be on the current, not future release.

[jvolkenant]

ssl_dh_parameters_length is a pre 2.3 option https://doc.dovecot.org/configuration_manual/dovecot_ssl_configuration/

It results in a log warning to /var/log/maillog

Sep 29 16:14:43 config: Warning: Obsolete setting in /etc/dovecot/conf.d/10-ssl.conf:91: ssl_dh_parameters_length is no longer needed

Should we exclude the option?, I imagine it now uses the length from ssl_dh

[JoshData]

Merged!

Announcement at https://discourse.mailinabox.email/t/version-60-for-ubuntu-22-04-is-about-to-be-released/9558.

Fingers crossed it works. Please report any critical installation bugs here (or open a new PR if you have a fix). Anything else can go in the discussion link above or a new issue here on GitHub, probably. I'll update the website tomorrow.

Thanks to everyone who participated, especially everyone who opened pull requests. I really appreciated it.