Stable updates regression Rockstor 4 release candidate

[phillxnet]

Thanks to forum member GeoffA for highlighting this issue. During the development of Rockstor 4, currently in release candidate phase, the Stable repository remained empty. We have as of yet no 'Built on openSUSE' release. This now looks to have been a mistake on my part as it obscured what looks like an omission of a required auth parameter to zypper addrepo which has differing defaults with regard to basic http auth used by the stable channel than our prior yum (CentOS) base.

Note that we still use yum, in parallel, or rather dnf-yum, to drive Web-UI notification of a rockstor package update, and our Web-UI pending changelog of the same is also yum provided, as zypper doesn't yet have this capability. But for the actual updates themselves we only use zypper.

This issue surfaces to the user as an available Stable channel update (dnf-yum informed) that silently fails to install (the zypper mis-configuration). There is also, following a cli zypper refresh, a persistent user (auto filled with appliance ID), and consequent password prompt. These are currently believed to be cli artifacts of the same zypper mis-configuration.

N.B. there is also a related zypper/libzypp upstream fix re basic auth reliability that we depend upon here. This is reported to have been resolved in libzypp version 17.24.1: See libzypp changelog available here:
https://build.opensuse.org/package/view_file/zypp:Head/libzypp/libzypp.changes?expand=1
This fix rolled out around mid/late August this year:
https://lists.suse.com/pipermail/sle-updates/2020-August/015787.html
https://www.suse.com/support/update/announcement/2020/suse-ru-20202279-1/

As part of our field testing of this issue the Stable Channel was populated with our latest Release Candidate 4 rpm:
4.0.3-0, this is not an official stable release but has assisted in our diagnosis of this issue.

This issue does not affect the testing channel updates as they are not http authed.

Workaround

The fix for this pre stable issue is now included in 4.0.4-0 - available in the testing channel which is not affected by this bug.
N.B. if you are reading this past our official Release in Stable (planned to be 4.0.4-0 currently) then you may have to use the command line (not the Web Shell) to specify an update to only 4.0.4-0

systemctl stop rockstor
zypper in rockstor-4.0.4-0
systemctl start rockstor

You can then switch back to the Stable channel where you will again only be offered field tested rpms.

Alternative workaround:

Again for stable subscribers running < 4.0.4 (i.e. DIY installers running Release Candidate rockstor versions):

zypper refresh
# "User Name: Appliance ID" will appear: <Press Enter>
Password: <Enter activation code>
systemctl stop rockstor
zypper --non-interactive up rockstor
systemctl start rockstor

• Subscribe to Testing then directly back to Stable

As of writing there are only Release Candidate placeholder rpms in the Stable channel. In part to avoid a repeat of this same issue and to get further field testing of Stable updates prior to our official Stable rpm announcement.

[phillxnet]

I am currently working on this issue and a fix is 'in development'.

[phillxnet]

There is also, following a cli zypper refresh, a persistent user (auto filled with appliance ID), and consequent password prompt.

This behaviour can be caused by incorrect credentials or, in our case, a general failure in the authentication system.

[phillxnet]

Progress to date, we appear to have 2 elements at play.

1: The additional use of:

?auth=basic

to our Stable channel url helps with enforcing a proactive use of the supplied credentials.
(This has a working fix in developement)

2: Having established (1) above, there was proper function from cli but failure to retrieve supplied credentials from within the Web-UI hosted zypper commands thus:

From: /var/log/zypper.log

Web-UI initiated zypper activities:

2020-10-16 18:52:44 <1> rleap15.2(1959) [zypp++] CredentialManager.cc(init_globalCredentials):140 global cred file does not existGot 0 global records.
2020-10-16 18:52:44 <1> rleap15.2(1959) [zypp++] CredentialManager.cc(init_userCredentials):162 user cred file not knownGot 0 user records.
2020-10-16 18:52:44 <1> rleap15.2(1959) [zypp++] CredentialManager.cc(getCred):216 No credentials for 'http://User@updates.rockstor.com:8999/rockstor-stable/leap/15.2?auth=basic'
2020-10-16 18:52:44 <1> rleap15.2(1959) [zypp++] MediaCurl.cc(setupEasy):419 Enabling HTTP authentication methods: basic (CURLOPT_HTTPAUTH=1)
2020-10-16 18:52:44 <1> rleap15.2(1959) [zypp++] MediaCurl.cc(setupEasy):472 Proxy: not explicitly set
2020-10-16 18:52:44 <1> rleap15.2(1959) [zypp++] MediaCurl.cc(setupEasy):473 Proxy: libcurl may look into the environment

CLI initiated 'zypper refresh':

2020-10-16 19:09:07 <1> rleap15.2(3497) [zypp++] CredentialManager.cc(init_globalCredentials):140 global cred file does not existGot 0 global records.
2020-10-16 19:09:07 <1> rleap15.2(3497) [zypp] IniParser.cc(parse):84 Start parsing /root/.zypp/credentials.cat[g___]
2020-10-16 19:09:07 <1> rleap15.2(3497) [zypp] IniParser.cc(parse):138 Done parsing /root/.zypp/credentials.cat[_eF_]
2020-10-16 19:09:07 <1> rleap15.2(3497) [Progress++] ProgressData.cc(report):88 {#15|/root/.zypp/credentials.cat}END

2020-10-16 19:09:07 <1> rleap15.2(3497) [zypp++] CredentialManager.cc(init_userCredentials):162 Got 3 user records.
2020-10-16 19:09:07 <1> rleap15.2(3497) [zypp++] CredentialManager.cc(getCred):214 Found credentials for 'http://User@updates.rockstor.com:8999/rockstor-stable/leap/15.2?auth=basic':
2020-10-16 19:09:07 <1> rleap15.2(3497) [zypp++] CredentialManager.cc(getCred):214 [http://updates.rockstor.com:8999/rockstor-stable]
2020-10-16 19:09:07 <1> rleap15.2(3497) [zypp++] CredentialManager.cc(getCred):214 username: 'User'
2020-10-16 19:09:07 <1> rleap15.2(3497) [zypp++] MediaCurl.cc(setupEasy):419 password: <non-empty>Enabling HTTP authentication methods: basic (CURLOPT_HTTPAUTH=1)

Indicating environmental factors breaking Web-UI initiated zypper's password retrieval.

Thanks to @FroggyFlox for some super helpful discussion and researched pointers on this one. Specifically we have the following prior observation:

https://tickets.puppetlabs.com/browse/PUP-1124

Unlike other comparable utilities for other distributions, zypper stores such credentials information in a file in $HOME, $HOME/.zypp/credentials.cat to be precise. However, this requires the environment variable $HOME to be set, which is currently not the case, rendering the zypper package provider unable to access a required password.

My current assumption and line of investigation is to find a simple work around for this behaviour. Once resolved I can post a pull request.