Apply erratas to Debian / Ubuntu systems

[rpasche]

Hi Uyuni,

I just wanted to share my knowledge, of how I'm currently - more or less - able to apply erratas to Ubuntu / Debian systems and this hopefully helps to implement this.

Currently the aptpkg salt module is not able to list errata information from repositories, because this information is simply missing for Debian / Ubuntu repositories. (For RPM based repos (and tools) like zypper, yum and dnf, this information is availble within repositories).

Because of this lack of information you cannot apply errata to a Debian / Ubuntu salt client, because the functions are missing.

Due to this, I just now copied the functionality of the errata.py script (you may find at https://github.com/philicious/spacewalk-scripts/blob/master/errata.py (and which is just a copy from an RPM based traditional client). For this to work, I needed to modify the Uyuni server (salt master and also patch the aptpkg.py salt module on the Debian / Ubuntu clients.

Modifications on server:

• created the folder /srv/salt/_modules/runners
• downloaded the spacewalk runner code from https://github.com/saltstack/salt/blob/master/salt/runners/spacewalk.py and placed it in above folder
• created a read-only API user in Uyuni (to retrieve errata information)
• created a custom salt configuration file like /etc/salt/master.d/errata.conf

# debug log
#log_level: debug

# add below directory to load custom modules from there
module_dirs:
  - /srv/salt/_modules

# allow all minions to run all functions in the 'spacewalk' runner module
# CURRENTLY TESTING. SHOULD BE STRIPPED DOWN TO QUERY ERRATA INFORMATION ONLY!!!
peer_run:
  .*:
    - spacewalk.*

# configure access to API
spacewalk:
  <uyuni_salt_maste_name>:
    username: '<read_only_user>'
    password: '<read_only_user_pwd>'

<uyuni_salt_master_name> is the FQDN of the uyuni server which is set within the salt minion configuration.
User and PW should be clear.

• Restart the salt master service for the changes to take effect.

Modifications on Ubuntu / Debian clients:

Note: The salt client needs to be one of the salt packages you can get here https://build.opensuse.org/project/subprojects/systemsmanagement:Uyuni:Stable (already contains further modifications for Debian / Ubuntu)

• apply patch below to aptpkg.py (in /usr/lib/python2.7/dist-packages/salt/modules/)

--- aptpkg.py.orig      2019-05-07 18:03:52.000000000 +0200
+++ aptpkg.py   2020-03-22 16:07:33.899722527 +0100
@@ -44,8 +44,11 @@
 import salt.utils.versions
 import salt.utils.yaml
 import salt.utils.environment
+import salt.crypt
+import salt.payload
+import salt.transport.client
 from salt.exceptions import (
-    CommandExecutionError, MinionError, SaltInvocationError
+    CommandExecutionError, MinionError, SaltInvocationError, SaltReqTimeoutError
 )

 log = logging.getLogger(__name__)
@@ -565,6 +568,18 @@
                        'new': '<new-version>'}}
     '''
     _refresh_db = False
+
+    if 'advisory_ids' in kwargs:
+        if pkgs:
+            pkgs.update(_runner_errata_packagelist(kwargs['advisory_ids']))
+        else:
+            pkgs = _runner_errata_packagelist(kwargs['advisory_ids'])
+
+        # we need to remove the advisory_ids again, after we have converted
+        # them to packages (pkgs)
+        del kwargs['advisory_ids']
+
+
     if salt.utils.data.is_true(refresh):
         _refresh_db = True
         if 'version' in kwargs and kwargs['version']:
@@ -2916,3 +2931,98 @@
             )

     return http_proxy_url
+
+def _parse_args(arg):
+    '''
+    Also copied from the publish module
+
+    yamlify `arg` and ensure it's outermost datatype is a list
+    '''
+    yaml_args = salt.utils.args.yamlify_arg(arg)
+
+    if yaml_args is None:
+        return []
+    elif not isinstance(yaml_args, list):
+        return [yaml_args]
+    else:
+        return yaml_args
+
+def _runner_errata_packagelist(advisories=None, timeout=5):
+    '''
+    More or less copied form the modules/publish.runner code
+
+    Call a runner on master server to retrieve package list
+    of advisories
+    '''
+    arg = _parse_args(advisories)
+
+    '''
+    Trying to pass unicode strings to the API via the runner always failed.
+    Therefore, converting to strings. That works
+    '''
+
+    arg2 = [x.encode('utf-8') for x in arg]
+
+    if 'master_uri' not in __opts__:
+        return 'No access to master. If using salt-call with --local, please remove.'
+    #log.info('Publishing runner \'%s\' to %s', fun, __opts__['master_uri'])
+    auth = salt.crypt.SAuth(__opts__)
+    tok = auth.gen_token(b'salt')
+    load = {'cmd': 'minion_runner',
+            'fun': 'spacewalk.api',
+            'arg': [
+                '{0}={1}'.format('server', __opts__['master']),
+                '{0}={1}'.format('command', 'errata.listPackages'),
+                '{0}={1}'.format('arguments', arg2)],
+            'tok': tok,
+            'tmo': timeout,
+            'id': __opts__['id'],
+            'no_parse': __opts__.get('no_parse', [])}
+
+    channel = salt.transport.client.ReqChannel.factory(__opts__)
+    pkglist = []
+    try:
+        pkglist = channel.send(load)
+    except SaltReqTimeoutError:
+        return '\'{0}\' runner publish timed out'.format(fun)
+    finally:
+        channel.close()
+
+    # the runner returns a list of data included within
+    # and object with name of the function that was called
+    pkglist = pkglist.values()[0]
+    cur_pkgs = list_pkgs().keys()
+
+    # return the package name and version for each package already installed on system
+    # we do not want to install 'extra' packages
+    pkgs = []
+    for p in pkglist:
+        if p['name'] in cur_pkgs:
+            if p['epoch']:
+                pkgs.append({p['name']: '{0}:{1}-{2}'.format(p['epoch'], p['version'], p['release'])})
+            else:
+                pkgs.append({p['name']: '{0}-{1}'.format(p['version'], p['release'])})
+
+    return pkgs
+
+def _get_patches():
+    '''
+    still to do
+    '''
+
+    return {}
+
+def list_patches():
+    '''
+    do something
+    '''
+
+    return _get_patches()
+
+def list_installed_patches():
+    '''
+    List the currently installed advisories
+    '''
+
+    return _get_patches()
+

I know....this is by far not final and there are still bugs but with this modification, I can at least apply errata to systems and they can be installed (if no further dependency problems arise on the client). The problem with dependencies on the client is, that for installation of the packet, dpkg seems to be used, wheras apt-get install or àpt install` would be better (at least my opinion).

• delete the compiled version of aptpkg.py (aptpkg.pyc), so it gets recompiled.
• restart the salt-minion on the client (just to be sure)

So before all these modifications, trying to apply an errata to a client looks like this in WebUI:

{
  "pkg_|-mgr_regular_patches_|-mgr_regular_patches_|-patch_installed": {
    "comment": "The pkg.patch_installed state is not available on this platform",
    "name": "mgr_regular_patches",
    "start_time": "13:36:31.570342",
    "result": false,
    "duration": 9.34,
    "__run_num__": 2.0,
    "__sls__": "packages.patchinstall",
    "changes": {},
    "__id__": "mgr_regular_patches"
  },
...
...

After modification like this:

{
  "pkg_|-mgr_regular_patches_|-mgr_regular_patches_|-patch_installed": {
    "comment": "",
    "name": "mgr_regular_patches",
    "start_time": "16:37:48.064025",
    "result": true,
    "duration": 3786.456,
    "__run_num__": 2.0,
    "__sls__": "packages.patchinstall",
    "changes": {
      "libexpat1": {
        "new": "2.2.6-2+deb10u1",
        "old": "2.2.6-2"
      }
    },
    "__id__": "mgr_regular_patches"
  },
...
...

I am also looking to @isbm , because he already helped me to integrate other functions into the salt package the patch would also have to - further fixed - and maybe later integrated into salt.

Kind regards,
rpasche

[paususe]

Hi Robert

Do I understand it correctly that you have essentially re-implemented errata.py in Salt, so errata.py is no longer necessary?

[rpasche]

Yes, at least for a salt client. For a traditional client you still need it. ⁣sent from my mobile device​

…

-------- Originale Nachricht -------- Von: Pau Garcia Quiles <notifications@github.com> Gesendet: Sat Mar 28 00:37:26 GMT+01:00 2020 An: uyuni-project/uyuni <uyuni@noreply.github.com> Cc: Robert Paschedag <robert.paschedag@web.de>, Author <author@noreply.github.com> Betreff: Re: [uyuni-project/uyuni] Apply erratas to Debian / Ubuntu systems (#2068) Hi Robert Do I understand it correctly that you have essentially re-implemented `errata.py` in Salt, so `errata.py` is no longer necessary?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #2068 (comment)

[rpasche]

Please find an updated version of the patch below. It now supports multiple erratas to be applied.

--- aptpkg.py.orig      2019-05-07 18:03:52.000000000 +0200
+++ aptpkg.py   2020-03-29 20:34:32.484062347 +0200
@@ -44,8 +44,11 @@
 import salt.utils.versions
 import salt.utils.yaml
 import salt.utils.environment
+import salt.crypt
+import salt.payload
+import salt.transport.client
 from salt.exceptions import (
-    CommandExecutionError, MinionError, SaltInvocationError
+    CommandExecutionError, MinionError, SaltInvocationError, SaltReqTimeoutError
 )

 log = logging.getLogger(__name__)
@@ -565,6 +568,18 @@
                        'new': '<new-version>'}}
     '''
     _refresh_db = False
+
+    if 'advisory_ids' in kwargs:
+        if pkgs:
+            pkgs.append(_runner_errata_packagelist(kwargs['advisory_ids']))
+        else:
+            pkgs = _runner_errata_packagelist(kwargs['advisory_ids'])
+
+        # we need to remove the advisory_ids again, after we have converted
+        # them to packages (pkgs)
+        del kwargs['advisory_ids']
+
+
     if salt.utils.data.is_true(refresh):
         _refresh_db = True
         if 'version' in kwargs and kwargs['version']:
@@ -2916,3 +2931,110 @@
             )

     return http_proxy_url
+
+def _parse_args(arg):
+    '''
+    Also copied from the publish module
+
+    yamlify `arg` and ensure it's outermost datatype is a list
+    '''
+    yaml_args = salt.utils.args.yamlify_arg(arg)
+
+    if yaml_args is None:
+        return []
+    elif not isinstance(yaml_args, list):
+        return [yaml_args]
+    else:
+        return yaml_args
+
+def _get_packages_from_errata(advisory, timeout=5):
+    '''
+    Retrieve a list of packages from one errata
+    '''
+
+    auth = salt.crypt.SAuth(__opts__)
+    tok = auth.gen_token(b'salt')
+    load = {'cmd': 'minion_runner',
+            'fun': 'spacewalk.api',
+            'arg': [
+                '{0}={1}'.format('server', __opts__['master']),
+                '{0}={1}'.format('command', 'errata.listPackages'),
+                '{0}={1}'.format('arguments', [advisory])],
+            'tok': tok,
+            'tmo': timeout,
+            'id': __opts__['id'],
+            'no_parse': __opts__.get('no_parse', [])}
+
+    channel = salt.transport.client.ReqChannel.factory(__opts__)
+    pkglist = []
+    try:
+        pkglist = channel.send(load)
+    except SaltReqTimeoutError:
+        return []
+    finally:
+        channel.close()
+
+    return pkglist.values()[0]
+
+def _runner_errata_packagelist(advisories=None, timeout=5):
+    '''
+    More or less copied form the modules/publish.runner code
+
+    Call a runner on master server to retrieve package list
+    of advisories
+    '''
+    tmparg = _parse_args(advisories)
+
+    '''
+    Trying to pass unicode strings to the API via the runner always failed.
+    Therefore, converting to strings. That works
+    '''
+
+    arg = [x.encode('utf-8') for x in tmparg]
+
+    if 'master_uri' not in __opts__:
+        return 'No access to master. If using salt-call with --local, please remove.'
+    #log.info('Publishing runner \'%s\' to %s', fun, __opts__['master_uri'])
+    # the runner returns a list of data included within
+    # and object with name of the function that was called
+
+    pkglist = []
+
+    for adv in arg:
+        pkglist = pkglist + _get_packages_from_errata(adv)
+
+    cur_pkgs = list_pkgs().keys()
+
+    # return the package name and version for each package already installed on system
+    # we do not want to install 'extra' packages
+    pkgs = []
+    for p in pkglist:
+        if p['name'] in cur_pkgs:
+            if p['epoch']:
+                pkgs.append({p['name']: '{0}:{1}-{2}'.format(p['epoch'], p['version'], p['release'])})
+            else:
+                pkgs.append({p['name']: '{0}-{1}'.format(p['version'], p['release'])})
+
+    return pkgs
+
+def _get_patches():
+    '''
+    still to do
+    '''
+
+    return {}
+
+def list_patches():
+    '''
+    do something
+    '''
+
+    return _get_patches()
+
+def list_installed_patches():
+    '''
+    List the currently installed advisories
+    '''
+
+    return _get_patches()
+

Example results

{
  "pkg_|-mgr_regular_patches_|-mgr_regular_patches_|-patch_installed": {
    "comment": "",
    "name": "mgr_regular_patches",
    "start_time": "12:20:58.315473",
    "result": true,
    "duration": 6720.789,
    "__run_num__": 2.0,
    "__sls__": "packages.patchinstall",
    "changes": {
      "libsasl2-2": {
        "new": "2.1.27+dfsg-1+deb10u1",
        "old": "2.1.27+dfsg-1"
      },
      "libsasl2-modules": {
        "new": "2.1.27+dfsg-1+deb10u1",
        "old": "2.1.27+dfsg-1"
      },
      "libexpat1": {
        "new": "2.2.6-2+deb10u1",
        "old": "2.2.6-2"
      },
      "libsasl2-modules-db": {
        "new": "2.1.27+dfsg-1+deb10u1",
        "old": "2.1.27+dfsg-1"
      }
    },
    "__id__": "mgr_regular_patches"
  }...

[paususe]

@rpasche
Could you please submit a pull request? It'll be easier to review and manage this that way

Thanks for the contribution!

[Shirocco88]

@rpasche
Could you please add label "ubuntu", "Debian"

Thanks in Advance.

[rpasche]

@rpasche
Could you please submit a pull request? It'll be easier to review and manage this that way

Thanks for the contribution!

@paususe Where should I create the PR to? Uyuni or - I guess - saltstack/salt? Should I somehow flag it, that this needs to go into the "uyuni" version of salt (I know you are using a specially patched version). Thanks for any hints.

[paususe]

@brejoc
Could you please advise @rpasche on what's the best lace to create the PR? I think a PR to Uyuni (to susemanager-sls) may be the right place but I'd rather have an expert confirm :-)

[brejoc]

@rpasche Thanks for taking care of that. I'd say saltstack/salt should be the first target for this. As soon as we have green light there, we can cherry pick this to openSUSE/salt. But you don't have to do that. Just let us know and we'll take care of that.

But if you'd like to do the PR to openSUSE/salt yourself, then you can do that of course. Just open a PR with the changes, add a link to the upstream PR and we'll take it from there. The branch that you'd have to target is openSUSE-3000 for the current stable version or openSUSE-3000.3 for the next release.

[rpasche]

@brejoc

@rpasche Thanks for taking care of that. I'd say saltstack/salt should be the first target for this. As soon as we have green light there, we can cherry pick this to openSUSE/salt. But you don't have to do that. Just let us know and we'll take care of that.

The thing is, how can I test the latest stable salt client, which is not compatible to Uyuni? Currently, because the "default" salt client is not compatible to Uyuni, I'm unable to fully test this. This "errata" fix is code is special to Uyuni only and it also depends on another module spacewalk.api that has to be set. So it is really special to Uyuni.

But if you'd like to do the PR to openSUSE/salt yourself, then you can do that of course. Just open a PR with the changes, add a link to the upstream PR and we'll take it from there. The branch that you'd have to target is openSUSE-3000 for the current stable version or openSUSE-3000.3 for the next release.

Are there any packages already available based on openSUSE-3000.3 that can be used on a Uyuni client. So this can be tested?

[brejoc]

@rpasche Sorry for the huge delay! I wanted to suggest sumaform for that, but I think you can't get a nightly Uyuni with it. Checking with @moio right now. He should know better than me.

[vega82]

Is there a timeline when automatic patching on Ubuntu/Debian will be available?

[an0nz]

+1 for this feature, we primarily use Ubuntu and have errata loaded in but cannot apply patches which would make everything so much simpler.

[an0nz]

For any others waiting on this feature, pull request #4733 by @lucidd has been merged in for a future release that provides errata and patch support for Ubuntu.

[juliogonzalez]

I think this is already fixed? Can we close @vega82 / @an0nz ?

[vega82]

yes it's fixed, you can close this issue, i think so