errata is associated with only one channel

[angystardust]

Hi @stevemeier and first of all thanks for you very useful script!
Yesterday I noticed a very strange problem on my Spacewalk 2.7 setup with both Centos6 and CentOS-7 channels synced.
Let's take for example CESA-2018-0093. This is the relevant entry in the errata.latest.xml:

 <CESA-2018--0093 description="Not available" from="centos-announce@centos.org" issue_date="2018-01-17 14:59:21" multirelease="1" notes="Not available" product="CentOS Linux" references="https://access.redhat.com/errata/RHSA-2018:0093 https://lists.centos.org/pipermail/centos-announce/2018-January/022709.html https://lists.centos.org/pipermail/centos-announce/2018-January/022710.html" release="2" severity="Important" solution="Not available" synopsis="Important CentOS microcode_ctl Security Update" topic="Not available" type="Security Advisory">
    <os_arch>i686</os_arch>
    <os_arch>x86_64</os_arch>
    <os_release>6</os_release>
    <os_release>7</os_release>
    <packages>microcode_ctl-1.17-25.4.el6_9.i686.rpm</packages>
    <packages>microcode_ctl-1.17-25.4.el6_9.src.rpm</packages>
    <packages>microcode_ctl-1.17-25.4.el6_9.x86_64.rpm</packages>
    <packages>microcode_ctl-2.1-22.5.el7_4.src.rpm</packages>
    <packages>microcode_ctl-2.1-22.5.el7_4.x86_64.rpm</packages>
  </CESA-2018--0093>

If I run your script with the following parameters:

errata-import.pl --server localhost --errata $TMPDIR/errata.latest.xml --include-channels=centos6-x86_64,centos6-x86_64-updates,centos7-x86_64,centos7-x86_64-updates --rhsa-oval=/$TMPDIR/com.redhat.rhsa-all.xml --publish

at the end of the script execution, I can successfully see the errata published on my Spacewalk instance but it's only linked with one channel (centos7-x86_64) instead of both centos6-x86_64 and centos7-x86_64-updates channels
You can see from the output of spacecmd that the affected packages are both linked to the errata but the linked channel is only one:

spacecmd {SSM:0}> errata_details CESA-2018:0093
Name:       CESA-2018:0093
Product:    CentOS Linux
Type:       Security Advisory
Issue Date: 1/17/18

Topic
-----
Not available

Description
-----------
Not available

Notes
-----
Not available

CVEs
----

Solution
--------
Not available

References
----------
https://access.redhat.com/errata/RHSA-2018:0093
https://lists.centos.org/pipermail/centos-
announce/2018-January/022709.html https://lists.centos.org/pipermail
/centos-announce/2018-January/022710.html

Affected Channels
-----------------
centos7-x86_64-updates

Affected Systems
----------------
2

Affected Packages
-----------------
microcode_ctl-1.17-25.4.el6_9:1.x86_64
microcode_ctl-2.1-22.5.el7_4:2.x86_64

This is the what the script logs if I put it in debug mode:

DEBUG: Processing CESA-2018:0093
DEBUG: Processing CESA-2018:0093 -- OVAL ID is oval:com.redhat.rhsa:def:20180093
INFO: Errata for CESA-2018:0093 already exists
DEBUG: CESA-2018:0093 packages: 18430
DEBUG: Package: microcode_ctl-1.17-25.4.el6_9.i686.rpm not found
DEBUG: Package: microcode_ctl-1.17-25.4.el6_9.src.rpm not found
DEBUG: Package: microcode_ctl-1.17-25.4.el6_9.x86_64.rpm -> 57623 -> centos6-x86_64-updates
DEBUG: Package: microcode_ctl-2.1-22.5.el7_4.src.rpm not found
DEBUG: Package: microcode_ctl-2.1-22.5.el7_4.x86_64.rpm -> 18430 -> centos7-x86_64-updates
INFO: Adding packages to CESA-2018:0093

Is it the expected behaviour? Is there a way to link both channels to I can see the relevant errata applicable for all the relevant systems?
Thanks a lot for your help

[stevemeier]

I just ran an import on a fresh install of Spacewalk 2.7 with only the Updates channels synced for CentOS 6 and 7. This is the result I got:

spacecmd {SSM:0}> errata_details CESA-2018:0093
Name:       CESA-2018:0093
Product:    CentOS Linux
Type:       Security Advisory
Issue Date: 1/17/18

Topic
-----
Not available

Description
-----------
The microcode_ctl packages provide microcode updates for Intel and AMD
processors.  This update supersedes microcode provided by Red Hat with
the CVE-2017-5715 ("Spectre") CPU branch injection vulnerability
mitigation. (Historically, Red Hat has provided updated microcode,
developed by our microprocessor partners, as a customer convenience.)
Further testing has uncovered problems with the microcode provided
along with the "Spectre" mitigation that could lead to system
instabilities. As a result, Red Hat is providing an microcode update
that reverts to the last known good microcode version dated before 03
January 2018. Red Hat strongly recommends that customers contact their
hardware provider for the latest microcode updates.  IMPORTANT:
Customers using Intel Skylake-, Broadwell-, and Haswell-based
platforms must obtain and install updated microcode from their
hardware vendor immediately. The "Spectre" mitigation requires both an
updated kernel from Red Hat and updated microcode from your hardware
vendor.

Notes
-----
The description and CVE numbers have been taken from Red Hat OVAL
definitions.  Copyright 2018 Red Hat, Inc.

CVEs
----


Solution
--------
Not available

References
----------
https://access.redhat.com/errata/RHSA-2018:0093
https://lists.centos.org/pipermail/centos-
announce/2018-January/022709.html https://lists.centos.org/pipermail
/centos-announce/2018-January/022710.html

Affected Channels
-----------------
centos6-i386-updates
centos6-x86_64-updates
centos7-x86_64-updates

Affected Systems
----------------
0

Affected Packages
-----------------
microcode_ctl-1.17-25.4.el6_9:1.i686
microcode_ctl-1.17-25.4.el6_9:1.x86_64
microcode_ctl-2.1-22.5.el7_4:2.x86_64

This looks fine.

The only difference I see in your import is that you were updating an errata (possibly because of newly imported packages after a sync). This should not behave differently, but I will give it a try.

[stevemeier]

OK, I think I have found the issue.

When the errata-import script detects that an errata already exists but finds new packages that belong to it (e.g. after a sync), it adds them using the addPackages API call.

The missing step after this seems to be a "re-publish", which is not initiated by addPackages.

This should be a fairly straight-forward addition to errata-import.pl.
I will look at the required changes over the next few days and post an update.

[stevemeier]

I couldn't go to sleep with this bug still open.
The errata-import.pl script has been updated to handle this.
@angystardust: Please test and post some feedback here. Thanks!

[angystardust]

Wow! I didn't expect a so fast reply and solution, @stevemeier ! I feel guilty for you lost sleep :)
Anyway, I tested the patch and I can say that it works as expected (I can now see all the errata <-> channel association) but...

Before the patch

INFO: Errata created: 0

real	9m19.268s
user	1m36.624s
sys	0m7.724s

After the patch:

INFO: Errata created: 0
INFO: Errata updated: 0

real	1205m43.098s
user	2m10.792s
sys	0m13.098s

As you can see, after the patch the script takes nearly 20 hours (!) to complete the execution.
Is this expected? It seems that each errata is republished even when no change is done

A minor glitch: the counter "errata updated" doesn't seem to work.

[stevemeier]

The re-publishing should now be fixed.
Please check again, if possible. Thanks!

[angystardust]

Great! I'm trying the new version now. I'll let you know as soon as it finish ;)

[angystardust]

Sorry for the waiting, @stevemeier. I re-tested the script with the last patch and now it's fast again but in my last run there wasn't any new errata to create or something to update.

INFO: Errata created: 0
INFO: Errata updated: 0

real	9m28.145s
user	1m34.634s
sys	0m7.774s

I also want to test the script in case of an update to an errata.
So, before launching the script again I modified by hand an errata (CEEA-2018:0232) through the WebGUI: I un-flagged one of the 2 channels (centos7-x86_64-updates) in order to disassociate it from the errata and then I also removed the 2 relevant packages (tzdata-java-2018c-1.el7.noarch and tzdata-2018c-1.el7.noarch) using spacecmd. Obliviously I did a spacewalk-reposync of the centos7-x86_64-updates repository in order to upload back the 2 removed packages in the spacewalk database.
The goal was testing if the script would be able to update the errata with all their relevant packages and channels. Here's what happened:

INFO: Errata for CEEA-2018:0232 already exists
INFO: Adding packages to CEEA-2018:0232
INFO: Republishing CEEA-2018:0232
500 Internal Server Error

real	9m4.893s
user	1m33.974s
sys	0m7.755s
It seems like there was a problem while publishing the most recent errata...

Here is the full strack-trace found in rhn_web_api.log

[2018-03-07 17:54:55,320] INFO  - REQUESTED FROM: 127.0.0.1 CALL: errata.list_packages(errata, CEEA-2018:0232) CALLER: (errata) TIME: 0.015 seconds
[2018-03-07 17:54:55,378] INFO  - REQUESTED FROM: 127.0.0.1 CALL: errata.add_packages(errata, CEEA-2018:0232, [57647, 57624, 57625, 19086]) CALLER: (errata) TIME: 0.051 seconds
[2018-03-07 17:54:56,163] ERROR - REQUESTED FROM: 127.0.0.1 CALL: errata.publish(errata, CEEA-2018:0232, [centos7-x86_64-updates]) CALLER: (errata) TIME: 0.777 seconds
redstone.xmlrpc.XmlRpcFault: unhandled internal exception: could not insert: [com.redhat.rhn.domain.errata.impl.PublishedErrataFile]
	at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:179)
	at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)
	at com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54)
	at com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

Strangely enough, the errata is successfully updated even if the script ends with an error.
If i re-run the script again, it completes successfully.
Do you have any ideas why this happens?

[stevemeier]

The documentation of the Spacewalk API is, unfortunately, not great.
It documents the calls and parameters but does not contain information on limitations or requirements.

When the republishing is done in line 656, the scripts attempts to publish the errata to all channels that contain an affected RPM. This includes the channel(s) it has previously been published too.
I think Spacewalk is chocking on this and throws the exception you are seeing.

To fix this, I would have to make the code a lot smarter. I will see if that's possible.

[angystardust]

Thanks for taking care of this, @stevemeier!

[angystardust]

Hi @stevemeier just to help you troubleshoot and analyse the issue, I'll paste here a snippet of the stack error that the errata-import.pl produces:

[2018-03-22 02:56:04,765] ERROR - REQUESTED FROM: 127.0.0.1 CALL: errata.create(errata, {advisory_name=CEEA-2018:0405, product=CentOS Linux, notes=Not available, solution=Not available, references=https://access.redhat.com/errata/RHEA-2018:0405 https://lists.centos.org/pipermail/centos-announce/2018-March/022777.html, advisory_type=Product Enhancement Advisory, advisory_release=1, topic=Not available, description=Not available, synopsis=CentOS fence-agents Enhancement Update}, [], [], [57899, 57916, 57905, 57920, 57910, 57915, 57898, 57913, 57914, 57893, 57892, 57907, 57889, 57903, 57894, 57918, 57911, 57895, 57904, 57906, 57917, 57896, 57902, 57909, 57901, 57919, 57921, 57890, 57912, 57891, 57897, 57908, 57900], false, [centos7-x86_64-updates]) CALLER: (errata) TIME: 0.028 seconds
redstone.xmlrpc.XmlRpcFault: Errata already exists with advisory CEEA-2018:0405
	at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:171)
	at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)
	at com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54)
	at com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

P.s.: associating by hand the relevant channel to the errata from the webgui and re-launching the script fixed the issue (I did this task for every errata that was breaking the script)

[cheeseburger12]

Hi,

i have the same problem since i updates spacewalk from 2.6 to 2.7. I get the folowing error:
INFO: Creating errata for CEBA-2014:1045

then on the console:

"Fault returned from XML RPC Server, fault code 2601"

ERROR - REQUESTED FROM: 127.0.0.1 CALL: errata.create

[stevemeier]

@cheeseburger12: Can you please post the full stack trace? I don't really get any info from the piece you posted.