New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

errata is associated with only one channel #4

Closed
angystardust opened this Issue Mar 4, 2018 · 17 comments

Comments

Projects
None yet
3 participants
@angystardust
Contributor

angystardust commented Mar 4, 2018

Hi @stevemeier and first of all thanks for you very useful script!
Yesterday I noticed a very strange problem on my Spacewalk 2.7 setup with both Centos6 and CentOS-7 channels synced.
Let's take for example CESA-2018-0093. This is the relevant entry in the errata.latest.xml:

 <CESA-2018--0093 description="Not available" from="centos-announce@centos.org" issue_date="2018-01-17 14:59:21" multirelease="1" notes="Not available" product="CentOS Linux" references="https://access.redhat.com/errata/RHSA-2018:0093 https://lists.centos.org/pipermail/centos-announce/2018-January/022709.html https://lists.centos.org/pipermail/centos-announce/2018-January/022710.html" release="2" severity="Important" solution="Not available" synopsis="Important CentOS microcode_ctl Security Update" topic="Not available" type="Security Advisory">
    <os_arch>i686</os_arch>
    <os_arch>x86_64</os_arch>
    <os_release>6</os_release>
    <os_release>7</os_release>
    <packages>microcode_ctl-1.17-25.4.el6_9.i686.rpm</packages>
    <packages>microcode_ctl-1.17-25.4.el6_9.src.rpm</packages>
    <packages>microcode_ctl-1.17-25.4.el6_9.x86_64.rpm</packages>
    <packages>microcode_ctl-2.1-22.5.el7_4.src.rpm</packages>
    <packages>microcode_ctl-2.1-22.5.el7_4.x86_64.rpm</packages>
  </CESA-2018--0093>

If I run your script with the following parameters:

errata-import.pl --server localhost --errata $TMPDIR/errata.latest.xml --include-channels=centos6-x86_64,centos6-x86_64-updates,centos7-x86_64,centos7-x86_64-updates --rhsa-oval=/$TMPDIR/com.redhat.rhsa-all.xml --publish

at the end of the script execution, I can successfully see the errata published on my Spacewalk instance but it's only linked with one channel (centos7-x86_64) instead of both centos6-x86_64 and centos7-x86_64-updates channels
You can see from the output of spacecmd that the affected packages are both linked to the errata but the linked channel is only one:

spacecmd {SSM:0}> errata_details CESA-2018:0093
Name:       CESA-2018:0093
Product:    CentOS Linux
Type:       Security Advisory
Issue Date: 1/17/18

Topic
-----
Not available

Description
-----------
Not available

Notes
-----
Not available

CVEs
----

Solution
--------
Not available

References
----------
https://access.redhat.com/errata/RHSA-2018:0093
https://lists.centos.org/pipermail/centos-
announce/2018-January/022709.html https://lists.centos.org/pipermail
/centos-announce/2018-January/022710.html

Affected Channels
-----------------
centos7-x86_64-updates

Affected Systems
----------------
2

Affected Packages
-----------------
microcode_ctl-1.17-25.4.el6_9:1.x86_64
microcode_ctl-2.1-22.5.el7_4:2.x86_64

This is the what the script logs if I put it in debug mode:

DEBUG: Processing CESA-2018:0093
DEBUG: Processing CESA-2018:0093 -- OVAL ID is oval:com.redhat.rhsa:def:20180093
INFO: Errata for CESA-2018:0093 already exists
DEBUG: CESA-2018:0093 packages: 18430
DEBUG: Package: microcode_ctl-1.17-25.4.el6_9.i686.rpm not found
DEBUG: Package: microcode_ctl-1.17-25.4.el6_9.src.rpm not found
DEBUG: Package: microcode_ctl-1.17-25.4.el6_9.x86_64.rpm -> 57623 -> centos6-x86_64-updates
DEBUG: Package: microcode_ctl-2.1-22.5.el7_4.src.rpm not found
DEBUG: Package: microcode_ctl-2.1-22.5.el7_4.x86_64.rpm -> 18430 -> centos7-x86_64-updates
INFO: Adding packages to CESA-2018:0093

Is it the expected behaviour? Is there a way to link both channels to I can see the relevant errata applicable for all the relevant systems?
Thanks a lot for your help

@stevemeier

This comment has been minimized.

Owner

stevemeier commented Mar 5, 2018

I just ran an import on a fresh install of Spacewalk 2.7 with only the Updates channels synced for CentOS 6 and 7. This is the result I got:

spacecmd {SSM:0}> errata_details CESA-2018:0093
Name:       CESA-2018:0093
Product:    CentOS Linux
Type:       Security Advisory
Issue Date: 1/17/18

Topic
-----
Not available

Description
-----------
The microcode_ctl packages provide microcode updates for Intel and AMD
processors.  This update supersedes microcode provided by Red Hat with
the CVE-2017-5715 ("Spectre") CPU branch injection vulnerability
mitigation. (Historically, Red Hat has provided updated microcode,
developed by our microprocessor partners, as a customer convenience.)
Further testing has uncovered problems with the microcode provided
along with the "Spectre" mitigation that could lead to system
instabilities. As a result, Red Hat is providing an microcode update
that reverts to the last known good microcode version dated before 03
January 2018. Red Hat strongly recommends that customers contact their
hardware provider for the latest microcode updates.  IMPORTANT:
Customers using Intel Skylake-, Broadwell-, and Haswell-based
platforms must obtain and install updated microcode from their
hardware vendor immediately. The "Spectre" mitigation requires both an
updated kernel from Red Hat and updated microcode from your hardware
vendor.

Notes
-----
The description and CVE numbers have been taken from Red Hat OVAL
definitions.  Copyright 2018 Red Hat, Inc.

CVEs
----


Solution
--------
Not available

References
----------
https://access.redhat.com/errata/RHSA-2018:0093
https://lists.centos.org/pipermail/centos-
announce/2018-January/022709.html https://lists.centos.org/pipermail
/centos-announce/2018-January/022710.html

Affected Channels
-----------------
centos6-i386-updates
centos6-x86_64-updates
centos7-x86_64-updates

Affected Systems
----------------
0

Affected Packages
-----------------
microcode_ctl-1.17-25.4.el6_9:1.i686
microcode_ctl-1.17-25.4.el6_9:1.x86_64
microcode_ctl-2.1-22.5.el7_4:2.x86_64

This looks fine.

The only difference I see in your import is that you were updating an errata (possibly because of newly imported packages after a sync). This should not behave differently, but I will give it a try.

@stevemeier

This comment has been minimized.

Owner

stevemeier commented Mar 5, 2018

OK, I think I have found the issue.

When the errata-import script detects that an errata already exists but finds new packages that belong to it (e.g. after a sync), it adds them using the addPackages API call.

The missing step after this seems to be a "re-publish", which is not initiated by addPackages.

This should be a fairly straight-forward addition to errata-import.pl.
I will look at the required changes over the next few days and post an update.

stevemeier added a commit that referenced this issue Mar 5, 2018

@stevemeier

This comment has been minimized.

Owner

stevemeier commented Mar 5, 2018

I couldn't go to sleep with this bug still open.
The errata-import.pl script has been updated to handle this.
@angystardust: Please test and post some feedback here. Thanks!

@stevemeier stevemeier self-assigned this Mar 6, 2018

@angystardust

This comment has been minimized.

Contributor

angystardust commented Mar 7, 2018

Wow! I didn't expect a so fast reply and solution, @stevemeier ! I feel guilty for you lost sleep :)
Anyway, I tested the patch and I can say that it works as expected (I can now see all the errata <-> channel association) but...

Before the patch

INFO: Errata created: 0

real	9m19.268s
user	1m36.624s
sys	0m7.724s

After the patch:

INFO: Errata created: 0
INFO: Errata updated: 0

real	1205m43.098s
user	2m10.792s
sys	0m13.098s

As you can see, after the patch the script takes nearly 20 hours (!) to complete the execution.
Is this expected? It seems that each errata is republished even when no change is done

A minor glitch: the counter "errata updated" doesn't seem to work.

@stevemeier

This comment has been minimized.

Owner

stevemeier commented Mar 7, 2018

The re-publishing should now be fixed.
Please check again, if possible. Thanks!

@angystardust

This comment has been minimized.

Contributor

angystardust commented Mar 7, 2018

Great! I'm trying the new version now. I'll let you know as soon as it finish ;)

@angystardust

This comment has been minimized.

Contributor

angystardust commented Mar 9, 2018

Sorry for the waiting, @stevemeier. I re-tested the script with the last patch and now it's fast again but in my last run there wasn't any new errata to create or something to update.

INFO: Errata created: 0
INFO: Errata updated: 0

real	9m28.145s
user	1m34.634s
sys	0m7.774s

I also want to test the script in case of an update to an errata.
So, before launching the script again I modified by hand an errata (CEEA-2018:0232) through the WebGUI: I un-flagged one of the 2 channels (centos7-x86_64-updates) in order to disassociate it from the errata and then I also removed the 2 relevant packages (tzdata-java-2018c-1.el7.noarch and tzdata-2018c-1.el7.noarch) using spacecmd. Obliviously I did a spacewalk-reposync of the centos7-x86_64-updates repository in order to upload back the 2 removed packages in the spacewalk database.
The goal was testing if the script would be able to update the errata with all their relevant packages and channels. Here's what happened:

INFO: Errata for CEEA-2018:0232 already exists
INFO: Adding packages to CEEA-2018:0232
INFO: Republishing CEEA-2018:0232
500 Internal Server Error

real	9m4.893s
user	1m33.974s
sys	0m7.755s
It seems like there was a problem while publishing the most recent errata...

Here is the full strack-trace found in rhn_web_api.log

[2018-03-07 17:54:55,320] INFO  - REQUESTED FROM: 127.0.0.1 CALL: errata.list_packages(errata, CEEA-2018:0232) CALLER: (errata) TIME: 0.015 seconds
[2018-03-07 17:54:55,378] INFO  - REQUESTED FROM: 127.0.0.1 CALL: errata.add_packages(errata, CEEA-2018:0232, [57647, 57624, 57625, 19086]) CALLER: (errata) TIME: 0.051 seconds
[2018-03-07 17:54:56,163] ERROR - REQUESTED FROM: 127.0.0.1 CALL: errata.publish(errata, CEEA-2018:0232, [centos7-x86_64-updates]) CALLER: (errata) TIME: 0.777 seconds
redstone.xmlrpc.XmlRpcFault: unhandled internal exception: could not insert: [com.redhat.rhn.domain.errata.impl.PublishedErrataFile]
	at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:179)
	at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)
	at com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54)
	at com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

Strangely enough, the errata is successfully updated even if the script ends with an error.
If i re-run the script again, it completes successfully.
Do you have any ideas why this happens?

@stevemeier

This comment has been minimized.

Owner

stevemeier commented Mar 9, 2018

The documentation of the Spacewalk API is, unfortunately, not great.
It documents the calls and parameters but does not contain information on limitations or requirements.

When the republishing is done in line 656, the scripts attempts to publish the errata to all channels that contain an affected RPM. This includes the channel(s) it has previously been published too.
I think Spacewalk is chocking on this and throws the exception you are seeing.

To fix this, I would have to make the code a lot smarter. I will see if that's possible.

@angystardust

This comment has been minimized.

Contributor

angystardust commented Mar 9, 2018

Thanks for taking care of this, @stevemeier!

@angystardust

This comment has been minimized.

Contributor

angystardust commented Mar 22, 2018

Hi @stevemeier just to help you troubleshoot and analyse the issue, I'll paste here a snippet of the stack error that the errata-import.pl produces:

[2018-03-22 02:56:04,765] ERROR - REQUESTED FROM: 127.0.0.1 CALL: errata.create(errata, {advisory_name=CEEA-2018:0405, product=CentOS Linux, notes=Not available, solution=Not available, references=https://access.redhat.com/errata/RHEA-2018:0405 https://lists.centos.org/pipermail/centos-announce/2018-March/022777.html, advisory_type=Product Enhancement Advisory, advisory_release=1, topic=Not available, description=Not available, synopsis=CentOS fence-agents Enhancement Update}, [], [], [57899, 57916, 57905, 57920, 57910, 57915, 57898, 57913, 57914, 57893, 57892, 57907, 57889, 57903, 57894, 57918, 57911, 57895, 57904, 57906, 57917, 57896, 57902, 57909, 57901, 57919, 57921, 57890, 57912, 57891, 57897, 57908, 57900], false, [centos7-x86_64-updates]) CALLER: (errata) TIME: 0.028 seconds
redstone.xmlrpc.XmlRpcFault: Errata already exists with advisory CEEA-2018:0405
	at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:171)
	at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)
	at com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54)
	at com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

P.s.: associating by hand the relevant channel to the errata from the webgui and re-launching the script fixed the issue (I did this task for every errata that was breaking the script)

@cheeseburger12

This comment has been minimized.

cheeseburger12 commented Mar 26, 2018

Hi,

i have the same problem since i updates spacewalk from 2.6 to 2.7. I get the folowing error:
INFO: Creating errata for CEBA-2014:1045

then on the console:

"Fault returned from XML RPC Server, fault code 2601"

ERROR - REQUESTED FROM: 127.0.0.1 CALL: errata.create

@stevemeier

This comment has been minimized.

Owner

stevemeier commented Mar 27, 2018

@cheeseburger12: Can you please post the full stack trace? I don't really get any info from the piece you posted.

@stevemeier

This comment has been minimized.

Owner

stevemeier commented Mar 27, 2018

@angystardust, @cheeseburger12
I have just committed a new version (20180327) that is more selective when doing the republishing.
I think this should fix the issue but I was not able to reproduce it recently so I am not 100% sure.
Please test it and let me know if it works. Thanks!

@angystardust

This comment has been minimized.

Contributor

angystardust commented Mar 28, 2018

Thank you a lot @stevemeier. I'll test during the next weeks and let you know ;)

@cheeseburger12

This comment has been minimized.

cheeseburger12 commented Mar 29, 2018

I've all times the problem that the "errata-import" script stops with:
Fault returned from XML RPC Server, fault code 2601: redstone.xmlrpc.XmlRpcFault: Errata already exists with advisory CEBA-2014:1081

I also clear the hole erratas with spacecmd(spacecmd -y "errata_delete *") and your wipe script. After that i see the issue again.

In the frontend I could not see any more "erratas", but in the database.

That's why I deleted the remaining "erratas" directly from the database:
rhnschema=# delete from rhnerrata where advisory IN (select advisory from rhnerrata where advisory like 'CEBA-2014:1081');

now it seems to work and i will test it to other channel.

Thank you steve

@stevemeier

This comment has been minimized.

Owner

stevemeier commented Mar 29, 2018

@cheeseburger12 Your issue sounds like a database inconsistency. My scripts asks the API for a list of existing Errata before trying to create them. So in your case the API did not report this Errata as existing but then complained when the script tried to create it. Unfortunately, such a thing is almost impossible to reproduce.

@cheeseburger12

This comment has been minimized.

cheeseburger12 commented Mar 29, 2018

Absolutely right. Thank you very much. I assumed that it is on the second channel

@stevemeier stevemeier added the bug label May 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment