Channel repo sync schedule change fails during automatic synchronization and deletes existing sync schedule

[santeri3700]

Problem description

Channel repo sync schedule can't be changed while Taskomatic is (automatically) running a "repo-sync" task for the specific channel at the moment.
Attempting to change the sync schedule will result in error and the previously configured schedule to be deleted/lost.

I would not expect the schedule change to fail and especially result in the previously set schedule to be deleted/lost.

Version of Uyuni Server and Proxy (if used)

Reproduced on two separate Uyuni Servers on versions 2022.05 and 2022.06 (updated from 2022.05). I do not know if this affects older versions.

Information for package Uyuni-Server-release:
---------------------------------------------
Repository     : Uyuni Server Stable
Name           : Uyuni-Server-release
Version        : 2022.06-183.9.uyuni2
Arch           : x86_64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level  : Level 3
Installed Size : 1.4 KiB
Installed      : Yes (automatically)
Status         : up-to-date
Source package : Uyuni-Server-release-2022.06-183.9.uyuni2.src
Summary        : Uyuni Server
Description    : 
    Uyuni lets you efficiently manage physical, virtual,
    and cloud-based Linux systems. It provides automated and cost-effective
    configuration and software management, asset management, and system
    provisioning.

Details about the issue

I discovered this issue about a week before opening this issue. I was trying to mass-update channel sync schedules via spacecmd and some channels repeatedly resulted in error. It later came apparent that the channels in question were being automatically synchronized at the moment by Taskomatic due to the daily schedule setting.
These channels are both added via spacewalk-common-channels or manually via Web UI or spacecmd and function fine otherwise.

2022-06-29: I have so far only observed this issue on DEB repositories (Debian & Ubuntu), but this could be because the RPM repositories in my environment(s) are automatically synchronized earlier in the morning when I haven't done any tests.
This affects RPM and DEB based channels.

How to reproduce

• Set up an Uyuni Server version 2022.06 with some channels added via spacewalk-common-channels etc.
  These channels contain packages and patches/errata from previous automatic synchronizations.
• Set a daily sync schedule for a channel and wait for Taskomatic to initiate "repo-sync" at the specified time (I've used Daily at 11:00 schedule for this test)
  • Web UI: Software -> Channels -> [CHANNEL] -> Repositories -> Sync -> Select a Schedule: "Daily at 11:00"
  • CLI: spacecmd 'softwarechannel_setsyncschedule CHANNEL-LABEL 0 0 11 ? * *'
• Inspect the Taskomatic Runtime Status to see when the repo-sync task has been initiated by Taskomatic at about 11:00.
  • Web UI: Admin -> Task Engine Status -> Runtime Status -> Filter by name: "repo-sync"
  • CLI: taskotop -e (taskotop is provided by the spacewalk-utils-extras package)
• Attempt to change the existing sync schedule for the channel currently being synchronized (I've usually tried changing the schedule when the task has been running for 1 or more minutes).
• It will always result in error "Schedule of Repo Sync failed. Check logs for more information." (via Web UI) or "Taskomatic is not available." (via spacecmd)
• Existing sync schedule (0 0 11 ? * *) is deleted at this point and appears unset/disabled when inspecting via Web UI or spacecmd.

Logs and error messages

#Automatically scheduled sync is currently being run for the channel by Taskomatic. It is 11:09~ at this time.
$ spacecmd softwarechannel_listsyncschedule | grep 'ubuntu-1804-amd64-multiverse-security-uyuni'
INFO: Connected to http://localhost/rpc/api as admin
  123  ubuntu-1804-amd64-multiverse-security-uyuni 0 0 11 ? * *

#Attempting to change the sync schedule from 11:00 to 20:00.
$ spacecmd 'softwarechannel_setsyncschedule ubuntu-1804-amd64-multiverse-security-uyuni 0 0 20 ? * *'
INFO: Connected to http://localhost/rpc/api as admin
ERROR: redstone.xmlrpc.XmlRpcFault: Taskomatic is not available.

#Previous schedule change failed and the existing schedule (`0 0 11 ? * *`) was deleted without the new schedule being set.
$ spacecmd softwarechannel_listsyncschedule | grep 'ubuntu-1804-amd64-multiverse-security-uyuni'
INFO: Connected to http://localhost/rpc/api as admin
  123  ubuntu-1804-amd64-multiverse-security-uyuni

/var/log/rhn/rhn_taskomatic_daemon.log:

2022-06-29 11:09:06,311 [Thread-18] WARN  com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job repo-sync-1-123 failed to schedule.

/var/log/messages:

2022-06-29T11:09:06.328960+03:00 uyuni-server server[6462]: [Fatal Error] :-1:-1: Premature end of file.
2022-06-29T11:09:06.329935+03:00 uyuni-server server[6462]: 29-Jun-2022 11:01:52.629 WARNING [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-2] redstone.xmlrpc.XmlRpcDispatcher.writeError redstone.xmlrpc.XmlRpcFault: Taskomatic is not available.

Task Engine Runtime Status at the time:

Task Id	Task Name	Start Time	End Time	Elapsed Time	Status	Data
85789	repo-sync	11:00		549 seconds	running	Ubuntu 18.04 LTS AMD64 Multiverse Security for Uyuni

The task ended successfully later and there were no new packages or patches/errata to be imported.

[mbussolotto]

Thanks for reporting @santeri3700 , if you have the chance it would be interesting to understand if it happens just for RPM (I guess so).

[santeri3700]

@mbussolotto I just tested the issue with RPM based channels and got same results. So it doesn't seem to be specific to RPM or DEB based channels.
Also, I think it is not actually necessary to have the "repo-sync" task start automatically, I just tested this with the "Sync now" button.

[krax13]

Hi all,

I have been facing the same error message with the latest version of SUSE Manager (4.3.5).
Manually repo sync not working on the Web UI, and the automatic mgr-sync-refresh schedule also run into errors.

The full error message in /var/log/rhn/rhn_taskomatic.log:

2023-06-09 14:20:53,031 [DefaultQuartzScheduler_Worker-1] DEBUG com.redhat.rhn.taskomatic.task.MgrSyncRefresh - Refreshing mgr-sync data
…
2023-06-09 14:21:43,943 [DefaultQuartzScheduler_Worker-1] DEBUG com.redhat.rhn.taskomatic.task.MgrSyncRefresh - Scheduling synchronization of all vendor channels
2023-06-09 14:21:43,986 [Thread-8] WARN  com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job single-repo-sync-bunch-1 failed to schedule.
2023-06-09 14:21:44,015 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.taskomatic.task.MgrSyncRefresh - Executing a task threw an exception: org.quartz.JobExecutionException
org.quartz.JobExecutionException: com.redhat.rhn.taskomatic.TaskomaticApiException: redstone.xmlrpc.XmlRpcException: The response could not be parsed.
        at com.redhat.rhn.taskomatic.task.MgrSyncRefresh.execute(MgrSyncRefresh.java:127) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:56) [rhn.jar:?]
        at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:155) [rhn.jar:?]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.3.0.jar:?]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.3.0.jar:?]
Caused by: com.redhat.rhn.taskomatic.TaskomaticApiException: redstone.xmlrpc.XmlRpcException: The response could not be parsed.
        at com.redhat.rhn.taskomatic.TaskomaticApi.invoke(TaskomaticApi.java:92) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.TaskomaticApi.scheduleSingleRepoSync(TaskomaticApi.java:172) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.task.MgrSyncRefresh.execute(MgrSyncRefresh.java:118) ~[rhn.jar:?]
        ... 4 more
Caused by: redstone.xmlrpc.XmlRpcException: The response could not be parsed.
        at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:435) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:209) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at com.redhat.rhn.taskomatic.TaskomaticApi.invoke(TaskomaticApi.java:89) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.TaskomaticApi.scheduleSingleRepoSync(TaskomaticApi.java:172) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.task.MgrSyncRefresh.execute(MgrSyncRefresh.java:118) ~[rhn.jar:?]
        ... 4 more
Caused by: org.xml.sax.SAXParseException: Premature end of file.
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) ~[xerces-j2.jar:?]
        at redstone.xmlrpc.XmlRpcParser.parse(XmlRpcParser.java:158) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:420) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:209) ~[redstone-xmlrpc-client-1.1_20071120.jar:?]
        at com.redhat.rhn.taskomatic.TaskomaticApi.invoke(TaskomaticApi.java:89) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.TaskomaticApi.scheduleSingleRepoSync(TaskomaticApi.java:172) ~[rhn.jar:?]
        at com.redhat.rhn.taskomatic.task.MgrSyncRefresh.execute(MgrSyncRefresh.java:118) ~[rhn.jar:?]
        ... 4 more

• The journalctl -u tomcat shows these error messages:

Jun 09 14:45:10 node01 server[12314]: [Fatal Error] :-1:-1: Premature end of file.
Jun 09 14:47:24 node01 server[12314]: [Fatal Error] :-1:-1: Premature end of file.

• Taskomatic logs shows these errors at the same time:

2023-06-09 14:45:10,183 [Thread-10] WARN  com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job single-repo-sync-bunch-1 failed to schedule.
2023-06-09 14:47:24,907 [Thread-11] WARN  com.redhat.rhn.taskomatic.TaskoQuartzHelper - Job single-repo-sync-bunch-1 failed to schedule.

The custom and vendor channels sync are working via spacewalk-repo-sync but not manually from Web UI.
Vendor channels sync are not working automatically.

Do you have any suggestion of this issue?

[avshiliaev]

@wweellddeerr can this one also be related to https://github.com/SUSE/spacewalk/issues/22195 ?

[santeri3700]

This issue has been resolved at some point and doesn't seem to affect Uyuni 2025.07.