New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fedora templates do not update the Qubes' repos through Whonix update proxy (R4.0-RC5) : " Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current' " #3737

Closed
n1m1 opened this Issue Mar 23, 2018 · 43 comments

Comments

@n1m1

n1m1 commented Mar 23, 2018

Qubes OS version:

R4.0-rc5

Affected component(s):

Fedora templates.


Steps to reproduce the behavior:

In a Fedora template type:
sudo dnf update

Qubes is configured in order to update the templates through sys-whonix-14. Below the conf file in Dom0' (/etc/qubes-rpc/policy/qubes.UpdatesProxy)


$type:TemplateVM $default allow,target=sys-whonix-14
$tag:whonix-updatevm $default allow,target=sys-whonix-14
$tag:whonix-updatevm $anyvm deny
## Note that policy parsing stops at the first match,
## so adding anything below "$anyvm $anyvm action" line will have no effect

## Please use a single # to start your custom comments

# Default rule for all TemplateVMs - direct the connection to sys-net
$type:TemplateVM $default allow,target=sys-net

$anyvm $anyvm deny

Expected behavior:

The Fedora template is updated

Actual behavior:

The Fedora repos properly work, but the Qubes' ones do not. Here is the output I get.

Fedora 26 - x86_64 - Updates                    561 kB/s |  21 MB     00:38    
Fedora 26 - x86_64                              916 kB/s |  53 MB     00:59    
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

General notes:

  • I started to experience this behavior only after updating to Qubes 4.0-rc5. This issue was not present in Qubes 4.0-rc4
  • I do not have this issue with Debian-based templates. apt update && apt upgrade just work.
  • qubes-dom0-update just works.
  • Qubes-updates-proxy is installed in sys-whonix and sys-net
  • If I change this line $type:TemplateVM $default allow,target=sys-whonix-14 in $type:TemplateVM $default allow,target=sys-net everything works as expected.
  • This is the output of systemctl status qubes-updates-proxy in sys-whonix-14
● qubes-updates-proxy.service - Qubes updates proxy (tinyproxy)
   Loaded: loaded (/lib/systemd/system/qubes-updates-proxy.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/qubes-updates-proxy.service.d
           └─40_qubes-whonix.conf
   Active: active (running) since Fri 2018-03-23 14:37:20 UTC; 2h 10min ago
  Process: 887 ExecStartPre=/usr/bin/install -d --owner tinyproxy --group tinyproxy /var/run/tinyproxy (code=exited, status=0/SUCCESS)
 Main PID: 892 (tinyproxy)
    Tasks: 8 (limit: 4915)
   CGroup: /system.slice/qubes-updates-proxy.service
           ├─ 892 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─ 898 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─ 899 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─3367 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─3396 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─3455 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           ├─8643 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf
           └─9233 /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf

Mar 23 14:36:56 host tinyproxy[892]: Waiting servers (0) is less than MinSpareServers (2). Creating new child.
Mar 23 14:37:01 host tinyproxy[892]: Waiting servers (0) is less than MinSpareServers (2). Creating new child.                                                                                                                                                
Mar 23 14:37:06 host tinyproxy[892]: Waiting servers (0) is less than MinSpareServers (2). Creating new child.                                                                                                                                                
Mar 23 14:38:05 host tinyproxy[898]: Proxying refused on filtered domain "127.0.0.1"                                                                                                                                                                          
Mar 23 16:39:03 host tinyproxy[892]: Waiting servers (1) is less than MinSpareServers (2). Creating new child.                                                                                                                                                
Mar 23 16:39:35 host tinyproxy[3455]: Error reading readable client_fd 10                                                                                                                                                                                     
Mar 23 16:39:35 host tinyproxy[3455]: Could not retrieve request entity                                                                                                                                                                                       
Mar 23 16:39:35 host tinyproxy[8643]: Error reading readable client_fd 10                                                                                                                                                                                     
Mar 23 16:39:35 host tinyproxy[8643]: Could not retrieve request entity                                                                                                                                                                                       
Mar 23 16:42:33 host tinyproxy[892]: Waiting servers (0) is less than MinSpareServers (2). Creating new child.  
  • Running sudo systemctl restart qubes-updates-proxy in sys-whonix-14 did not sort effect.

Related issues:

#3557 and #3135 ?

@mirrorway

This comment has been minimized.

mirrorway commented Mar 23, 2018

Same issue here with jessie-based sys-whonix and fedora-26 current-testing.

$ sudo dnf update
Fedora 26 - x86_64 - Updates                    644 kB/s |  21 MB     00:33    
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current-testing'

debian, dom0 updates not affected.
fedora-26 updates properly if changed to use sys-net in /etc/qubes-rpc/policy/qubes.UpdatesProxy.

@andrewdavidwong

This comment has been minimized.

Member

andrewdavidwong commented Mar 24, 2018

Duplicate of #1352 (comment)

@andrewdavidwong andrewdavidwong marked this as a duplicate of #1352 Mar 24, 2018

@andrewdavidwong

This comment has been minimized.

Member

andrewdavidwong commented Mar 24, 2018

This appears to be a duplicate of an existing issue. If you believe this is not really a duplicate, please leave a comment briefly explaining why. We'll be happy to take another look and, if appropriate, reopen this issue. Thank you.

@marmarek

This comment has been minimized.

Member

marmarek commented Mar 24, 2018

I'm not so sure about this being duplicate - fedora repositories do not use onion service by default.

@marmarek marmarek reopened this Mar 24, 2018

@n1m1

This comment has been minimized.

n1m1 commented Mar 24, 2018

Thanks @andrewdavidwong and @marmarek for your kind and prompt answers. I am not using onion services on Qubes 4.0: the problem is not related to them.

P.s: just a note, not related with this issue. I use onion services on a different laptop where I run Qubes 3.2 and, until few days ago, I've never ever experienced a problem with them.

@marmarek

This comment has been minimized.

Member

marmarek commented Mar 24, 2018

Does dnf update --refresh change anything?

@n1m1

This comment has been minimized.

n1m1 commented Mar 24, 2018

Unfortunately it does not.

[user@fedora-26 ~]$ sudo dnf update --refresh
Fedora 26 - x86_64 - Updates                    265 kB/s |  21 MB     01:20    
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

I tried also with:


[user@fedora-26 ~]$ sudo dnf clean all
18 files removed
[user@fedora-26 ~]$ sudo dnf update
Fedora 26 - x86_64 - Updates                    172 kB/s |  21 MB     02:04    
Fedora 26 - x86_64                              351 kB/s |  53 MB     02:35    
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

Some more information here:

[user@fedora-26 ~]$ sudo dnf update -v
Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, qubes-hooks, repoclosure, repograph, repomanage, reposync
DNF version: 2.7.5
cachedir: /var/cache/dnf
repo: using cache for: updates
updates: using metadata from Sat Mar 24 04:21:18 2018.
repo: using cache for: fedora
not found deltainfo for: Fedora 26 - x86_64
not found updateinfo for: Fedora 26 - x86_64
fedora: using metadata from Wed Jul  5 21:31:38 2017.
Cannot download 'http://yum.qubes-os.org/r4.0/current/vm/fc26': repomd.xml parser error: Parse error at line: 1 (not well-formed (invalid token)).
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

May be it is not necessary, but let me emphasize this one more time: this issue occurs only with the Qubes' repos on Fedora templates when updating through sys-whonix. With Qubes' repos on Debian-based templates (8,9.Whonix 13 and 14) I do not have this problem.

@marmarek

This comment has been minimized.

Member

marmarek commented Mar 24, 2018

Can you access that http://yum.qubes-os.org/r4.0/current/vm/fc26/repodata/repomd.xml from sys-whonix (using wget or curl)?

@n1m1

This comment has been minimized.

n1m1 commented Mar 24, 2018

Yes, I can.

user@host:~$ wget http://yum.qubes-os.org/r4.0/current/vm/fc26/repodata/repomd.xml
--2018-03-24 14:47:49--  http://yum.qubes-os.org/r4.0/current/vm/fc26/repodata/repomd.xml
Resolving yum.qubes-os.org (yum.qubes-os.org)... 82.94.215.165
Connecting to yum.qubes-os.org (yum.qubes-os.org)|82.94.215.165|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3689 (3.6K) [application/xml]
Saving to: 'repomd.xml'

repomd.xml                                                      100%[=====================================================================================================================================================>]   3.60K  --.-KB/s    in 0.02s   

2018-03-24 14:47:50 (219 KB/s) - 'repomd.xml' saved [3689/3689]

@andrewdavidwong andrewdavidwong marked this as not a duplicate of #1352 Mar 24, 2018

@andrewdavidwong andrewdavidwong added this to the Release 4.0 milestone Mar 24, 2018

@n1m1

This comment has been minimized.

n1m1 commented Mar 24, 2018

Have you modified something on your side? I've done several tests on my Fedora templates and now everything works as expected.

Here, I just run in Dom0:

sudo qubesctl state.sls qvm.updates-via-whonix

However, the configuration in /etc/qubes-rpc/policy/qubes.UpdatesProxy has not changed.

@mirrorway ?

@marmarek

This comment has been minimized.

Member

marmarek commented Mar 24, 2018

No changes from me.

@SSPJ

This comment has been minimized.

SSPJ commented Mar 24, 2018

I am also experiencing this on Qubes 3.2 trying to update Fedora 23 template.
Error: Failed to synchronize cache for repo 'qubes-vm-r3.2-current'
I am using whatever the default updating mechanism on 3.2 is.

@mirrorway

This comment has been minimized.

mirrorway commented Mar 25, 2018

The wget from sys-whonix also works for me. One tweak with my sys-whonix is, the firewall tab has traffic restricted to my tor entry nodes. Not sure if disabling those rules would help, but I'm a little scared to try...

@Caesarwm1

This comment has been minimized.

Caesarwm1 commented Mar 26, 2018

Hello!

Here, it's nearly the same as with user @n1m1. I'm quite sure I checked (since many, many days) every single dnf possibilitiy like --refresh --clean all, etc., as well as issues #3135, #1352, and setting RestartSec=0s in etc/systemd/system/multi-user.target.wants/qubes-updates-proxy.service.

I can access http://yum.qubes-os.org/r4.0/current/vm/fc26/repodata/repomd.xml from sys-whonix, if I connect sys-whonix directly to Chris Laprise'/tasket's OpenVPN setup, not also to sys-firewall. But the fedora-26 template dnf issue (Qubes repos only; Fedora updates properly if I set Qubes repos enabled=0) existed before this VPN setup, so I don't think it's related.

sudo qubesctl state.sls qvm.updates-via-whonix in dom0 succeeds.

caesarwm1_qubes_ctl_issue3737

More information:

[user@fedora-26 ~]$ sudo dnf update -v
Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, qubes-hooks, repoclosure, repograph, repomanage, reposync
DNF version: 2.7.5
cachedir: /var/cache/dnf
repo: using cache for: updates
updates: using metadata from Mon Mar 26 19:23:26 2018.
repo: using cache for: fedora
not found deltainfo for: Fedora 26 - x86_64
not found updateinfo for: Fedora 26 - x86_64
fedora: using metadata from Wed Jul  5 22:31:38 2017.
Cannot download 'http://yum.qubes-os.org/r4.0/current/vm/fc26': repomd.xml parser error: Parse error at line: 1 (not well-formed (invalid token)).
Error: Failed to synchronize cache for repo 'qubes-vm-r4.0-current'

I tried every other possibility here (sys-net and/or sys-firewall):

caesarwm1_updatesproxy_issue3737

Also, I tested both onion/hidden service addresses; at the moment I use the clearnet addresses.

Best regards!

@marmarek

This comment has been minimized.

Member

marmarek commented Mar 26, 2018

Try to modify /etc/yum.repos.d/qubes-r4.repo to use https instead of http.

@Caesarwm1

This comment has been minimized.

Caesarwm1 commented Mar 27, 2018

Oh my … That – dead ;) – simple. I didnt' read about this nowhere. And I read a lot.
Thank you so much, marmarek!
Best regards.

screenshot_2018-03-27_03-07-24

@mirrorway

This comment has been minimized.

mirrorway commented Mar 27, 2018

Change to https fixes it for me too :)

@awokd

This comment has been minimized.

awokd commented Apr 4, 2018

This is an issue on a fresh install of 4.0 final as well, with the default fedora-26 template. The https fix works, although those checksum errors are worrisome. Probably shouldn't expect users to edit files out of the box to get updates working. Is the http -> https redirect breaking it? Can 4.0 final be updated to include the https fix?

@andrewdavidwong

This comment has been minimized.

Member

andrewdavidwong commented Apr 22, 2018

@awokd: Thank you for providing these steps. Before I send an edited version of these steps to the MLs and post it on the website, I want to confirm with @marmarek and/or @rootkovska on a couple of points:

  1. Can you confirm that these are actually steps that all current 4.0 users should perform?
  2. Should this be a QSB? It is highly unusual for us to post an important announcement that requires user action that is not a QSB. (I can't currently recall another case.) For example, it seems like this announcement should go to qubes-announce, even if it's not a QSB.
@awokd

This comment has been minimized.

awokd commented Apr 22, 2018

I'm having difficulty recreating the dom0 update issue on my test install- they've been working pretty reliably there. My other systems might have been operating against an old cache, but neither of them showed updates until yesterday when I changed them to https.

@andrewdavidwong

This comment has been minimized.

Member

andrewdavidwong commented Apr 26, 2018

It looks like @marmarek and @rootkovska are too busy to respond to this, so I'm just going to opt for a conservative course of action by sending the announcement to qubes-users and qubes-devel (but not qubes-announce or posting it on the website). If, for some reason, we later come to believe that sending it to qubes-users and qubes-devel was insufficient, we can always send or post it in additional places.

@andrewdavidwong

This comment has been minimized.

Member

andrewdavidwong commented Apr 26, 2018

We should also update this FAQ entry:

After thinking about this some more, I don't think it's necessary to update the FAQ entry. The current content still applies in the general case, and the proposed addition wouldn't apply after this fix has been implemented. Stating the fix in the announcement, installation guide, and release notes should be sufficient to spread awareness to users.

@adrelanos

This comment has been minimized.

Member

adrelanos commented Apr 29, 2018

I think the only realistic way to have most users who have this issue having it fixed for them is a dom0 upgrade which then uses salt to make the change for the user inside the template.

(But I am aware that the general development time scarcity as well.)

@qubesos-bot

This comment has been minimized.

qubesos-bot commented May 2, 2018

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 testing repository for the CentOS centos7 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.0-current-testing

Changes included in this update

@qubesos-bot

This comment has been minimized.

qubesos-bot commented May 14, 2018

Automated announcement from builder-github

The package pykickstart-2.32-4.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

@qubesos-bot

This comment has been minimized.

qubesos-bot commented May 21, 2018

Automated announcement from builder-github

The component core-agent-linux (including package python2-dnf-plugins-qubes-hooks-4.0.28-1.fc26) has been pushed to the r4.0 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@qubesos-bot

This comment has been minimized.

qubesos-bot commented May 21, 2018

Automated announcement from builder-github

The package core-agent-linux has been pushed to the r4.0 stable repository for the Fedora centos7 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@qubesos-bot

This comment has been minimized.

qubesos-bot commented May 21, 2018

Automated announcement from builder-github

The package qubes-core-agent_4.0.28-1+deb9u1 has been pushed to the r4.0 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

marmarek added a commit to QubesOS/qubes-installer-qubes-os that referenced this issue May 28, 2018

qubes-release: switch qubes-dom0 repositories to https
This is needed for more reliable updates over tor, and also gives some
more privacy.

Fixes QubesOS/qubes-issues#3737

(cherry picked from commit 512af4b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment