SALT fails with "Failed to return clean data" since Dispvm switch

[ekaflaer]

Qubes OS version: 4.0

Affected component(s):

• Qubes Salt stack
• Qrexec policy

---

Steps to reproduce the behavior:

1. Change your dispvm as described in the wiki (in my case the dispvm used the same template as the one before)
2. Set it as default dispvm
3. Remove the old one
4. Run sudo qubesctl --templates state.highstate to apply
   `

Expected behavior:

Updates are applied according to my salt scripts which worked before with the old dispvm with the same template.

Actual behavior:

The dispvm is working fine if I start it from another AppVM or if I start Firefox from the XFCE menu but if I apply salt I get the follwing error in the mgmt log of the template. The regular output just shows "Error":

_error:
   Failed to return clean data
retcode:
   126
stderr:
  Request refused
stdout:
exit code: 20

Oh and switching back to the old dispvm did not fix the issue. So it seems something broke or left files behind which prevents proper salt execution.

General notes:

I also can not copy files with qvm-copy or the Nautilus pendant anymore. I always get the message Request refused.

My system and dom0 is up to date and I have restarted it.
Appvm:

rpm -q qubes-core-agent
qubes-core-agent-4.0.31-1.fc28.x86_64

rpm -q qubes-utils
qubes-utils-4.0.19-1.fc28.x86_64

Dom0:

rpm -q qubes-core-dom0
qubes-core-dom0-4.0.27-1.fc25.x86_64

I am not sure if I am just feeling this way but it is really hard to debug issues like this. The dispvm for salt just disappears if an error appears, the salt output just prints Error, the mgmt log of the template just shows the simple error message and I did not find any other information in the qubes log. I have even moved every log file in the qubes log directory away to only have fresh files and find something but I was unable to.

So it would be great if you could update or add a wiki page for debugging salt issues and where to look if something fails with qrexec policy. Thanks in advance and thank you for creating such a great operating system.

[marmarek]

Check journalctl -b in dom0 - especially messages about qrexec calls. Both problems looks related to policy denying the calls.

[marmarek]

Check journalctl -b in dom0 - especially messages about qrexec calls. Both problems looks related to policy denying the calls.

[ekaflaer]

Thanks for your hint.
If I copy a file the error message in journalctl looks like
dom0 qrexec [...]: qubes.Filecopy: test -> @default: error loading policy: /etc/qubes-rpc/policy/qubes.Filecopy:20 wrong number of fields

The file looked like:

[some old vm names before I upgraded to Fedora 28]
# default comment
$anyvm $anyvm ask
m          ask
k
m          ask

Removing the three lines at the bottom fixes the copying issue, thanks you for the help. I have no clue where they came from, I do not remember changing this file.

Anyway it does not fix the salt issue. I will follow up on this because this needs more debugging.

[ekaflaer]

Thanks for your hint.
If I copy a file the error message in journalctl looks like
dom0 qrexec [...]: qubes.Filecopy: test -> @default: error loading policy: /etc/qubes-rpc/policy/qubes.Filecopy:20 wrong number of fields

The file looked like:

[some old vm names before I upgraded to Fedora 28]
# default comment
$anyvm $anyvm ask
m          ask
k
m          ask

Removing the three lines at the bottom fixes the copying issue, thanks you for the help. I have no clue where they came from, I do not remember changing this file.

Anyway it does not fix the salt issue. I will follow up on this because this needs more debugging.

[ekaflaer]

@marmarek I have fixed the salt issue by removing all files in this policy directory besides the qubes.ReceiveUpdates and reinstalling all the packages which provided the files there.

If it is interesting for you I can try to diff all the files and check what the relevant difference might be but if not I will leave it like this since it is working. I do not think I changed the files manually.

[ekaflaer]

@marmarek I have fixed the salt issue by removing all files in this policy directory besides the qubes.ReceiveUpdates and reinstalling all the packages which provided the files there.

If it is interesting for you I can try to diff all the files and check what the relevant difference might be but if not I will leave it like this since it is working. I do not think I changed the files manually.

[marmarek]

I remember some bug resulting in empty policy file, but that was in testing framework (#3535).

[marmarek]

I remember some bug resulting in empty policy file, but that was in testing framework (#3535).