4.2: user_salt isn't working #8491

rapenne-s · 2023-09-05T11:58:17Z

Qubes OS release

Fresh install of 4.2-RC3

Brief summary

I installed 4.2-RC3, enabled user-dirs in salt, imported my salt repo which work on other Qubes OS deployments (that are not fresh 4.2-RC3 but upgraded from 4.1), it fails in weird ways.

dom0 related configuration are working 50% of the time

non-dom0 configuration aren't working, qubesctl report that no top file are matching their name, while it works on other computers. Although, adding saltenv=user make it work!

Steps to reproduce

Enable user-dirs with qubesctl state.sls qubes.user-dirs
Modify /srv/user_salt/top.sls to look like

user:
  'dom0':
    - qubes-vm

Create /srv/user_salt/qubes-vm.sls with this content:

create-qubes-www:
  qvm.vm:
    - name: WWW
    - present:
      - template: fedora-38
      - label: orange
    - prefs:
      - label: orange

Run qubesctl state.highstate

First run fails

[root@dom0 srv]# qubesctl --targets dom0 --show-output state.apply
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
[ERROR   ] State 'qvm.vm' was not found in SLS 'qubes-vm'
Reason: 'qvm.vm' is not available.

local:
----------
          ID: create-qubes-www
    Function: qvm.vm
        Name: WWW
      Result: False
     Comment: State 'qvm.vm' was not found in SLS 'qubes-vm'
              Reason: 'qvm.vm' is not available.
     Changes:   

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
DOM0 configuration failed, not continuing

2nd run works

[root@dom0 srv]# qubesctl --targets dom0 --show-output state.apply
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
local:
----------
          ID: topd-always-passes
    Function: test.succeed_without_changes
        Name: foo
      Result: True
     Comment: Success!
     Started: 10:23:00.000115
    Duration: 0.588 ms
     Changes:   
----------
          ID: create-qubes-www
    Function: qvm.vm
        Name: WWW
      Result: True
     Comment: ====== ['present'] ======
              [SKIP] A VM with the name 'WWW' already exists.
              
              ====== ['prefs'] ======
              [SKIP] label              : orange
     Started: 10:23:00.031787
    Duration: 413.524 ms
     Changes:   

Summary for local
------------
Succeeded: 2
Failed:    0
------------
Total states run:     2
Total run time: 414.112 ms

3rd run doesn't work

[root@dom0 srv]# qubesctl --targets dom0 --show-output state.apply
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
[ERROR   ] State 'qvm.vm' was not found in SLS 'qubes-vm'
Reason: 'qvm.vm' is not available.

local:
----------
          ID: create-qubes-www
    Function: qvm.vm
        Name: WWW
      Result: False
     Comment: State 'qvm.vm' was not found in SLS 'qubes-vm'
              Reason: 'qvm.vm' is not available.
     Changes:   

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
DOM0 configuration failed, not continuing

4th run works

[root@dom0 srv]# qubesctl --targets dom0 --show-output state.apply
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
local:
----------
          ID: topd-always-passes
    Function: test.succeed_without_changes
        Name: foo
      Result: True
     Comment: Success!
     Started: 10:23:15.758674
    Duration: 0.636 ms
     Changes:   
----------
          ID: create-qubes-www
    Function: qvm.vm
        Name: WWW
      Result: True
     Comment: ====== ['present'] ======
              [SKIP] A VM with the name 'WWW' already exists.
              
              ====== ['prefs'] ======
              [SKIP] label              : orange
     Started: 10:23:15.788875
    Duration: 328.364 ms
     Changes:   

Summary for local
------------
Succeeded: 2
Failed:    0
------------
Total states run:     2
Total run time: 329.000 ms

Reproducer for template

user_salt for templates is completely broken:

Enable user-dirs with qubesctl state.sls qubes.user-dirs
Modify /srv/user_salt/top.sls to look like

user:
  'fedora-38':
    - package

Create /srv/user_salt/package.sls with this content:

jq:
  pkg.installed

Run qubesctl --targets fedora-38 --skip-dom0 --show-output state.highstate

[root@dom0 srv]# qubesctl --targets fedora-38 --skip-dom0  --show-output state.highstate
fedora-38:
      An Exception occurred while executing state.highstate: Unable to render top file. No targets found.
  [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
  [ERROR   ] An Exception occurred while executing state.highstate: Unable to render top file. No targets found.
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.

It's possible to fix the command by adding saltenv=user:

Run qubesctl --targets fedora-38 --skip-dom0 --show-output state.highstate saltenv=user

[root@dom0 srv]# qubesctl --targets fedora-38 --skip-dom0  --show-output state.highstate saltenv=user
fedora-38:
  ----------
            ID: jq
      Function: pkg.installed
        Result: True
       Comment: All specified packages are already installed
       Started: 10:36:50.037392
      Duration: 3369.414 ms
       Changes:   
  
  Summary for fedora-38
  ------------
  Succeeded: 1
  Failed:    0
  ------------
  Total states run:     1
  Total run time:   3.369 s
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.

Expected behavior

It works

Actual behavior

mix of half working for dom0 and not working for non-dom0

The text was updated successfully, but these errors were encountered:

rapenne-s · 2023-09-05T12:57:18Z

Hold on, it may be a mistake of mine, I'm trying to figure what's going on here 🙏

edit: it often fails with the error above, but sometimes it just works and create the qube 🤷‍♀️

rapenne-s · 2023-09-05T14:35:29Z

Ok! I've been able to make a reproducer for dom0 and fedora 38 template.

I compared my /srv and /etc/salt files with a working Qubes OS running 4.2-RC3, they are 100% identical, yet it works on the other computers.

The line [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable seems related to the issue.

DemiMarie · 2023-09-05T15:14:48Z

@rapenne-s does sudo qubesctl saltutil.sync_all help? It did for me. I’m not sure if that should be necessary, though.

rapenne-s · 2023-09-05T16:17:28Z

@rapenne-s does sudo qubesctl saltutil.sync_all help? It did for me. I’m not sure if that should be necessary, though.

this didn't change anything

DemiMarie · 2023-09-05T18:28:18Z

Does --target=fedora-38 (instead of --targets) help?

rapenne-s · 2023-09-05T20:01:28Z

Does --target=fedora-38 (instead of --targets) help?

no, while qubesctl man pages states --targets foo,bar,stuff is the correct syntax, for some reason --target=foo or --target foo also work, but they all produce the same result in that case

rapenne-s · 2023-09-15T22:07:45Z

@andrewdavidwong isn't this a blocker?

andrewdavidwong · 2023-09-16T12:54:26Z

Sorry, I'm not very familiar with Salt. Can you explain in simple terms why it's a blocker? For example, which use cases does it break that will affect most users?

gonzalo-bulnes · 2023-09-16T13:15:36Z

@andrewdavidwong When using Salt to automate configuration, you've got the choice of placing your personal "files" in one of two environments: base or user. Both exist in the default setup of Salt in Qubes OS.

base is where the Salt "files" created by the Qubes OS team live, you may put your own there, but there is a risk that a system update will overwrite them
the user environment is guaranteed not to be overwritten by system updates

If you're like me, you do what @rapenne-s did: enable the so-called "user directories" and write your stuff in the user environment.

Is that most users? (I don't know.) Is that most users who use Salt? (I don't know either.) What I know is that a few of us would see our workflows reliability go down significantly* by not being able to prevent conflicts between our personal automation efforts and the base ones. Does this make sense?

* At some point in the history of Qubes OS, this was determined to be significant enough for the user environment to be set up in the first place, and qubes.user-dirs to be written for the purpose of making enabling those user directories a very straightforward operation.

andrewdavidwong · 2023-09-16T13:25:55Z

@gonzalo-bulnes: Thanks for the explanation! I suppose if the user environment is supposed to be "guaranteed" to work in the manner you describe (and currently doesn't), then that would be a pretty strong reason for considering this a blocker, but it's @marmarek's call either way.

rapenne-s · 2023-09-16T14:28:43Z

thanks @gonzalo-bulnes for the explanation! I'd add that a lot of community work is provided through Salt (like sys-vpn), which is broken in this case.

rapenne-s · 2023-09-25T09:23:26Z

if this helps, when looking at pillar items on both my systems, I found a huge difference:

4.1 -> 4.2-RC3 system that work

# qubesctl --target TheQube --skip-dom0 --show-output pillar.items
TheQube:
      ----------
      qubes:
          ----------
          netvm:
              sys-firewall
          template:
              fedora-38
          type:
              app
      salt-modules:
          ----------
          topd:
              ----------
              topd_base_pillar:
                  /srv/pillar
              topd_base_state:
                  /srv/salt
              topd_dir_name:
                  _tops

The fresh 4.2-RC3 which have issues

# qubesctl --target TheQube --skip-dom0 --show-output pillar.items
TheQube:
      ----------
      qubes:
          ----------
          netvm:
              sys-firewall
          template:
              fedora-38
          type:
              app

rapenne-s · 2023-09-25T10:00:25Z

Investigating further, it seems qubesctl saltutil.sync_all isn't idempotent on the fresh 4.2-RC3, while the upgraded one doesn't show this random behavior.

When running the qubesctl saltutil.sync_all and then qubesctl --target TheQube --skip-dom0 --show-output pillar.items

sometimes it returns only the qubes structure
sometimes only the salt-modules structure
sometimes both

rapenne-s · 2023-09-25T10:05:11Z

I found a fix for the issue, but is it really that?

When comparing my two salt directories, the system with the issue was missing a link in /src/_tops/base/qubes.user-dirs.top to /srv/salt/qubes/user-dirs.top, despite running qubesctl state.sls qubes.user-dirs (this command is in in /srv/salt/qubes/user-dirs.sls)

Making the link allows me to deploy using my user defined salt states, removing the link make the error appearing again.

DemiMarie · 2023-09-25T15:02:23Z

That could well be it @rapenne-s!

feskehau · 2023-10-02T23:03:15Z

Had the same issue on 4.2-rc3

ln -s /srv/salt/qubes/user-dirs.top /srv/salt/_tops/base/user-dirs.top

Solved it for me. Thanks:)

edit, corrected path...

marmarek · 2023-12-13T12:55:18Z

I just checked on a fresh 4.1 installation, and it is this way there too, so it isn't regression.

The simplest way to create those dirs is to run qubesctl state.apply qubes.user-dirs. That probably needs to be documented somewhere (on https://www.qubes-os.org/doc/salt/ ?). But it definitely doesn't qualify as release blocker.

qubesos-bot · 2024-02-01T22:03:04Z

Automated announcement from builder-github

The component mgmt-salt (including package mgmt-salt) has been pushed to the r4.2 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

qubesos-bot · 2024-02-01T22:16:19Z

Automated announcement from builder-github

The component mgmt-salt (including package mgmt-salt) has been pushed to the r4.2 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

qubesos-bot · 2024-02-13T01:04:46Z

Automated announcement from builder-github

The package mgmt-salt has been pushed to the r4.2 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot · 2024-02-13T01:05:05Z

Automated announcement from builder-github

The package mgmt-salt has been pushed to the r4.2 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot · 2024-02-13T01:27:33Z

Automated announcement from builder-github

The component mgmt-salt (including package mgmt-salt) has been pushed to the r4.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

qubesos-bot · 2024-02-13T01:27:58Z

Automated announcement from builder-github

The component mgmt-salt (including package mgmt-salt) has been pushed to the r4.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

qubesos-bot · 2024-02-13T01:28:19Z

Automated announcement from builder-github

The component mgmt-salt (including package mgmt-salt) has been pushed to the r4.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

rapenne-s · 2024-04-12T07:18:35Z

I leave this in case it's useful to someone one day, I had the error [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable on a working 4.2 (I certainly did something wrong), preventing dom0 to be updated from the GUI updater.

The recommend command qubesctl saltutil.sync_all solved it.

rapenne-s added P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists. labels Sep 5, 2023

rapenne-s changed the title ~~4.2-RC3: salt error with when using qvm module~~ 4.2-RC3: user_salt isn't working Sep 5, 2023

andrewdavidwong added C: mgmt needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. affects-4.2 This issue affects Qubes OS 4.2. labels Sep 5, 2023

DemiMarie added P: blocker Priority: blocker. Prevents release or would have prevented release if known prior to release. and removed P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. labels Sep 15, 2023

andrewdavidwong added diagnosed Technical diagnosis has been performed (see issue comments). and removed needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. labels Oct 4, 2023

marmarek added P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. C: doc and removed P: blocker Priority: blocker. Prevents release or would have prevented release if known prior to release. labels Dec 13, 2023

qubesos-bot added the r4.2-vm-fc37-cur-test label Feb 1, 2024

qubesos-bot added the r4.2-vm-fc38-cur-test label Feb 1, 2024

qubesos-bot added the r4.2-vm-fc39-cur-test label Feb 1, 2024

qubesos-bot added r4.2-vm-bookworm-stable and removed r4.2-vm-bookworm-cur-test labels Feb 13, 2024

qubesos-bot added r4.2-vm-trixie-stable r4.2-host-stable and removed r4.2-vm-trixie-cur-test r4.2-host-cur-test labels Feb 13, 2024

qubesos-bot added r4.2-vm-fc37-stable and removed r4.2-vm-fc37-cur-test labels Feb 13, 2024

qubesos-bot added r4.2-vm-fc38-stable and removed r4.2-vm-fc38-cur-test labels Feb 13, 2024

qubesos-bot added r4.2-vm-fc39-stable and removed r4.2-vm-fc39-cur-test labels Feb 13, 2024

qubesos-bot mentioned this issue Mar 15, 2024

core-admin v4.2.21 (r4.2) QubesOS/updates-status#4458

Closed

gonzalo-bulnes mentioned this issue Apr 12, 2024

Testing of qvm-tags-in-pillar under R4.2 is needed! gonzalo-bulnes/qubes-mgmt-salt-user#27

Closed

This was referenced Apr 17, 2024

core-admin v4.2.21 (r4.3) QubesOS/updates-status#4513

Closed

mgmt-salt v4.2.2 (r4.3) QubesOS/updates-status#4545

Open

rocodes mentioned this issue May 31, 2024

[Refactor] Move all Salt provisioning files into /srv/salt/securedrop_salt freedomofpress/securedrop-workstation#1048

Merged

7 tasks

qubesos-bot mentioned this issue Aug 19, 2024

mgmt-salt v4.2.2 (r4.2) QubesOS/updates-status#5027

Closed

rapenne-s mentioned this issue Sep 26, 2024

top file in user_salt isn't working with explicit saltenv=user in dom0 #9468

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

4.2: user_salt isn't working #8491

4.2: user_salt isn't working #8491

rapenne-s commented Sep 5, 2023 •

edited

Loading

rapenne-s commented Sep 5, 2023 •

edited

Loading

rapenne-s commented Sep 5, 2023

DemiMarie commented Sep 5, 2023

rapenne-s commented Sep 5, 2023

DemiMarie commented Sep 5, 2023

rapenne-s commented Sep 5, 2023

rapenne-s commented Sep 15, 2023

andrewdavidwong commented Sep 16, 2023

gonzalo-bulnes commented Sep 16, 2023

andrewdavidwong commented Sep 16, 2023

rapenne-s commented Sep 16, 2023

rapenne-s commented Sep 25, 2023

rapenne-s commented Sep 25, 2023

rapenne-s commented Sep 25, 2023

DemiMarie commented Sep 25, 2023

feskehau commented Oct 2, 2023 •

edited

Loading

marmarek commented Dec 13, 2023

qubesos-bot commented Feb 1, 2024

qubesos-bot commented Feb 1, 2024

qubesos-bot commented Feb 13, 2024

qubesos-bot commented Feb 13, 2024

qubesos-bot commented Feb 13, 2024

qubesos-bot commented Feb 13, 2024

qubesos-bot commented Feb 13, 2024

rapenne-s commented Apr 12, 2024

4.2: user_salt isn't working #8491

4.2: user_salt isn't working #8491

Comments

rapenne-s commented Sep 5, 2023 • edited Loading

Qubes OS release

Brief summary

Steps to reproduce

Reproducer for template

Expected behavior

Actual behavior

rapenne-s commented Sep 5, 2023 • edited Loading

rapenne-s commented Sep 5, 2023

DemiMarie commented Sep 5, 2023

rapenne-s commented Sep 5, 2023

DemiMarie commented Sep 5, 2023

rapenne-s commented Sep 5, 2023

rapenne-s commented Sep 15, 2023

andrewdavidwong commented Sep 16, 2023

gonzalo-bulnes commented Sep 16, 2023

andrewdavidwong commented Sep 16, 2023

rapenne-s commented Sep 16, 2023

rapenne-s commented Sep 25, 2023

rapenne-s commented Sep 25, 2023

rapenne-s commented Sep 25, 2023

DemiMarie commented Sep 25, 2023

feskehau commented Oct 2, 2023 • edited Loading

marmarek commented Dec 13, 2023

qubesos-bot commented Feb 1, 2024

qubesos-bot commented Feb 1, 2024

qubesos-bot commented Feb 13, 2024

qubesos-bot commented Feb 13, 2024

qubesos-bot commented Feb 13, 2024

qubesos-bot commented Feb 13, 2024

qubesos-bot commented Feb 13, 2024

rapenne-s commented Apr 12, 2024

rapenne-s commented Sep 5, 2023 •

edited

Loading

rapenne-s commented Sep 5, 2023 •

edited

Loading

feskehau commented Oct 2, 2023 •

edited

Loading