Fetching require data during salt states run, after previous states executed.

[hackerwin7]

HI,
I am writing some salt states into SLS file and encounter a simple common case is that:
In a sls file, if state.2 need state.1 generated data.
such as:

download-pkg:
  cmd.run:
    - name: curl -s -o {{ pkg_pah }} {{ pkg_url }}

{%- set dir_name = salt['cmd.run'](' tar tf ' + pkg_path + ' | sed -e "s@/.*@@" | uniq') %}

other-states:
  need {{ dir_name }} value

other-states need {{ dir_name }} value to continue to execute.
After viewing the following links:
Jinja variables are set before the state will be executed
#12470

Is there any way to fit this case by using salt states or other flexible solutions for this.

[garethgreenaway]

@hackerwin7 I think what you might want is to use a combination of include and require inside the second state file and have it require the first state file sls. Something like this:

include:
  - state_1

other-states:
  something:
    - require:
      - sls: state_1

[hackerwin7]

@garethgreenaway Thanks your reply, the key problem is how can I get the run-time and dynamic result that state_1 produced ?
Such as following case:

state_1

#jinja-call-salt-modules-get-produced-result, the jinja must call after the state_1 executed
{% set data = salt[modules]() %}

other-states:
  something:
    - some-args: {{ data }}
    - other-args

The data is run-time and dynamic, we can not previously set the value in pillar or grains, the data is produced by state_1, so we must get the run-time value after the state_1 executed and before the other states run. however, the execution order above is Jinja evaluated before states executed.
I test you solution for include and require. any way, jinja still evaluated before state_1 run.
My puzzle is that how can I get the run-time result between the states execution and use it for subsequent execution of states?

[garethgreenaway]

@hackerwin7 ahh that makes more sense. Perhaps the orchastrate module will accomplish what you're trying to do?

[hackerwin7]

@garethgreenaway Thanks your advice very much!
After a simple glance of orchestration, I changed my states file, such as following:

state_1

salt.function

other-states:
  something:
    - some-args: {{ data }}
    - other-args

However, a few puzzles appears:

1. How can I get the result that salt.function produced ? such as {%- set data = salt['cmd.run'](some command)%} ,while jinja render before states run.
2. Orchestration generally applied in the case between the minions, while I just only execute these states in one single minion and one single sls file.
3. Is there any solution for one minion and single sls which implement the following case in the order literally:

state_1

# execute a cmd.run or file.exists or other execution modules and retrieve return data
result = execution_modules

# return data as subsequent states arg
state_2:
   args: {{ result }}

As you see, first I want to use jinja to implement result = execution_modules, but failed because of render order for jinja and states.

[hackerwin7]

After reviewing my question, my key problem is that get a run-time value produced by the states in a single minion and single sls file.
There is a instance for this:

# such as download file or other
download-tar-file:
  cmd.run:
     - name: curl -s -o /path/filename.tar.gz http:/file.url.xxx/filename.tar.gz

# get downloaded tar archive file items
# first, I want to use Jinja, but failed that jinja evaluated before states download-tar-file run
{#  {%- set count = salt['cmd.run']('tar tvf /path/filename.tar.gz | wc -l') %} #}

count is a run-time value, I can not previously set in pillar or grains, this value is produced by the states doanload-tar-file.
Why not single sls file support this combination of states and execution modules in the literally order ?

[hackerwin7]

@garethgreenaway Is there any flexible solution for this? I find this is very similar to me.
Maybe I Should use writing custom python modules (_modules/) for this case. even though I still want to do this by salt state.

[nuharaf]

I have similar problem to this, tough I don't know if there is possible workaround for my problem.
I need to extract an archive, and the set the environment variable based on the top level directory of the archive.

[andreasnuesslein]

same here. I need to create a user, create their ssh-keys, then read the pubkey and use it in a http.query state. is there a nice solution to this?

[dgengtek]

Found this issue when I was faced with the same problem. Creating a user with a formula and querying user data of the same newly created user in another formula during a highstate.

Since the query happens when the highstate gets rendered the highstate will obviously fail with no data for the new user - since the user did not exist at that time.

Getting dynamic results which depend on a previous salt states result in the renderer is impossible since everything gets rendered before the states gets executed. This has to be done when executing a state. Namely doing this with custom states. https://docs.saltstack.com/en/latest/ref/states/writing.html

The only way to do this is by wrapping the states with your custom state modules defined in _states with the required logic.

All you would need to do is for example

# _states/somefilename.py
__virtualname__ = 'mystate'

def __virtual__():
    return __virtualname__

def wrap_whatever(name, username, **kwargs):
    user_data = __salt__.user.info(username) # or some other module which queries some dynamic data which was created in a previous run or query some central datasource
    # do whatever you need to with the dynamically created user data for the required state
    kwargs["name"] = name
    return __states__["somestate.function"](**kwargs)

then call the state in the formula + sync custom states beforehand

my custom wrapped state id:
  mystate.wrap_whatever:
    - name: 'somethingsomething'
    - username: 'someuser'
    - arg_of_wrapped_state1: value1
    - arg_of_wrapped_state2: value2
   ...

This would allow one to create a new user and query its data in the same highstate and update kwargs with whatever is needed which one would pass to the original state which requires this data.

If one does not want to wrap their states then the only sane way to achieve this is by having some central datasource which gets updated via custom external pillars so that all states in a highstate work with the same required data.

[prometheanfire]

I'm wondering if this is the same issue I'm seeing

FOO:
  location:
{% if grains['id'].endswith('.fmt1.foo.com') %}
    site: fmt1
{% else %}
    site: unknown
{% endif %}

nameservers:
{% if salt['pillar.get']('FOO:location:site', 'unknown') == 'fmt1' %}
{% if salt['network.in_subnet']('1.2.12.0/23') %}
  - '1.2.12.16'
{% endif %}
  - '1.2.13.16'
  - '1.2.14.16'
{% else %}
  # WARNING we don't know which sites name servers to use.
  - '8.8.8.8'
  - '8.8.4.4'
{% endif %}

and querying

# salt 'bar.fmt1.foo.com' pillar.get nameservers
bar.fmt1.foo.com:
    - 1.2.8.16
    - 8.8.8.8
    - 8.8.4.4
# salt 'bar.fmt1.foo.com' pillar.get FOO:location:site
bar.fmt1.foo.com:
fmt1

[nnsense]

Same here: a custom exec module is fetching a file from internet, and it returns its name and location. I should use the returned data to install such package during the next state.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[prometheanfire]

reopen, still seems like an issue

on another note, not sure how ansible solves it, but I know you can use output of previous ansible commands in subsequent commands.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[andreasnuesslein]

reopen, still seems like an issue

on another note, not sure how ansible solves it, but I know you can use output of previous ansible commands in subsequent commands.

Yeah ansible is designed different that way. Afaik Salt lacks such a feature and it would be really helpful to have it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[prometheanfire]

been another month, please reopen...

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[prometheanfire]

been another month, please reopen...

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[andreasnuesslein]

don't bother stalebot, this is still an issue.

[sooraj2589]

any update on this? facing the same issue

[Elufimov]

I think this sep describe this issue problem saltstack/salt-enhancement-proposals#33

[anitakrueger]

Just wanted to post a workaround that we are using in our production system which has worked fairly well.
When we need to run a 2nd state that requires the output of the 1st one, we send an event to the event bus from the first state like this:

event_send_{{ var }}_apply_followup_state:
  event.send:
    - name: service/apply_followup_state
    - data:
        state: my.custom.state
        customvariable: {{ var }}
    - onchanges:
      - file: file_managed_{{ var }}

The salt master has this configured in the reactor states:

reactor:
  - 'service/apply_followup_state':
    - salt://reactor/apply_followup_state.sls

The reactor file itself looks like this and injects the custom variables into pillar for the next state to use:

{%- set server = data.get('id', '') %}
apply-follow-up-state-{{ server }}:
  local.state.apply:
    - tgt: {{ server }}
    - args:
      - mods: {{ data.data.get('state' ) }}
      - queue: True
      - test: {{ data.data.get('test', False) }}
      - pillar:
{%- for p,v in data.data.items() %}
          {{ p }}: {{ v | json }}
{%- endfor %}

Whichever state was defined in the first event message, will then be executed on the same minion. That state can then run an execution module before rendering and gather the info from the previous state run.

[matthewsht]

I'm also facing this problem - in my case, globbing to pkg.removed. (i'm trying to cleanup no longer needed rpms).

What I want is a a salt state that gets all the packages with pkg.list_pkgs, grep out a pattern (glob would also be acceptable here), and then pkg.removed that list.

[Darkentik]

Hey, i just want to let my experience here and hope that it helps some of you.
Story: I am develop a saltstack formula for setting up a complex icinga2 env with icinga2-master, icinga2-agents and so on.

You have different ways how to accomplish your goal.
Let us start:

1. runtime calling cmd.run and save output to jinja var**
   Example description:
   Check on Windows salt-minion if the "IcingaForWindows-PowershellFramework" already installed the Icinga2 software.
   We use Get-WmiObject -Class Win32_Product with some filtering and want the output of the package name.
   In our case it is "Icinga 2".

my .sls:

...snip...
{%- set ifw_install_state = salt['cmd.run']('Get-WmiObject -Class Win32_Product | where vendor -eq \"Icinga GmbH\" | select Name -ExpandProperty Name',shell='powershell') %}

icinga2-legacy_debug_01:
  cmd.run:
    - name: 'echo "here the content of grain icinga2-legacy:ifw_install_state : {{ ifw_install_state }}" '
...snip...

Additionally important here in this powershell case is to escape the double quotes in the command that should be executed and the option "shell" to tell cmd.run to use powershell.

Here is the output:

----------
          ID: icinga2-legacy_manage_windows_installer_script
    Function: file.managed
        Name: C:\Program Files\Salt Project\Salt\IcingaForWindows.ps1
      Result: True
     Comment: File C:\Program Files\Salt Project\Salt\IcingaForWindows.ps1 is in the correct state
     Started: 09:47:24.111623
    Duration: 187.5 ms
     Changes:   
----------
          ID: icinga2-legacy_manage_windows_installer_answer_file
    Function: file.managed
        Name: C:\ProgramData\Salt Project\IfW_answer.json
      Result: True
     Comment: File C:\ProgramData\Salt Project\IfW_answer.json is in the correct state
     Started: 09:47:24.299123
    Duration: 62.501 ms
     Changes:   
----------
          ID: icinga2-legacy_debug_01
    Function: cmd.run
        Name: echo "here the content of grain icinga2-legacy:ifw_install_state : Icinga 2" 
      Result: True
     Comment: Command "echo "here the content of grain icinga2-legacy:ifw_install_state : Icinga 2" " run
     Started: 09:47:24.439749
    Duration: 31.256 ms
     Changes:   
              ----------
              pid:
                  760
              retcode:
                  0
              stderr:
              stdout:
                  "here the content of grain icinga2-legacy:ifw_install_state : Icinga 2"

Summary for auto01-a.test.hal01.de.mycompany.local
------------
Succeeded: 3 (changed=1)
Failed:    0
------------
Total states run:     3
Total run time: 281.257 ms

Our debug state shows us that it is working.
Now i can go further and can use the jinja var in jinja if cases ot whereever i want to use it.

I post an other special solution for mysql_query.run in a second post. :)

[Darkentik]

Hey, i just want to let my experience here and hope that it helps some of you.
Story: I am develop a saltstack formula for setting up a complex icinga2 env with icinga2-master, icinga2-agents and so on.

As mentioned above here now a other special way for mysql_query.run.

Story of .sls:
I want only to import once the mysql schema from icinga2 to my mariadb.

Description of this code:
You can create a custom grain with the content of the output of a mysql query.

Let's go.

my .sls:

Check_icinga2_schema_exists:
  mysql_query.run:
    - database: {{ datamap.dbname }}
    - query: "SHOW tables;"
    - output: grain
    - grain: icinga2
    - key: mysql_check_tables

Check download folder for Icinga2 mysql schema is present:
  file.directory:
    - name: /srv/icinga2
    - user: root
    - group: root

{%-   set schema = salt['pillar.get']('icinga2:install_files:mysql_schema') %}

Download icinga2 MySQL Schema file from aptcache Server:
  cmd.run:
    - name: wget -O /srv/icinga2/icinga2.schema.sql {{ schema.srcpath }}{{ schema.filename }}
    - require:
      - file: Check download folder for Icinga2 mysql schema is present

{%-   set checkreturn = salt['grains.get']('icinga2:mysql_check_tables:rows returned') %}

{%-   if checkreturn is defined and checkreturn == 0 %}

Import DB schema for Icinga2:
  cmd.run:
    - name: mysql -u root -D icinga < /srv/icinga2/icinga2.schema.sql
    - onfail:
      - mysql_query: Check_icinga2_schema_exists
    - require:
      - cmd: "Download icinga2 MySQL Schema file from aptcache Server"

{%-   endif %}

Sadly i don't have an output to this highstate. But it worked for me at year 2020/21.
I think this is still working.