Minion did not return. [No response] - system.reboot

[ronak2303]

Description of Issue

i am trying to reboot linux host during orchestration ( deployment process).
i have tried multiple ways, as soon as it hit reboot.sls i got error message.

Setup

reboot.sls

reboot_minions:
  salt.function:
    - name: system.reboot
    - at_time: 1
    - tgt: {{pillar['match']|join(',')}}
    - tgt_type: list

wait_for_reboots:
  salt.wait_for_event:
    - name: salt/minion/*/start
    - id_list: {{pillar['match']}}
    - require:
      - salt: reboot_minions

also tried with code below and run via orchestration.
reboot.sls

final_housekeeping:
   module.run:
      - name: system.reboot
      - wait_for_reboot: True

init.sls

run minion restart 1:
  salt.state:
    - tgt: {{ target_minion }}
    - sls:
      - {{ slspath }}.reboot

wait for minion to restart 1:
  salt.wait_for_event:
    - name: salt/minion/*/start
    - id_list:
      - {{ target_minion }}
    - require:
      - salt: run minion restart 1

Steps to Reproduce Issue

[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: wh-140a786872.svr.us.jpmchase.net
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] LazyLoaded state.orchestrate
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: wh-140a786872.svr.us.jpmchase.net
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/run/20201230204901837092/new; data = {'fun': 'runner.state.orchestrate', 'jid': '20201230204901837092', 'user': 'root', 'fun_args': ['sqlprovision', {'pillar': {'target': 'wh-1c385b6305', 'login': 'xxxx', 'pass': 'xxxxx', 'sapass': 'xxxxxx', 'inputfile': '/home/SQLInstall/linux_gss_inputfile.txt'}, 'orchestration_jid': '20201230204901837092'}], '_stamp': '2020-12-30T20:49:04.312062'}
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: wh-140a786872.svr.us.jpmchase.net
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded state.sls
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] LazyLoaded config.get
[DEBUG   ] key: test, ret: _|-
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[DEBUG   ] Gathering pillar data for state run
[DEBUG   ] Determining pillar cache
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] compile template: /srv/pillar/top.sls
[DEBUG   ] Jinja search path: ['/srv/pillar', '/srv/spm/pillar']
[PROFILE ] Time (in seconds) to render '/srv/pillar/top.sls' using 'jinja' renderer: 0.011786699295043945
[DEBUG   ] Rendered data from file: /srv/pillar/top.sls:
base:
  '*':
    - InputFilePath

[DEBUG   ] Results of YAML rendering:
OrderedDict([('base', OrderedDict([('*', ['InputFilePath'])]))])
[PROFILE ] Time (in seconds) to render '/srv/pillar/top.sls' using 'yaml' renderer: 0.00030517578125
[DEBUG   ] LazyLoaded confirm_top.confirm_top
[DEBUG   ] LazyLoaded compound_match.match
[DEBUG   ] compound_match: wh-140a786872.svr.us.jpmchase.net_master ? *
[DEBUG   ] LazyLoaded glob_match.match
[DEBUG   ] compound_match wh-140a786872.svr.us.jpmchase.net_master ? "*" => "True"
[DEBUG   ] compile template: /srv/pillar/InputFilePath.sls
[DEBUG   ] Jinja search path: ['/srv/pillar', '/srv/spm/pillar']
[PROFILE ] Time (in seconds) to render '/srv/pillar/InputFilePath.sls' using 'jinja' renderer: 0.0006053447723388672
[DEBUG   ] Rendered data from file: /srv/pillar/InputFilePath.sls:
# Set path to input file to be stored on Pillar

inputfile: "/home/SQLInstall/linux_gss_inputfile.txt"

[DEBUG   ] Results of YAML rendering:
OrderedDict([('inputfile', '/home/SQLInstall/linux_gss_inputfile.txt')])
[PROFILE ] Time (in seconds) to render '/srv/pillar/InputFilePath.sls' using 'yaml' renderer: 0.00016188621520996094
[DEBUG   ] Finished gathering pillar data for state run
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Could not find file 'salt://sqlprovision.sls' in saltenv 'base'
[DEBUG   ] In saltenv 'base', looking at rel_path 'sqlprovision/init.sls' to resolve 'salt://sqlprovision/init.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/master/files/base/sqlprovision/init.sls' to resolve 'salt://sqlprovision/init.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://sqlprovision/init.sls'
[DEBUG   ] No dest file found
[INFO    ] Fetching file from saltenv 'base', ** done ** 'sqlprovision/init.sls'
[DEBUG   ] compile template: /var/cache/salt/master/files/base/sqlprovision/init.sls
[DEBUG   ] Jinja search path: ['/var/cache/salt/master/files/base']
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] LazyLoaded pillar.get
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/files/base/sqlprovision/init.sls' using 'jinja' renderer: 0.024741411209106445
[DEBUG   ] Rendered data from file: /var/cache/salt/master/files/base/sqlprovision/init.sls:







run minion restart 1:
  salt.state:
    - tgt: wh-1c385b6305
    - sls:
      - sqlprovision.minion_reboot

wait for minion to restart 1:
  salt.wait_for_event:
    - name: salt/minion/*/start
    - id_list:
      - wh-1c385b6305
    - require:
      - salt: run minion restart 1

[DEBUG   ] Results of YAML rendering:
OrderedDict([('run minion restart 1', OrderedDict([('salt.state', [OrderedDict([('tgt', 'wh-1c385b6305')]), OrderedDict([('sls', ['sqlprovision.minion_reboot'])])])])), ('wait for minion to restart 1', OrderedDict([('salt.wait_for_event', [OrderedDict([('name', 'salt/minion/*/start')]), OrderedDict([('id_list', ['wh-1c385b6305'])]), OrderedDict([('require', [OrderedDict([('salt', 'run minion restart 1')])])])])]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/master/files/base/sqlprovision/init.sls' using 'yaml' renderer: 0.0004239082336425781
[DEBUG   ] LazyLoaded config.option
[DEBUG   ] LazyLoaded salt.state
[INFO    ] Running state [run minion restart 1] at time 20:49:04.922186
[INFO    ] Executing state salt.state for [run minion restart 1]
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/run/20201230204901837092/args; data = {'type': 'state', 'tgt': 'wh-1c385b6305', 'name': 'run minion restart 1', 'args': {'arg': ['sqlprovision.minion_reboot'], 'kwarg': {'queue': False, 'concurrent': False}, 'ret': '', 'timeout': None, 'tgt_type': 'glob', 'ssh': False, 'expect_minions': True}, '_stamp': '2020-12-30T20:49:04.924076'}
[DEBUG   ] Closing IPCMessageClient instance
[DEBUG   ] LazyLoaded saltutil.cmd
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: wh-140a786872.svr.us.jpmchase.net
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'wh-140a786872.svr.us.jpmchase.net_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/39/691a1a433f982acb9f45475ee8a99d599155dae2f2fe816e871f253edd9257/.minions.p
[DEBUG   ] get_iter_returns for jid 20201230204904974268 sent to {'wh-1c385b6305'} will timeout at 20:49:09.978757
[DEBUG   ] Checking whether jid 20201230204904974268 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'wh-140a786872.svr.us.jpmchase.net_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] retcode missing from client return
[DEBUG   ] Closing IPCMessageSubscriber instance
[WARNING ] Output from salt state not highstate
[ERROR   ] {'out': 'highstate', 'ret': {'wh-1c385b6305': False}}
[INFO    ] Completed state [run minion restart 1] at time 20:49:20.059995 (duration_in_ms=15137.808)
[DEBUG   ] File /var/cache/salt/master/accumulator/140440397671224 does not exist, no need to cleanup
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] LazyLoaded local_cache.prep_jid
[DEBUG   ] Sending event: tag = salt/run/20201230204901837092/ret; data = {'fun': 'runner.state.orchestrate', 'jid': '20201230204901837092', 'user': 'root', 'fun_args': ['sqlprovision', {'pillar': {'target': 'wh-1c385b6305', 'login': ' xxxxx', 'pass': 'xxxx', 'sapass': 'xxxx', 'inputfile': '/home/SQLInstall/linux_gss_inputfile.txt'}, 'orchestration_jid': '20201230204901837092'}], '_stamp': '2020-12-30T20:49:20.063392', 'return': {'data': {'wh-140a786872.svr.us.jpmchase.net_master': {'salt_|-run minion restart 1_|-run minion restart 1_|-state': {'name': 'run minion restart 1', 'changes': {'out': 'highstate', 'ret': {'wh-1c385b6305': False}}, 'comment': 'Run failed on minions: wh-1c385b6305', 'result': False, '__sls__': 'sqlprovision', '__run_num__': 0, 'start_time': '20:49:04.922187', 'duration': 15137.808, '__id__': 'run minion restart 1'}, 'salt_|-wait for minion to restart 1_|-salt/minion/*/start_|-wait_for_event': {'changes': {}, 'result': False, 'duration': 0.004, 'start_time': '20:49:20.060410', 'comment': 'One or more requisite failed: sqlprovision.run minion restart 1', '__run_num__': 1, '__sls__': 'sqlprovision'}}}, 'outputter': 'highstate', 'retcode': 1}, 'success': False}
[DEBUG   ] LazyLoaded highstate.output
wh-140a786872.svr.us.jpmchase.net_master:
----------
          ID: run minion restart 1
    Function: salt.state
      Result: False
     Comment: Run failed on minions: wh-1c385b6305
     Started: 20:49:04.922187
    Duration: 15137.808 ms
     Changes:
              wh-1c385b6305:
                  False
----------
          ID: wait for minion to restart 1
    Function: salt.wait_for_event
        Name: salt/minion/*/start
      Result: False
     Comment: One or more requisite failed: sqlprovision.run minion restart 1
     Started: 20:49:20.060410
    Duration: 0.004 ms
     Changes:

Summary for wh-140a786872.svr.us.jpmchase.net_master
------------
Succeeded: 0 (changed=1)
Failed:    2
------------
Total states run:     2
Total run time:  15.138 s

Versions Report

Salt Version:
           Salt: 2019.2.7

Dependency Versions:
           cffi: 1.9.1
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.11.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Aug 13 2020, 07:46:32)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.7 Maipo
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.18.1.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.7 Maipo

[robertpenberthy]

We've been using this to reboot our Hosts and has been working for us for a while now. It's slightly different than what you have, but maybe you can tweak yours to work. The only real difference I see is you have at_time: 1 defined in your system.reboot statement and I'm passing the 1 as a positional argument, with the obvious difference also being how we determine for which hosts to reboot and wait.

{% set deployment = salt['pillar.get']('deployment', none) %}
{% set tgt = 'G@deployment:{0}'.format(deployment)  %}
{% set hosts = salt['saltutil.runner']('mine.get', tgt=tgt, fun='ip', tgt_type='compound') %}

{{ sls }}-hosts-restart:
  salt.function:
    - name: system.reboot
    - arg: [1]
    - tgt: {{ tgt }}
    - tgt_type: compound

{{ sls }}-hosts-wait-for-reboot:
  salt.wait_for_event:
    - name: salt/minion/*/start
    - id_list: {{ hosts.keys()|list|yaml }}
    - timeout: 600
    - require:
      - salt: {{ sls }}-hosts-restart

[ronak2303]

hi Robert ,
i am not to much familiar with salt, but how do i run this.

i see {{ sls }}, what this refer to and how can i create sls file which i can only pass minion

[robertpenberthy]

@ronak2303

{{ sls }} is just the path to the current sls file. It's just something we use in our organization for naming state ids. Helps us know where to look when there are problems with state ids. Feel free to use something else in your code.

https://docs.saltstack.com/en/latest/ref/states/vars.html#sls-only-variables

[ronak2303]

hi Robert,
should i keep my reboot.sls to as below.
final_housekeeping:
module.run:
- name: system.reboot
- wait_for_reboot: True

and update my init.sls with your code.

[ronak2303]

hi Robert,
I corrected my mistake and got to know how should i run my sls file.
your solution worked for me. i just replaced at_time: 1 with arg: [1] in my sls file and it is working well.

[ronak2303]

hi Robert,
when i run my sls file with
salt-run state.orchestrate rebootserver.reboot1 pillar='{"match": ["wh-12fb0f9f67"]}'
it runs good,

when i try to put it in init.sls, it failes.

OR how can i use my reboot1.sls file in orchastration.

[robertpenberthy]

Trying to figure out what you're saying. Looking back up at your original post, it looks like you're calling your reboot1.sls file as a state on the host. So I think your question is how do you execute another orchestration inside an orchestration command? Calling salt.state as I've understood it is like calling state.sls on the host and is not an orchestration.

I cannot think of how to execute a different orchestration during an orchestration or if that is possible, it's not something that I believe we've tried. What I would do is just include the reboot sls file in your init.sls orchestration file instead of trying to execute it with salt.state.

Beyond that you just might need someone else to reply. I'm just a fellow community member that thought I could provide an answer to your issue. We rarely if ever put any states into our init.sls file, but rather just have it call other sls files to complete the state with maybe some jinja that may exclude certain executing some states depending on minion grains.

I pulled and simplified the reboot example from our orchestration state for applying YUM updates to our hosts. When rendered, our init.sls file is simply:

include:
  - yum.orch.stop
  - yum.orch.backup
  - yum.orch.update
  - yum.orch.glusterfs
  - yum.orch.mysql
  - yum.orch.others
  - yum.orch.start

The orchestration stops the services running on the hosts, makes any necessary precautionary backups, applies the updates, then restarts our glusterfs VMs, then the database VMs, and finally all other VMs in the deployment. Once the restarts are finished, it starts up the services again.

There are many ways to accomplish things in SaltStack and maybe executing the reboot1.sls file of yours can be done just by including it in the init.sls file, but if you're looking for something else, I'm not sure I can help further at this time.