2019.2.0 some function throws in test=True mode exception "Passed invalid arguments: sequence item 0: expected string or Unicode, list found."

[Reiner030]

As from Github found already some years ago thrown in #31591

All I get it this not very useful thrown error:

[...]
[DEBUG   ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [salt-minion] at time 16:13:45.596428 (duration_in_ms=28.239)
[INFO    ] Running state [salt-minion] at time 16:13:45.599755
[INFO    ] Executing state service.running for [salt-minion]
[INFO    ] Executing command [u'systemctl', 'is-active', u'salt-minion.service'] in directory '/root'
[DEBUG   ] stdout: active
[INFO    ] Executing command [u'systemctl', 'is-enabled', u'salt-minion.service'] in directory '/root'
[DEBUG   ] stdout: enabled
[INFO    ] The service salt-minion is already running
[INFO    ] Completed state [salt-minion] at time 16:13:45.628544 (duration_in_ms=28.789)
[INFO    ] Running state [salt-cloud] at time 16:13:45.629082
[INFO    ] Executing state pkg.installed for [salt-cloud]
[DEBUG   ] Could not LazyLoad pkg.resolve_capabilities: 'pkg.resolve_capabilities' is not available.
[DEBUG   ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [salt-cloud] at time 16:13:45.669880 (duration_in_ms=40.797)
[INFO    ] Running state [awscli] at time 16:13:45.670374
[INFO    ] Executing state pip.installed for [awscli]
[DEBUG   ] Could not LazyLoad pip.mod_run_check: 'pip.mod_run_check' is not available.
[INFO    ] Executing command 'type aws' in directory '/root'
[DEBUG   ] stdout: aws is /usr/local/bin/aws
[DEBUG   ] Last command return code: 0
[INFO    ] [u'unless condition is true']
[INFO    ] Completed state [awscli] at time 16:13:45.694873 (duration_in_ms=24.498)

Passed invalid arguments: sequence item 0: expected string or Unicode, list found.

Usage:

    .. versionadded:: 2015.5.0

    This function will call :mod:`state.highstate
    <salt.modules.state.highstate>` or :mod:`state.sls
    <salt.modules.state.sls>` based on the arguments passed to this function.
    It exists as a more intuitive way of applying states.

[...]

        .. versionadded:: 2017.7.8,2018.3.3,2019.2.0

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 237, in call
    ret['return'] = self.minion.executors[fname](self.opts, data, func, args, kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 12, in execute
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 779, in apply_
    return highstate(**kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 1083, in highstate
    orchestration_jid=orchestration_jid)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 3966, in call_highstate
    return self.state.call_high(high, orchestration_jid)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2789, in call_high
    ret = self.call_chunks(chunks)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2151, in call_chunks
    running = self.call_chunk(low, running, chunks)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2596, in call_chunk
    running[tag] = self.call(low, chunks, running)
  File "/usr/lib/python2.7/dist-packages/salt/utils/decorators/state.py", line 30, in _func
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2036, in call
    low['retry']['splay'])])
TypeError: sequence item 0: expected string or Unicode, list found
[DEBUG   ] Closing AsyncZeroMQReqChannel instance

When running saltmaster states productive there were no errors:

[...]
[DEBUG   ] Could not LazyLoad pkg.resolve_capabilities: 'pkg.resolve_capabilities' is not available.
[DEBUG   ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [salt-minion] at time 16:32:00.645059 (duration_in_ms=41.759)
[INFO    ] Running state [salt-minion] at time 16:32:00.651175
[INFO    ] Executing state service.running for [salt-minion]
[INFO    ] Executing command [u'systemctl', 'status', u'salt-minion.service', u'-n', u'0'] in directory '/root'
[DEBUG   ] stdout: * salt-minion.service - The Salt Minion
   Loaded: loaded (/lib/systemd/system/salt-minion.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-05-02 16:02:24 UTC; 29min ago
     Docs: man:salt-minion(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltstack.com/en/latest/contents.html
 Main PID: 14018 (salt-minion)
   CGroup: /system.slice/salt-minion.service
           |-14018 /usr/bin/python2 /usr/bin/salt-minion
           |-14221 /usr/bin/python2 /usr/bin/salt-minion
           `-14289 /usr/bin/python2 /usr/bin/salt-minion
[INFO    ] Executing command [u'systemctl', 'is-active', u'salt-minion.service'] in directory '/root'
[DEBUG   ] stdout: active
[INFO    ] Executing command [u'systemctl', 'is-enabled', u'salt-minion.service'] in directory '/root'
[DEBUG   ] stdout: enabled
[INFO    ] The service salt-minion is already running
[INFO    ] Completed state [salt-minion] at time 16:32:00.695356 (duration_in_ms=44.182)
[INFO    ] Running state [salt-cloud] at time 16:32:00.695770
[INFO    ] Executing state pkg.installed for [salt-cloud]
[DEBUG   ] Could not LazyLoad pkg.resolve_capabilities: 'pkg.resolve_capabilities' is not available.
[DEBUG   ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [salt-cloud] at time 16:32:00.737360 (duration_in_ms=41.59)
[INFO    ] Running state [awscli] at time 16:32:00.737721
[INFO    ] Executing state pip.installed for [awscli]
[DEBUG   ] Could not LazyLoad pip.mod_run_check: 'pip.mod_run_check' is not available.
[INFO    ] Executing command 'type aws' in directory '/root'
[DEBUG   ] stdout: aws is /usr/local/bin/aws
[DEBUG   ] Last command return code: 0
[INFO    ] [u'unless condition is true']
[INFO    ] Completed state [awscli] at time 16:32:00.760679 (duration_in_ms=22.957)
[INFO    ] Running state [easy-rsa] at time 16:32:00.761167
[INFO    ] Executing state pkg.installed for [easy-rsa]
[DEBUG   ] Could not LazyLoad pkg.resolve_capabilities: 'pkg.resolve_capabilities' is not available.
[DEBUG   ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [easy-rsa] at time 16:32:00.802852 (duration_in_ms=41.685)
[INFO    ] Running state [pwgen] at time 16:32:00.803197
[INFO    ] Executing state pkg.installed for [pwgen]
[DEBUG   ] Could not LazyLoad pkg.resolve_capabilities: 'pkg.resolve_capabilities' is not available.
[DEBUG   ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [pwgen] at time 16:32:00.843328 (duration_in_ms=40.13)
[...]

Since the states should be run in same order I think this is caused somehow by state
easy-rsa:

[...]
salt-cloud-package:
  pkg.installed:
    - name: salt-cloud
    - retry:
        attempts: 20    # default: 2
        until: True     # default: True
        interval: 10    # default: 30
        splay: 10       # default: 0

awscli-pip:
  pip.installed:
    - name: awscli
    - retry:
        attempts: 20    # default: 2
        until: True     # default: True
        interval: 10    # default: 30
        splay: 10       # default: 0
    - unless: type aws

easy-rsa-package:
  pkg.installed:
    - name: easy-rsa
    - retry:
        attempts: 20    # default: 2
        until: True     # default: True
        interval: 10    # default: 30
        splay: 10       # default: 0

pwgen-package:
  pkg.installed:
    - name: pwgen
    - retry:
        attempts: 20    # default: 2
        until: True     # default: True
        interval: 10    # default: 30
        splay: 10       # default: 0
[...]

but the file is fine and this setup works since 1-2 years till version 2018.3.4 fine:

/srv/salt # file base/saltmaster/init.sls
base/saltmaster/init.sls: ASCII text

[waynew]

Seems unlikely but I wonder if it has anything to do with https://docs.saltstack.com/en/latest/topics/releases/2019.2.0.html#non-backward-compatible-change-to-yaml-renderer

[waynew]

If you remove the trailing comments in your - retry: does it work?

[Reiner030]

Hi, nice idea but

1. I have many comments in my states and all where fine processed
2. The state is running fine productive; only in test=True mode it fails
3. Here the final check - I remove in this sls all " * # default: .*" items:

   # salt-call state.apply saltmaster test=True
   
   Passed invalid arguments: sequence item 0: expected string or Unicode, list found.
   ....
     File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2036, in call
       low['retry']['splay'])])
   

4. I removed all the retry blocks completely
   => State is run correct in test=True
   ==> interesting question: why they fail only in this sls?
   On non-saltmaster instances which same retry options I have no problems.
   And the block was copy&pasted by my some months ago for all occurences ^^

So it must be somehow a failure in above state ?

salt/salt/state.py

Lines 2090 to 2100 in ffa4305

	else:
	ret['comment'] = ' '.join(
	['' if not ret['comment'] else ret['comment'],
	('The state would be retried every {1} seconds '
	'(with a splay of up to {3} seconds) '
	'a maximum of {0} times or until a result of {2} '
	'is returned').format(low['retry']['attempts'],
	low['retry']['interval'],
	low['retry']['until'],
	low['retry']['splay'])])
	return ret

Btw. I guess that's the same problem I had also with calculated values which a devops/developer initial setup for our site to calculate ports and database names...
I had to cast to |int and .decode('ascii') all UTF-8 strings in this array else I got in all ASCII config files (PHP, nginx, haproxy, ...) "nice" port numbers like u'20001' and database names like u'backend-staging'.

Sadly I had'nt got a simple example yet working for writing an issue here but if interested I can reference a github repo of a demo VM of a newer / branched system similar to our setup with these buggy port calculations in 2019.2.0.

[Orlando-Richards]

I think I've just run into something similar - submitted as #54501

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale]

Thank you for updating this issue. It is no longer marked as stale.