--static flag causes longer run time

[keagan-thinkst]

Description of Issue

When running a salt command with the --static flag it causes the run time of a module function to increases by order of magnitude.
eg.
Example function targeting approximately 1000 minions:

time sudo salt --out=json -C * module.module_func

<removed_sensitive data>
.
.
real	0m12.666s
user	0m2.512s
sys	0m0.380s

vs when adding the `--static flag:

time sudo salt --static--out=json -C * module.module_func

<removed_sensitive data>
.
.
real	1m36.855s
user	0m3.166s
sys	0m0.405s

Forcing a low timeout does not help the static case either, I am trying to use the --static flag to get valid/parseable JSON as a whole from the output.

This issue has arisen recently without any changes to our salt deployment other than adding in a new reactor that listens for high states:

reactor:
  - 'salt/job/*/new':
    - salt://_reactor/console-highstate.sls

console-highstate.sls:

{% if data['fun'] == 'state.highstate' %}
changed_name_for_senistive_data:
  runner.runner_name.runner_fun:
    - args:
      - target: {{ data['minions'] | json }}
{% endif %}

The current work around is to not use the --static flag and perform string manipulation to make the whole output valid JSON. Would like to know why the static flag is causing this large increase in time though. None of the minions are failing from the call in either case where the flag is set or not.

Versions Report

           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 2.0.5
      gitpython: 2.1.14
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: 1.3.10
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15+ (default, Jul  9 2019, 16:51:35)
   python-gnupg: 0.4.1
         PyYAML: 3.13
          PyZMQ: 18.1.0
           RAET: Not Installed
          smmap: 2.0.5
        timelib: Not Installed
        Tornado: 5.1.1
            ZMQ: 4.3.2

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-1032-aws
         system: Linux
        version: Ubuntu 18.04 bionic```

[waynew]

Thanks for the report! Working to reproduce this now.

[waynew]

I wasn't able to run this with 1,000 minions, but even with 25 minions there's a measurable difference between running with and without --static.