[BUG] Minion key listing is too slow on NFS mounts

[bluesliverx]

Description
We run Salt in a MoM/syndic configuration with ~35k minions in a single environment (with many syndics so that nothing is overloaded). We put our minion keys on a NAS mount so that all masters/syndics share the same keys as recommended by Salt docs (as of a few years ago). However, listing keys using salt-key or even on startup was extremely slow, taking minutes to startup the salt master or do a simple salt-key -L. We profiled the processes and found out that most of the time was spent in the os.isfile check for minion keys. For a small number of keys, it works great, but in the thousands of keys on a slow storage system like a NAS mount, this was untenable for us.

Setup
There is no specific setup for this except to put minion keys on a NAS/slow storage system and then have 10k+ minion keys accepted.

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

If more details are needed about setup, just let me know, but this may be difficult to reproduce without a full NAS setup.

Steps to Reproduce the behavior
Start salt-master with a huge number of keys on slow storage, observe it not starting. OR run salt-key -L.

Expected behavior
It should startup and return from salt-key quickly, within 10 seconds probably, instead of minutes.

Screenshots
n/a

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 1.0.4
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.15.0
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 6.0
         PyZMQ: 21.0.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.3

System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.36.2.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

Additional context
We have a fix for this, and I will be submitting a patch shortly for it.