Lazily gather the lists of available sls data from the master

[sathieu]

What does this PR do?

This improves significantly the performance of a state.apply when using
GitFS and a lot of branches.

What issues does this PR fix or reference?

none

Previous Behavior

3 minutes with 276 branches.

New Behavior

25 seconds

Tests written?

No. Already covered by existing tests.

Commits signed with GPG?

No

[sathieu]

Note : I have tested a similar patch for pillars (in salt/pillar/__init__.py:462 Pillar.__gather_avail()), but the performance improvement is not that significant. Probably because the loop in master-only, while the loop in BaseHighState is on the minion but talks to the master for each env.

[sathieu]

@twangboy Could you please review?

[sathieu]

@dwoz I've added tests

[waynew]

@thatch45 can you take a look at this? It looks pretty straightforward to me but I'm not 100% familiar with the implications of futzing with the states.

[thatch45]

I really like this approach, it is clean, and it can be pulled out if there are issues. But this does look like it should work.
I love simple, clean additions :)

[DmitryKuzmenko]

@twangboy your turn!

[DmitryKuzmenko]

@sathieu could you please resolve merge conflicts?

[DmitryKuzmenko]

@sathieu could you please resolve the merge conflicts and run the pre-commit script on your code to re-format it with isort and black we're using now.

pip install pre-commit
pre-commit install
FILES="$(git show --pretty="" --name-only master..HEAD)"
pre-commit run isort $FILES
pre-commit run black $FILES

next time you'll commit it will be ran automatically.

[sathieu]

@DmitryKuzmenko I've rebased and used pre-commit hook. I had to use virtualenv -p /usr/bin/python3 DESTDIR, otherwise python2 was used.

Then, it failed with:

  Attempting uninstall: wrapt
    Found existing installation: wrapt 1.10.11
ERROR: Cannot uninstall 'wrapt'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Session lint-tests-pre-commit failed.

I've commited again (black and isort already runt) with --no-verify and pushed here.

[DmitryKuzmenko]

@sathieu thank you for report. We'll work on it.
I've just checked your code formatting is good now.

[waynew]

@sathieu did you use pip or python3 -m pip install --user pre-commit? It should be defaulting to python3 when installing the hooks.

If just installing pre-commit that way doesn't work running rm -rf ~/.cache/pre-commit/ may help.

[DmitryKuzmenko]

@waynew ah, it looks it's my fault. I should recommend python3 pip next time.