Rewrite vault core, issue AppRoles to minions

[lkubb]

What does this PR do?

• Fundamentally rewrites the Vault integration core and provides higher-level abstractions to interact with Vault.
• Issues AppRoles to minions and manages their metadata, allowing much simpler ACL policies.
• Makes use of response wrapping to distribute secrets.
• Allows to configure token lifecycle, including renewals and revocations.
• Uses Salt cache classes instead of raw file operations.
• Changes the configuration format, the old format is translated automatically.
• Fixes some bugs with the current implementation (see below).
• Adds some low-effort miscellaneous new behaviors like being able to configure audit log metadata.
• Properly splits policy management into execution and state module parts.
• Corrects some docs, adds configuration examples and required policies.
• Removes __utils__ use.
• Migrates tests to pytest.
• Adds (a lot of) new (unit/integration) tests (accounts for ~75% of additions!).
• Tries to achieve all this while staying backwards-compatible: Keeps the previous runner endpoint as well as public utils module functions. Translates old vault configuration. Compatible with legacy peer_run config, but it should be updated to avoid unnecessary roundtrips/reduce network overhead.

Background

While working on #62674, I noticed the better approach would be to issue AppRoles to minions. Implementing that, I was a bit frustrated with the abstraction level and found myself in a yak shaving situation.
The missing abstraction causes issues such as #62651.

Builds on #62674.
See also: https://discuss.hashicorp.com/t/saltstack-vault-and-host-role-policies/19214

What issues does this PR fix or reference?

Fixes #62380
Fixes #58174
Fixes #62823
Fixes #62825
Fixes #60779
Fixes #57561
Fixes #62828
Fixes #58580
Fixes #43287
Fixes #51986
Fixes #63406
Fixes #63440
Fixes #64096
Fixes #64128
Fixes #64379

Included:
#62552
#59827

Previous Behavior

• Adding new Vault behavior is cumbersome.
• The master fetches tokens in plain text and sends them to minions, which are unknown to Vault (see Anti-Patterns).
• To securely assign ACL policies to issued tokens, you need to create a separate policy for each minion. This would be has been relieved a bit by the mentioned pillar templating PR, so that a separate policy is only necessary for each defined role.
• Tokens are always issued with the same minimum and maximum ttl and never renewed or revoked.
• Lease management is left to the user/author of custom modules.

New Behavior

• Adding new Vault behavior has a much lower entry barrier and potential for mistakes.
• Secrets are distributed via response wrapping tokens, ensuring integrity and secrecy. The master can be configured to issue AppRoles and entities to minions. Vault knows about them and associated pillars.
• In theory, a single policy for all salt minions is sufficient for most behaviors:

path "salt/data/minions/{{identity.entity.metadata.minion-id}}" {
    capabilities = ["read"]  # change capabilities as necessary
}

path "salt/data/roles/{{identity.entity.metadata.role}}" {
    capabilities = ["read"]
}

# ... match other custom metadata that is managed by Salt

• Tokens (and AppRole-issued tokens) can be configured with most parameters supported by Vault. Tokens are renewed when necessary and revoked when not in use anymore.
• Leases can be stored and retrieved easily by modules and are renewed automatically (when requested).

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

• Docs
• Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
• Tests written/updated

Commits signed with GPG?

Yes

[lukasraska]

From the looks of it this is breaking change right? Meaning that people that currently use Vault integration will need to adjust at least their configuration (or worse) to get it working again.

If that's the case, I wonder whether it wouldn't make sense deprecating the current modules and provide this as "v2" or similar.

[lkubb]

The intention is for this to be backwards-compatible. The current configuration scheme is translated and in case the master is upgraded before minions, the current runner endpoint is still available, same as with the current public utility functions in case custom modules are using those. They are marked as deprecated though.

[lkubb]

@dwoz I resolved the conflicts and fixed the tests, remaining failures are flaky ones imo. Can you restart the failed runs only? Not sure why it's not done automatically, this was the first run.

[lkubb]

@dwoz Today's test run is all green (and merge conflicts are resolved).

I have been using this code heavily since its inception and think it has at least a few less rough edges than the current one. Either way, the saltext will be published soon as well, so issues can be dealt with there. Sorry for pinging twice, wanted to give this a chance of landing in core before the final saltext migration.

[lkubb]

Update for anyone that followed this: The first release of saltext.vault has just been published to PyPi: https://pypi.org/project/saltext.vault/

You can migrate to the extension at your discretion, it's nearly identical* to what's found in 3007.0. The vault modules (together with many others) have been removed from the master branch, so from 3008.0 onward, the only option to use the Vault integration will be the extension.

* with some minor deprecations, a bugfix that's still pending for 3007 and a small QoL improvement.