Feature suggestion: Automatic parameter lookup - allow list for lookups

[dowlingw]

Suggest adding an optional allowed_keys to the lookup_options parameters - when specified, this would prevent lookups for anything that does not match entries (either strings or regular expressions) in the list.

Automatic parameter lookup will (by default) look up any variables not found in the hierarchy. Even on relatively small sites, this can mean hundreds or thousands of queries against the KeyVault when using this feature.

As an example, instrumenting azure_key_vault::lookup with the following code:

open('/tmp/lookup_trace', 'a') do |f|
      f.puts "get_access_token called for: " + secret_name
    end

Shows that when generating a catalog for a puppet master running Puppet Enterprise v2019.8.5, this results in 776 lookups against Azure KeyVault.

Azure KeyVault service limits allow a maximum of 2,000 lookups every 10 seconds against a key vault: https://docs.microsoft.com/en-us/azure/key-vault/general/service-limits#secrets-managed-storage-account-keys-and-vault-transactions

[TraGicCode]

Old module but example of what we are looking for

https://github.com/Accenture/hiera-aws-sm

[TraGicCode]

https://github.com/petems/petems-hiera_vault

[adelany]

Hi, Do you know when this might be available for use? At the moment my lookups are generating 100's of calls to Keyvault with lookups that are being generated by variables outside hiera that I am not interested in.

[TraGicCode]

Hi, Do you know when this might be available for use? At the moment my lookups are generating 100's of calls to Keyvault with lookups that are being generated by variables outside hiera that I am not interested in.

Hey @adelany ,

I can take a stab at this next weekend and let you know how it goes.

[TraGicCode]

Hey @adelany ,

Please give v2.0.0 a try with the new confine_to_keys hiera configuration option. Let me know if you have any issues or questions.

Thanks!

[adelany]

Hi,

I am getting the following error (not sure I am doing something wrong or it's my setup)

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, 'azure_key_vault::lookup' expects (Variant[String, Numeric] secret_name, Struct[{'vault_name' => String, 'vault_api_version' => String, 'metadata_api_version' => String, 'confine_to_keys' => Array[Regexp], Optional['key_replacement_token'] => String}] options, Object[{name => 'Puppet::LookupContext', parent => Any, attributes => {'environment_name' => {type => String[1], kind => derived}, 'module_name' => Variant[String[1], Undef]}, functions => {'not_found' => Callable[[0, 0], Undef], 'explain' => Callable[[0, 0, Callable[0, 0]], Undef], 'interpolate' => Callable[1, 1], 'cache' => Callable[Optional[Scalar], Any], 'cache_all' => Callable[[Hash[Optional[Scalar], Any]], Undef], 'cache_has_key' => Callable[[Optional[Scalar]], Boolean], 'cached_value' => Callable[Optional[Scalar]], 'cached_entries' => Variant[Callable[[0, 0, Callable[1, 1]], Undef], Callable[[0, 0, Callable[2, 2]], Undef], Callable[[0, 0], Iterable[Tuple[Optional[Scalar], Any]]]], 'cached_file_data' => Callable[String, Optional[Callable[Array[Integer]]]]}}] context)
rejected: parameter 'options' entry 'confine_to_keys' index 0 expects a Regexp value, got String
rejected: parameter 'options' entry 'confine_to_keys' index 1 expects a Regexp value, got String
rejected: parameter 'options' entry 'confine_to_keys' index 2 expects a Regexp value, got String
rejected: parameter 'options' entry 'confine_to_keys' index 3 expects a Regexp value, got String
rejected: parameter 'options' entry 'confine_to_keys' index 4 expects a Regexp value, got String (file: /etc/puppetlabs/code/environments/its_azplatform_rt_dev/manifests/site.pp, line: 3, column: 1) on

Line in site.pp is

lookup('classes', {merge => unique}).include

hiera.yaml : -

---

version: 5

defaults:
datadir: data
data_hash: yaml_data

hierarchy:

• name: 'Application Key Vault Secrets'
  lookup_key: azure_key_vault::lookup
  options:
  vault_name: 'devartifactorykvne'
  #vault_name: "%{facts.az_metadata.compute.tags.environment}%{facts.az_metadata.compute.tags.application}kvne"
  vault_api_version: '2016-10-01'
  metadata_api_version: '2018-04-02'
  key_replacement_token: '-'
  confine_to_keys:
  - '^profile.'
  - '^azure_.'
  - '^._password$'
  - '^password.'
• name: "Per-node data (eyaml version)"
  globs:
  • "per_region/%{facts.az_metadata.compute.location}/%{facts.az_metadata.compute.tags.environment}/%{facts.az_metadata.compute.tags.application}/*.yaml"
  • "per_region/%{facts.az_metadata.compute.location}/%{facts.az_metadata.compute.tags.service}/%{facts.az_metadata.compute.tags.application}/*.yaml"
  • "secrets/%{facts.az_metadata.compute.tags.environment}/%{facts.az_metadata.compute.tags.application}/*.yaml"
  • "secrets/%{facts.az_metadata.compute.tags.environment}/*.yaml"
  • "secrets/*.yaml"
  • "per_environment/%{facts.az_metadata.compute.tags.environment}/%{facts.az_metadata.compute.tags.application}/*.yaml"
  • "per_environment/%{facts.az_metadata.compute.tags.environment}/%{facts.az_metadata.compute.tags.environment}.yaml"
  • "per_service/%{facts.az_metadata.compute.tags.service}/%{facts.az_metadata.compute.tags.application}/%{facts.az_metadata.compute.location}/%{facts.az_metadata.compute.tags.environment}.yaml"
  • "per_service/%{facts.az_metadata.compute.tags.service}/%{facts.az_metadata.compute.tags.application}/%{facts.az_metadata.compute.tags.environment}.yaml"
  • "per_service/%{facts.az_metadata.compute.tags.service}/%{facts.az_metadata.compute.tags.environment}.yaml"
  • "per_role/%{trusted.extensions.pp_role}.yaml"
  • "per_os/%{facts.os.name}.yaml"
    lookup_key: eyaml_lookup_key
    options:
    pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/its_azplatform/private_key.pkcs7.pem
    pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/its_azplatform/public_key.pkcs7.pem
• name: common_data
  glob: 'common/*.yaml'
  lookup_key: eyaml_lookup_key
  options:
  pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/its_azplatform/private_key.pkcs7.pem
  pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/its_azplatform/public_key.pkcs7.pem

Key's I am looking up from a dev.yaml file within hiera data

profile::artifactory_app::artifactory_storage_accname: "%{lookup('artifactory-storage-account')}"
profile::artifactory_app::artifactory_storage_acckey: "%{lookup('artifactory-storage-key')}"
profile::artifactory_app::artifactory_storage_accurl: "%{lookup('artifactory-storage-url')}"
profile::artifactory_app::artifactory_storage_container: "%{lookup('artifactory-storage-container')}"

Any ideas why this isn't working

[TraGicCode]

Hey @adelany ,

I know what the issue is. I was hoping this would work but apparently it doesn't. Let me push out a new release.

[TraGicCode]

Hey @adelany ,

Please get 2.0.1 which resolves your issue.

[adelany]

Many thanks this resolved the issue now working perfectly.