From 683010b70b13154bbacbfc7733b9b1b611b87582 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 25 Jan 2017 14:42:44 +0000 Subject: [PATCH 01/62] snapshot --- language/resource-api/README.md | 52 ++++++ language/resource-api/examples.rb | 29 +++ language/resource-api/package.pp | 31 ++++ language/resource-api/simple_apt.rb | 230 ++++++++++++++++++++++++ language/resource-api/simpleresource.rb | 47 +++++ 5 files changed, 389 insertions(+) create mode 100644 language/resource-api/README.md create mode 100644 language/resource-api/examples.rb create mode 100644 language/resource-api/package.pp create mode 100644 language/resource-api/simple_apt.rb create mode 100644 language/resource-api/simpleresource.rb diff --git a/language/resource-api/README.md b/language/resource-api/README.md new file mode 100644 index 0000000..b32eb8f --- /dev/null +++ b/language/resource-api/README.md @@ -0,0 +1,52 @@ +Draft replacement for types and providers + +Hi *, + +I'm currently working on designing a nicer API to replace the current type&provider things. My primary goals are to provide a smooth and simple ruby developer experience for both scripters and coders. Secondary goals were to eliminate server side code, and make puppet 4 data types available. + +To showcase my vision, https://gist.github.com/DavidS/430330ae43ba4b51fe34bd27ddbe4bc7 has the apt_key resource from https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/type/apt_key.rb and https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/apt_key/apt_key.rb ported over. + +The `define()` call provides a data-only description of the Type. This is all that is needed on the server side to compile a manifest. Thanks to puppet 4 data type checking, this will already be much more strict (with less effort) than possible with the current APIs. + +The `implement()` call contains the `current_state = get()` and `set(current_state, target_state, noop)` implementations that will provide a data-driven API to working with resource instances. + +Details in no particular order: + +* All of this should fit on any unmodified puppet4 installation. It is completely additive and optional. Currently. + +* Type definition +** It is data-only. +** No code runs on the server. +** autorelations are restricted to unmodified attribute values and constant values. +** Refers to puppet data types. +** This information can be re-used in all tooling around displaying/working with types (e.g. puppet-strings, console, ENC, etc.). +** No more `validate` or `munge`! For the edge cases not covered by data types, runtime checking can happen in the implementation on the agent. There it can use local system state (e.g. different mysql versions have different max table length constraints), and it will only fail the part of the resource tree, that is dependent on this error. There is already ample precedent for runtime validation, as most remote resources do not try to replicate the validation their target is already doing anyways. +** It maps 1:1 to the capabilities of PCore, and is similar to the libral interface description (see [libral#1](https://github.com/puppetlabs/libral/pull/2)). This ensures future interoperability between the different parts of the ecosystem. +** Related types can share common attributes by sharing/merging the attribute hashes. + +* The implementation are two simple functions `current_state = get()`, and `set(current_state, target_state, noop)`. +** There is no direct dependency on puppet in the implementation. +** The dependencies on the `logger`, `commands`, and similar utilities can be supplied by a small utility library (TBD). +** Calling `r.set(r.get, r.get)` would ensure the current state. This should run without any changes, proving the idempotency of the implementation. +** `get` on its own is already useful for many things, like puppet resource. +** the `current_state` and `target_state` values are lists of simple data structures built up of primitives like strings, numbers, hashes and arrays. They match the schema defined in the type. +** `set` receives the current state from `get`. While this is necessary for proper operation there is a certain race condition there, if the system state changes between the calls. This is no different than the current state, and implementations are well-equipped to deal with this. +** `set` is called with a list of resources, and can do batching if it is beneficial. This is not yet supported by the agent. + +* The logging of updates to the transaction is only a sketch. See the usage of `logger` throughout the example. I've tried different styles for fit. + +* Obviously this is not sufficient to cover everything existing types and providers are able to do. For the first iteration we are choosing simplicity over functionality. +** Generating more resource instances for the catalog during compilation (e.g. file#recurse or concat) becomes impossible with a pure data-driven Type. There is still space in the API to add server-side code. +** Some resources (e.g. file, ssh_authorized_keys, concat) cannot or should not be prefetched. While it might not be convenient, a provider could always return nothing on the `get()` and do a more customized enforce motion in the `set()`. +** With current puppet versions, only "native" data types will be supported, as type aliases do not get pluginsynced. Yet. + +* There is some convenient infrastructure (e.g. parsedfile) that needs porting over to this model. + +* Testing becomes possible on a complete new level. The test library can know how data is transformed outside the API, and - using the shape of the type - start generating test cases, and checking the actions of the implementation. This will require developer help to isolate the implementation from real systems, but it should go a long way towards reducing the tedium in writing tests. + + +What do you think about this? + + +Cheers, David + diff --git a/language/resource-api/examples.rb b/language/resource-api/examples.rb new file mode 100644 index 0000000..8767a2d --- /dev/null +++ b/language/resource-api/examples.rb @@ -0,0 +1,29 @@ +SHARED_PACKAGE_ATTRIBUTES = { + name: { type: 'String' }, + ENSURE_ATTRIBUTE, + reinstall_on_refresh: { type: 'Boolean'}, +}.freeze + +LOCAL_PACKAGE_ATTRIBUTES = { + source: { type: 'String' }, +}.freeze + +VERSIONABLE_PACKAGE_ATTRIBUTES = { + version: { type: 'String' }, +}.freeze + +APT_PACKAGE_ATTRIBUTES = { + install_options: { type: 'String' }, + responsefile: { type: 'String' }, +}.freeze + +Puppet::SimpleResource.define( + name: 'package_rpm', + attributes: {}.merge(SHARED_PACKAGE_ATTRIBUTES).merge(LOCAL_PACKAGE_ATTRIBUTES), +) + +Puppet::SimpleResource.define( + name: 'package_apt', + attributes: {}.merge(SHARED_PACKAGE_ATTRIBUTES).merge(LOCAL_PACKAGE_ATTRIBUTES).merge(VERSIONABLE_PACKAGE_ATTRIBUTES).merge(APT_PACKAGE_ATTRIBUTES), +) + diff --git a/language/resource-api/package.pp b/language/resource-api/package.pp new file mode 100644 index 0000000..d91e7f4 --- /dev/null +++ b/language/resource-api/package.pp @@ -0,0 +1,31 @@ +define package ( + Ensure $ensure, + Enum[apt, rpm] $provider, + Optional[String] $source = undef, + Optional[String] $version = undef, + Optional[String] $install_options = undef, + Optional[String] $responsefile = undef, + Optional[Hash] $options = { }, +) { + case $provider { + apt: { + package_apt { $title: + ensure => $ensure, + source => $source, + version => $version, + install_options => $install_options, + responsefile => $responsefile, + * => $options, + } + } + rpm: { + package_rpm { $title: + ensure => $ensure, + source => $source, + * => $options, + } + if defined($version) { fail("RPM doesn't support \$version") } + # ... + } + } +} diff --git a/language/resource-api/simple_apt.rb b/language/resource-api/simple_apt.rb new file mode 100644 index 0000000..e5c72ee --- /dev/null +++ b/language/resource-api/simple_apt.rb @@ -0,0 +1,230 @@ +Puppet::SimpleResource.define( + name: 'apt_key', + docs: <<-EOS, + This type provides Puppet with the capabilities to manage GPG keys needed + by apt to perform package validation. Apt has it's own GPG keyring that can + be manipulated through the `apt-key` command. + + apt_key { '6F6B15509CF8E59E6E469F327F438280EF8D349F': + source => 'http://apt.puppetlabs.com/pubkey.gpg' + } + + **Autorequires**: + If Puppet is given the location of a key file which looks like an absolute + path this type will autorequire that file. + EOS + attributes: { + ensure: { + type: 'Enum[present, absent]', + docs: 'Whether this apt key should be present or absent on the target system.' + }, + id: { + type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]', + docs: 'The ID of the key you want to manage.', + namevar: true, + }, + content: { + type: 'Optional[String]', + docs: 'The content of, or string representing, a GPG key.', + }, + source: { + type: 'Variant[Stdlib::Absolutepath, Pattern[/\A(https?|ftp):\/\//]]', + docs: 'Location of a GPG key file, /path/to/file, ftp://, http:// or https://', + }, + server: { + type: 'Pattern[/\A((hkp|http|https):\/\/)?([a-z\d])([a-z\d-]{0,61}\.)+[a-z\d]+(:\d{2,5})?$/]', + docs: 'The key server to fetch the key from based on the ID. It can either be a domain name or url.', + default: 'keyserver.ubuntu.com' + }, + options: { + type: 'Optional[String]', + docs: 'Additional options to pass to apt-key\'s --keyserver-options.', + }, + fingerprint: { + type: 'String', + docs: 'The 40-digit hexadecimal fingerprint of the specified GPG key.', + read_only: true, + }, + long: { + type: 'String', + docs: 'The 16-digit hexadecimal id of the specified GPG key.', + read_only: true, + }, + short: { + type: 'String', + docs: 'The 8-digit hexadecimal id of the specified GPG key.', + read_only: true, + }, + expired: { + type: 'Boolean', + docs: 'Indicates if the key has expired.', + read_only: true, + }, + expiry: { + type: 'String', + docs: 'The date the key will expire, or nil if it has no expiry date, in ISO format.', + read_only: true, + }, + size: { + type: 'String', + docs: 'The key size, usually a multiple of 1024.', + read_only: true, + }, + type: { + type: 'String', + docs: 'The key type, one of: rsa, dsa, ecc, ecdsa.', + read_only: true, + }, + created: { + type: 'String', + docs: 'Date the key was created, in ISO format.', + read_only: true, + }, + }, + autorequires: { + file: '$source', # will evaluate to the value of the `source` attribute + package: 'apt', + }, +) + +Puppet::SimpleResource.implement('apt_key') do + commands apt_key: 'apt-key' + commands gpg: '/usr/bin/gpg' + + def get + cli_args = %w(adv --list-keys --with-colons --fingerprint --fixed-list-mode) + key_output = apt_key(cli_args).encode('UTF-8', 'binary', :invalid => :replace, :undef => :replace, :replace => '') + pub_line = nil + fpr_line = nil + + key_output.split("\n").collect do |line| + if line.start_with?('pub') + pub_line = line + elsif line.start_with?('fpr') + fpr_line = line + end + + next unless (pub_line and fpr_line) + + result = key_line_to_hash(pub_line, fpr_line) + + # reset everything + pub_line = nil + fpr_line = nil + + result + end.compact! + end + + def self.key_line_to_hash(pub_line, fpr_line) + pub_split = pub_line.split(':') + fpr_split = fpr_line.split(':') + + # set key type based on types defined in /usr/share/doc/gnupg/DETAILS.gz + key_type = case pub_split[3] + when '1' + :rsa + when '17' + :dsa + when '18' + :ecc + when '19' + :ecdsa + else + :unrecognized + end + + fingerprint = fpr_split.last + expiry = pub_split[6].empty? ? nil : Time.at(pub_split[6].to_i) + + { + name: fingerprint, + ensure: 'present', + fingerprint: fingerprint, + long: fingerprint[-16..-1], # last 16 characters of fingerprint + short: fingerprint[-8..-1], # last 8 characters of fingerprint + size: pub_split[2], + type: key_type, + created: Time.at(pub_split[5].to_i), + expiry: expiry, + expired: expiry && Time.now >= expiry, + } + end + + def set(current_state, target_state, noop = false) + existing_keys = Hash[current_state.collect { |k| [k[:name], k] }] + target_state.each do |key| + logger.warning(key[:name], 'The id should be a full fingerprint (40 characters) to avoid collision attacks, see the README for details.') if key[:name].length < 40 + if key[:source] and key[:content] + logger.fail(key[:name], 'The properties content and source are mutually exclusive') + next + end + + current = existing_keys[k[:name]] + if current && key[:ensure].to_s == 'absent' + logger.deleting(key[:name]) do + begin + apt_key('del', key[:short], noop: noop) + r = execute(["#{command(:apt_key)} list | grep '/#{resource.provider.short}\s'"], :failonfail => false) + end while r.exitstatus == 0 + end + elsif current && key[:ensure].to_s == 'present' + # No updating implemented + # update(key, noop: noop) + elsif !current && key[:ensure].to_s == 'present' + create(key, noop: noop) + end + end + end + + def create(key, noop = false) + logger.creating(key[:name]) do |logger| + if key[:source].nil? and key[:content].nil? + # Breaking up the command like this is needed because it blows up + # if --recv-keys isn't the last argument. + args = ['adv', '--keyserver', key[:server]] + if key[:options] + args.push('--keyserver-options', key[:options]) + end + args.push('--recv-keys', key[:id]) + apt_key(*args, noop: noop) + elsif key[:content] + temp_key_file(key[:content], logger) do |key_file| + apt_key('add', key_file, noop: noop) + end + elsif key[:source] + key_file = source_to_file(key[:source]) + apt_key('add', key_file.path, noop: noop) + # In case we really screwed up, better safe than sorry. + else + logger.fail("an unexpected condition occurred while trying to add the key: #{key[:id]} (content: #{key[:content].inspect}, source: #{key[:source].inspect})") + end + end + end + + # This method writes out the specified contents to a temporary file and + # confirms that the fingerprint from the file, matches the long key that is in the manifest + def temp_key_file(key, logger) + file = Tempfile.new('apt_key') + begin + file.write key[:content] + file.close + if name.size == 40 + if File.executable? command(:gpg) + extracted_key = execute(["#{command(:gpg)} --with-fingerprint --with-colons #{file.path} | awk -F: '/^fpr:/ { print $10 }'"], :failonfail => false) + extracted_key = extracted_key.chomp + + unless extracted_key.match(/^#{name}$/) + logger.fail("The id in your manifest #{key[:name]} and the fingerprint from content/source do not match. Please check there is not an error in the id or check the content/source is legitimate.") + end + else + logger.warning('/usr/bin/gpg cannot be found for verification of the id.') + end + end + yield file.path + ensure + file.close + file.unlink + end + end +end diff --git a/language/resource-api/simpleresource.rb b/language/resource-api/simpleresource.rb new file mode 100644 index 0000000..7282a7f --- /dev/null +++ b/language/resource-api/simpleresource.rb @@ -0,0 +1,47 @@ +Puppet::SimpleResource.define( + name: 'iis_application_pool', + docs: 'Manage an IIS application pool through a powershell proxy.', + attributes: { + ensure: { + type: 'Enum[present, absent]', + docs: 'Whether this ApplicationPool should be present or absent on the target system.' + }, + name: { + type: 'String', + docs: 'The name of the ApplicationPool.', + namevar: true, + }, + state: { + type: 'Enum[running, stopped]', + docs: 'The state of the ApplicationPool.', + default: 'running', + }, + managedpipelinemode: { + type: 'String', + docs: 'The managedPipelineMode of the ApplicationPool.', + }, + managedruntimeversion: { + type: 'String', + docs: 'The managedRuntimeVersion of the ApplicationPool.', + }, + } +) do + + require 'puppet/provider/iis_powershell' + include Puppet::Provider::IIS_PowerShell + + def get + result = run('fetch_application_pools.ps1', logger) # call out to powershell to talk to the API + + # returns an array of hashes with data according to the schema above + JSON.parse(result) + end + + def set(goals, noop = false) + result = run('enforce_application_pools.ps1', goals, logger, noop) # call out to powershell to talk to the API + + # returns an array of hashes with status data from the changes + JSON.parse(result) + end + +end From 7c694fa50afed89a85483fe9732c12e41908a1ef Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Fri, 27 Jan 2017 09:57:47 +0000 Subject: [PATCH 02/62] more design drafting --- language/resource-api/README.md | 1 + language/resource-api/tests.rb | 41 +++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) create mode 100644 language/resource-api/tests.rb diff --git a/language/resource-api/README.md b/language/resource-api/README.md index b32eb8f..f04a558 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -39,6 +39,7 @@ Details in no particular order: ** Generating more resource instances for the catalog during compilation (e.g. file#recurse or concat) becomes impossible with a pure data-driven Type. There is still space in the API to add server-side code. ** Some resources (e.g. file, ssh_authorized_keys, concat) cannot or should not be prefetched. While it might not be convenient, a provider could always return nothing on the `get()` and do a more customized enforce motion in the `set()`. ** With current puppet versions, only "native" data types will be supported, as type aliases do not get pluginsynced. Yet. +** With current puppet versions, `puppet resource` can't load the data types, and therefore will not be able to take full advantage of this. Yet. * There is some convenient infrastructure (e.g. parsedfile) that needs porting over to this model. diff --git a/language/resource-api/tests.rb b/language/resource-api/tests.rb new file mode 100644 index 0000000..60a9420 --- /dev/null +++ b/language/resource-api/tests.rb @@ -0,0 +1,41 @@ +import 'facts_db.pp' + +example 'mongodb::db' { + example 'default' { + $facts_db.each { |$loop_facts| + given 'default' ( + $facts = $loop_facts, + $modules = $default_modules, + ) { + mongodb::db { 'testdb' : + user => 'testuser', + password => 'testpass', + } + } + assert 'it contains mongodb_database with mongodb::server requirement' { + mongodb_database { 'testdb' : } + } + assert 'it contains mongodb_user with mongodb_database requirement' { + mongodb_user { 'User testuser on db testdb' : + username => 'testuser', + database => 'testdb', + require => Mongodb_database['testdb'], + } + } + } + } + example 'old modules' { + given 'default' ( + $facts = {} + $modules = {'puppetlabs-stdlib' => '4.4.0', }.merge($default_modules), + ) { + mongodb::db { 'testdb' : + user => 'testuser', + password => 'testpass', + } + } + assert { + # it compiles, at least + } + } +} From 154d75352742273473f7dce5e911846aeab7a8b4 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Fri, 27 Jan 2017 11:23:46 +0000 Subject: [PATCH 03/62] Final touches This is the version currently on https://gist.github.com/DavidS/430330ae43ba4b51fe34bd27ddbe4bc7 --- language/resource-api/README.md | 42 ++++++++++++++++----------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index f04a558..1a5914f 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -4,42 +4,42 @@ Hi *, I'm currently working on designing a nicer API to replace the current type&provider things. My primary goals are to provide a smooth and simple ruby developer experience for both scripters and coders. Secondary goals were to eliminate server side code, and make puppet 4 data types available. -To showcase my vision, https://gist.github.com/DavidS/430330ae43ba4b51fe34bd27ddbe4bc7 has the apt_key resource from https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/type/apt_key.rb and https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/apt_key/apt_key.rb ported over. +To showcase my vision, this [gist](https://gist.github.com/DavidS/430330ae43ba4b51fe34bd27ddbe4bc7) has the [apt_key type](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/type/apt_key.rb) and [provider](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/apt_key/apt_key.rb) ported over. The `define()` call provides a data-only description of the Type. This is all that is needed on the server side to compile a manifest. Thanks to puppet 4 data type checking, this will already be much more strict (with less effort) than possible with the current APIs. -The `implement()` call contains the `current_state = get()` and `set(current_state, target_state, noop)` implementations that will provide a data-driven API to working with resource instances. +The `implement()` call contains the `current_state = get()` and `set(current_state, target_state, noop)` implementations that will provide a data-driven API to working with resource instances. The state descriptions passed around are simple lists of primitive data describing resources. Details in no particular order: * All of this should fit on any unmodified puppet4 installation. It is completely additive and optional. Currently. * Type definition -** It is data-only. -** No code runs on the server. -** autorelations are restricted to unmodified attribute values and constant values. -** Refers to puppet data types. -** This information can be re-used in all tooling around displaying/working with types (e.g. puppet-strings, console, ENC, etc.). -** No more `validate` or `munge`! For the edge cases not covered by data types, runtime checking can happen in the implementation on the agent. There it can use local system state (e.g. different mysql versions have different max table length constraints), and it will only fail the part of the resource tree, that is dependent on this error. There is already ample precedent for runtime validation, as most remote resources do not try to replicate the validation their target is already doing anyways. -** It maps 1:1 to the capabilities of PCore, and is similar to the libral interface description (see [libral#1](https://github.com/puppetlabs/libral/pull/2)). This ensures future interoperability between the different parts of the ecosystem. -** Related types can share common attributes by sharing/merging the attribute hashes. + * It is data-only. + * No code runs on the server. + * autorelations are restricted to unmodified attribute values and constant values. + * Refers to puppet data types. + * This information can be re-used in all tooling around displaying/working with types (e.g. puppet-strings, console, ENC, etc.). + * No more `validate` or `munge`! For the edge cases not covered by data types, runtime checking can happen in the implementation on the agent. There it can use local system state (e.g. different mysql versions have different max table length constraints), and it will only fail the part of the resource tree, that is dependent on this error. There is already ample precedent for runtime validation, as most remote resources do not try to replicate the validation their target is already doing anyways. + * It maps 1:1 to the capabilities of PCore, and is similar to the libral interface description (see [libral#1](https://github.com/puppetlabs/libral/pull/2)). This ensures future interoperability between the different parts of the ecosystem. + * Related types can share common attributes by sharing/merging the attribute hashes. * The implementation are two simple functions `current_state = get()`, and `set(current_state, target_state, noop)`. -** There is no direct dependency on puppet in the implementation. -** The dependencies on the `logger`, `commands`, and similar utilities can be supplied by a small utility library (TBD). -** Calling `r.set(r.get, r.get)` would ensure the current state. This should run without any changes, proving the idempotency of the implementation. -** `get` on its own is already useful for many things, like puppet resource. -** the `current_state` and `target_state` values are lists of simple data structures built up of primitives like strings, numbers, hashes and arrays. They match the schema defined in the type. -** `set` receives the current state from `get`. While this is necessary for proper operation there is a certain race condition there, if the system state changes between the calls. This is no different than the current state, and implementations are well-equipped to deal with this. -** `set` is called with a list of resources, and can do batching if it is beneficial. This is not yet supported by the agent. + * There is no direct dependency on puppet in the implementation. + * The dependencies on the `logger`, `commands`, and similar utilities can be supplied by a small utility library (TBD). + * Calling `r.set(r.get, r.get)` would ensure the current state. This should run without any changes, proving the idempotency of the implementation. + * `get` on its own is already useful for many things, like puppet resource. + * the `current_state` and `target_state` values are lists of simple data structures built up of primitives like strings, numbers, hashes and arrays. They match the schema defined in the type. + * `set` receives the current state from `get`. While this is necessary for proper operation there is a certain race condition there, if the system state changes between the calls. This is no different than the current state, and implementations are well-equipped to deal with this. + * `set` is called with a list of resources, and can do batching if it is beneficial. This is not yet supported by the agent. * The logging of updates to the transaction is only a sketch. See the usage of `logger` throughout the example. I've tried different styles for fit. * Obviously this is not sufficient to cover everything existing types and providers are able to do. For the first iteration we are choosing simplicity over functionality. -** Generating more resource instances for the catalog during compilation (e.g. file#recurse or concat) becomes impossible with a pure data-driven Type. There is still space in the API to add server-side code. -** Some resources (e.g. file, ssh_authorized_keys, concat) cannot or should not be prefetched. While it might not be convenient, a provider could always return nothing on the `get()` and do a more customized enforce motion in the `set()`. -** With current puppet versions, only "native" data types will be supported, as type aliases do not get pluginsynced. Yet. -** With current puppet versions, `puppet resource` can't load the data types, and therefore will not be able to take full advantage of this. Yet. + * Generating more resource instances for the catalog during compilation (e.g. file#recurse or concat) becomes impossible with a pure data-driven Type. There is still space in the API to add server-side code. + * Some resources (e.g. file, ssh_authorized_keys, concat) cannot or should not be prefetched. While it might not be convenient, a provider could always return nothing on the `get()` and do a more customized enforce motion in the `set()`. + * With current puppet versions, only "native" data types will be supported, as type aliases do not get pluginsynced. Yet. + * With current puppet versions, `puppet resource` can't load the data types, and therefore will not be able to take full advantage of this. Yet. * There is some convenient infrastructure (e.g. parsedfile) that needs porting over to this model. From 7ac03ba06778dc4a9f8a65bc5211d46c26706e76 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 31 Jan 2017 15:44:13 +0000 Subject: [PATCH 04/62] Finalise Resource API draft for community review --- language/resource-api/README.md | 43 +++-- language/resource-api/apt_key_get.rb | 120 ++++++++++++ language/resource-api/apt_key_set.rb | 239 ++++++++++++++++++++++++ language/resource-api/simple_apt.rb | 2 +- language/resource-api/simpleresource.rb | 31 +-- 5 files changed, 404 insertions(+), 31 deletions(-) create mode 100755 language/resource-api/apt_key_get.rb create mode 100644 language/resource-api/apt_key_set.rb diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 1a5914f..395da62 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -2,48 +2,59 @@ Draft replacement for types and providers Hi *, -I'm currently working on designing a nicer API to replace the current type&provider things. My primary goals are to provide a smooth and simple ruby developer experience for both scripters and coders. Secondary goals were to eliminate server side code, and make puppet 4 data types available. +The type and provider API has been the bane of my existence since I [started writing native resources](https://github.com/DavidS/puppet-mysql-old/commit/d33c7aa10e3a4bd9e97e947c471ee3ed36e9d1e2). Now, finally, we'll do something about it. I'm currently working on designing a nicer API for types and providers. My primary goals are to provide a smooth and simple ruby developer experience for both scripters and coders. Secondary goals were to eliminate server side code, and make puppet 4 data types available. Currently this is completely aspirational (i.e. no real code has been written), but early private feedback was encouraging. -To showcase my vision, this [gist](https://gist.github.com/DavidS/430330ae43ba4b51fe34bd27ddbe4bc7) has the [apt_key type](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/type/apt_key.rb) and [provider](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/apt_key/apt_key.rb) ported over. +To showcase my vision, this [gist](https://gist.github.com/DavidS/430330ae43ba4b51fe34bd27ddbe4bc7) has the [apt_key type](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/type/apt_key.rb) and [provider](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/apt_key/apt_key.rb) ported over to my proposal. The second example there is a more long-term teaser on what would become possible with such an API. -The `define()` call provides a data-only description of the Type. This is all that is needed on the server side to compile a manifest. Thanks to puppet 4 data type checking, this will already be much more strict (with less effort) than possible with the current APIs. +The new API, like the existing, has two parts: the implementation that interacts with the actual resources, a.k.a. the provider, and information about what the implementation is all about. Due to the different usage patterns of the two parts, they need to be passed to puppet in two different calls: + +The `Puppet::SimpleResource.implement()` call receives the `current_state = get()` and `set(current_state, target_state, noop)` methods. `get` returns a list of discovered resources, while `set` takes the target state and enforces those goals on the subject. There is only a single (ruby) object throughout an agent run, that can easily do caching and what ever else is required for a good functioning of the provider. The state descriptions passed around are simple lists of key/value hashes describing resources. This will allow the implementation wide latitude in how to organise itself for simplicity and efficiency. + +The `Puppet::SimpleResource.define()` call provides a data-only description of the Type. This is all that is needed on the server side to compile a manifest. Thanks to puppet 4 data type checking, this will already be much more strict (with less effort) than possible with the current APIs, while providing more automatically readable documentation about the meaning of the attributes. -The `implement()` call contains the `current_state = get()` and `set(current_state, target_state, noop)` implementations that will provide a data-driven API to working with resource instances. The state descriptions passed around are simple lists of primitive data describing resources. Details in no particular order: -* All of this should fit on any unmodified puppet4 installation. It is completely additive and optional. Currently. +* All of this should fit on any unmodified puppet4 installation. It is completely additive and optional. Currently. -* Type definition +* The Type definition * It is data-only. - * No code runs on the server. - * autorelations are restricted to unmodified attribute values and constant values. * Refers to puppet data types. + * No code runs on the server. * This information can be re-used in all tooling around displaying/working with types (e.g. puppet-strings, console, ENC, etc.). + * autorelations are restricted to unmodified attribute values and constant values. * No more `validate` or `munge`! For the edge cases not covered by data types, runtime checking can happen in the implementation on the agent. There it can use local system state (e.g. different mysql versions have different max table length constraints), and it will only fail the part of the resource tree, that is dependent on this error. There is already ample precedent for runtime validation, as most remote resources do not try to replicate the validation their target is already doing anyways. * It maps 1:1 to the capabilities of PCore, and is similar to the libral interface description (see [libral#1](https://github.com/puppetlabs/libral/pull/2)). This ensures future interoperability between the different parts of the ecosystem. * Related types can share common attributes by sharing/merging the attribute hashes. + * `defaults`, `read_only`, and similar data about attributes in the definition are mostly aesthetic at the current point in time, but will make for better documentation, and allow more intelligence built on top of this later. * The implementation are two simple functions `current_state = get()`, and `set(current_state, target_state, noop)`. - * There is no direct dependency on puppet in the implementation. - * The dependencies on the `logger`, `commands`, and similar utilities can be supplied by a small utility library (TBD). - * Calling `r.set(r.get, r.get)` would ensure the current state. This should run without any changes, proving the idempotency of the implementation. * `get` on its own is already useful for many things, like puppet resource. - * the `current_state` and `target_state` values are lists of simple data structures built up of primitives like strings, numbers, hashes and arrays. They match the schema defined in the type. - * `set` receives the current state from `get`. While this is necessary for proper operation there is a certain race condition there, if the system state changes between the calls. This is no different than the current state, and implementations are well-equipped to deal with this. + * `set` receives the current state from `get`. While this is necessary for proper operation, there is a certain race condition there, if the system state changes between the calls. This is no different than what current implementations face, and they are well-equipped to deal with this. * `set` is called with a list of resources, and can do batching if it is beneficial. This is not yet supported by the agent. + * the `current_state` and `target_state` values are lists of simple data structures built up of primitives like strings, numbers, hashes and arrays. They match the schema defined in the type. + * Calling `r.set(r.get, r.get)` would ensure the current state. This should run without any changes, proving the idempotency of the implementation. + * The ruby instance hosting the `get` and `set` functions is only alive for the duration of an agent transaction. An implementation can provide a `initialize` method to read credentials from the system, and setup other things as required. The single instance is used for all instances of the resource. + * There is no direct dependency on puppet core libraries in the implementation. + * While implementations can use utility functions, they are completely optional. + * The dependencies on the `logger`, `commands`, and similar utilities can be supplied by a small utility library (TBD). -* The logging of updates to the transaction is only a sketch. See the usage of `logger` throughout the example. I've tried different styles for fit. +* Having a well-defined small API makes remoting, stacking, proxying, batching, interactive use, and other shenanigans possible, which will make for a interesting time ahead. +* The logging of updates to the transaction is only a sketch. See the usage of `logger` throughout the example. I've tried different styles for fit. + * the `logger` is the primary way of reporting back information to the log, and the report. + * results can be streamed for immediate feedback + * block-based constructs allow detailed logging with little code ("Started X", "X: Doing Something", "X: Success|Failure", with one or two calls, and only one reference to X) + * Obviously this is not sufficient to cover everything existing types and providers are able to do. For the first iteration we are choosing simplicity over functionality. * Generating more resource instances for the catalog during compilation (e.g. file#recurse or concat) becomes impossible with a pure data-driven Type. There is still space in the API to add server-side code. * Some resources (e.g. file, ssh_authorized_keys, concat) cannot or should not be prefetched. While it might not be convenient, a provider could always return nothing on the `get()` and do a more customized enforce motion in the `set()`. * With current puppet versions, only "native" data types will be supported, as type aliases do not get pluginsynced. Yet. * With current puppet versions, `puppet resource` can't load the data types, and therefore will not be able to take full advantage of this. Yet. -* There is some convenient infrastructure (e.g. parsedfile) that needs porting over to this model. +* There is some "convenient" infrastructure (e.g. parsedfile) that needs porting over to this model. -* Testing becomes possible on a complete new level. The test library can know how data is transformed outside the API, and - using the shape of the type - start generating test cases, and checking the actions of the implementation. This will require developer help to isolate the implementation from real systems, but it should go a long way towards reducing the tedium in writing tests. +* Testing becomes possible on a completely new level. The test library can know how data is transformed outside the API, and - using the shape of the type - start generating test cases, and checking the actions of the implementation. This will require developer help to isolate the implementation from real systems, but it should go a long way towards reducing the tedium in writing tests. What do you think about this? diff --git a/language/resource-api/apt_key_get.rb b/language/resource-api/apt_key_get.rb new file mode 100755 index 0000000..2613a0e --- /dev/null +++ b/language/resource-api/apt_key_get.rb @@ -0,0 +1,120 @@ +#!/usr/bin/ruby + +require 'json' + + +def key_line_to_hash(pub_line, fpr_line) + pub_split = pub_line.split(':') + fpr_split = fpr_line.split(':') + + # set key type based on types defined in /usr/share/doc/gnupg/DETAILS.gz + key_type = case pub_split[3] + when '1' + :rsa + when '17' + :dsa + when '18' + :ecc + when '19' + :ecdsa + else + :unrecognized + end + + fingerprint = fpr_split.last + expiry = pub_split[6].empty? ? nil : Time.at(pub_split[6].to_i) + + { + name: fingerprint, + ensure: 'present', + fingerprint: fingerprint, + long: fingerprint[-16..-1], # last 16 characters of fingerprint + short: fingerprint[-8..-1], # last 8 characters of fingerprint + size: pub_split[2], + type: key_type, + created: Time.at(pub_split[5].to_i), + expiry: expiry, + expired: !!(expiry && Time.now >= expiry), + } +end + +key_output = <: +sub:-:2048:1:4AB781597254279C:1370645731::::::e:::::: +fpr:::::::::B71ACDE6B52658D12C3106F44AB781597254279C: +pub:-:1024:17:A040830F7FAC5991:1173385030:::-:::scESC::::::: +fpr:::::::::4CCA1EAF950CEE4AB83976DCA040830F7FAC5991: +uid:-::::1175811711::0F5F08408BC3D293942A5E5A2D1AE1BD277FF5DB::Google, Inc. Linux Package Signing Key : +sub:-:2048:16:4F30B6B4C07CB649:1173385035::::::e:::::: +fpr:::::::::9534C9C4130B4DC9927992BF4F30B6B4C07CB649: +pub:-:4096:1:7638D0442B90D010:1416603673:1668891673::-:::scSC::::::: +rvk:::1::::::309911BEA966D0613053045711B4E5FF15B0FD82:80: +rvk:::1::::::FBFABDB541B5DC955BD9BA6EDB16CF5BB12525C4:80: +rvk:::1::::::80E976F14A508A48E9CA3FE9BC372252CA1CF964:80: +fpr:::::::::126C0D24BD8A2942CC7DF8AC7638D0442B90D010: +uid:-::::1416603673::15C761B84F0C9C293316B30F007E34BE74546B48::Debian Archive Automatic Signing Key (8/jessie) : +pub:-:4096:1:9D6D8F6BC857C906:1416604417:1668892417::-:::scSC::::::: +rvk:::1::::::FBFABDB541B5DC955BD9BA6EDB16CF5BB12525C4:80: +rvk:::1::::::309911BEA966D0613053045711B4E5FF15B0FD82:80: +rvk:::1::::::80E976F14A508A48E9CA3FE9BC372252CA1CF964:80: +fpr:::::::::D21169141CECD440F2EB8DDA9D6D8F6BC857C906: +uid:-::::1416604417::088FA6B00E33BCC6F6EB4DFEFAC591F9940E06F0::Debian Security Archive Automatic Signing Key (8/jessie) : +pub:-:4096:1:CBF8D6FD518E17E1:1376739416:1629027416::-:::scSC::::::: +fpr:::::::::75DDC3C4A499F1A18CB5F3C8CBF8D6FD518E17E1: +uid:-::::1376739416::2D9AEBB80FC7D1724686A20DC5712C7D0DC07AF6::Jessie Stable Release Key : +pub:-:4096:1:AED4B06F473041FA:1282940623:1520281423::-:::scSC::::::: +fpr:::::::::9FED2BCBDCD29CDF762678CBAED4B06F473041FA: +uid:-::::1282940896::CED55047A1889F383B10CE9D04346A5CA12E2445::Debian Archive Automatic Signing Key (6.0/squeeze) : +pub:-:4096:1:64481591B98321F9:1281140461:1501892461::-:::scSC::::::: +fpr:::::::::0E4EDE2C7F3E1FC0D033800E64481591B98321F9: +uid:-::::1281140461::BB638CC58BB7B36929C2C6DEBE580CC46FC94B36::Squeeze Stable Release Key : +pub:-:4096:1:8B48AD6246925553:1335553717:1587841717::-:::scSC::::::: +fpr:::::::::A1BD8E9D78F7FE5C3E65D8AF8B48AD6246925553: +uid:-::::1335553717::BCBD552DFB543AADFE3812AF631B17F5EDEF820E::Debian Archive Automatic Signing Key (7.0/wheezy) : +pub:-:4096:1:6FB2A1C265FFB764:1336489909:1557241909::-:::scSC::::::: +fpr:::::::::ED6D65271AACF0FF15D123036FB2A1C265FFB764: +uid:-::::1336489909::0BB8E4C85595D59CE65881DDD593ECBAE583607B::Wheezy Stable Release Key : +pub:e:4096:1:1054B7A24BD6EC30:1278720832:1483574797::-:::sc::::::: +fpr:::::::::47B320EB4C7C375AA9DAE1A01054B7A24BD6EC30: +uid:e::::1460074501::BA4BCA138CEBDF8444241CE928DEE1AD79612E6C::Puppet Labs Release Key (Puppet Labs Release Key) : +pub:-:4096:1:B8F999C007BB6C57:1360109177:1549910347::-:::scESC::::::: +fpr:::::::::8735F5AF62A99A628EC13377B8F999C007BB6C57: +uid:-::::1455302347::A8FC88656336852AD4301DF059CEE6134FD37C21::Puppet Labs Nightly Build Key (Puppet Labs Nightly Build Key) : +uid:-::::1455302347::4EF2A82F1FF355343885012A832C628E1A4F73A8::Puppet Labs Nightly Build Key (Puppet Labs Nightly Build Key) : +sub:-:4096:1:AE8282E5A5FC3E74:1360109177:1549910293:::::e:::::: +fpr:::::::::F838D657CCAF0E4A6375B0E9AE8282E5A5FC3E74: +pub:-:4096:1:7F438280EF8D349F:1471554366:1629234366::-:::scESC::::::: +fpr:::::::::6F6B15509CF8E59E6E469F327F438280EF8D349F: +uid:-::::1471554366::B648B946D1E13EEA5F4081D8FE5CF4D001200BC7::Puppet, Inc. Release Key (Puppet, Inc. Release Key) : +sub:-:4096:1:A2D80E04656674AE:1471554366:1629234366:::::e:::::: +fpr:::::::::07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE: +EOM + + + +pub_line = nil +fpr_line = nil + +instances = key_output.split("\n").collect do |line| + if line.start_with?('pub') + pub_line = line + elsif line.start_with?('fpr') + fpr_line = line + end + + next unless (pub_line and fpr_line) + + result = key_line_to_hash(pub_line, fpr_line) + + # reset everything + pub_line = nil + fpr_line = nil + + result +end.compact! + +puts JSON.generate(instances) diff --git a/language/resource-api/apt_key_set.rb b/language/resource-api/apt_key_set.rb new file mode 100644 index 0000000..b3fa347 --- /dev/null +++ b/language/resource-api/apt_key_set.rb @@ -0,0 +1,239 @@ +#!/usr/bin/ruby + +require 'json' + +current_state_json = < false) + end while r.exitstatus == 0 + end + elsif current && key[:ensure].to_s == 'present' + # No updating implemented + # update(key, noop: noop) + elsif !current && key[:ensure].to_s == 'present' + create(key, noop: noop) + end + end +end + +def create(key, noop = false) + logger.creating(key[:name]) do |logger| + if key[:source].nil? and key[:content].nil? + # Breaking up the command like this is needed because it blows up + # if --recv-keys isn't the last argument. + args = ['adv', '--keyserver', key[:server]] + if key[:options] + args.push('--keyserver-options', key[:options]) + end + args.push('--recv-keys', key[:id]) + apt_key(*args, noop: noop) + elsif key[:content] + temp_key_file(key[:content], logger) do |key_file| + apt_key('add', key_file, noop: noop) + end + elsif key[:source] + key_file = source_to_file(key[:source]) + apt_key('add', key_file.path, noop: noop) + # In case we really screwed up, better safe than sorry. + else + logger.fail("an unexpected condition occurred while trying to add the key: #{key[:id]} (content: #{key[:content].inspect}, source: #{key[:source].inspect})") + end + end +end + +# This method writes out the specified contents to a temporary file and +# confirms that the fingerprint from the file, matches the long key that is in the manifest +def temp_key_file(key, logger) + file = Tempfile.new('apt_key') + begin + file.write key[:content] + file.close + if name.size == 40 + if File.executable? command(:gpg) + extracted_key = execute(["#{command(:gpg)} --with-fingerprint --with-colons #{file.path} | awk -F: '/^fpr:/ { print $10 }'"], :failonfail => false) + extracted_key = extracted_key.chomp + + unless extracted_key.match(/^#{name}$/) + logger.fail("The id in your manifest #{key[:name]} and the fingerprint from content/source do not match. Please check there is not an error in the id or check the content/source is legitimate.") + end + else + logger.warning('/usr/bin/gpg cannot be found for verification of the id.') + end + end + yield file.path + ensure + file.close + file.unlink + end +end + + +set(current_state, target_state) diff --git a/language/resource-api/simple_apt.rb b/language/resource-api/simple_apt.rb index e5c72ee..e848c68 100644 --- a/language/resource-api/simple_apt.rb +++ b/language/resource-api/simple_apt.rb @@ -147,7 +147,7 @@ def self.key_line_to_hash(pub_line, fpr_line) type: key_type, created: Time.at(pub_split[5].to_i), expiry: expiry, - expired: expiry && Time.now >= expiry, + expired: !!(expiry && Time.now >= expiry), } end diff --git a/language/resource-api/simpleresource.rb b/language/resource-api/simpleresource.rb index 7282a7f..4dac5f9 100644 --- a/language/resource-api/simpleresource.rb +++ b/language/resource-api/simpleresource.rb @@ -1,22 +1,22 @@ Puppet::SimpleResource.define( - name: 'iis_application_pool', - docs: 'Manage an IIS application pool through a powershell proxy.', + name: 'iis_application_pool', + docs: 'Manage an IIS application pool through a powershell proxy.', attributes: { - ensure: { + ensure: { type: 'Enum[present, absent]', docs: 'Whether this ApplicationPool should be present or absent on the target system.' }, - name: { - type: 'String', - docs: 'The name of the ApplicationPool.', + name: { + type: 'String', + docs: 'The name of the ApplicationPool.', namevar: true, }, - state: { - type: 'Enum[running, stopped]', - docs: 'The state of the ApplicationPool.', + state: { + type: 'Enum[running, stopped]', + docs: 'The state of the ApplicationPool.', default: 'running', }, - managedpipelinemode: { + managedpipelinemode: { type: 'String', docs: 'The managedPipelineMode of the ApplicationPool.', }, @@ -25,20 +25,23 @@ docs: 'The managedRuntimeVersion of the ApplicationPool.', }, } -) do +) +Puppet::SimpleResource.implement('iis_application_pool') do require 'puppet/provider/iis_powershell' include Puppet::Provider::IIS_PowerShell def get - result = run('fetch_application_pools.ps1', logger) # call out to powershell to talk to the API + # call out to PowerShell to talk to the API + result = run('fetch_application_pools.ps1', logger) # returns an array of hashes with data according to the schema above JSON.parse(result) end - def set(goals, noop = false) - result = run('enforce_application_pools.ps1', goals, logger, noop) # call out to powershell to talk to the API + def set(current_state, target_state, noop = false) + # call out to PowerShell to talk to the API + result = run('enforce_application_pools.ps1', JSON.generate(current_state), JSON.generate(target_state), logger, noop) # returns an array of hashes with status data from the changes JSON.parse(result) From 5ee1e2209bb5191d2c7f86b022d1b2073e3eb01f Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 31 Jan 2017 15:52:05 +0000 Subject: [PATCH 05/62] More touchups --- language/resource-api/README.md | 2 +- language/resource-api/simpleresource.rb | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 395da62..0110ba4 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -1,4 +1,4 @@ -Draft replacement for types and providers +Draft for new type and provider API Hi *, diff --git a/language/resource-api/simpleresource.rb b/language/resource-api/simpleresource.rb index 4dac5f9..7ded072 100644 --- a/language/resource-api/simpleresource.rb +++ b/language/resource-api/simpleresource.rb @@ -28,6 +28,7 @@ ) Puppet::SimpleResource.implement('iis_application_pool') do + # hiding all the nasty bits require 'puppet/provider/iis_powershell' include Puppet::Provider::IIS_PowerShell From ecc3b4ec5ee864bf405b687e15b52be5c14132ae Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Sun, 19 Feb 2017 14:19:13 +0000 Subject: [PATCH 06/62] Add proper README for resource API; update test implementations --- language/resource-api/README.md | 257 +++++++++++++++++++++++++++ language/resource-api/apt_key_set.rb | 17 ++ language/resource-api/simple_apt.rb | 98 +++++----- 3 files changed, 325 insertions(+), 47 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 0110ba4..07f768d 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -1,3 +1,260 @@ +# Resource API + +A *resource* is the basic thing that is managed by puppet. Each resource has a set of attributes describing its current state. Some of the attributes can be changed throughout the life-time of the resource, some attributes only report back, but cannot be changed (see `read_only`)others can only be set once during initial creation (see `init_only`). To gather information about those resources, and to enact changes in the real world, puppet requires a piece of code to *implement* this interaction. The implementation can have parameters that influence its mode of operation (see `operational_parameters`). To describe all these parts to the infrastructure, and the consumers, the resource *Definition* (f.k.a. 'type') contains definitions for all of them. The *Implementation* (f.k.a. 'provider') contains code to *get* and *set* the system state. + +# Resource Definition + +```ruby +Puppet::ResourceDefinition.register( + name: 'apt_key', + docs: <<-EOS, + This type provides Puppet with the capabilities to manage GPG keys needed + by apt to perform package validation. Apt has it's own GPG keyring that can + be manipulated through the `apt-key` command. + + apt_key { '6F6B15509CF8E59E6E469F327F438280EF8D349F': + source => 'http://apt.puppetlabs.com/pubkey.gpg' + } + + **Autorequires**: + If Puppet is given the location of a key file which looks like an absolute + path this type will autorequire that file. + EOS + attributes: { + ensure: { + type: 'Enum[present, absent]', + docs: 'Whether this apt key should be present or absent on the target system.' + }, + id: { + type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]', + docs: 'The ID of the key you want to manage.', + namevar: true, + }, + # ... + created: { + type: 'String', + docs: 'Date the key was created, in ISO format.', + read_only: true, + }, + }, + autorequires: { + file: '$source', # will evaluate to the value of the `source` attribute + package: 'apt', + }, +) +``` + +The `Puppet::ResourceDefinition.register(options)` function takes a Hash with the following top-level keys: + +* `name`: the name of the resource. For autoloading to work, the whole function call needs to go into `lib/puppet/type/.rb`. +* `docs`: a doc string that describes the overall working of the type, gives examples, and explains pre-requisites as well as known issues. +* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `docs` string, and whether the attribute is the `namevar`, `read_only`, and/or `init_only`. +* `operational_parameters`: a hash mapping parameter names to puppet data `type`s and `docs` strings. +* `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a Hash mapping resource types to titles. Currently the titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. + +# Resource Implementation + +At runtime the current and intended system states for a single resource instance are always represented as ruby Hashes of the resource's attributes, and applicable operational parameters. + +The two fundamental operations to manage resources are reading and writing system state. These operations are implemented in the `ResourceImplementation` as `get` and `set`: + +```ruby +Puppet::ResourceImplementation.register('apt_key') do + def get + [ + 'title': { + name: 'title', + # ... + }, + ] + end + + def set(current_state, target_state, noop: false) + target_state.each do |title, resource| + # ... + end + end +end +``` + +The `get` method returns a Hash of all resources currently available, keyed by their title. If the `get` method raises an exception, the implementation is marked as unavailable during the current run, and all resources of its type will fail. The error message will be reported to the user. + +The `set` method updates resources to the state defined in `target_state`. For convenience, `current_state` contains the last available system state from a prior `get` call. When `noop` is set to true, the implementation must not change the system state, but only report what it would change. + + +# Runtime Environment + +The primary runtime environment for the implementation is the puppet agent, a long-running daemon process. The implementation can also be used in the puppet apply command, a one-shot version of the agent, or the puppet resource command, a short-lived CLI process for listing or managing a single resource type. Other callers who want to access the implementation will have to emulate those environments. In any case the registered block will be surfaced in a clean class which will be instantiated once for each transaction. The implementation can define any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for an implementation, and use it immediately, the provider is instantiated as late as possible. A transaction will usually call `get` once, and may call set any number of times to effect change. The host object can be used to cache ephemeral state during execution. The implementation should not try to cache state beyond the transaction, to avoid interfering with the agent daemon. In many other cases caching beyond the transaction won't help anyways, as the hosting process will only manage a single transaction. + +## Utilities + +The runtime environment provides some utilities to make the implementation's life easier, and provide a uniform experience for its users. + +### Commands + +To use CLI commands in a safe and comfortable manner, the implementation can use the `commands` method to access shell commands. You can either use a full path, or a bare command name. In the latter case puppet will use the system PATH setting to search for the command. If the commands are not available, an error will be raised and the resources will fail in this run. The commands are aware of whether noop is in effect or not, and will skip the actual execution if necessary. + +```ruby +Puppet::ResourceImplementation.register('apt_key') do + commands apt_key: '/usr/bin/apt-key' + commands gpg: 'gpg' +``` + +This will create methods called `apt_get`, and `gpg`, which will take CLI arguments without any escaping, and run them in a safe environment (clean working directory, clean environment). For example to call `apt-key` to delete a specific key by id: + +```ruby +apt_key 'del', key_id +``` + +By default the stdout of the command is logged to debug, while the stderr is logged to warning. To access the stdout in the implementation, use the command name with `_lines` appended, and process it through the returned [Enumerable](http://ruby-doc.org/core/Enumerable.html) line-by-line. For example, to process the list of all apt keys: + +```ruby +apt_key_lines(%w{adv --list-keys --with-colons --fingerprint --fixed-list-mode}).collect do |line| + # process each line here, and return a result +end +``` + +> Note: the output of the command is streamed through the Enumerable. If the implementation requires the exit value of the command before processing, or wants to cache the output, use `to_a` to read the complete stream in one go. + +If the command returns a non-zero exit code, an error is signalled to puppet. If this happens during `get`, all managed resources of this type will fail. If this happens during a `set`, all resources that have been scheduled for processing in this call, but not yet have been marked as a success will be marked as failed. To avoid this behaviour, call the `try_` prefix variant. In this (hypothetical) example, `apt-key` signals already deleted keys with an exit code of `1`, which is OK when the implementation is trying to delete the key: + +```ruby +try_apt_key 'del', key_id + +if [0, 1].contains $?.exitstatus + # success, or already deleted +else + # fail +end +``` + +The exit code is signalled through the ruby standard variable `$?` as a [`Process::Status` object](https://ruby-doc.org/core/Process/Status.html) + +### Logging and Reporting + +The implementation needs to signal changes, successes and failures to the runtime environment. The `logger` provides a structured way to do so. + +#### General messages + +To provide feedback about the overall operation of the implementation, the logger provides the usual set of [loglevel](https://docs.puppet.com/puppet/latest/metaparameter.html#loglevel) methods that take a string, and pass that up to puppet's logging infrastructure: + +```ruby +logger.warning("Unexpected state detected, continuing in degraded mode.") +``` + +will result in the following message: + +```text +Warning: apt_key: Unexpected state detected, continuing in degraded mode. +``` + +* debug: detailed messages to understand everything that is happening at runtime; only shown on request +* info: high level progress messages; especially useful before long-running operations, or before operations that can fail, to provide context to interactive users +* notice: use this loglevel to indicate state changes and similar events of notice from the regular operations of the implementation +* warning: signal error conditions that do not (yet) prohibit execution of the main part of the implementation; for example deprecation warnings, temporary errors. +* err: signal error conditions that have caused normal operations to fail +* critical/alert/emerg: should not be used by resource implementations + +See [wikipedia](https://en.wikipedia.org/wiki/Syslog#Severity_level) and [RFC424](https://tools.ietf.org/html/rfc5424) for more details. + +#### Logging contexts + +Most of an implementation's messages are expected to be relative to a specific resource instance, and a specific operation on that instance. For example, to report the change of an attribute: + +```ruby +logger.attribute_changed(title:, attribute:, old_value:, new_value:, message: "Changed #{attribute} from #{old_value.inspect} to #{newvalue.inspect}") +``` + +To enable detailed logging without repeating key arguments, and provide consistent error logging, the logger provides *logging context* methods that capture the current action and resource instance. + +```ruby +logger.updating(title: title) do + if key_not_found + logger.warning('Original key not found') + end + + # Update the key by calling CLI tool + apt_key(...) + + logger.attribute_changed( + attribute: 'content', + old_value: nil, + new_value: content_hash, + message: "Created with content hash #{content_hash}") +end +``` + +will result in the following messages (of course, with the `#{}` sequences replaced by the true values): + +```text +Debug: Apt_key[#{title}]: Started updating +Warning: Apt_key[#{title}]: Updating: Original key not found +Debug: Apt_key[#{title}]: Executing 'apt-key ...' +Debug: Apt_key[#{title}]: Successfully executed 'apt-key ...' +Notice: Apt_key[#{title}]: Updating content: Created with content hash #{content_hash} +Notice: Apt_key[#{title}]: Successfully updated +# TODO: update messages to match current log message formats for resource messages +``` + +In the case of an exception escaping the block, the error is logged appropriately: + +```text +Debug: Apt_key[#{title}]: Started updating +Warning: Apt_key[#{title}]: Updating: Original key not found +Error: Apt_key[#{title}]: Updating failed: #{exception message} +# TODO: update messages to match current log message formats for resource messages +``` + +Logging contexts process all exceptions. [`StandardError`s](https://ruby-doc.org/core/StandardError.html) are assumed to be regular failures in handling a resources, and they are swallowed after logging. Everything else is assumed to be a fatal application-level issue, and is passed up the stack, ending execution. See the [ruby documentation](https://ruby-doc.org/core/Exception.html) for details on which exceptions are not `StandardError`s. + +The equivalent long-hand form with manual error handling: + +```ruby +logger.updating(title: title) +begin + if key_not_found + logger.warning(title: title, message: 'Original key not found') + end + + # Update the key by calling CLI tool + try_apt_key(...) + + if $?.exitstatus != 0 + logger.error(title: title, "Failed executing apt-key #{...}") + else + logger.attribute_changed( + title: title, + attribute: 'content', + old_value: nil, + new_value: content_hash, + message: "Created with content hash #{content_hash}") + end + logger.changed(title: title) +rescue StandardError => e + logger.error(title: title, exception: e, message: 'Updating failed') + raise unless e.is_a? StandardError +end +``` + +This example is only for demonstration purposes. In the normal course of operations, implementations should always use the utility functions. + +#### Logging reference + +The following action/context methods are available: + +* `creating(title:, message: 'Creating', &block)` +* `updating(title:, message: 'Updating', &block)` +* `deleting(title:, message: 'Deleting', &block)` +* `attribute_changed(title:, attribute:, old_value:, new_value:, message: nil)` + +* `created(title:, message: 'Created')` +* `updated(title:, message: 'Updated')` +* `deleted(title:, message: 'Deleted')` + +* `fail(title:, message:)` - abort the current context with an error + +# Earlier notes + Draft for new type and provider API Hi *, diff --git a/language/resource-api/apt_key_set.rb b/language/resource-api/apt_key_set.rb index b3fa347..6f1f760 100644 --- a/language/resource-api/apt_key_set.rb +++ b/language/resource-api/apt_key_set.rb @@ -184,6 +184,23 @@ def set(current_state, target_state, noop = false) end end +def set(current_state, target_state, noop = false) + existing_keys = Hash[current_state.collect { |k| [k[:name], k] }] + target_state.each do |resource| + # additional validation for this resource goes here + + current = existing_keys[resource[:name]] + if current && resource[:ensure].to_s == 'absent' + # delete the resource + elsif current && resource[:ensure].to_s == 'present' + # update the resource + elsif !current && resource[:ensure].to_s == 'present' + # create the resource + end + end +end + + def create(key, noop = false) logger.creating(key[:name]) do |logger| if key[:source].nil? and key[:content].nil? diff --git a/language/resource-api/simple_apt.rb b/language/resource-api/simple_apt.rb index e848c68..1346444 100644 --- a/language/resource-api/simple_apt.rb +++ b/language/resource-api/simple_apt.rb @@ -41,17 +41,17 @@ docs: 'Additional options to pass to apt-key\'s --keyserver-options.', }, fingerprint: { - type: 'String', + type: 'String[40, 40]', docs: 'The 40-digit hexadecimal fingerprint of the specified GPG key.', read_only: true, }, long: { - type: 'String', + type: 'String[16, 16]', docs: 'The 16-digit hexadecimal id of the specified GPG key.', read_only: true, }, short: { - type: 'String', + type: 'String[8, 8]', docs: 'The 8-digit hexadecimal id of the specified GPG key.', read_only: true, }, @@ -66,7 +66,7 @@ read_only: true, }, size: { - type: 'String', + type: 'Integer', docs: 'The key size, usually a multiple of 1024.', read_only: true, }, @@ -97,7 +97,7 @@ def get pub_line = nil fpr_line = nil - key_output.split("\n").collect do |line| + kv_pairs = key_output.split("\n").collect do |line| if line.start_with?('pub') pub_line = line elsif line.start_with?('fpr') @@ -106,7 +106,7 @@ def get next unless (pub_line and fpr_line) - result = key_line_to_hash(pub_line, fpr_line) + result = key_line_to_kv_pair(pub_line, fpr_line) # reset everything pub_line = nil @@ -114,9 +114,11 @@ def get result end.compact! + + Hash[kv_pairs] end - def self.key_line_to_hash(pub_line, fpr_line) + def self.key_line_to_kv_pair(pub_line, fpr_line) pub_split = pub_line.split(':') fpr_split = fpr_line.split(':') @@ -137,77 +139,79 @@ def self.key_line_to_hash(pub_line, fpr_line) fingerprint = fpr_split.last expiry = pub_split[6].empty? ? nil : Time.at(pub_split[6].to_i) - { - name: fingerprint, - ensure: 'present', - fingerprint: fingerprint, - long: fingerprint[-16..-1], # last 16 characters of fingerprint - short: fingerprint[-8..-1], # last 8 characters of fingerprint - size: pub_split[2], - type: key_type, - created: Time.at(pub_split[5].to_i), - expiry: expiry, - expired: !!(expiry && Time.now >= expiry), - } + [ + fingerprint, + { + ensure: 'present', + id: fingerprint, + fingerprint: fingerprint, + long: fingerprint[-16..-1], # last 16 characters of fingerprint + short: fingerprint[-8..-1], # last 8 characters of fingerprint + size: pub_split[2].to_i, + type: key_type, + created: Time.at(pub_split[5].to_i), + expiry: expiry, + expired: !!(expiry && Time.now >= expiry), + } + ] end def set(current_state, target_state, noop = false) - existing_keys = Hash[current_state.collect { |k| [k[:name], k] }] - target_state.each do |key| - logger.warning(key[:name], 'The id should be a full fingerprint (40 characters) to avoid collision attacks, see the README for details.') if key[:name].length < 40 - if key[:source] and key[:content] - logger.fail(key[:name], 'The properties content and source are mutually exclusive') + target_state.each do |title, resource| + logger.warning(title, 'The id should be a full fingerprint (40 characters) to avoid collision attacks, see the README for details.') if title.length < 40 + if resource[:source] and resource[:content] + logger.fail(title, 'The properties content and source are mutually exclusive') next end - current = existing_keys[k[:name]] - if current && key[:ensure].to_s == 'absent' - logger.deleting(key[:name]) do + current = current_state[title] + if current && resource[:ensure].to_s == 'absent' + logger.deleting(title) do begin - apt_key('del', key[:short], noop: noop) - r = execute(["#{command(:apt_key)} list | grep '/#{resource.provider.short}\s'"], :failonfail => false) + apt_key('del', resource[:short], noop: noop) + r = execute(["#{command(:apt_key)} list | grep '/#{resource[:short]}\s'"], :failonfail => false) end while r.exitstatus == 0 end - elsif current && key[:ensure].to_s == 'present' + elsif current && resource[:ensure].to_s == 'present' # No updating implemented # update(key, noop: noop) - elsif !current && key[:ensure].to_s == 'present' - create(key, noop: noop) + elsif !current && resource[:ensure].to_s == 'present' + create(title, resource, noop: noop) end end end - def create(key, noop = false) - logger.creating(key[:name]) do |logger| - if key[:source].nil? and key[:content].nil? + def create(title, resource, noop = false) + logger.creating(title) do |logger| + if resource[:source].nil? and resource[:content].nil? # Breaking up the command like this is needed because it blows up # if --recv-keys isn't the last argument. - args = ['adv', '--keyserver', key[:server]] - if key[:options] - args.push('--keyserver-options', key[:options]) + args = ['adv', '--keyserver', resource[:server]] + if resource[:options] + args.push('--keyserver-options', resource[:options]) end - args.push('--recv-keys', key[:id]) + args.push('--recv-keys', resource[:id]) apt_key(*args, noop: noop) - elsif key[:content] - temp_key_file(key[:content], logger) do |key_file| + elsif resource[:content] + temp_key_file(resource[:content], logger) do |key_file| apt_key('add', key_file, noop: noop) end - elsif key[:source] - key_file = source_to_file(key[:source]) + elsif resource[:source] + key_file = source_to_file(resource[:source]) apt_key('add', key_file.path, noop: noop) # In case we really screwed up, better safe than sorry. else - logger.fail("an unexpected condition occurred while trying to add the key: #{key[:id]} (content: #{key[:content].inspect}, source: #{key[:source].inspect})") + logger.fail("an unexpected condition occurred while trying to add the key: #{title} (content: #{resource[:content].inspect}, source: #{resource[:source].inspect})") end end end # This method writes out the specified contents to a temporary file and # confirms that the fingerprint from the file, matches the long key that is in the manifest - def temp_key_file(key, logger) + def temp_key_file(resource, logger) file = Tempfile.new('apt_key') begin - file.write key[:content] + file.write resource[:content] file.close if name.size == 40 if File.executable? command(:gpg) @@ -215,7 +219,7 @@ def temp_key_file(key, logger) extracted_key = extracted_key.chomp unless extracted_key.match(/^#{name}$/) - logger.fail("The id in your manifest #{key[:name]} and the fingerprint from content/source do not match. Please check there is not an error in the id or check the content/source is legitimate.") + logger.fail("The id in your manifest #{resource[:id]} and the fingerprint from content/source do not match. Please check there is not an error in the id or check the content/source is legitimate.") end else logger.warning('/usr/bin/gpg cannot be found for verification of the id.') From c05f1f2adf475b0f08ebcaaa7ab5529006de3863 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Sun, 19 Feb 2017 17:56:22 +0000 Subject: [PATCH 07/62] apt_key prototype shim This works for puppet resource and puppet apply. --- language/resource-api/apt_key.rb | 364 +++++++++++++++++++++++++++++++ 1 file changed, 364 insertions(+) create mode 100644 language/resource-api/apt_key.rb diff --git a/language/resource-api/apt_key.rb b/language/resource-api/apt_key.rb new file mode 100644 index 0000000..59eb510 --- /dev/null +++ b/language/resource-api/apt_key.rb @@ -0,0 +1,364 @@ +require 'puppet/pops/patterns' +require 'puppet/pops/utils' + +require 'pry' + +DEFINITION = { + name: 'apt_key', + docs: <<-EOS, + This type provides Puppet with the capabilities to manage GPG keys needed + by apt to perform package validation. Apt has it's own GPG keyring that can + be manipulated through the `apt-key` command. + + apt_key { '6F6B15509CF8E59E6E469F327F438280EF8D349F': + source => 'http://apt.puppetlabs.com/pubkey.gpg' + } + + **Autorequires**: + If Puppet is given the location of a key file which looks like an absolute + path this type will autorequire that file. + EOS + attributes: { + ensure: { + type: 'Enum[present, absent]', + docs: 'Whether this apt key should be present or absent on the target system.' + }, + id: { + type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]', + docs: 'The ID of the key you want to manage.', + namevar: true, + }, + content: { + type: 'Optional[String]', + docs: 'The content of, or string representing, a GPG key.', + }, + source: { + type: 'Variant[Stdlib::Absolutepath, Pattern[/\A(https?|ftp):\/\//]]', + docs: 'Location of a GPG key file, /path/to/file, ftp://, http:// or https://', + }, + server: { + type: 'Pattern[/\A((hkp|http|https):\/\/)?([a-z\d])([a-z\d-]{0,61}\.)+[a-z\d]+(:\d{2,5})?$/]', + docs: 'The key server to fetch the key from based on the ID. It can either be a domain name or url.', + default: 'keyserver.ubuntu.com' + }, + options: { + type: 'Optional[String]', + docs: 'Additional options to pass to apt-key\'s --keyserver-options.', + }, + fingerprint: { + type: 'String', + docs: 'The 40-digit hexadecimal fingerprint of the specified GPG key.', + read_only: true, + }, + long: { + type: 'String', + docs: 'The 16-digit hexadecimal id of the specified GPG key.', + read_only: true, + }, + short: { + type: 'String', + docs: 'The 8-digit hexadecimal id of the specified GPG key.', + read_only: true, + }, + expired: { + type: 'Boolean', + docs: 'Indicates if the key has expired.', + read_only: true, + }, + expiry: { + # TODO: should be DateTime + type: 'String', + docs: 'The date the key will expire, or nil if it has no expiry date, in ISO format.', + read_only: true, + }, + size: { + type: 'Integer', + docs: 'The key size, usually a multiple of 1024.', + read_only: true, + }, + type: { + type: 'String', + docs: 'The key type, one of: rsa, dsa, ecc, ecdsa.', + read_only: true, + }, + created: { + type: 'String', + docs: 'Date the key was created, in ISO format.', + read_only: true, + }, + }, + autorequires: { + file: '$source', # will evaluate to the value of the `source` attribute + package: 'apt', + }, +} + +module Puppet::SimpleResource + class TypeShim + attr_reader :values + + def initialize(title, resource_hash) + # internalize and protect - needs to go deeper + @values = resource_hash.dup + # "name" is a privileged key + @values[:name] = title + @values.freeze + end + + def to_resource + ResourceShim.new(@values) + end + + def name + values[:name] + end + end + + class ResourceShim + attr_reader :values + + def initialize(resource_hash) + @values = resource_hash.dup.freeze # whatevs + end + + def title + values[:name] + end + + def prune_parameters(*args) + puts "not pruning #{args.inspect}" if args.length > 0 + self + end + + def to_manifest + [ + "apt_key { #{values[:name].inspect}: ", + ] + values.keys.select { |k| k != :name }.collect { |k| " #{k} => #{values[k].inspect}," } + ['}'] + end + end +end + +Puppet::Type.newtype(DEFINITION[:name].to_sym) do + @doc = DEFINITION[:docs] + + has_namevar = false + + DEFINITION[:attributes].each do |name, options| + puts "#{name}: #{options.inspect}" + + # TODO: using newparam everywhere would suppress change reporting + # that would allow more fine-grained reporting through logger, + # but require more invest in hooking up the infrastructure to emulate existing data + param_or_property = if options[:read_only] || options[:namevar] + :newparam + else + :newproperty + end + send(param_or_property, name.to_sym) do + unless options[:type] + fail("#{DEFINITION[:name]}.#{name} has no type") + end + + if options[:docs] + desc "#{options[:docs]} (a #{options[:type]}" + else + warn("#{DEFINITION[:name]}.#{name} has no docs") + end + + if options[:namevar] + puts 'setting namevar' + isnamevar + has_namevar = true + end + + # read-only values do not need type checking + if not options[:read_only] + # TODO: this should use Pops infrastructure to avoid hardcoding stuff, and enhance type fidelity + case options[:type] + when 'String' + # require any string value + newvalue // do + end + when 'Boolean' + ['true', 'false', :true, :false, true, false].each do |v| + newvalue v do + end + end + + munge do |v| + case v + when 'true', :true + true + when 'false', :false + false + else + v + end + end + when 'Integer' + newvalue /^\d+$/ do + end + munge do |v| + Puppet::Pops::Utils.to_n(v) + end + when 'Float', 'Numeric' + newvalue Puppet::Pops::Patterns::NUMERIC do + end + munge do |v| + Puppet::Pops::Utils.to_n(v) + end + when 'Enum[present, absent]' + newvalue :absent do + end + newvalue :present do + end + when 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]' + # the namevar needs to be a Parameter, which only has newvalue*s* + newvalues(/\A(0x)?[0-9a-fA-F]{8}\Z/, /\A(0x)?[0-9a-fA-F]{16}\Z/, /\A(0x)?[0-9a-fA-F]{40}\Z/) + when 'Optional[String]' + newvalue :undef do + end + newvalue // do + end + when 'Variant[Stdlib::Absolutepath, Pattern[/\A(https?|ftp):\/\//]]' + # TODO: this is wrong, but matches original implementation + [/^\//, /\A(https?|ftp):\/\//].each do |v| + newvalue v do + end + end + when /^(Enum|Optional|Variant)/ + fail("#{$1} is not currently supported") + end + end + end + end + + unless has_namevar + fail("#{DEFINITION[:name]} has no namevar") + end + + def self.fake_system_state + @fake_system_state ||= { + 'BBCB188AD7B3228BCF05BD554C0BE21B5FF054BD' => { + ensure: :present, + fingerprint: 'BBCB188AD7B3228BCF05BD554C0BE21B5FF054BD', + long: '4C0BE21B5FF054BD', + short: '5FF054BD', + size: 2048, + type: :rsa, + created: '2013-06-07 23:55:31 +0100', + expiry: nil, + expired: false, + }, + 'B71ACDE6B52658D12C3106F44AB781597254279C' => { + ensure: :present, + fingerprint: 'B71ACDE6B52658D12C3106F44AB781597254279C', + long: '4AB781597254279C', + short: '7254279C', + size: 1024, + type: :dsa, + created: '2007-03-08 20:17:10 +0000', + expiry: nil, + expired: false + }, + '9534C9C4130B4DC9927992BF4F30B6B4C07CB649' => { + ensure: :present, + fingerprint: '9534C9C4130B4DC9927992BF4F30B6B4C07CB649', + long: '4F30B6B4C07CB649', + short: 'C07CB649', + size: 4096, + type: :rsa, + created: '2014-11-21 21:01:13 +0000', + expiry: '2022-11-19 21:01:13 +0000', + expired: false + }, + '126C0D24BD8A2942CC7DF8AC7638D0442B90D010' => { + ensure: :present, + fingerprint: '126C0D24BD8A2942CC7DF8AC7638D0442B90D010', + long: '7638D0442B90D010', + short: '2B90D010', + size: 4096, + type: :rsa, + created: '2014-11-21 21:13:37 +0000', + expiry: '2022-11-19 21:13:37 +0000', + expired: false + }, + 'ED6D65271AACF0FF15D123036FB2A1C265FFB764' => { + ensure: :present, + fingerprint: 'ED6D65271AACF0FF15D123036FB2A1C265FFB764', + long: '6FB2A1C265FFB764', + short: '65FFB764', + size: 4096, + type: :rsa, + created: '2010-07-10 01:13:52 +0100', + expiry: '2017-01-05 00:06:37 +0000', + expired: true + }, + } + end + + def self.get + puts 'get' + fake_system_state + end + + def self.set(current_state, target_state, noop = false) + puts "enforcing change from #{current_state} to #{target_state} (noop=#{noop})" + target_state.each do |title, resource| + # additional validation for this resource goes here + + # set default value + resource[:ensure] ||= :present + + current = current_state[title] + if current && resource[:ensure].to_s == 'absent' + # delete the resource + puts "deleting #{title}" + fake_system_state.delete_if { |k, _| k==title } + elsif current && resource[:ensure].to_s == 'present' + # update the resource + puts "updating #{title}" + resource = current.merge(resource) + fake_system_state[title] = resource.dup + elsif !current && resource[:ensure].to_s == 'present' + # create the resource + puts "creating #{title}" + fake_system_state[title] = resource.dup + end + # TODO: update Type's notion of reality to ensure correct puppet resource output with all available attributes + end + end + + def self.instances + puts 'instances' + # klass = Puppet::Type.type(:api) + get.collect do |title, resource_hash| + Puppet::SimpleResource::TypeShim.new(title, resource_hash) + end + end + + def retrieve + puts 'retrieve' + result = Puppet::Resource.new(self.class, title) + current_state = self.class.get[title] + + if current_state + current_state.each do |k, v| + result[k]=v + end + else + result[:ensure] = :absent + end + + @rapi_current_state = current_state + result + end + + def flush + puts 'flush' + # binding.pry + target_state = Hash[@parameters.collect { |k, v| [k, v.value] }] + self.class.set({title => @rapi_current_state}, {title => target_state}, false) + end + +end From 4fcc5249471f02d19a52bc5d94123b8ecfae6dc7 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Fri, 3 Mar 2017 13:31:24 +0000 Subject: [PATCH 08/62] Update readme with feedback from puppet-dev --- language/resource-api/README.md | 83 ++++++++++++++++++++++++++++++--- 1 file changed, 77 insertions(+), 6 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 07f768d..a820072 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -1,6 +1,6 @@ # Resource API -A *resource* is the basic thing that is managed by puppet. Each resource has a set of attributes describing its current state. Some of the attributes can be changed throughout the life-time of the resource, some attributes only report back, but cannot be changed (see `read_only`)others can only be set once during initial creation (see `init_only`). To gather information about those resources, and to enact changes in the real world, puppet requires a piece of code to *implement* this interaction. The implementation can have parameters that influence its mode of operation (see `operational_parameters`). To describe all these parts to the infrastructure, and the consumers, the resource *Definition* (f.k.a. 'type') contains definitions for all of them. The *Implementation* (f.k.a. 'provider') contains code to *get* and *set* the system state. +A *resource* is the basic thing that is managed by puppet. Each resource has a set of attributes describing its current state. Some of the attributes can be changed throughout the life-time of the resource, some attributes only report back, but cannot be changed (see `read_only`) others can only be set once during initial creation (see `init_only`). To gather information about those resources, and to enact changes in the real world, puppet requires a piece of code to *implement* this interaction. The implementation can have parameters that influence its mode of operation (see `parameter`). To describe all these parts to the infrastructure, and the consumers, the resource *Definition* (f.k.a. 'type') contains definitions for all of them. The *Implementation* (f.k.a. 'provider') contains code to *get* and *set* the system state. # Resource Definition @@ -48,8 +48,11 @@ The `Puppet::ResourceDefinition.register(options)` function takes a Hash with th * `name`: the name of the resource. For autoloading to work, the whole function call needs to go into `lib/puppet/type/.rb`. * `docs`: a doc string that describes the overall working of the type, gives examples, and explains pre-requisites as well as known issues. -* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `docs` string, and whether the attribute is the `namevar`, `read_only`, and/or `init_only`. -* `operational_parameters`: a hash mapping parameter names to puppet data `type`s and `docs` strings. +* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `docs` string, and whether the attribute is the `namevar`, `read_only`, `init_only`, or a `parameter`. + * `namevar`: marks an attribute as part of the "primary key", or "identity" of the resource. A given set of namevar values needs to distinctively identify a instance. + * `init_only`: this attribute can only be set during creation of the resource. Its value will be reported going forward, but trying to change it later will lead to an error. For example, the base image for a VM, or the UID of a user. + * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example the checksum of a file, or the MAC address of a network interface. + * `parameter`: these attributes influence how the implementation behaves, and cannot be read from the target system. For example, the target file on inifile, or credentials to access an API. * `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a Hash mapping resource types to titles. Currently the titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. # Resource Implementation @@ -84,7 +87,7 @@ The `set` method updates resources to the state defined in `target_state`. For c # Runtime Environment -The primary runtime environment for the implementation is the puppet agent, a long-running daemon process. The implementation can also be used in the puppet apply command, a one-shot version of the agent, or the puppet resource command, a short-lived CLI process for listing or managing a single resource type. Other callers who want to access the implementation will have to emulate those environments. In any case the registered block will be surfaced in a clean class which will be instantiated once for each transaction. The implementation can define any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for an implementation, and use it immediately, the provider is instantiated as late as possible. A transaction will usually call `get` once, and may call set any number of times to effect change. The host object can be used to cache ephemeral state during execution. The implementation should not try to cache state beyond the transaction, to avoid interfering with the agent daemon. In many other cases caching beyond the transaction won't help anyways, as the hosting process will only manage a single transaction. +The primary runtime environment for the implementation is the puppet agent, a long-running daemon process. The implementation can also be used in the puppet apply command, a one-shot version of the agent, or the puppet resource command, a short-lived CLI process for listing or managing a single resource type. Other callers who want to access the implementation will have to emulate those environments. In any case the registered block will be surfaced in a clean class which will be instantiated once for each transaction. The implementation can define any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for an implementation, and use it immediately, the provider is instantiated as late as possible. A transaction will usually call `get` once, and may call set any number of times to effect change. The object instance hosting the `get` and `set` methods can be used to cache ephemeral state during execution. The implementation should not try to cache state beyond the transaction, to avoid interfering with the agent daemon. In many other cases caching beyond the transaction won't help anyways, as the hosting process will only manage a single transaction. ## Utilities @@ -92,7 +95,7 @@ The runtime environment provides some utilities to make the implementation's lif ### Commands -To use CLI commands in a safe and comfortable manner, the implementation can use the `commands` method to access shell commands. You can either use a full path, or a bare command name. In the latter case puppet will use the system PATH setting to search for the command. If the commands are not available, an error will be raised and the resources will fail in this run. The commands are aware of whether noop is in effect or not, and will skip the actual execution if necessary. +To use CLI commands in a safe and comfortable manner, the implementation can use the `commands` method to access shell commands. You can either use a full path, or a bare command name. In the latter case puppet will use the system PATH setting to search for the command. If the commands are not available, an error will be raised and the resources will fail in this run. The commands are aware of whether noop is in effect or not, and will signal success while skipping the real execution if necessary. ```ruby Puppet::ResourceImplementation.register('apt_key') do @@ -100,12 +103,18 @@ Puppet::ResourceImplementation.register('apt_key') do commands gpg: 'gpg' ``` -This will create methods called `apt_get`, and `gpg`, which will take CLI arguments without any escaping, and run them in a safe environment (clean working directory, clean environment). For example to call `apt-key` to delete a specific key by id: +This will create methods called `apt_get`, and `gpg`, which will take CLI arguments in an array, and execute the command directly without any shell processing in a safe environment (clean working directory, clean environment). For example to call `apt-key` to delete a specific key by id: ```ruby apt_key 'del', key_id ``` +To pass additional environment variables through to the command, pass a hash of them as `env:`: + +```ruby +apt_key 'del', key_id, env: { 'LC_ALL': 'C' } +``` + By default the stdout of the command is logged to debug, while the stderr is logged to warning. To access the stdout in the implementation, use the command name with `_lines` appended, and process it through the returned [Enumerable](http://ruby-doc.org/core/Enumerable.html) line-by-line. For example, to process the list of all apt keys: ```ruby @@ -130,6 +139,8 @@ end The exit code is signalled through the ruby standard variable `$?` as a [`Process::Status` object](https://ruby-doc.org/core/Process/Status.html) + + ### Logging and Reporting The implementation needs to signal changes, successes and failures to the runtime environment. The `logger` provides a structured way to do so. @@ -253,6 +264,66 @@ The following action/context methods are available: * `fail(title:, message:)` - abort the current context with an error +# Known Limitations + +This API is not a full replacement for the power of 3.x style types and providers. Here is a (incomplete) list of missing pieces and thoughts on how to go about solving these. In the end, the goal of the new Resource API is not to be a complete replacement of prior art, but a cleaner way to get good results for the majority of simple cases. + +## Multiple implementations + +The previous version of this API allowed multiple implementations for the same resource type. This leads to the following problems: + +* attribute sprawl +* missing features +* convoluted implementations + +puppet DSL already can address this: + +```puppet +define package ( + Ensure $ensure, + Enum[apt, rpm] $provider, # have a hiera 5 dynamic binding to a function choosing a sensible default for the current system + Optional[String] $source = undef, + Optional[String] $version = undef, + Optional[Hash] $options = { }, +) { + case $provider { + apt: { + package_apt { $title: + ensure => $ensure, + source => $source, + version => $version, + * => $options, + } + } + rpm: { + package_rpm { $title: + ensure => $ensure, + source => $source, + * => $options, + } + if defined($version) { fail("RPM doesn't support \$version") } + # ... + } + } +} +``` + +## Composite namevars + +The current API does not provide a way to specify composite namevars. [`title_patterns`](https://github.com/puppetlabs/puppet-specifications/blob/master/language/resource_types.md#title-patterns) are already very data driven, and will be easy to add at a later point. + +## Puppet 4 data types + +Currently anywhere "puppet 4 data types" are mentioned, only the built-in types are usable. This is because the type information is required on the agent, but puppet doesn't make it available yet. This work is tracked in [PUP-7197](https://tickets.puppetlabs.com/browse/PUP-7197), but even once that is implemented, modules will have to wait until the functionality is widely available, before being able to rely on that. + +## Resources that can't be enumerated + +Some resources, like files, cannot (or should not) be completely enumerated each time puppet runs. In some cases, the runtime environment knows that it doesn't require all resource instances. The current API does not provide a way to support those use-cases. An easy way forward would be to add a `find(title)` method that would return data for a single resource instance. A more involved solution my leverage PQL, but would require a much more sophisticated implementation. This also interacts with composite namevars. + +## Catalog access + +There is no way to access the catalog from the implementation. Several existing types rely on this to implement advanced functionality. Some of those use-cases would be better suited to be implemented as "external" catalog transformations, instead of munging the catalog from within the compilation process. + # Earlier notes Draft for new type and provider API From b076462ce69f30fad6eb835c437c4e2dc24c8698 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Mon, 6 Mar 2017 12:35:14 +0000 Subject: [PATCH 09/62] Updates from internal feedback --- language/resource-api/README.md | 79 ++++++++++++++++++--------------- 1 file changed, 42 insertions(+), 37 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index a820072..d9025b7 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -60,20 +60,22 @@ The `Puppet::ResourceDefinition.register(options)` function takes a Hash with th At runtime the current and intended system states for a single resource instance are always represented as ruby Hashes of the resource's attributes, and applicable operational parameters. The two fundamental operations to manage resources are reading and writing system state. These operations are implemented in the `ResourceImplementation` as `get` and `set`: - + ```ruby Puppet::ResourceImplementation.register('apt_key') do def get - [ + { 'title': { name: 'title', # ... }, - ] + } end - - def set(current_state, target_state, noop: false) - target_state.each do |title, resource| + + def set(changes, noop: false) + changes.each do |title, change| + current = change.has_key? :current ? change[:current] : get_single(title) + target = change[:target] # ... end end @@ -82,8 +84,8 @@ end The `get` method returns a Hash of all resources currently available, keyed by their title. If the `get` method raises an exception, the implementation is marked as unavailable during the current run, and all resources of its type will fail. The error message will be reported to the user. -The `set` method updates resources to the state defined in `target_state`. For convenience, `current_state` contains the last available system state from a prior `get` call. When `noop` is set to true, the implementation must not change the system state, but only report what it would change. - +The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by resource title. Each value is another hash with a `:target` key, and an optional `:current` key. Those values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:target` value. For convenience, `:current` may contain the last available system state from a prior `get` call. If the `:current` value is `nil`, the resources was not found by `get`. If there is no `:current` key, the runtime did not have a cached state available. When `noop` is set to true, the implementation must not change the system state, but only report what it would change. The `set` method should always return `nil`. Any progress signalling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. + # Runtime Environment @@ -95,7 +97,7 @@ The runtime environment provides some utilities to make the implementation's lif ### Commands -To use CLI commands in a safe and comfortable manner, the implementation can use the `commands` method to access shell commands. You can either use a full path, or a bare command name. In the latter case puppet will use the system PATH setting to search for the command. If the commands are not available, an error will be raised and the resources will fail in this run. The commands are aware of whether noop is in effect or not, and will signal success while skipping the real execution if necessary. +To use CLI commands in a safe and comfortable manner, the implementation can use the `commands` method to access shell commands. You can either use a full path, or a bare command name. In the latter case puppet will use the system PATH setting to search for the command. If the commands are not available, an error will be raised and the resources will fail in this run. The commands are aware of whether noop is in effect or not, and will signal success while skipping the real execution if necessary. ```ruby Puppet::ResourceImplementation.register('apt_key') do @@ -104,7 +106,7 @@ Puppet::ResourceImplementation.register('apt_key') do ``` This will create methods called `apt_get`, and `gpg`, which will take CLI arguments in an array, and execute the command directly without any shell processing in a safe environment (clean working directory, clean environment). For example to call `apt-key` to delete a specific key by id: - + ```ruby apt_key 'del', key_id ``` @@ -165,7 +167,7 @@ Warning: apt_key: Unexpected state detected, continuing in degraded mode. * warning: signal error conditions that do not (yet) prohibit execution of the main part of the implementation; for example deprecation warnings, temporary errors. * err: signal error conditions that have caused normal operations to fail * critical/alert/emerg: should not be used by resource implementations - + See [wikipedia](https://en.wikipedia.org/wiki/Syslog#Severity_level) and [RFC424](https://tools.ietf.org/html/rfc5424) for more details. #### Logging contexts @@ -176,21 +178,21 @@ Most of an implementation's messages are expected to be relative to a specific r logger.attribute_changed(title:, attribute:, old_value:, new_value:, message: "Changed #{attribute} from #{old_value.inspect} to #{newvalue.inspect}") ``` -To enable detailed logging without repeating key arguments, and provide consistent error logging, the logger provides *logging context* methods that capture the current action and resource instance. - +To enable detailed logging without repeating key arguments, and provide consistent error logging, the logger provides *logging context* methods that capture the current action and resource instance. + ```ruby logger.updating(title: title) do if key_not_found - logger.warning('Original key not found') + logger.warning('Original key not found') end - + # Update the key by calling CLI tool apt_key(...) - + logger.attribute_changed( - attribute: 'content', + attribute: 'content', old_value: nil, - new_value: content_hash, + new_value: content_hash, message: "Created with content hash #{content_hash}") end ``` @@ -253,14 +255,15 @@ This example is only for demonstration purposes. In the normal course of operati The following action/context methods are available: -* `creating(title:, message: 'Creating', &block)` -* `updating(title:, message: 'Updating', &block)` -* `deleting(title:, message: 'Deleting', &block)` -* `attribute_changed(title:, attribute:, old_value:, new_value:, message: nil)` +* `creating(title, message: 'Creating', &block)` +* `updating(title, message: 'Updating', &block)` +* `deleting(title, message: 'Deleting', &block)` +* `attribute_changed(title, attribute, old_value:, new_value:, message: nil)` -* `created(title:, message: 'Created')` -* `updated(title:, message: 'Updated')` -* `deleted(title:, message: 'Deleted')` +* `created(title, message: 'Created')` +* `updated(title, message: 'Updated')` +* `deleted(title, message: 'Deleted')` +* `unchanged(title, message: 'Unchanged')`: the resource did not require a change * `fail(title:, message:)` - abort the current context with an error @@ -322,30 +325,33 @@ Some resources, like files, cannot (or should not) be completely enumerated each ## Catalog access -There is no way to access the catalog from the implementation. Several existing types rely on this to implement advanced functionality. Some of those use-cases would be better suited to be implemented as "external" catalog transformations, instead of munging the catalog from within the compilation process. +There is no way to access the catalog from the implementation. Several existing types rely on this to implement advanced functionality. Some of those use-cases would be better suited to be implemented as "external" catalog transformations, instead of munging the catalog from within the compilation process. + +## Logging for unmanaged instances + +The implementation could provide log messages for resource instances that were not passed into the `set` call. In the current implementation those will be reported to the log, but will not cause the same resource-based reporting as a managed resource. How this is handeled in the future might change drastically. -# Earlier notes -Draft for new type and provider API +# Earlier notes -Hi *, +## Draft for new type and provider API The type and provider API has been the bane of my existence since I [started writing native resources](https://github.com/DavidS/puppet-mysql-old/commit/d33c7aa10e3a4bd9e97e947c471ee3ed36e9d1e2). Now, finally, we'll do something about it. I'm currently working on designing a nicer API for types and providers. My primary goals are to provide a smooth and simple ruby developer experience for both scripters and coders. Secondary goals were to eliminate server side code, and make puppet 4 data types available. Currently this is completely aspirational (i.e. no real code has been written), but early private feedback was encouraging. -To showcase my vision, this [gist](https://gist.github.com/DavidS/430330ae43ba4b51fe34bd27ddbe4bc7) has the [apt_key type](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/type/apt_key.rb) and [provider](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/apt_key/apt_key.rb) ported over to my proposal. The second example there is a more long-term teaser on what would become possible with such an API. +To showcase my vision, this [gist](https://gist.github.com/DavidS/430330ae43ba4b51fe34bd27ddbe4bc7) has the [apt_key type](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/type/apt_key.rb) and [provider](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/apt_key/apt_key.rb) ported over to my proposal. The second example there is a more long-term teaser on what would become possible with such an API. -The new API, like the existing, has two parts: the implementation that interacts with the actual resources, a.k.a. the provider, and information about what the implementation is all about. Due to the different usage patterns of the two parts, they need to be passed to puppet in two different calls: +The new API, like the existing, has two parts: the implementation that interacts with the actual resources, a.k.a. the provider, and information about what the implementation is all about. Due to the different usage patterns of the two parts, they need to be passed to puppet in two different calls: The `Puppet::SimpleResource.implement()` call receives the `current_state = get()` and `set(current_state, target_state, noop)` methods. `get` returns a list of discovered resources, while `set` takes the target state and enforces those goals on the subject. There is only a single (ruby) object throughout an agent run, that can easily do caching and what ever else is required for a good functioning of the provider. The state descriptions passed around are simple lists of key/value hashes describing resources. This will allow the implementation wide latitude in how to organise itself for simplicity and efficiency. -The `Puppet::SimpleResource.define()` call provides a data-only description of the Type. This is all that is needed on the server side to compile a manifest. Thanks to puppet 4 data type checking, this will already be much more strict (with less effort) than possible with the current APIs, while providing more automatically readable documentation about the meaning of the attributes. - +The `Puppet::SimpleResource.define()` call provides a data-only description of the Type. This is all that is needed on the server side to compile a manifest. Thanks to puppet 4 data type checking, this will already be much more strict (with less effort) than possible with the current APIs, while providing more automatically readable documentation about the meaning of the attributes. + Details in no particular order: * All of this should fit on any unmodified puppet4 installation. It is completely additive and optional. Currently. -* The Type definition +* The Type definition * It is data-only. * Refers to puppet data types. * No code runs on the server. @@ -373,9 +379,9 @@ Details in no particular order: * the `logger` is the primary way of reporting back information to the log, and the report. * results can be streamed for immediate feedback * block-based constructs allow detailed logging with little code ("Started X", "X: Doing Something", "X: Success|Failure", with one or two calls, and only one reference to X) - + * Obviously this is not sufficient to cover everything existing types and providers are able to do. For the first iteration we are choosing simplicity over functionality. - * Generating more resource instances for the catalog during compilation (e.g. file#recurse or concat) becomes impossible with a pure data-driven Type. There is still space in the API to add server-side code. + * Generating more resource instances for the catalog during compilation (e.g. file#recurse or concat) becomes impossible with a pure data-driven Type. There is still space in the API to add server-side code. * Some resources (e.g. file, ssh_authorized_keys, concat) cannot or should not be prefetched. While it might not be convenient, a provider could always return nothing on the `get()` and do a more customized enforce motion in the `set()`. * With current puppet versions, only "native" data types will be supported, as type aliases do not get pluginsynced. Yet. * With current puppet versions, `puppet resource` can't load the data types, and therefore will not be able to take full advantage of this. Yet. @@ -389,4 +395,3 @@ What do you think about this? Cheers, David - From c491dacca248bd4b77179ec70cb3969dbc10af7b Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Mon, 6 Mar 2017 14:18:55 +0000 Subject: [PATCH 10/62] Sketch of pops based validation --- language/resource-api/apt_key.rb | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/language/resource-api/apt_key.rb b/language/resource-api/apt_key.rb index 59eb510..e1b33cb 100644 --- a/language/resource-api/apt_key.rb +++ b/language/resource-api/apt_key.rb @@ -174,6 +174,17 @@ def to_manifest # read-only values do not need type checking if not options[:read_only] # TODO: this should use Pops infrastructure to avoid hardcoding stuff, and enhance type fidelity + # validate do |v| + # type = Puppet::Pops::Types::TypeParser.singleton.parse(options[:type]).normalize + # if type.instance?(v) + # return true + # else + # inferred_type = Puppet::Pops::Types::TypeCalculator.infer_set(value) + # error_msg = Puppet::Pops::Types::TypeMismatchDescriber.new.describe_mismatch("#{DEFINITION[:name]}.#{name}", type, inferred_type) + # raise Puppet::ResourceError, error_msg + # end + # end + case options[:type] when 'String' # require any string value From 2e8882518d710e2dedb903e920f31292ff1d1b40 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 7 Mar 2017 14:46:33 +0000 Subject: [PATCH 11/62] Add `processed` logging for very simple implementations --- language/resource-api/README.md | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index d9025b7..b75a3a1 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -170,15 +170,24 @@ Warning: apt_key: Unexpected state detected, continuing in degraded mode. See [wikipedia](https://en.wikipedia.org/wiki/Syslog#Severity_level) and [RFC424](https://tools.ietf.org/html/rfc5424) for more details. -#### Logging contexts +#### Signalling resource status -Most of an implementation's messages are expected to be relative to a specific resource instance, and a specific operation on that instance. For example, to report the change of an attribute: +In many simple cases, an implementation can pass off the real work to a external tool, detailed logging happens there, and reporting back to puppet only requires acknowledging those changes. In these situations, signalling can be as easy as this: -```ruby -logger.attribute_changed(title:, attribute:, old_value:, new_value:, message: "Changed #{attribute} from #{old_value.inspect} to #{newvalue.inspect}") +``` +apt_key action, key_id +logger.processed(key_id, is, should) ``` -To enable detailed logging without repeating key arguments, and provide consistent error logging, the logger provides *logging context* methods that capture the current action and resource instance. +This will report all changes from `is` to `should`, using default messages. + +Implementations that want to have more control over the logging throughout the processing can use the more specific `created(title)`, `updated(title)`, `deleted(title)`, `unchanged(title)` methods for that. To report the change of an attribute, the `logger` provides a `attribute_changed(title, attribute, old_value, new_value)` method. + +> Note: Implementations making use of those primitive functions need to take care to + +#### Logging contexts + +Most of those messages are expected to be relative to a specific resource instance, and a specific operation on that instance. To enable detailed logging without repeating key arguments, and provide consistent error logging, the logger provides *logging context* methods that capture the current action and resource instance. ```ruby logger.updating(title: title) do @@ -258,14 +267,19 @@ The following action/context methods are available: * `creating(title, message: 'Creating', &block)` * `updating(title, message: 'Updating', &block)` * `deleting(title, message: 'Deleting', &block)` -* `attribute_changed(title, attribute, old_value:, new_value:, message: nil)` +* `attribute_changed(attribute, old_value:, new_value:, message: nil)`: default to the title from the context * `created(title, message: 'Created')` * `updated(title, message: 'Updated')` * `deleted(title, message: 'Deleted')` -* `unchanged(title, message: 'Unchanged')`: the resource did not require a change +* `unchanged(title, message: 'Unchanged')`: the resource did not require a change - emit no logging +* `processed(title, is, should)`: the resource has been processed - emit default logging for the resource and each attribute +* `failed(title:, message:)`: the resource has not been updated successfully +* `attribute_changed(title, attribute, old_value:, new_value:, message: nil)` + +* `fail(message:)`: abort the current context with an error -* `fail(title:, message:)` - abort the current context with an error +`title` can be a single identifier for a resource instance, or an array of values, if the following block batch-processes multiple resources in one pass. If that processing is not atomic, implementations should instead use the non-block forms of logging, and provide accurate status reporting on the individual parts of update operations. # Known Limitations From f60df1d4d17f98574b0b7ddf7afea7c1402782fe Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 7 Mar 2017 15:15:15 +0000 Subject: [PATCH 12/62] More logging updates * describe required call sequences * note the uniqueness requirements of status reporting per resource instance * use is/should for old and new attribute values --- language/resource-api/README.md | 61 ++++++++++++++++++++------------- 1 file changed, 37 insertions(+), 24 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index b75a3a1..3d3abea 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -198,10 +198,7 @@ logger.updating(title: title) do # Update the key by calling CLI tool apt_key(...) - logger.attribute_changed( - attribute: 'content', - old_value: nil, - new_value: content_hash, + logger.attribute_changed('content', nil, content_hash, message: "Created with content hash #{content_hash}") end ``` @@ -213,7 +210,7 @@ Debug: Apt_key[#{title}]: Started updating Warning: Apt_key[#{title}]: Updating: Original key not found Debug: Apt_key[#{title}]: Executing 'apt-key ...' Debug: Apt_key[#{title}]: Successfully executed 'apt-key ...' -Notice: Apt_key[#{title}]: Updating content: Created with content hash #{content_hash} +Notice: Apt_key[#{title}]: Updating content: Created with content hash { a: hash } Notice: Apt_key[#{title}]: Successfully updated # TODO: update messages to match current log message formats for resource messages ``` @@ -223,7 +220,7 @@ In the case of an exception escaping the block, the error is logged appropriatel ```text Debug: Apt_key[#{title}]: Started updating Warning: Apt_key[#{title}]: Updating: Original key not found -Error: Apt_key[#{title}]: Updating failed: #{exception message} +Error: Apt_key[#{title}]: Updating failed: Something went wrong # TODO: update messages to match current log message formats for resource messages ``` @@ -244,12 +241,8 @@ begin if $?.exitstatus != 0 logger.error(title: title, "Failed executing apt-key #{...}") else - logger.attribute_changed( - title: title, - attribute: 'content', - old_value: nil, - new_value: content_hash, - message: "Created with content hash #{content_hash}") + logger.attribute_changed(title, 'content', nil, content_hash, + message: "Created with content hash #{content_hash}") end logger.changed(title: title) rescue StandardError => e @@ -264,23 +257,43 @@ This example is only for demonstration purposes. In the normal course of operati The following action/context methods are available: -* `creating(title, message: 'Creating', &block)` -* `updating(title, message: 'Updating', &block)` -* `deleting(title, message: 'Deleting', &block)` -* `attribute_changed(attribute, old_value:, new_value:, message: nil)`: default to the title from the context - -* `created(title, message: 'Created')` -* `updated(title, message: 'Updated')` -* `deleted(title, message: 'Deleted')` -* `unchanged(title, message: 'Unchanged')`: the resource did not require a change - emit no logging -* `processed(title, is, should)`: the resource has been processed - emit default logging for the resource and each attribute -* `failed(title:, message:)`: the resource has not been updated successfully -* `attribute_changed(title, attribute, old_value:, new_value:, message: nil)` +* Context functions +** `creating(title, message: 'Creating', &block)` +** `updating(title, message: 'Updating', &block)` +** `deleting(title, message: 'Deleting', &block)` +** `processing(title, is, should, message: 'Processing', &block)` +** `failing(title, message: 'Failing', &block)`: unlikely to be used often, but provided for completeness +** `attribute_changed(attribute, is, should, message: nil)`: default to the title from the context + +* Action functions +** `created(title, message: 'Created')` +** `updated(title, message: 'Updated')` +** `deleted(title, message: 'Deleted')` +** `unchanged(title, message: 'Unchanged')`: the resource did not require a change - emit no logging +** `processed(title, is, should)`: the resource has been processed - emit default logging for the resource and each attribute +** `failed(title:, message:)`: the resource has not been updated successfully +** `attribute_changed(title, attribute, is, should, message: nil)`: use outside of a context, or in a context with multiple resources * `fail(message:)`: abort the current context with an error +* Plain messages +** `debug(message)` +** `debug(title:, message:)` +** `info(message)` +** `info(title:, message:)` +** `notice(message)` +** `notice(title:, message:)` +** `warning(message)` +** `warning(title:, message:)` +** `err(message)` +** `err(title:, message:)` + `title` can be a single identifier for a resource instance, or an array of values, if the following block batch-processes multiple resources in one pass. If that processing is not atomic, implementations should instead use the non-block forms of logging, and provide accurate status reporting on the individual parts of update operations. +A single `set()` execution may only log messages for instances it has been passed as part of the `changes` to process. Logging for foreign instances will cause an exception, as the runtime environment is not prepared for other resources to change. + +The implementation is free to call different logging methods for different resources in any order it needs to. The only ordering restriction is for all calls specifying the same `title`. The `attribute_changed` logging needs to be done before that resource's action logging, and if a context is opened, it needs to be opened before any other logging for this resource. + # Known Limitations This API is not a full replacement for the power of 3.x style types and providers. Here is a (incomplete) list of missing pieces and thoughts on how to go about solving these. In the end, the goal of the new Resource API is not to be a complete replacement of prior art, but a cleaner way to get good results for the majority of simple cases. From 9de46f43b1c405daf4b3af99660eb0a6df150a00 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 8 Mar 2017 14:40:01 +0000 Subject: [PATCH 13/62] Update language to be closer to what experienced developers would expect To make the transition easier, this version re-uses existing language as far as possible. --- language/resource-api/README.md | 224 ++++++++++++-------------------- 1 file changed, 80 insertions(+), 144 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 3d3abea..baf0839 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -1,11 +1,11 @@ # Resource API -A *resource* is the basic thing that is managed by puppet. Each resource has a set of attributes describing its current state. Some of the attributes can be changed throughout the life-time of the resource, some attributes only report back, but cannot be changed (see `read_only`) others can only be set once during initial creation (see `init_only`). To gather information about those resources, and to enact changes in the real world, puppet requires a piece of code to *implement* this interaction. The implementation can have parameters that influence its mode of operation (see `parameter`). To describe all these parts to the infrastructure, and the consumers, the resource *Definition* (f.k.a. 'type') contains definitions for all of them. The *Implementation* (f.k.a. 'provider') contains code to *get* and *set* the system state. +A *resource* is the basic thing that is managed by puppet. Each resource has a set of attributes describing its current state. Some of the attributes can be changed throughout the life-time of the resource, some attributes are only reported back, but cannot be changed (see `read_only`) others can only be set once during initial creation (see `init_only`). To gather information about those resources, and to enact changes in the real world, puppet requires a *provider* to implement this interaction. The provider can have parameters that influence its mode of operation (see `parameter`). To describe all these parts to the infrastructure, and the consumers, the resource *type* defines the all the metadata, including the list of the attributes. The *provider* contains the code to *get* and *set* the system state. # Resource Definition ```ruby -Puppet::ResourceDefinition.register( +Puppet::ResourceType.register( name: 'apt_key', docs: <<-EOS, This type provides Puppet with the capabilities to manage GPG keys needed @@ -44,60 +44,60 @@ Puppet::ResourceDefinition.register( ) ``` -The `Puppet::ResourceDefinition.register(options)` function takes a Hash with the following top-level keys: +The `Puppet::ResourceType.register(options)` function takes a Hash with the following top-level keys: -* `name`: the name of the resource. For autoloading to work, the whole function call needs to go into `lib/puppet/type/.rb`. -* `docs`: a doc string that describes the overall working of the type, gives examples, and explains pre-requisites as well as known issues. +* `name`: the name of the resource type. For autoloading to work, the function call needs to go into `lib/puppet/type/.rb`. +* `docs`: a doc string that describes the overall working of the resource type, gives examples, and explains pre-requisites as well as known issues. * `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `docs` string, and whether the attribute is the `namevar`, `read_only`, `init_only`, or a `parameter`. * `namevar`: marks an attribute as part of the "primary key", or "identity" of the resource. A given set of namevar values needs to distinctively identify a instance. * `init_only`: this attribute can only be set during creation of the resource. Its value will be reported going forward, but trying to change it later will lead to an error. For example, the base image for a VM, or the UID of a user. * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example the checksum of a file, or the MAC address of a network interface. - * `parameter`: these attributes influence how the implementation behaves, and cannot be read from the target system. For example, the target file on inifile, or credentials to access an API. -* `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a Hash mapping resource types to titles. Currently the titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. + * `parameter`: these attributes influence how the provider behaves, and cannot be read from the target system. For example, the target file on inifile, or credentials to access an API. +* `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a Hash mapping resource types to titles. Currently the titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, puppet will automatically create the relationsships requested here. -# Resource Implementation +# Resource Provider -At runtime the current and intended system states for a single resource instance are always represented as ruby Hashes of the resource's attributes, and applicable operational parameters. +At runtime the current and intended system states for a specific resource are always represented as ruby Hashes of the resource's attributes, and applicable operational parameters. -The two fundamental operations to manage resources are reading and writing system state. These operations are implemented in the `ResourceImplementation` as `get` and `set`: +The two fundamental operations to manage resources are reading and writing system state. These operations are implemented in the `ResourceProvider` as `get` and `set`: ```ruby -Puppet::ResourceImplementation.register('apt_key') do +Puppet::ResourceProvider.register('apt_key') do def get { - 'title': { - name: 'title', + 'name': { + name: 'name', # ... }, } end def set(changes, noop: false) - changes.each do |title, change| - current = change.has_key? :current ? change[:current] : get_single(title) - target = change[:target] + changes.each do |name, change| + is = change.has_key? :is ? change[:is] : get_single(name) + should = change[:should] # ... end end end ``` -The `get` method returns a Hash of all resources currently available, keyed by their title. If the `get` method raises an exception, the implementation is marked as unavailable during the current run, and all resources of its type will fail. The error message will be reported to the user. +The `get` method returns a Hash of all resources currently available, keyed by their name. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The error message will be reported to the user. -The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by resource title. Each value is another hash with a `:target` key, and an optional `:current` key. Those values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:target` value. For convenience, `:current` may contain the last available system state from a prior `get` call. If the `:current` value is `nil`, the resources was not found by `get`. If there is no `:current` key, the runtime did not have a cached state available. When `noop` is set to true, the implementation must not change the system state, but only report what it would change. The `set` method should always return `nil`. Any progress signalling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. +The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by the resource's name. Each value is another hash with a `:should` key, and an optional `:is` key. Those values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. For convenience, `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources was not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. When `noop` is set to true, the provider must not change the system state, but only report what it would change. The `set` method should always return `nil`. Any progress signalling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. # Runtime Environment -The primary runtime environment for the implementation is the puppet agent, a long-running daemon process. The implementation can also be used in the puppet apply command, a one-shot version of the agent, or the puppet resource command, a short-lived CLI process for listing or managing a single resource type. Other callers who want to access the implementation will have to emulate those environments. In any case the registered block will be surfaced in a clean class which will be instantiated once for each transaction. The implementation can define any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for an implementation, and use it immediately, the provider is instantiated as late as possible. A transaction will usually call `get` once, and may call set any number of times to effect change. The object instance hosting the `get` and `set` methods can be used to cache ephemeral state during execution. The implementation should not try to cache state beyond the transaction, to avoid interfering with the agent daemon. In many other cases caching beyond the transaction won't help anyways, as the hosting process will only manage a single transaction. +The primary runtime environment for the provider is the puppet agent, a long-running daemon process. The provider can also be used in the puppet apply command, a one-shot version of the agent, or the puppet resource command, a short-lived CLI process for listing or managing a single resource type. Other callers who want to access the provider will have to emulate those environments. The primary lifecycle of resource managment in each of those tools is the *transaction*, a single set of changes (e.g. a catalog, or a CLI invocation) to work on. In any case the registered block will be surfaced in a clean class which will be instantiated once for each transaction. The provider can define any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for an provider, and use it immediately, the provider is instantiated as late as possible. A transaction will usually call `get` once, and may call `set` any number of times to effect change. The object instance hosting the `get` and `set` methods can be used to cache ephemeral state during execution. The provider should not try to cache state beyond the transaction, to avoid interfering with the agent daemon. In many other cases caching beyond the transaction won't help anyways, as the hosting process will only manage a single transaction. ## Utilities -The runtime environment provides some utilities to make the implementation's life easier, and provide a uniform experience for its users. +The runtime environment provides some utilities to make the providers's life easier, and provide a uniform experience for its users. ### Commands -To use CLI commands in a safe and comfortable manner, the implementation can use the `commands` method to access shell commands. You can either use a full path, or a bare command name. In the latter case puppet will use the system PATH setting to search for the command. If the commands are not available, an error will be raised and the resources will fail in this run. The commands are aware of whether noop is in effect or not, and will signal success while skipping the real execution if necessary. +To use CLI commands in a safe and comfortable manner, the provider can use the `commands` method to access shell commands. You can either specify a full path, or a bare command name. In the latter case puppet will use the system's `PATH` setting to search for the command. If the commands are not available, an error will be raised and the resources will fail in this run. The commands are aware of whether noop is in effect or not, and will signal success while skipping the real execution if necessary. Using these methods also causes the provider's actions to be logged at the appropriate levels. ```ruby Puppet::ResourceImplementation.register('apt_key') do @@ -117,7 +117,7 @@ To pass additional environment variables through to the command, pass a hash of apt_key 'del', key_id, env: { 'LC_ALL': 'C' } ``` -By default the stdout of the command is logged to debug, while the stderr is logged to warning. To access the stdout in the implementation, use the command name with `_lines` appended, and process it through the returned [Enumerable](http://ruby-doc.org/core/Enumerable.html) line-by-line. For example, to process the list of all apt keys: +By default the `stdout` of the command is logged to debug, while the `stderr` is logged to warning. To access the `stdout` in the provider, use the command name with `_lines` appended, and process it through the returned [Enumerable](http://ruby-doc.org/core/Enumerable.html) line-by-line. For example, to process the list of all apt keys: ```ruby apt_key_lines(%w{adv --list-keys --with-colons --fingerprint --fixed-list-mode}).collect do |line| @@ -127,7 +127,7 @@ end > Note: the output of the command is streamed through the Enumerable. If the implementation requires the exit value of the command before processing, or wants to cache the output, use `to_a` to read the complete stream in one go. -If the command returns a non-zero exit code, an error is signalled to puppet. If this happens during `get`, all managed resources of this type will fail. If this happens during a `set`, all resources that have been scheduled for processing in this call, but not yet have been marked as a success will be marked as failed. To avoid this behaviour, call the `try_` prefix variant. In this (hypothetical) example, `apt-key` signals already deleted keys with an exit code of `1`, which is OK when the implementation is trying to delete the key: +If the command returns a non-zero exit code, an error is signalled to puppet. If this happens during `get`, all managed resources of this type will fail. If this happens during a `set`, all resources that have been scheduled for processing in this call, but not yet have been marked as a success will be marked as failed. To avoid this behaviour, call the `try_` prefix variant. In this (hypothetical) example, `apt-key` signals already deleted keys with an exit code of `1`, which is still OK when the provider is trying to delete the key: ```ruby try_apt_key 'del', key_id @@ -141,15 +141,18 @@ end The exit code is signalled through the ruby standard variable `$?` as a [`Process::Status` object](https://ruby-doc.org/core/Process/Status.html) - + ### Logging and Reporting -The implementation needs to signal changes, successes and failures to the runtime environment. The `logger` provides a structured way to do so. +The provider needs to signal changes, successes and failures to the runtime environment. The `logger` is the primary way to do so. It provides a single interface for both the detailed technical information ofr later automatic processing, as well as human readable progress and status messages for operators. #### General messages -To provide feedback about the overall operation of the implementation, the logger provides the usual set of [loglevel](https://docs.puppet.com/puppet/latest/metaparameter.html#loglevel) methods that take a string, and pass that up to puppet's logging infrastructure: +To provide feedback about the overall operation of the provider, the logger has the usual set of [loglevel](https://docs.puppet.com/puppet/latest/metaparameter.html#loglevel) methods that take a string, and pass that up to runtime environment's logging infrastructure: ```ruby logger.warning("Unexpected state detected, continuing in degraded mode.") @@ -162,17 +165,17 @@ Warning: apt_key: Unexpected state detected, continuing in degraded mode. ``` * debug: detailed messages to understand everything that is happening at runtime; only shown on request -* info: high level progress messages; especially useful before long-running operations, or before operations that can fail, to provide context to interactive users -* notice: use this loglevel to indicate state changes and similar events of notice from the regular operations of the implementation -* warning: signal error conditions that do not (yet) prohibit execution of the main part of the implementation; for example deprecation warnings, temporary errors. +* info: regular progress and status messages; especially useful before long-running operations, or before operations that can fail, to provide context to interactive users +* notice: indicates state changes and other events of notice from the regular operations of the provider +* warning: signals error conditions that do not (yet) prohibit execution of the main part of the provider; for example deprecation warnings, temporary errors * err: signal error conditions that have caused normal operations to fail -* critical/alert/emerg: should not be used by resource implementations +* critical/alert/emerg: should not be used by resource providers See [wikipedia](https://en.wikipedia.org/wiki/Syslog#Severity_level) and [RFC424](https://tools.ietf.org/html/rfc5424) for more details. #### Signalling resource status -In many simple cases, an implementation can pass off the real work to a external tool, detailed logging happens there, and reporting back to puppet only requires acknowledging those changes. In these situations, signalling can be as easy as this: +In many simple cases, a provider can pass off the real work to a external tool, detailed logging happens there, and reporting back to puppet only requires acknowledging those changes. In these situations, signalling can be as easy as this: ``` apt_key action, key_id @@ -181,16 +184,14 @@ logger.processed(key_id, is, should) This will report all changes from `is` to `should`, using default messages. -Implementations that want to have more control over the logging throughout the processing can use the more specific `created(title)`, `updated(title)`, `deleted(title)`, `unchanged(title)` methods for that. To report the change of an attribute, the `logger` provides a `attribute_changed(title, attribute, old_value, new_value)` method. - -> Note: Implementations making use of those primitive functions need to take care to +Providers that want to have more control over the logging throughout the processing can use the more specific `created(title)`, `updated(title)`, `deleted(title)`, `unchanged(title)` methods for that. To report the change of an attribute, the `logger` provides a `attribute_changed(title, attribute, old_value, new_value, message)` method. #### Logging contexts Most of those messages are expected to be relative to a specific resource instance, and a specific operation on that instance. To enable detailed logging without repeating key arguments, and provide consistent error logging, the logger provides *logging context* methods that capture the current action and resource instance. ```ruby -logger.updating(title: title) do +logger.updating(title) do if key_not_found logger.warning('Original key not found') end @@ -199,28 +200,28 @@ logger.updating(title: title) do apt_key(...) logger.attribute_changed('content', nil, content_hash, - message: "Created with content hash #{content_hash}") + message: "Replaced with content hash #{content_hash}") end ``` -will result in the following messages (of course, with the `#{}` sequences replaced by the true values): +will result in the following messages: ```text -Debug: Apt_key[#{title}]: Started updating -Warning: Apt_key[#{title}]: Updating: Original key not found -Debug: Apt_key[#{title}]: Executing 'apt-key ...' -Debug: Apt_key[#{title}]: Successfully executed 'apt-key ...' -Notice: Apt_key[#{title}]: Updating content: Created with content hash { a: hash } -Notice: Apt_key[#{title}]: Successfully updated +Debug: Apt_key[F1D2D2F9]: Started updating +Warning: Apt_key[F1D2D2F9]: Updating: Original key not found +Debug: Apt_key[F1D2D2F9]: Executing 'apt-key ...' +Debug: Apt_key[F1D2D2F9]: Successfully executed 'apt-key ...' +Notice: Apt_key[F1D2D2F9]: Updating content: Replaced with content hash E242ED3B +Notice: Apt_key[F1D2D2F9]: Successfully updated # TODO: update messages to match current log message formats for resource messages ``` In the case of an exception escaping the block, the error is logged appropriately: ```text -Debug: Apt_key[#{title}]: Started updating -Warning: Apt_key[#{title}]: Updating: Original key not found -Error: Apt_key[#{title}]: Updating failed: Something went wrong +Debug: Apt_keyF1D2D2F9]: Started updating +Warning: Apt_key[F1D2D2F9]: Updating: Original key not found +Error: Apt_key[F1D2D2F9]: Updating failed: Something went wrong # TODO: update messages to match current log message formats for resource messages ``` @@ -229,78 +230,78 @@ Logging contexts process all exceptions. [`StandardError`s](https://ruby-doc.org The equivalent long-hand form with manual error handling: ```ruby -logger.updating(title: title) +logger.updating(title) begin if key_not_found - logger.warning(title: title, message: 'Original key not found') + logger.warning(title, message: 'Original key not found') end # Update the key by calling CLI tool try_apt_key(...) if $?.exitstatus != 0 - logger.error(title: title, "Failed executing apt-key #{...}") + logger.error(title, "Failed executing apt-key #{...}") else logger.attribute_changed(title, 'content', nil, content_hash, - message: "Created with content hash #{content_hash}") + message: "Replaced with content hash #{content_hash}") end - logger.changed(title: title) -rescue StandardError => e - logger.error(title: title, exception: e, message: 'Updating failed') + logger.changed(title) +rescue Exception => e + logger.error(title, e, message: 'Updating failed') raise unless e.is_a? StandardError end ``` -This example is only for demonstration purposes. In the normal course of operations, implementations should always use the utility functions. +This example is only for demonstration purposes. In the normal course of operations, providers should always use the utility functions. #### Logging reference The following action/context methods are available: * Context functions -** `creating(title, message: 'Creating', &block)` -** `updating(title, message: 'Updating', &block)` -** `deleting(title, message: 'Deleting', &block)` -** `processing(title, is, should, message: 'Processing', &block)` -** `failing(title, message: 'Failing', &block)`: unlikely to be used often, but provided for completeness +** `creating(titles, message: 'Creating', &block)` +** `updating(titles, message: 'Updating', &block)` +** `deleting(titles, message: 'Deleting', &block)` +** `processing(titles, is, should, message: 'Processing', &block)` +** `failing(titles, message: 'Failing', &block)`: unlikely to be used often, but provided for completeness ** `attribute_changed(attribute, is, should, message: nil)`: default to the title from the context * Action functions -** `created(title, message: 'Created')` -** `updated(title, message: 'Updated')` -** `deleted(title, message: 'Deleted')` -** `unchanged(title, message: 'Unchanged')`: the resource did not require a change - emit no logging -** `processed(title, is, should)`: the resource has been processed - emit default logging for the resource and each attribute -** `failed(title:, message:)`: the resource has not been updated successfully -** `attribute_changed(title, attribute, is, should, message: nil)`: use outside of a context, or in a context with multiple resources +** `created(titles, message: 'Created')` +** `updated(titles, message: 'Updated')` +** `deleted(titles, message: 'Deleted')` +** `unchanged(titles, message: 'Unchanged')`: the resource did not require a change - emit no logging +** `processed(titles, is, should)`: the resource has been processed - emit default logging for the resource and each attribute +** `failed(titles, message:)`: the resource has not been updated successfully +** `attribute_changed(titles, attribute, is, should, message: nil)`: use outside of a context, or in a context with multiple resources -* `fail(message:)`: abort the current context with an error +* `fail(message)`: abort the current context with an error * Plain messages ** `debug(message)` -** `debug(title:, message:)` +** `debug(titles, message:)` ** `info(message)` -** `info(title:, message:)` +** `info(titles, message:)` ** `notice(message)` -** `notice(title:, message:)` +** `notice(titles, message:)` ** `warning(message)` -** `warning(title:, message:)` +** `warning(titles, message:)` ** `err(message)` -** `err(title:, message:)` +** `err(titles, message:)` -`title` can be a single identifier for a resource instance, or an array of values, if the following block batch-processes multiple resources in one pass. If that processing is not atomic, implementations should instead use the non-block forms of logging, and provide accurate status reporting on the individual parts of update operations. +`titles` can be a single identifier for a resource, or an array of values, if the following block batch-processes multiple resources in one pass. If that processing is not atomic, providers should instead use the non-block forms of logging, and provide accurate status reporting on the individual parts of update operations. A single `set()` execution may only log messages for instances it has been passed as part of the `changes` to process. Logging for foreign instances will cause an exception, as the runtime environment is not prepared for other resources to change. -The implementation is free to call different logging methods for different resources in any order it needs to. The only ordering restriction is for all calls specifying the same `title`. The `attribute_changed` logging needs to be done before that resource's action logging, and if a context is opened, it needs to be opened before any other logging for this resource. +The provider is free to call different logging methods for different resources in any order it needs to. The only ordering restriction is for all calls specifying the same `title`. The `attribute_changed` logging needs to be done before that resource's action logging, and if a context is opened, it needs to be opened before any other logging for this resource. # Known Limitations This API is not a full replacement for the power of 3.x style types and providers. Here is a (incomplete) list of missing pieces and thoughts on how to go about solving these. In the end, the goal of the new Resource API is not to be a complete replacement of prior art, but a cleaner way to get good results for the majority of simple cases. -## Multiple implementations +## Multiple providers for the same type -The previous version of this API allowed multiple implementations for the same resource type. This leads to the following problems: +The previous version of this API allowed multiple providers for the same resource type. This leads to the following problems: * attribute sprawl * missing features @@ -348,77 +349,12 @@ Currently anywhere "puppet 4 data types" are mentioned, only the built-in types ## Resources that can't be enumerated -Some resources, like files, cannot (or should not) be completely enumerated each time puppet runs. In some cases, the runtime environment knows that it doesn't require all resource instances. The current API does not provide a way to support those use-cases. An easy way forward would be to add a `find(title)` method that would return data for a single resource instance. A more involved solution my leverage PQL, but would require a much more sophisticated implementation. This also interacts with composite namevars. +Some resources, like files, cannot (or should not) be completely enumerated each time puppet runs. In some cases, the runtime environment knows that it doesn't require all resource instances. The current API does not provide a way to support those use-cases. An easy way forward would be to add a `find(title)` method that would return data for a single resource instance. A more involved solution my leverage PQL, but would require a much more sophisticated provider implementation. This also interacts with composite namevars. ## Catalog access -There is no way to access the catalog from the implementation. Several existing types rely on this to implement advanced functionality. Some of those use-cases would be better suited to be implemented as "external" catalog transformations, instead of munging the catalog from within the compilation process. +There is no way to access the catalog from the provider. Several existing types rely on this to implement advanced functionality. Some of those use-cases would be better suited to be implemented as "external" catalog transformations, instead of munging the catalog from within the compilation process. ## Logging for unmanaged instances -The implementation could provide log messages for resource instances that were not passed into the `set` call. In the current implementation those will be reported to the log, but will not cause the same resource-based reporting as a managed resource. How this is handeled in the future might change drastically. - - -# Earlier notes - -## Draft for new type and provider API - -The type and provider API has been the bane of my existence since I [started writing native resources](https://github.com/DavidS/puppet-mysql-old/commit/d33c7aa10e3a4bd9e97e947c471ee3ed36e9d1e2). Now, finally, we'll do something about it. I'm currently working on designing a nicer API for types and providers. My primary goals are to provide a smooth and simple ruby developer experience for both scripters and coders. Secondary goals were to eliminate server side code, and make puppet 4 data types available. Currently this is completely aspirational (i.e. no real code has been written), but early private feedback was encouraging. - -To showcase my vision, this [gist](https://gist.github.com/DavidS/430330ae43ba4b51fe34bd27ddbe4bc7) has the [apt_key type](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/type/apt_key.rb) and [provider](https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/apt_key/apt_key.rb) ported over to my proposal. The second example there is a more long-term teaser on what would become possible with such an API. - -The new API, like the existing, has two parts: the implementation that interacts with the actual resources, a.k.a. the provider, and information about what the implementation is all about. Due to the different usage patterns of the two parts, they need to be passed to puppet in two different calls: - -The `Puppet::SimpleResource.implement()` call receives the `current_state = get()` and `set(current_state, target_state, noop)` methods. `get` returns a list of discovered resources, while `set` takes the target state and enforces those goals on the subject. There is only a single (ruby) object throughout an agent run, that can easily do caching and what ever else is required for a good functioning of the provider. The state descriptions passed around are simple lists of key/value hashes describing resources. This will allow the implementation wide latitude in how to organise itself for simplicity and efficiency. - -The `Puppet::SimpleResource.define()` call provides a data-only description of the Type. This is all that is needed on the server side to compile a manifest. Thanks to puppet 4 data type checking, this will already be much more strict (with less effort) than possible with the current APIs, while providing more automatically readable documentation about the meaning of the attributes. - - -Details in no particular order: - -* All of this should fit on any unmodified puppet4 installation. It is completely additive and optional. Currently. - -* The Type definition - * It is data-only. - * Refers to puppet data types. - * No code runs on the server. - * This information can be re-used in all tooling around displaying/working with types (e.g. puppet-strings, console, ENC, etc.). - * autorelations are restricted to unmodified attribute values and constant values. - * No more `validate` or `munge`! For the edge cases not covered by data types, runtime checking can happen in the implementation on the agent. There it can use local system state (e.g. different mysql versions have different max table length constraints), and it will only fail the part of the resource tree, that is dependent on this error. There is already ample precedent for runtime validation, as most remote resources do not try to replicate the validation their target is already doing anyways. - * It maps 1:1 to the capabilities of PCore, and is similar to the libral interface description (see [libral#1](https://github.com/puppetlabs/libral/pull/2)). This ensures future interoperability between the different parts of the ecosystem. - * Related types can share common attributes by sharing/merging the attribute hashes. - * `defaults`, `read_only`, and similar data about attributes in the definition are mostly aesthetic at the current point in time, but will make for better documentation, and allow more intelligence built on top of this later. - -* The implementation are two simple functions `current_state = get()`, and `set(current_state, target_state, noop)`. - * `get` on its own is already useful for many things, like puppet resource. - * `set` receives the current state from `get`. While this is necessary for proper operation, there is a certain race condition there, if the system state changes between the calls. This is no different than what current implementations face, and they are well-equipped to deal with this. - * `set` is called with a list of resources, and can do batching if it is beneficial. This is not yet supported by the agent. - * the `current_state` and `target_state` values are lists of simple data structures built up of primitives like strings, numbers, hashes and arrays. They match the schema defined in the type. - * Calling `r.set(r.get, r.get)` would ensure the current state. This should run without any changes, proving the idempotency of the implementation. - * The ruby instance hosting the `get` and `set` functions is only alive for the duration of an agent transaction. An implementation can provide a `initialize` method to read credentials from the system, and setup other things as required. The single instance is used for all instances of the resource. - * There is no direct dependency on puppet core libraries in the implementation. - * While implementations can use utility functions, they are completely optional. - * The dependencies on the `logger`, `commands`, and similar utilities can be supplied by a small utility library (TBD). - -* Having a well-defined small API makes remoting, stacking, proxying, batching, interactive use, and other shenanigans possible, which will make for a interesting time ahead. - -* The logging of updates to the transaction is only a sketch. See the usage of `logger` throughout the example. I've tried different styles for fit. - * the `logger` is the primary way of reporting back information to the log, and the report. - * results can be streamed for immediate feedback - * block-based constructs allow detailed logging with little code ("Started X", "X: Doing Something", "X: Success|Failure", with one or two calls, and only one reference to X) - -* Obviously this is not sufficient to cover everything existing types and providers are able to do. For the first iteration we are choosing simplicity over functionality. - * Generating more resource instances for the catalog during compilation (e.g. file#recurse or concat) becomes impossible with a pure data-driven Type. There is still space in the API to add server-side code. - * Some resources (e.g. file, ssh_authorized_keys, concat) cannot or should not be prefetched. While it might not be convenient, a provider could always return nothing on the `get()` and do a more customized enforce motion in the `set()`. - * With current puppet versions, only "native" data types will be supported, as type aliases do not get pluginsynced. Yet. - * With current puppet versions, `puppet resource` can't load the data types, and therefore will not be able to take full advantage of this. Yet. - -* There is some "convenient" infrastructure (e.g. parsedfile) that needs porting over to this model. - -* Testing becomes possible on a completely new level. The test library can know how data is transformed outside the API, and - using the shape of the type - start generating test cases, and checking the actions of the implementation. This will require developer help to isolate the implementation from real systems, but it should go a long way towards reducing the tedium in writing tests. - - -What do you think about this? - - -Cheers, David +The provider could provide log messages for resource instances that were not passed into the `set` call. In the current implementation those will cause an error. How this is handeled in the future might change drastically. From 4d974476dc25db4363d4174e5afb117a1ae2a3eb Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 8 Mar 2017 15:03:35 +0000 Subject: [PATCH 14/62] Allow `get()` to provide filtering capabilities --- language/resource-api/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index baf0839..8dc5e22 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -63,7 +63,7 @@ The two fundamental operations to manage resources are reading and writing syste ```ruby Puppet::ResourceProvider.register('apt_key') do - def get + def get(names = nil) { 'name': { name: 'name', @@ -82,7 +82,7 @@ Puppet::ResourceProvider.register('apt_key') do end ``` -The `get` method returns a Hash of all resources currently available, keyed by their name. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The error message will be reported to the user. +The `get` method reports the current state of the managed resources. It is called with an array of resource names, or `nil`. It is expected to return a Hash of resources keyed by their name. These resources must at least contain the ones mentioned in the `names` array, but may contain more than those. As a special case, if the `names` parameter is `nil`, all existing resources should be returned. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The error message will be reported to the user. The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by the resource's name. Each value is another hash with a `:should` key, and an optional `:is` key. Those values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. For convenience, `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources was not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. When `noop` is set to true, the provider must not change the system state, but only report what it would change. The `set` method should always return `nil`. Any progress signalling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. From 2956b710d7af88f97a54594814472d84e720cf51 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 8 Mar 2017 15:15:18 +0000 Subject: [PATCH 15/62] Change get return type to an array Returning a hash was intended to protect against duplicate resources, but it is a hassle to implement, and duplicate resources at this point were never an issue in the first place. --- language/resource-api/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 8dc5e22..cde8208 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -64,12 +64,12 @@ The two fundamental operations to manage resources are reading and writing syste ```ruby Puppet::ResourceProvider.register('apt_key') do def get(names = nil) - { - 'name': { + [ + { name: 'name', # ... }, - } + ] end def set(changes, noop: false) @@ -82,7 +82,7 @@ Puppet::ResourceProvider.register('apt_key') do end ``` -The `get` method reports the current state of the managed resources. It is called with an array of resource names, or `nil`. It is expected to return a Hash of resources keyed by their name. These resources must at least contain the ones mentioned in the `names` array, but may contain more than those. As a special case, if the `names` parameter is `nil`, all existing resources should be returned. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The error message will be reported to the user. +The `get` method reports the current state of the managed resources. It is called with an array of resource names, or `nil`. It is expected to return an Array of resources. These resources must at least contain the ones mentioned in the `names` array, but may contain more than those. As a special case, if the `names` parameter is `nil`, all existing resources should be returned. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The error message will be reported to the user. The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by the resource's name. Each value is another hash with a `:should` key, and an optional `:is` key. Those values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. For convenience, `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources was not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. When `noop` is set to true, the provider must not change the system state, but only report what it would change. The `set` method should always return `nil`. Any progress signalling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. From faf46c4bd234b044fce4200b609f3d5aff489088 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 8 Mar 2017 15:16:09 +0000 Subject: [PATCH 16/62] Explicitly call out missing resources from get() return value --- language/resource-api/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index cde8208..6202c92 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -82,7 +82,7 @@ Puppet::ResourceProvider.register('apt_key') do end ``` -The `get` method reports the current state of the managed resources. It is called with an array of resource names, or `nil`. It is expected to return an Array of resources. These resources must at least contain the ones mentioned in the `names` array, but may contain more than those. As a special case, if the `names` parameter is `nil`, all existing resources should be returned. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The error message will be reported to the user. +The `get` method reports the current state of the managed resources. It is called with an array of resource names, or `nil`. It is expected to return an Array of resources. These resources must at least contain the ones mentioned in the `names` array, but may contain more than those. As a special case, if the `names` parameter is `nil`, all existing resources should be returned. If a requested resource is not listed in the result, it is considered to not exist on the system. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The error message will be reported to the user. The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by the resource's name. Each value is another hash with a `:should` key, and an optional `:is` key. Those values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. For convenience, `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources was not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. When `noop` is set to true, the provider must not change the system state, but only report what it would change. The `set` method should always return `nil`. Any progress signalling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. From d394c29b6ee1bfb9d6118716214b0dace124297e Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 4 Apr 2017 17:27:50 +0100 Subject: [PATCH 17/62] Update experimental apt_key implementation * add explanatory text * tighten up read_only attribute types --- language/resource-api/apt_key.rb | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/language/resource-api/apt_key.rb b/language/resource-api/apt_key.rb index e1b33cb..47949b0 100644 --- a/language/resource-api/apt_key.rb +++ b/language/resource-api/apt_key.rb @@ -1,3 +1,7 @@ + +# This is a experimental hardcoded implementation of what will be come the Resource API's runtime +# environment. This code is used as test-bed to see that the proposal is technically feasible. + require 'puppet/pops/patterns' require 'puppet/pops/utils' @@ -46,17 +50,17 @@ docs: 'Additional options to pass to apt-key\'s --keyserver-options.', }, fingerprint: { - type: 'String', + type: 'Pattern[/[a-f]{40}/]', docs: 'The 40-digit hexadecimal fingerprint of the specified GPG key.', read_only: true, }, long: { - type: 'String', + type: 'Pattern[/[a-f]{16}/]', docs: 'The 16-digit hexadecimal id of the specified GPG key.', read_only: true, }, short: { - type: 'String', + type: 'Pattern[/[a-f]{8}/]', docs: 'The 8-digit hexadecimal id of the specified GPG key.', read_only: true, }, From 959411ee8eebc42d7b622c44b27d8f21633fa7af Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 4 Apr 2017 17:28:51 +0100 Subject: [PATCH 18/62] Switch to unified attribute `kind` The different kinds of attributes are mutually exclusive. Changing this to a single key makes it easier to read, and reason about. --- language/resource-api/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 6202c92..0f3e8b0 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -26,15 +26,15 @@ Puppet::ResourceType.register( docs: 'Whether this apt key should be present or absent on the target system.' }, id: { - type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]', - docs: 'The ID of the key you want to manage.', - namevar: true, + type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]', + kind: :namevar, + docs: 'The ID of the key you want to manage.', }, # ... created: { - type: 'String', - docs: 'Date the key was created, in ISO format.', - read_only: true, + type: 'String', + kind: :read_only, + docs: 'Date the key was created, in ISO format.', }, }, autorequires: { @@ -48,7 +48,7 @@ The `Puppet::ResourceType.register(options)` function takes a Hash with the foll * `name`: the name of the resource type. For autoloading to work, the function call needs to go into `lib/puppet/type/.rb`. * `docs`: a doc string that describes the overall working of the resource type, gives examples, and explains pre-requisites as well as known issues. -* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `docs` string, and whether the attribute is the `namevar`, `read_only`, `init_only`, or a `parameter`. +* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `docs` string, and the `kind` of the attribute: `namevar`, `read_only`, `init_only`, or a `parameter`. * `namevar`: marks an attribute as part of the "primary key", or "identity" of the resource. A given set of namevar values needs to distinctively identify a instance. * `init_only`: this attribute can only be set during creation of the resource. Its value will be reported going forward, but trying to change it later will lead to an error. For example, the base image for a VM, or the UID of a user. * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example the checksum of a file, or the MAC address of a network interface. From 952e31312dbce552b82f1fedbb557e5e49302bac Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 4 Apr 2017 18:45:05 +0100 Subject: [PATCH 19/62] Add provider features and pull out the first two optional parts This will make the first step to a new provider smaller, while still providing the power and flexibility if required. --- language/resource-api/README.md | 53 ++++++++++++++++++++++++++++++--- 1 file changed, 49 insertions(+), 4 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 0f3e8b0..c691468 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -54,6 +54,7 @@ The `Puppet::ResourceType.register(options)` function takes a Hash with the foll * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example the checksum of a file, or the MAC address of a network interface. * `parameter`: these attributes influence how the provider behaves, and cannot be read from the target system. For example, the target file on inifile, or credentials to access an API. * `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a Hash mapping resource types to titles. Currently the titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, puppet will automatically create the relationsships requested here. +* `provider_features`: a list of feature names, specifying which optional parts of this spec the provider supports. Currently there are two defined: `simple_get_filter`, and `noop_handler`. See below for details. # Resource Provider @@ -63,7 +64,7 @@ The two fundamental operations to manage resources are reading and writing syste ```ruby Puppet::ResourceProvider.register('apt_key') do - def get(names = nil) + def get() [ { name: 'name', @@ -72,7 +73,7 @@ Puppet::ResourceProvider.register('apt_key') do ] end - def set(changes, noop: false) + def set(changes) changes.each do |name, change| is = change.has_key? :is ? change[:is] : get_single(name) should = change[:should] @@ -82,10 +83,54 @@ Puppet::ResourceProvider.register('apt_key') do end ``` -The `get` method reports the current state of the managed resources. It is called with an array of resource names, or `nil`. It is expected to return an Array of resources. These resources must at least contain the ones mentioned in the `names` array, but may contain more than those. As a special case, if the `names` parameter is `nil`, all existing resources should be returned. If a requested resource is not listed in the result, it is considered to not exist on the system. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The error message will be reported to the user. +The `get` method reports the current state of the managed resources. It is expected to return an Array of resources. Each resource is a Hash with attribute names as keys, and their respective values as values. It is an error to return values not matching the type specified in the resource type. If a requested resource is not listed in the result, it is considered to not exist on the system. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The error message will be reported to the user. + +The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by the resource's name. Each value is another hash with a `:should` key, and an optional `:is` key. Those values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. For convenience, `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources was not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. The `set` method should always return `nil`. Any progress signaling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the `set` method if there are changes to be made. Especially in the case of resources marked with `noop => true` (either locally, or through a global flag), the runtime will not pass them to `set`. See `noop_handler` below for changing this behaviour if required. + +## Provider Feature: simple_get_filter + +```ruby +Puppet::ResourceType.register( + name: 'apt_key', + features: [ 'simple_get_filter' ], +) + +Puppet::ResourceProvider.register('apt_key') do + def get(names = []) + [ + { + name: 'name', + # ... + }, + ] + end +``` + +Some resources are very expensive to enumerate. In this case the provider can implement `simple_get_filter` to signal extended capabilities of the `get` method to address this. The provider's `get` method will be called with an array of resource names, or `nil`. The `get` method must at least return the resources mentioned in the `names` array, but may return more than those. As a special case, if the `names` parameter is `nil`, all existing resources should be returned. To support simple runtimes, the `names` parameter should default to `[]`, to avoid unnecessary work if the runtime does not specify a filter at all. + +The runtime environment should call `get` with a minimal set of names it is interested in, and should keep track of additional instances returned, to avoid double querying. + +## Provider Feature: noop_handler -The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by the resource's name. Each value is another hash with a `:should` key, and an optional `:is` key. Those values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. For convenience, `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources was not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. When `noop` is set to true, the provider must not change the system state, but only report what it would change. The `set` method should always return `nil`. Any progress signalling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. +```ruby +Puppet::ResourceType.register( + name: 'apt_key', + features: [ 'noop_handler' ], +) + +Puppet::ResourceProvider.register('apt_key') do + def set(changes, noop: false) + changes.each do |name, change| + is = change.has_key? :is ? change[:is] : get_single(name) + should = change[:should] + # ... + do_something unless noop + end + end +end +``` +When a resource is marked with `noop => true`, either locally, or through a global flag, the standard runtime will emit the default change report with a `noop` flag set. In some cases an implementation can provide additional information (e.g. commands that would get executed), or requires additional evaluation before determining the effective changes (e.g. `exec`'s `onlyif` attribute). In those cases, the resource type can specify the `noop_handler` feature to have `set` called for all resources, even those flagged with `noop`. When the `noop` parameter is set to true, the provider must not change the system state, but only report what it would change. The `noop` parameter should default to `false` to allow simple runtimes to ignore this feature. # Runtime Environment From 36af9fadfaa46d159ea5a4a27106403b010c77d2 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 12 Apr 2017 16:15:30 +0100 Subject: [PATCH 20/62] Fix typo in `features` description --- language/resource-api/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index c691468..02d9cb5 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -54,7 +54,7 @@ The `Puppet::ResourceType.register(options)` function takes a Hash with the foll * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example the checksum of a file, or the MAC address of a network interface. * `parameter`: these attributes influence how the provider behaves, and cannot be read from the target system. For example, the target file on inifile, or credentials to access an API. * `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a Hash mapping resource types to titles. Currently the titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, puppet will automatically create the relationsships requested here. -* `provider_features`: a list of feature names, specifying which optional parts of this spec the provider supports. Currently there are two defined: `simple_get_filter`, and `noop_handler`. See below for details. +* `features`: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently there are two defined: `simple_get_filter`, and `noop_handler`. See below for details. # Resource Provider From d7a5d61b2e572f3a4e06df6242d0d20984cb1239 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Thu, 13 Apr 2017 14:26:28 +0100 Subject: [PATCH 21/62] Add the canonicalization feature --- language/resource-api/README.md | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 02d9cb5..ebf0b48 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -54,7 +54,7 @@ The `Puppet::ResourceType.register(options)` function takes a Hash with the foll * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example the checksum of a file, or the MAC address of a network interface. * `parameter`: these attributes influence how the provider behaves, and cannot be read from the target system. For example, the target file on inifile, or credentials to access an API. * `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a Hash mapping resource types to titles. Currently the titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, puppet will automatically create the relationsships requested here. -* `features`: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently there are two defined: `simple_get_filter`, and `noop_handler`. See below for details. +* `features`: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently defined: features: `canonicalize`, `simple_get_filter`, and `noop_handler`. See below for details. # Resource Provider @@ -87,6 +87,33 @@ The `get` method reports the current state of the managed resources. It is expec The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by the resource's name. Each value is another hash with a `:should` key, and an optional `:is` key. Those values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. For convenience, `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources was not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. The `set` method should always return `nil`. Any progress signaling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the `set` method if there are changes to be made. Especially in the case of resources marked with `noop => true` (either locally, or through a global flag), the runtime will not pass them to `set`. See `noop_handler` below for changing this behaviour if required. +## Provider Feature: canonicalize + +```ruby +Puppet::ResourceType.register( + name: 'apt_key', + features: [ 'canonicalize' ], +) + +Puppet::ResourceProvider.register('apt_key') do + def canonicalize(resources) + resources.collect do |r| + r[:name] = if r[:name].start_with?('0x') + r[:name][2..-1].upcase + else + r[:name].upcase + end + r + end + end +``` + +The runtime environment requires a provider to always use the same format for values to be able to correctly detect changes, and not produce false positives. In the example, the `apt_key` name is a hexadecimal number that can be written with, and without, the `'0x'` prefix, and the casing of the digits is irrelevant. The implementation has chosen to always use all upper case, and no prefix. To avoid inflicting a unneccessarily strict form on users, the `canonicalize` function transforms all allowed formats into the standard format. The only argument to `canonicalize` is a list of resource hashes matching the structure returned by `get`. The function should transform all values in those hashes into the canonical format returned by get. The runtime environment must use `canonicalize` before comparing user input values with values returned from get. The runtime environment must protect itself from modifications to the object passed in as `resources`, if it requires the original values later in its processing. + +> Note: When the provider implements canonicalisation, it should strive for always logging canonicalized values. By virtue of `get`, and `set` always producing and consuming canonically formatted values, this is not expected to pose extra overhead. + +> Note: A interesting side-effect of these rules is the fact that the canonicalization of `get`'s return value must not change the processed values. Runtime environments may have strict or development modes that check this property. + ## Provider Feature: simple_get_filter ```ruby From 0b05cc94e129f778136a456c681f557f8b05968d Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 21 Jun 2017 15:07:08 +0100 Subject: [PATCH 22/62] Clarify the `canonicalize` feature --- language/resource-api/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index ebf0b48..c91d259 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -108,7 +108,7 @@ Puppet::ResourceProvider.register('apt_key') do end ``` -The runtime environment requires a provider to always use the same format for values to be able to correctly detect changes, and not produce false positives. In the example, the `apt_key` name is a hexadecimal number that can be written with, and without, the `'0x'` prefix, and the casing of the digits is irrelevant. The implementation has chosen to always use all upper case, and no prefix. To avoid inflicting a unneccessarily strict form on users, the `canonicalize` function transforms all allowed formats into the standard format. The only argument to `canonicalize` is a list of resource hashes matching the structure returned by `get`. The function should transform all values in those hashes into the canonical format returned by get. The runtime environment must use `canonicalize` before comparing user input values with values returned from get. The runtime environment must protect itself from modifications to the object passed in as `resources`, if it requires the original values later in its processing. +The runtime environment needs to compare user input from the manifest (the desired state) with values returned from `get` (the actual state) to determine whether or not changes need to be effected. In simple cases, a provider will only accept values from the manifest in the same format as `get` would return. In this case no extra work is required, as a trivial value comparison will behave correctly. In many cases this would place a high burden on the user to provide values in an unnaturally constrained format. In the example, the `apt_key` name is a hexadecimal number that can be written with, and without, the `'0x'` prefix, and the casing of the digits is irrelevant. A trivial value comparison on the strings would cause false positives, when the user input format does not match. In this case the provider can specify the `canonicalize` feature and implement the `canonicalize` method to transform incoming values into the standard format required by the rest of the provider. The only argument to `canonicalize` is a list of resource hashes matching the structure returned by `get`. It returns all passed values in the same structure, with the required transformations applied. The runtime environment must use `canonicalize` before comparing user input values with values returned from get. The runtime environment must protect itself from modifications to the object passed in as `resources`, if it requires the original values later in its processing. > Note: When the provider implements canonicalisation, it should strive for always logging canonicalized values. By virtue of `get`, and `set` always producing and consuming canonically formatted values, this is not expected to pose extra overhead. From bd772fed1ebbd2ab5d8f842d25f845eb1e540e1a Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 21 Jun 2017 15:14:22 +0100 Subject: [PATCH 23/62] Fix canonicalization example --- language/resource-api/README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index c91d259..c5b7c1a 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -97,13 +97,12 @@ Puppet::ResourceType.register( Puppet::ResourceProvider.register('apt_key') do def canonicalize(resources) - resources.collect do |r| + resources.each do |r| r[:name] = if r[:name].start_with?('0x') r[:name][2..-1].upcase else r[:name].upcase end - r end end ``` From 7e8bbf2c286340a92324b2f495dc974999f30479 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 21 Jun 2017 18:19:03 +0100 Subject: [PATCH 24/62] fixup --- language/resource-api/apt_key.rb | 2 -- 1 file changed, 2 deletions(-) diff --git a/language/resource-api/apt_key.rb b/language/resource-api/apt_key.rb index 47949b0..f67060a 100644 --- a/language/resource-api/apt_key.rb +++ b/language/resource-api/apt_key.rb @@ -5,8 +5,6 @@ require 'puppet/pops/patterns' require 'puppet/pops/utils' -require 'pry' - DEFINITION = { name: 'apt_key', docs: <<-EOS, From 5526216695bfc6f7451893a728f686cddc99f6d4 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 26 Jul 2017 15:39:32 +0100 Subject: [PATCH 25/62] Fix syntax --- language/resource-api/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index c5b7c1a..833ea08 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -191,7 +191,7 @@ apt_key 'del', key_id, env: { 'LC_ALL': 'C' } By default the `stdout` of the command is logged to debug, while the `stderr` is logged to warning. To access the `stdout` in the provider, use the command name with `_lines` appended, and process it through the returned [Enumerable](http://ruby-doc.org/core/Enumerable.html) line-by-line. For example, to process the list of all apt keys: ```ruby -apt_key_lines(%w{adv --list-keys --with-colons --fingerprint --fixed-list-mode}).collect do |line| +apt_key_lines('adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode').collect do |line| # process each line here, and return a result end ``` From 5968441345ef4787b4201e1435fb952e83e20b97 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 26 Jul 2017 16:40:29 +0100 Subject: [PATCH 26/62] Change `kind` to `behaviour` to avoid confusion In preliminary testing `type` and `kind` have been too close for easy distinction between the two concepts. --- language/resource-api/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 833ea08..2f6cc02 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -27,13 +27,13 @@ Puppet::ResourceType.register( }, id: { type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]', - kind: :namevar, + behaviour: :namevar, docs: 'The ID of the key you want to manage.', }, # ... created: { type: 'String', - kind: :read_only, + behaviour: :read_only, docs: 'Date the key was created, in ISO format.', }, }, @@ -48,7 +48,7 @@ The `Puppet::ResourceType.register(options)` function takes a Hash with the foll * `name`: the name of the resource type. For autoloading to work, the function call needs to go into `lib/puppet/type/.rb`. * `docs`: a doc string that describes the overall working of the resource type, gives examples, and explains pre-requisites as well as known issues. -* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `docs` string, and the `kind` of the attribute: `namevar`, `read_only`, `init_only`, or a `parameter`. +* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `docs` string, and the `behaviour` of the attribute: `namevar`, `read_only`, `init_only`, or a `parameter`. * `namevar`: marks an attribute as part of the "primary key", or "identity" of the resource. A given set of namevar values needs to distinctively identify a instance. * `init_only`: this attribute can only be set during creation of the resource. Its value will be reported going forward, but trying to change it later will lead to an error. For example, the base image for a VM, or the UID of a user. * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example the checksum of a file, or the MAC address of a network interface. From bbf7292f1bd1607e48f9aacea77e367f35c6b641 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 2 Aug 2017 10:39:17 +0100 Subject: [PATCH 27/62] Improve wording around "Multiple providers for the same type" Acknowledge the shortcomings of the DSL solution. There are now two options for future implementation on the table, that both can address that. --- language/resource-api/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 2f6cc02..5809223 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -372,13 +372,13 @@ This API is not a full replacement for the power of 3.x style types and provider ## Multiple providers for the same type -The previous version of this API allowed multiple providers for the same resource type. This leads to the following problems: +The predecessor of this API allows multiple providers for the same resource type. This leads to the following problems: * attribute sprawl * missing features * convoluted implementations -puppet DSL already can address this: +While the puppet DSL can address this, it is cumbersome, and does not provide the same look-up capabilities as a type with multiple providers. ```puppet define package ( @@ -410,6 +410,8 @@ define package ( } ``` +Options for the future include forward-porting the status quo through enabling multiple Implementations to register for the same Definition, or allowing Definitions to declare (partial) equivalence to other Definitions (ala "`apt::package` is a `package`"). The former option is quite simple to implement, but carry forward the issues described above. The latter option will require more implementation work, but allows Definitions to stay at arms length of each other, providing better decoupling. + ## Composite namevars The current API does not provide a way to specify composite namevars. [`title_patterns`](https://github.com/puppetlabs/puppet-specifications/blob/master/language/resource_types.md#title-patterns) are already very data driven, and will be easy to add at a later point. From 30de9f1dae7d47361d1ffdc27571be283042856b Mon Sep 17 00:00:00 2001 From: Reid Vandewiele Date: Fri, 4 Aug 2017 15:00:33 +0100 Subject: [PATCH 28/62] Copy-edit multi-provider description --- language/resource-api/README.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 5809223..483ec8d 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -372,13 +372,17 @@ This API is not a full replacement for the power of 3.x style types and provider ## Multiple providers for the same type -The predecessor of this API allows multiple providers for the same resource type. This leads to the following problems: +The original Puppet Type and Provider API allows multiple providers for the same resource type. This allows the creation of abstract resource types, such as package, which can span multiple operating systems. Automatic selection of an os-appropriate provider means less work for the user, as they don't have to address in their code whether the package needs to be managed using apt, or managed using yum. -* attribute sprawl -* missing features -* convoluted implementations +Allowing multiple providers doesn't come for free though and in the previous implementation it incurs a number of complexity costs to be shouldered by the type or provider developer. -While the puppet DSL can address this, it is cumbersome, and does not provide the same look-up capabilities as a type with multiple providers. + attribute sprawl + disparate feature sets between the different providers for the same abstract type + complexity in implementation of both the type and provider pieces stemming from the two issues above + +The Resource API will not implement support for multiple providers at this time. + +Today, should support for multiple providers be highly desirable for a given type, the two options are: 1) use the older, more complex API. 2) implement multiple similar types using the Resource API, and select the platform-appropriate type in Puppet code. For example: ```puppet define package ( @@ -410,7 +414,7 @@ define package ( } ``` -Options for the future include forward-porting the status quo through enabling multiple Implementations to register for the same Definition, or allowing Definitions to declare (partial) equivalence to other Definitions (ala "`apt::package` is a `package`"). The former option is quite simple to implement, but carry forward the issues described above. The latter option will require more implementation work, but allows Definitions to stay at arms length of each other, providing better decoupling. +Neither of these options is ideal, thus it is documented as a limitation today. Ideas for the future include forward-porting the status quo through enabling multiple Implementations to register for the same Definition, or allowing Definitions to declare (partial) equivalence to other Definitions (ala "apt::package is a package"). ## Composite namevars From b3fcddaa2a26bcb0b15813b82e6bc98b56a6c491 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 29 Aug 2017 19:39:03 +0100 Subject: [PATCH 29/62] Another editing pass over the specification * Added some explanatory text for the main sections * Renamed to Puppet::ResourceAPI.register_(type|provider) to simplify namespaces. * Renamed `docs` to `desc` to match practices in ruby and puppet ecosystem * Changed a few cases from Array to Enumerable to allow more flexibility * Changed the default for simple_get_filter from `[]` to `nil` to allow simple runtime environments to continue to work * Removed the Known Limitation section on enumeration, as simple_get_filter now provides a solution to this --- language/resource-api/README.md | 145 ++++++++++++++++++-------------- 1 file changed, 80 insertions(+), 65 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 483ec8d..c6c0c8f 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -1,54 +1,58 @@ -# Resource API +# Puppet Resource API + +This libarary provides a simple way to write new native resources for [puppet](https://puppet.com). A *resource* is the basic thing that is managed by puppet. Each resource has a set of attributes describing its current state. Some of the attributes can be changed throughout the life-time of the resource, some attributes are only reported back, but cannot be changed (see `read_only`) others can only be set once during initial creation (see `init_only`). To gather information about those resources, and to enact changes in the real world, puppet requires a *provider* to implement this interaction. The provider can have parameters that influence its mode of operation (see `parameter`). To describe all these parts to the infrastructure, and the consumers, the resource *type* defines the all the metadata, including the list of the attributes. The *provider* contains the code to *get* and *set* the system state. -# Resource Definition +# Resource Definition ("Type") + +To make the resource known to the puppet ecosystem, its definition ("type") needs to be registered with puppet: ```ruby -Puppet::ResourceType.register( - name: 'apt_key', - docs: <<-EOS, - This type provides Puppet with the capabilities to manage GPG keys needed - by apt to perform package validation. Apt has it's own GPG keyring that can - be manipulated through the `apt-key` command. - - apt_key { '6F6B15509CF8E59E6E469F327F438280EF8D349F': - source => 'http://apt.puppetlabs.com/pubkey.gpg' - } +Puppet::ResourceApi.register_type( + name: 'apt_key', + desc: <<-EOS, + This type provides Puppet with the capabilities to manage GPG keys needed + by apt to perform package validation. Apt has it's own GPG keyring that can + be manipulated through the `apt-key` command. - **Autorequires**: - If Puppet is given the location of a key file which looks like an absolute - path this type will autorequire that file. - EOS - attributes: { - ensure: { - type: 'Enum[present, absent]', - docs: 'Whether this apt key should be present or absent on the target system.' - }, - id: { - type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]', - behaviour: :namevar, - docs: 'The ID of the key you want to manage.', - }, - # ... - created: { - type: 'String', - behaviour: :read_only, - docs: 'Date the key was created, in ISO format.', - }, + apt_key { '6F6B15509CF8E59E6E469F327F438280EF8D349F': + source => 'http://apt.puppetlabs.com/pubkey.gpg' + } + + **Autorequires**: + If Puppet is given the location of a key file which looks like an absolute + path this type will autorequire that file. + EOS + attributes: { + ensure: { + type: 'Enum[present, absent]', + desc: 'Whether this apt key should be present or absent on the target system.' + }, + id: { + type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]', + behaviour: :namevar, + desc: 'The ID of the key you want to manage.', }, - autorequires: { - file: '$source', # will evaluate to the value of the `source` attribute - package: 'apt', + # ... + created: { + type: 'String', + behaviour: :read_only, + desc: 'Date the key was created, in ISO format.', }, + }, + autorequires: { + file: '$source', # will evaluate to the value of the `source` attribute + package: 'apt', + }, ) ``` -The `Puppet::ResourceType.register(options)` function takes a Hash with the following top-level keys: +The `Puppet::ResourceApi.register_type(options)` function takes the following keyword arguments: -* `name`: the name of the resource type. For autoloading to work, the function call needs to go into `lib/puppet/type/.rb`. -* `docs`: a doc string that describes the overall working of the resource type, gives examples, and explains pre-requisites as well as known issues. -* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `docs` string, and the `behaviour` of the attribute: `namevar`, `read_only`, `init_only`, or a `parameter`. +* `name`: the name of the resource type. +* `desc`: a doc string that describes the overall working of the resource type, gives examples, and explains pre-requisites as well as known issues. +* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `desc` string, and the `behaviour` of the attribute: `namevar`, `read_only`, `init_only`, or a `parameter`. * `namevar`: marks an attribute as part of the "primary key", or "identity" of the resource. A given set of namevar values needs to distinctively identify a instance. * `init_only`: this attribute can only be set during creation of the resource. Its value will be reported going forward, but trying to change it later will lead to an error. For example, the base image for a VM, or the UID of a user. * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example the checksum of a file, or the MAC address of a network interface. @@ -56,20 +60,25 @@ The `Puppet::ResourceType.register(options)` function takes a Hash with the foll * `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a Hash mapping resource types to titles. Currently the titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, puppet will automatically create the relationsships requested here. * `features`: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently defined: features: `canonicalize`, `simple_get_filter`, and `noop_handler`. See below for details. -# Resource Provider +For autoloading to work, this code needs to go into `lib/puppet/type/.rb` in your module. -At runtime the current and intended system states for a specific resource are always represented as ruby Hashes of the resource's attributes, and applicable operational parameters. +# Resource Implementation ("Provider") -The two fundamental operations to manage resources are reading and writing system state. These operations are implemented in the `ResourceProvider` as `get` and `set`: +To effect changes on the real world, a resource also requires an implementation that makes the universe's state available to puppet, and causes the changes to bring reality to whatever state is requested in the catalog. The two fundamental operations to manage resources are reading and writing system state. These operations are implemented as `get` and `set`. + +At runtime the current and intended system states for a specific resource are always represented as ruby Hashes of the resource's attributes, and applicable operational parameters. ```ruby -Puppet::ResourceProvider.register('apt_key') do +Puppet::ResourceApi.register_provider('apt_key') do def get() [ { name: 'name', + ensure: 'present', + created: '2017-01-01', # ... }, + # ... ] end @@ -83,19 +92,25 @@ Puppet::ResourceProvider.register('apt_key') do end ``` -The `get` method reports the current state of the managed resources. It is expected to return an Array of resources. Each resource is a Hash with attribute names as keys, and their respective values as values. It is an error to return values not matching the type specified in the resource type. If a requested resource is not listed in the result, it is considered to not exist on the system. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The error message will be reported to the user. +The `get` method reports the current state of the managed resources. It returns an Enumerable of all existing resources. Each resource is a Hash with attribute names as keys, and their respective values as values. It is an error to return values not matching the type specified in the resource type. If a requested resource is not listed in the result, it is considered to not exist on the system. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The exception's message will be reported to the user. + +The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by the resource's name. Each value is another hash with the optional `:is` and `:should` keys. At least one of the two has to be specified. The values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. As a special case, a missing `:should` entry indicates that a resource should be removed from the system. Even a type implementing the `ensure => [present, absent]` attribute pattern for its human consumers, still has to react correctly on a missing `:should` entry. For convenience, and performance, `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources was not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. The `set` method should always return `nil`. Any progress signaling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the `set` method if there are changes to be made. Especially in the case of resources marked with `noop => true` (either locally, or through a global flag), the runtime will not pass them to `set`. See `noop_handler` below for changing this behaviour if required. -The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by the resource's name. Each value is another hash with a `:should` key, and an optional `:is` key. Those values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. For convenience, `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources was not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. The `set` method should always return `nil`. Any progress signaling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the `set` method if there are changes to be made. Especially in the case of resources marked with `noop => true` (either locally, or through a global flag), the runtime will not pass them to `set`. See `noop_handler` below for changing this behaviour if required. +## Provider Features + +There are some common cases where an implementation might want to provide a better experience in specific usecases than the default runtime environment can provide. To avoid burdening the simplest providers with that additional complexity, these cases are hidden behind feature flags. To enable the special handling, the Resource Definition has a `feature` key to list all features implemented by the provider. ## Provider Feature: canonicalize +Allows the provider to accept a wide range of formats for values without confusing the user. + ```ruby -Puppet::ResourceType.register( +Puppet::ResourceApi.register_type( name: 'apt_key', features: [ 'canonicalize' ], ) -Puppet::ResourceProvider.register('apt_key') do +Puppet::ResourceApi.register_provider('apt_key') do def canonicalize(resources) resources.each do |r| r[:name] = if r[:name].start_with?('0x') @@ -107,7 +122,9 @@ Puppet::ResourceProvider.register('apt_key') do end ``` -The runtime environment needs to compare user input from the manifest (the desired state) with values returned from `get` (the actual state) to determine whether or not changes need to be effected. In simple cases, a provider will only accept values from the manifest in the same format as `get` would return. In this case no extra work is required, as a trivial value comparison will behave correctly. In many cases this would place a high burden on the user to provide values in an unnaturally constrained format. In the example, the `apt_key` name is a hexadecimal number that can be written with, and without, the `'0x'` prefix, and the casing of the digits is irrelevant. A trivial value comparison on the strings would cause false positives, when the user input format does not match. In this case the provider can specify the `canonicalize` feature and implement the `canonicalize` method to transform incoming values into the standard format required by the rest of the provider. The only argument to `canonicalize` is a list of resource hashes matching the structure returned by `get`. It returns all passed values in the same structure, with the required transformations applied. The runtime environment must use `canonicalize` before comparing user input values with values returned from get. The runtime environment must protect itself from modifications to the object passed in as `resources`, if it requires the original values later in its processing. +The runtime environment needs to compare user input from the manifest (the desired state) with values returned from `get` (the actual state) to determine whether or not changes need to be effected. In simple cases, a provider will only accept values from the manifest in the same format as `get` would return. Then no extra work is required, as a trivial value comparison will suffice. In many cases this places a high burden on the user to provide values in an unnaturally constrained format. In the example, the `apt_key` name is a hexadecimal number that can be written with, and without, the `'0x'` prefix, and the casing of the digits is irrelevant. A trivial value comparison on the strings would cause false positives, when the user input format does not match, and there is no Hexadecimal type in the Puppet language. In this case the provider can specify the `canonicalize` feature and implement the `canonicalize` method. + +The `canonicalize` method transforms its arguments into the standard format required by the rest of the provider. The only argument to `canonicalize` is an Enumerable of resource hashes matching the structure returned by `get`. It returns all passed values in the same structure, with the required transformations applied. It is free to re-use, or recreate the data structures passed in as arguments. The runtime environment must use `canonicalize` before comparing user input values with values returned from `get`. The runtime environment must always pass canonicalized values into `set`. If the runtime environment must requires the original values for later processing, it must protect itself from modifications to the objects passed into `canonicalize`, for example through creating a deep copy of the objects. > Note: When the provider implements canonicalisation, it should strive for always logging canonicalized values. By virtue of `get`, and `set` always producing and consuming canonically formatted values, this is not expected to pose extra overhead. @@ -115,14 +132,16 @@ The runtime environment needs to compare user input from the manifest (the desir ## Provider Feature: simple_get_filter +Allows for more efficient querying of the system state when only specific bits are required. + ```ruby -Puppet::ResourceType.register( +Puppet::ResourceApi.register_type( name: 'apt_key', features: [ 'simple_get_filter' ], ) -Puppet::ResourceProvider.register('apt_key') do - def get(names = []) +Puppet::ResourceApi.register_provider('apt_key') do + def get(names = nil) [ { name: 'name', @@ -132,19 +151,19 @@ Puppet::ResourceProvider.register('apt_key') do end ``` -Some resources are very expensive to enumerate. In this case the provider can implement `simple_get_filter` to signal extended capabilities of the `get` method to address this. The provider's `get` method will be called with an array of resource names, or `nil`. The `get` method must at least return the resources mentioned in the `names` array, but may return more than those. As a special case, if the `names` parameter is `nil`, all existing resources should be returned. To support simple runtimes, the `names` parameter should default to `[]`, to avoid unnecessary work if the runtime does not specify a filter at all. +Some resources are very expensive to enumerate. In this case the provider can implement `simple_get_filter` to signal extended capabilities of the `get` method to address this. The provider's `get` method will be called with an Array of resource names, or `nil`. The `get` method must at least return the resources mentioned in the `names` Array, but may return more than those. As a special case, if the `names` parameter is `nil`, all existing resources should be returned. The `names` parameter should default to `nil` to allow simple runtimes to ignore this feature. -The runtime environment should call `get` with a minimal set of names it is interested in, and should keep track of additional instances returned, to avoid double querying. +The runtime environment should call `get` with a minimal set of names it is interested in, and should keep track of additional instances returned, to avoid double querying. To reap the most benefits from batching implementations, the runtime should minimize the number of calls into `get`. ## Provider Feature: noop_handler ```ruby -Puppet::ResourceType.register( +Puppet::ResourceApi.register_type( name: 'apt_key', features: [ 'noop_handler' ], ) -Puppet::ResourceProvider.register('apt_key') do +Puppet::ResourceApi.register_provider('apt_key') do def set(changes, noop: false) changes.each do |name, change| is = change.has_key? :is ? change[:is] : get_single(name) @@ -171,12 +190,12 @@ The runtime environment provides some utilities to make the providers's life eas To use CLI commands in a safe and comfortable manner, the provider can use the `commands` method to access shell commands. You can either specify a full path, or a bare command name. In the latter case puppet will use the system's `PATH` setting to search for the command. If the commands are not available, an error will be raised and the resources will fail in this run. The commands are aware of whether noop is in effect or not, and will signal success while skipping the real execution if necessary. Using these methods also causes the provider's actions to be logged at the appropriate levels. ```ruby -Puppet::ResourceImplementation.register('apt_key') do +Puppet::ResourceApi.register_provider('apt_key') do commands apt_key: '/usr/bin/apt-key' commands gpg: 'gpg' ``` -This will create methods called `apt_get`, and `gpg`, which will take CLI arguments in an array, and execute the command directly without any shell processing in a safe environment (clean working directory, clean environment). For example to call `apt-key` to delete a specific key by id: +This will create methods called `apt_get`, and `gpg`, which will take CLI arguments in an Array, and execute the command directly without any shell processing in a safe environment (clean working directory, clean environment). For example to call `apt-key` to delete a specific key by id: ```ruby apt_key 'del', key_id @@ -198,7 +217,7 @@ end > Note: the output of the command is streamed through the Enumerable. If the implementation requires the exit value of the command before processing, or wants to cache the output, use `to_a` to read the complete stream in one go. -If the command returns a non-zero exit code, an error is signalled to puppet. If this happens during `get`, all managed resources of this type will fail. If this happens during a `set`, all resources that have been scheduled for processing in this call, but not yet have been marked as a success will be marked as failed. To avoid this behaviour, call the `try_` prefix variant. In this (hypothetical) example, `apt-key` signals already deleted keys with an exit code of `1`, which is still OK when the provider is trying to delete the key: +If the command returns a non-zero exit code, an error is raised. If this happens during `get`, all managed resources of this type will fail. If this happens during a `set`, all resources that have been scheduled for processing in this call, but not yet have been marked as a success will be marked as failed. To avoid this behaviour, call the `try_` prefix variant. In this (hypothetical) example, `apt-key` signals already deleted keys with an exit code of `1`, which is still OK when the provider is trying to delete the key: ```ruby try_apt_key 'del', key_id @@ -219,7 +238,7 @@ The exit code is signalled through the ruby standard variable `$?` as a [`Proces ### Logging and Reporting -The provider needs to signal changes, successes and failures to the runtime environment. The `logger` is the primary way to do so. It provides a single interface for both the detailed technical information ofr later automatic processing, as well as human readable progress and status messages for operators. +The provider needs to signal changes, successes and failures to the runtime environment. The `logger` is the primary way to do so. It provides a single interface for both the detailed technical information for later automatic processing, as well as human readable progress and status messages for operators. #### General messages @@ -360,7 +379,7 @@ The following action/context methods are available: ** `err(message)` ** `err(titles, message:)` -`titles` can be a single identifier for a resource, or an array of values, if the following block batch-processes multiple resources in one pass. If that processing is not atomic, providers should instead use the non-block forms of logging, and provide accurate status reporting on the individual parts of update operations. +`titles` can be a single identifier for a resource, or an Array of values, if the following block batch-processes multiple resources in one pass. If that processing is not atomic, providers should instead use the non-block forms of logging, and provide accurate status reporting on the individual parts of update operations. A single `set()` execution may only log messages for instances it has been passed as part of the `changes` to process. Logging for foreign instances will cause an exception, as the runtime environment is not prepared for other resources to change. @@ -422,11 +441,7 @@ The current API does not provide a way to specify composite namevars. [`title_pa ## Puppet 4 data types -Currently anywhere "puppet 4 data types" are mentioned, only the built-in types are usable. This is because the type information is required on the agent, but puppet doesn't make it available yet. This work is tracked in [PUP-7197](https://tickets.puppetlabs.com/browse/PUP-7197), but even once that is implemented, modules will have to wait until the functionality is widely available, before being able to rely on that. - -## Resources that can't be enumerated - -Some resources, like files, cannot (or should not) be completely enumerated each time puppet runs. In some cases, the runtime environment knows that it doesn't require all resource instances. The current API does not provide a way to support those use-cases. An easy way forward would be to add a `find(title)` method that would return data for a single resource instance. A more involved solution my leverage PQL, but would require a much more sophisticated provider implementation. This also interacts with composite namevars. +Currently anywhere "puppet 4 data types" are mentioned, only the built-in types are usable. This is because the type information is required on the agent, but puppet doesn't make it available yet. This work is tracked in [PUP-7197](https://tickets.puppetlabs.com/browse/PUP-7197). Even once that is implemented, modules will have to wait until the functionality is widely available, before being able to rely on that. ## Catalog access From 7cf9ff6986d873ebc954a69044fb2aaaad638fb3 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Mon, 4 Sep 2017 16:45:05 +0100 Subject: [PATCH 30/62] Add a note on code sharing between providers --- language/resource-api/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index c6c0c8f..bc80091 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -450,3 +450,7 @@ There is no way to access the catalog from the provider. Several existing types ## Logging for unmanaged instances The provider could provide log messages for resource instances that were not passed into the `set` call. In the current implementation those will cause an error. How this is handeled in the future might change drastically. + +## Sharing code between providers + +Providers in the old API share code through inheritance, using the `:parent` key in the `provide()` call. To reduce entanglement between the business end of code, and the required interactions with the Resource API, it is recommended to put shared code in separate classes, that are used directly, instead of inheriting their contents. This can either happen through normal instantiation and usage, or for small chunks of code through a `Module`, and `include`. From 81b6abd44ec91fbbe1c4834b8137a5440dd628a4 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 5 Sep 2017 12:15:18 +0100 Subject: [PATCH 31/62] Improve description of the spec around attribute definition --- language/resource-api/README.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index bc80091..77bf8c9 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -53,10 +53,13 @@ The `Puppet::ResourceApi.register_type(options)` function takes the following ke * `name`: the name of the resource type. * `desc`: a doc string that describes the overall working of the resource type, gives examples, and explains pre-requisites as well as known issues. * `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `desc` string, and the `behaviour` of the attribute: `namevar`, `read_only`, `init_only`, or a `parameter`. - * `namevar`: marks an attribute as part of the "primary key", or "identity" of the resource. A given set of namevar values needs to distinctively identify a instance. - * `init_only`: this attribute can only be set during creation of the resource. Its value will be reported going forward, but trying to change it later will lead to an error. For example, the base image for a VM, or the UID of a user. - * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example the checksum of a file, or the MAC address of a network interface. - * `parameter`: these attributes influence how the provider behaves, and cannot be read from the target system. For example, the target file on inifile, or credentials to access an API. + * `type`: the puppet 4 data type allowed in this attribute. + * `desc`: a string describing this attribute. This is used in creating the automated API docs with [puppet-strings](https://github.com/puppetlabs/puppet-strings). + * `behaviour`/`behavior`: how the attribute behaves. Currently available values: + * `namevar`: marks an attribute as part of the "primary key", or "identity" of the resource. A given set of namevar values needs to distinctively identify a instance. + * `init_only`: this attribute can only be set during creation of the resource. Its value will be reported going forward, but trying to change it later will lead to an error. For example, the base image for a VM, or the UID of a user. + * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example the checksum of a file, or the MAC address of a network interface. + * `parameter`: these attributes influence how the provider behaves, and cannot be read from the target system. For example, the target file on inifile, or credentials to access an API. * `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a Hash mapping resource types to titles. Currently the titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, puppet will automatically create the relationsships requested here. * `features`: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently defined: features: `canonicalize`, `simple_get_filter`, and `noop_handler`. See below for details. From f8d4dde100a75d8afc292f46fe5359ed2545ff4f Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 5 Sep 2017 12:17:54 +0100 Subject: [PATCH 32/62] Define a `default` value for attributes --- language/resource-api/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 77bf8c9..3f52bdc 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -52,9 +52,10 @@ The `Puppet::ResourceApi.register_type(options)` function takes the following ke * `name`: the name of the resource type. * `desc`: a doc string that describes the overall working of the resource type, gives examples, and explains pre-requisites as well as known issues. -* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `desc` string, and the `behaviour` of the attribute: `namevar`, `read_only`, `init_only`, or a `parameter`. +* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `desc` string, a `default` value, and the `behaviour` of the attribute: `namevar`, `read_only`, `init_only`, or a `parameter`. * `type`: the puppet 4 data type allowed in this attribute. * `desc`: a string describing this attribute. This is used in creating the automated API docs with [puppet-strings](https://github.com/puppetlabs/puppet-strings). + * `default`: a default value that will be used by the runtime environment, whenever the caller doesn't specify a value for this attribute. * `behaviour`/`behavior`: how the attribute behaves. Currently available values: * `namevar`: marks an attribute as part of the "primary key", or "identity" of the resource. A given set of namevar values needs to distinctively identify a instance. * `init_only`: this attribute can only be set during creation of the resource. Its value will be reported going forward, but trying to change it later will lead to an error. For example, the base image for a VM, or the UID of a user. From 66ec5bd0efdf10d5d955e2dac2c948a49a3efb48 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Thu, 7 Sep 2017 15:21:34 +0100 Subject: [PATCH 33/62] Change the Implementation to a plain class By providing the implementation as a plain class, the complexity of the runtime environment is gretly reduced. This makes testing much easier, and removes a big chunk of magic from the stack. The required follow-on change is to remove the `commands`, and `logger` DSL from the provider environment. `commands` are replaced by a much nicer explicit API based on the `childprocess` gem, enabling proper background processing for those who need it, while preserving simple calling conventions for common use-cases. The `logger` is replaced by a libral-like `context` object that is passed through to `get`/`set`. --- language/resource-api/README.md | 154 +++++++++++++++++++++----------- 1 file changed, 100 insertions(+), 54 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 3f52bdc..6977e50 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -68,13 +68,13 @@ For autoloading to work, this code needs to go into `lib/puppet/type/.rb` # Resource Implementation ("Provider") -To effect changes on the real world, a resource also requires an implementation that makes the universe's state available to puppet, and causes the changes to bring reality to whatever state is requested in the catalog. The two fundamental operations to manage resources are reading and writing system state. These operations are implemented as `get` and `set`. +To effect changes on the real world, a resource also requires an implementation that makes the universe's state available to puppet, and causes the changes to bring reality to whatever state is requested in the catalog. The two fundamental operations to manage resources are reading and writing system state. These operations are implemented as `get` and `set`. The implementation itself is a basic Ruby class in the `Puppet::Provider` namespace, named after the Type using CamelCase. At runtime the current and intended system states for a specific resource are always represented as ruby Hashes of the resource's attributes, and applicable operational parameters. ```ruby -Puppet::ResourceApi.register_provider('apt_key') do - def get() +class Puppet::Provider::AptKey + def get(context) [ { name: 'name', @@ -86,7 +86,7 @@ Puppet::ResourceApi.register_provider('apt_key') do ] end - def set(changes) + def set(context, changes) changes.each do |name, change| is = change.has_key? :is ? change[:is] : get_single(name) should = change[:should] @@ -100,6 +100,8 @@ The `get` method reports the current state of the managed resources. It returns The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by the resource's name. Each value is another hash with the optional `:is` and `:should` keys. At least one of the two has to be specified. The values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. As a special case, a missing `:should` entry indicates that a resource should be removed from the system. Even a type implementing the `ensure => [present, absent]` attribute pattern for its human consumers, still has to react correctly on a missing `:should` entry. For convenience, and performance, `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources was not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. The `set` method should always return `nil`. Any progress signaling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the `set` method if there are changes to be made. Especially in the case of resources marked with `noop => true` (either locally, or through a global flag), the runtime will not pass them to `set`. See `noop_handler` below for changing this behaviour if required. +Both methods take a `context` parameter which provides utilties from the Runtime Environment, and is decribed in more detail there. + ## Provider Features There are some common cases where an implementation might want to provide a better experience in specific usecases than the default runtime environment can provide. To avoid burdening the simplest providers with that additional complexity, these cases are hidden behind feature flags. To enable the special handling, the Resource Definition has a `feature` key to list all features implemented by the provider. @@ -114,8 +116,8 @@ Puppet::ResourceApi.register_type( features: [ 'canonicalize' ], ) -Puppet::ResourceApi.register_provider('apt_key') do - def canonicalize(resources) +class Puppet::Provider::AptKey + def canonicalize(context, resources) resources.each do |r| r[:name] = if r[:name].start_with?('0x') r[:name][2..-1].upcase @@ -128,7 +130,9 @@ Puppet::ResourceApi.register_provider('apt_key') do The runtime environment needs to compare user input from the manifest (the desired state) with values returned from `get` (the actual state) to determine whether or not changes need to be effected. In simple cases, a provider will only accept values from the manifest in the same format as `get` would return. Then no extra work is required, as a trivial value comparison will suffice. In many cases this places a high burden on the user to provide values in an unnaturally constrained format. In the example, the `apt_key` name is a hexadecimal number that can be written with, and without, the `'0x'` prefix, and the casing of the digits is irrelevant. A trivial value comparison on the strings would cause false positives, when the user input format does not match, and there is no Hexadecimal type in the Puppet language. In this case the provider can specify the `canonicalize` feature and implement the `canonicalize` method. -The `canonicalize` method transforms its arguments into the standard format required by the rest of the provider. The only argument to `canonicalize` is an Enumerable of resource hashes matching the structure returned by `get`. It returns all passed values in the same structure, with the required transformations applied. It is free to re-use, or recreate the data structures passed in as arguments. The runtime environment must use `canonicalize` before comparing user input values with values returned from `get`. The runtime environment must always pass canonicalized values into `set`. If the runtime environment must requires the original values for later processing, it must protect itself from modifications to the objects passed into `canonicalize`, for example through creating a deep copy of the objects. +The `canonicalize` method transforms its `resources` argument into the standard format required by the rest of the provider. The `resources` argument to `canonicalize` is an Enumerable of resource hashes matching the structure returned by `get`. It returns all passed values in the same structure, with the required transformations applied. It is free to re-use, or recreate the data structures passed in as arguments. The runtime environment must use `canonicalize` before comparing user input values with values returned from `get`. The runtime environment must always pass canonicalized values into `set`. If the runtime environment must requires the original values for later processing, it must protect itself from modifications to the objects passed into `canonicalize`, for example through creating a deep copy of the objects. + +The `context` parameter is the same as passed to `get` and `set` which provides utilties from the Runtime Environment, and is decribed in more detail there. > Note: When the provider implements canonicalisation, it should strive for always logging canonicalized values. By virtue of `get`, and `set` always producing and consuming canonically formatted values, this is not expected to pose extra overhead. @@ -144,8 +148,8 @@ Puppet::ResourceApi.register_type( features: [ 'simple_get_filter' ], ) -Puppet::ResourceApi.register_provider('apt_key') do - def get(names = nil) +class Puppet::Provider::AptKey + def get(context, names = nil) [ { name: 'name', @@ -167,8 +171,8 @@ Puppet::ResourceApi.register_type( features: [ 'noop_handler' ], ) -Puppet::ResourceApi.register_provider('apt_key') do - def set(changes, noop: false) +class Puppet::Provider::AptKey + def set(context, changes, noop: false) changes.each do |name, change| is = change.has_key? :is ? change[:is] : get_single(name) should = change[:should] @@ -191,65 +195,107 @@ The runtime environment provides some utilities to make the providers's life eas ### Commands -To use CLI commands in a safe and comfortable manner, the provider can use the `commands` method to access shell commands. You can either specify a full path, or a bare command name. In the latter case puppet will use the system's `PATH` setting to search for the command. If the commands are not available, an error will be raised and the resources will fail in this run. The commands are aware of whether noop is in effect or not, and will signal success while skipping the real execution if necessary. Using these methods also causes the provider's actions to be logged at the appropriate levels. +To use CLI commands in a safe and comfortable manner, the Resource API provides a thin wrapper around the excellent [childprocess gem](https://rubygems.org/gems/childprocess) to address the most common use-cases. Through using the library commands and their arguments are never passed through the shell leading to a safer execution environment (no funny parsing), and faster execution times (no extra processes). + +#### Creating a Command + +To create a re-usable command, create a new instance of `Puppet::ResourceApi::Command` passing in the command. You can either specify a full path, or a bare command name. In the latter case the Command will use the system's `PATH` setting to search for the command. ```ruby -Puppet::ResourceApi.register_provider('apt_key') do - commands apt_key: '/usr/bin/apt-key' - commands gpg: 'gpg' +class Puppet::Provider::AptKey + def initialize + @apt_key_cmd = Puppet::ResourceApi::Command.new('/usr/bin/apt-key') + @gpg_cmd = Puppet::ResourceApi::Command.new('gpg') + end ``` -This will create methods called `apt_get`, and `gpg`, which will take CLI arguments in an Array, and execute the command directly without any shell processing in a safe environment (clean working directory, clean environment). For example to call `apt-key` to delete a specific key by id: +It is recommended to create the command in the `initialize` function of the provider, and store them in a member named after the command, with the `_cmd` suffix. This makes it easy to re-use common settings throughout the provider. + +You can set default environment variables on the `@cmd.environment` Hash, and a default working directory using `@cmd.cwd=`. + +#### Running simple commands + +The `run(*args)` method takes any number of arguments, and executes the command using them. For example to call `apt-key` to delete a specific key by id: ```ruby -apt_key 'del', key_id +class Puppet::Provider::AptKey + def set(context, changes, noop: false) + # ... + @apt_key_cmd.run(context, 'del', key_id) ``` -To pass additional environment variables through to the command, pass a hash of them as `env:`: +If the command is not available, a `Puppet::ResourceApi::CommandNotFoundError` will be raised. This can be easily used to fail the resources for a specific run, if the requirements for the provider are not yet met. + +The call will only return after the command has finished executing. If the command exits with a exitstatus indicating an error condition (that is non-zero), a `Puppet::ResourceApi::CommandExecutionError` is raised, containing the details of the command, and exit status. + +Through the context, the commands are aware of whether noop is in effect or not, and will signal success while skipping the real execution if necessary. Using these methods also causes the provider's actions to be logged at the appropriate levels. + +To pass additional environment variables through to the command, pass a hash of them as `environment:`: ```ruby -apt_key 'del', key_id, env: { 'LC_ALL': 'C' } +@apt_key_cmd.run('del', key_id, environment: { 'LC_ALL': 'C' }) ``` -By default the `stdout` of the command is logged to debug, while the `stderr` is logged to warning. To access the `stdout` in the provider, use the command name with `_lines` appended, and process it through the returned [Enumerable](http://ruby-doc.org/core/Enumerable.html) line-by-line. For example, to process the list of all apt keys: +By default the `stdout` of the command is logged to debug, while the `stderr` is logged to warning. + +#### Processing commands + +For more involved scenarios, variants of `@cmd.start` take the same arguments as `run`, but will start the command in the background, and return a handle to that process. The different variants have different defaults in how the process is set up. The handle provides functionality to interact with the command, and query its state. + +To use a command to read information from the system, `start_read` does not allow input to the process, and its `stderr` is logged at the warning level. The handle's `stdout` attribute can be used to access the normal output of the command through an [`IO`](https://ruby-doc.org/core/IO.html) object. For example, to process the list of all apt keys: ```ruby -apt_key_lines('adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode').collect do |line| - # process each line here, and return a result -end +class Puppet::Provider::AptKey + def get(context) + @apt_key_cmd.start_read(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode') do |handle| + handle.stdout.each_line.collect do |line| + # process each line here, and compute a Hash + end + end + end ``` -> Note: the output of the command is streamed through the Enumerable. If the implementation requires the exit value of the command before processing, or wants to cache the output, use `to_a` to read the complete stream in one go. - -If the command returns a non-zero exit code, an error is raised. If this happens during `get`, all managed resources of this type will fail. If this happens during a `set`, all resources that have been scheduled for processing in this call, but not yet have been marked as a success will be marked as failed. To avoid this behaviour, call the `try_` prefix variant. In this (hypothetical) example, `apt-key` signals already deleted keys with an exit code of `1`, which is still OK when the provider is trying to delete the key: +To use a command to write to, `start_write` allows input into the process, but will only log its output like `run` does. For example, to provide a key on stdin to the apt-key tool: ```ruby -try_apt_key 'del', key_id +class Puppet::Provider::AptKey + def set(context, changes) + # ... + @apt_key_cmd.start_write(context, 'add', '-') do |handle| + handle.stdin.puts the_key + end + end +``` -if [0, 1].contains $?.exitstatus - # success, or already deleted -else - # fail -end +Like the `run` method, the block forms of `start` will wait after the block has finished processing, to make sure that the command has exited cleanly, and will raise an error if the command returns a non-zero exit code. + +#### Advanced scenarios + +For advanced scenarios, the plain `start` method returns a handle with the `stdin`, `stdout`, and `stderr` pipes open, and unhandled. + +This can be particularily useful together with providing your own `IO` objects, by using the `stdin:`, `stdout:`, and `stderr:` keyword arguments. For example redirecting the output of a command to a temporary file: + +```ruby +error_out = Tempfile.new('err') +@apt_key_cmd.start('add', '-', stdin: File.open('/tmp/key_in.gpg'), stdout: nil, stderr: error_out) ``` -The exit code is signalled through the ruby standard variable `$?` as a [`Process::Status` object](https://ruby-doc.org/core/Process/Status.html) +> Note that due to buffering on the OS level (or lack thereof), bidirectional communication with that command can randomly hang your process, unless you take extra care only using the non-blocking methods on `IO`. Depending on your needs, you can also go straight to the childprocess gem. + +The handle also can be used to query whether the process is still running with `alive?`, and `exited?`, access the `exit_code` of the command, `wait` for it to finish, or poll for it to exit using `poll_for_exit(seconds)`, and `stop` the process. All those methods correspond to their respective counterparts on [`ChildProcess::AbstractProcess`](http://www.rubydoc.info/gems/childprocess/ChildProcess/AbstractProcess). - +> Note: If you don't provide a block to the `start` methods, you will have to take care of exit code handling yourself. ### Logging and Reporting -The provider needs to signal changes, successes and failures to the runtime environment. The `logger` is the primary way to do so. It provides a single interface for both the detailed technical information for later automatic processing, as well as human readable progress and status messages for operators. +The provider needs to signal changes, successes and failures to the runtime environment. The `context` is the primary way to do so. It provides a single interface for both the detailed technical information for later automatic processing, as well as human readable progress and status messages for operators. #### General messages -To provide feedback about the overall operation of the provider, the logger has the usual set of [loglevel](https://docs.puppet.com/puppet/latest/metaparameter.html#loglevel) methods that take a string, and pass that up to runtime environment's logging infrastructure: +To provide feedback about the overall operation of the provider, the `context` has the usual set of [loglevel](https://docs.puppet.com/puppet/latest/metaparameter.html#loglevel) methods that take a string, and pass that up to runtime environment's logging infrastructure: ```ruby -logger.warning("Unexpected state detected, continuing in degraded mode.") +context.warning("Unexpected state detected, continuing in degraded mode.") ``` will result in the following message: @@ -271,29 +317,29 @@ See [wikipedia](https://en.wikipedia.org/wiki/Syslog#Severity_level) and [RFC424 In many simple cases, a provider can pass off the real work to a external tool, detailed logging happens there, and reporting back to puppet only requires acknowledging those changes. In these situations, signalling can be as easy as this: -``` -apt_key action, key_id -logger.processed(key_id, is, should) +```ruby +@apt_key_cmd.run(context, action, key_id) +context.processed(key_id, is, should) ``` This will report all changes from `is` to `should`, using default messages. -Providers that want to have more control over the logging throughout the processing can use the more specific `created(title)`, `updated(title)`, `deleted(title)`, `unchanged(title)` methods for that. To report the change of an attribute, the `logger` provides a `attribute_changed(title, attribute, old_value, new_value, message)` method. +Providers that want to have more control over the logging throughout the processing can use the more specific `created(title)`, `updated(title)`, `deleted(title)`, `unchanged(title)` methods for that. To report the change of an attribute, the `context` provides a `attribute_changed(title, attribute, old_value, new_value, message)` method. #### Logging contexts -Most of those messages are expected to be relative to a specific resource instance, and a specific operation on that instance. To enable detailed logging without repeating key arguments, and provide consistent error logging, the logger provides *logging context* methods that capture the current action and resource instance. +Most of those messages are expected to be relative to a specific resource instance, and a specific operation on that instance. To enable detailed logging without repeating key arguments, and provide consistent error logging, the context provides *logging context* methods that capture the current action and resource instance. ```ruby -logger.updating(title) do +context.updating(title) do if key_not_found - logger.warning('Original key not found') + context.warning('Original key not found') end # Update the key by calling CLI tool apt_key(...) - logger.attribute_changed('content', nil, content_hash, + context.attribute_changed('content', nil, content_hash, message: "Replaced with content hash #{content_hash}") end ``` @@ -324,24 +370,24 @@ Logging contexts process all exceptions. [`StandardError`s](https://ruby-doc.org The equivalent long-hand form with manual error handling: ```ruby -logger.updating(title) +context.updating(title) begin if key_not_found - logger.warning(title, message: 'Original key not found') + context.warning(title, message: 'Original key not found') end # Update the key by calling CLI tool try_apt_key(...) if $?.exitstatus != 0 - logger.error(title, "Failed executing apt-key #{...}") + context.error(title, "Failed executing apt-key #{...}") else - logger.attribute_changed(title, 'content', nil, content_hash, + context.attribute_changed(title, 'content', nil, content_hash, message: "Replaced with content hash #{content_hash}") end - logger.changed(title) + context.changed(title) rescue Exception => e - logger.error(title, e, message: 'Updating failed') + context.error(title, e, message: 'Updating failed') raise unless e.is_a? StandardError end ``` From 57fca3507eca095f19ca4b28592614c1ca795212 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Thu, 7 Sep 2017 16:38:19 +0100 Subject: [PATCH 34/62] Add a note on puppet's requirements around autoloading While the original formulation would be preferrable, in this case the puppet code base forces my hand, and I consider it preferrable in this case to be a bit ugly, but consistent rather than layering more surprises on top. --- language/resource-api/README.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 6977e50..df46083 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -68,12 +68,14 @@ For autoloading to work, this code needs to go into `lib/puppet/type/.rb` # Resource Implementation ("Provider") -To effect changes on the real world, a resource also requires an implementation that makes the universe's state available to puppet, and causes the changes to bring reality to whatever state is requested in the catalog. The two fundamental operations to manage resources are reading and writing system state. These operations are implemented as `get` and `set`. The implementation itself is a basic Ruby class in the `Puppet::Provider` namespace, named after the Type using CamelCase. +To effect changes on the real world, a resource also requires an implementation that makes the universe's state available to puppet, and causes the changes to bring reality to whatever state is requested in the catalog. The two fundamental operations to manage resources are reading and writing system state. These operations are implemented as `get` and `set`. The implementation itself is a basic Ruby class in the `Puppet::Provider` namespace, named after the Type using CamelCase. + +> Note: Due to the way puppet autoload works, this has to be in a file called `puppet/provider//.rb` and the class will also have the CamelCased type name twice. At runtime the current and intended system states for a specific resource are always represented as ruby Hashes of the resource's attributes, and applicable operational parameters. ```ruby -class Puppet::Provider::AptKey +class Puppet::Provider::AptKey::AptKey def get(context) [ { @@ -116,7 +118,7 @@ Puppet::ResourceApi.register_type( features: [ 'canonicalize' ], ) -class Puppet::Provider::AptKey +class Puppet::Provider::AptKey::AptKey def canonicalize(context, resources) resources.each do |r| r[:name] = if r[:name].start_with?('0x') @@ -148,7 +150,7 @@ Puppet::ResourceApi.register_type( features: [ 'simple_get_filter' ], ) -class Puppet::Provider::AptKey +class Puppet::Provider::AptKey::AptKey def get(context, names = nil) [ { @@ -171,7 +173,7 @@ Puppet::ResourceApi.register_type( features: [ 'noop_handler' ], ) -class Puppet::Provider::AptKey +class Puppet::Provider::AptKey::AptKey def set(context, changes, noop: false) changes.each do |name, change| is = change.has_key? :is ? change[:is] : get_single(name) @@ -202,7 +204,7 @@ To use CLI commands in a safe and comfortable manner, the Resource API provides To create a re-usable command, create a new instance of `Puppet::ResourceApi::Command` passing in the command. You can either specify a full path, or a bare command name. In the latter case the Command will use the system's `PATH` setting to search for the command. ```ruby -class Puppet::Provider::AptKey +class Puppet::Provider::AptKey::AptKey def initialize @apt_key_cmd = Puppet::ResourceApi::Command.new('/usr/bin/apt-key') @gpg_cmd = Puppet::ResourceApi::Command.new('gpg') @@ -218,7 +220,7 @@ You can set default environment variables on the `@cmd.environment` Hash, and a The `run(*args)` method takes any number of arguments, and executes the command using them. For example to call `apt-key` to delete a specific key by id: ```ruby -class Puppet::Provider::AptKey +class Puppet::Provider::AptKey::AptKey def set(context, changes, noop: false) # ... @apt_key_cmd.run(context, 'del', key_id) @@ -245,7 +247,7 @@ For more involved scenarios, variants of `@cmd.start` take the same arguments as To use a command to read information from the system, `start_read` does not allow input to the process, and its `stderr` is logged at the warning level. The handle's `stdout` attribute can be used to access the normal output of the command through an [`IO`](https://ruby-doc.org/core/IO.html) object. For example, to process the list of all apt keys: ```ruby -class Puppet::Provider::AptKey +class Puppet::Provider::AptKey::AptKey def get(context) @apt_key_cmd.start_read(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode') do |handle| handle.stdout.each_line.collect do |line| @@ -258,7 +260,7 @@ class Puppet::Provider::AptKey To use a command to write to, `start_write` allows input into the process, but will only log its output like `run` does. For example, to provide a key on stdin to the apt-key tool: ```ruby -class Puppet::Provider::AptKey +class Puppet::Provider::AptKey::AptKey def set(context, changes) # ... @apt_key_cmd.start_write(context, 'add', '-') do |handle| From e9744649876ece0c9fd4319e92a54b0a6ba37769 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Fri, 8 Sep 2017 09:02:59 +0100 Subject: [PATCH 35/62] Remove the automatic noop handling of Commands This removes another bit of magic, and allows developers to use commands, even under noop, if they are required for correct logging. Using an explicit keyword parameter avoids the need for too many conditionals or nesting. --- language/resource-api/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index df46083..9e5b1a8 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -223,14 +223,16 @@ The `run(*args)` method takes any number of arguments, and executes the command class Puppet::Provider::AptKey::AptKey def set(context, changes, noop: false) # ... - @apt_key_cmd.run(context, 'del', key_id) + @apt_key_cmd.run(context, 'del', key_id, noop: noop) ``` If the command is not available, a `Puppet::ResourceApi::CommandNotFoundError` will be raised. This can be easily used to fail the resources for a specific run, if the requirements for the provider are not yet met. The call will only return after the command has finished executing. If the command exits with a exitstatus indicating an error condition (that is non-zero), a `Puppet::ResourceApi::CommandExecutionError` is raised, containing the details of the command, and exit status. -Through the context, the commands are aware of whether noop is in effect or not, and will signal success while skipping the real execution if necessary. Using these methods also causes the provider's actions to be logged at the appropriate levels. +The commands take a `noop:` keyword argument, and will signal success while skipping the real execution if necessary. + +Using these methods also causes the provider's actions to be logged at the appropriate levels. To pass additional environment variables through to the command, pass a hash of them as `environment:`: From 027aa8e20327bacbdd8ea6e5d9a6ef423a48eb5a Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Mon, 11 Sep 2017 18:56:13 +0100 Subject: [PATCH 36/62] Add missing `context` argument in example --- language/resource-api/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 9e5b1a8..e10207b 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -237,7 +237,7 @@ Using these methods also causes the provider's actions to be logged at the appro To pass additional environment variables through to the command, pass a hash of them as `environment:`: ```ruby -@apt_key_cmd.run('del', key_id, environment: { 'LC_ALL': 'C' }) +@apt_key_cmd.run(context, 'del', key_id, environment: { 'LC_ALL': 'C' }) ``` By default the `stdout` of the command is logged to debug, while the `stderr` is logged to warning. From 4e9198368739712c8d16e75ace29f2d9384e24da Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 12 Sep 2017 13:26:32 +0100 Subject: [PATCH 37/62] Update the process handle to be a straight up childprocess process --- language/resource-api/README.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index e10207b..e83e001 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -244,16 +244,16 @@ By default the `stdout` of the command is logged to debug, while the `stderr` is #### Processing commands -For more involved scenarios, variants of `@cmd.start` take the same arguments as `run`, but will start the command in the background, and return a handle to that process. The different variants have different defaults in how the process is set up. The handle provides functionality to interact with the command, and query its state. +For more involved scenarios, variants of `@cmd.start` take the same arguments as `run`, but will start the command in the background, and return a handle to that process. The different variants have different defaults in how the process is set up. The `process` handle provides functionality to interact with the command, and query its state. -To use a command to read information from the system, `start_read` does not allow input to the process, and its `stderr` is logged at the warning level. The handle's `stdout` attribute can be used to access the normal output of the command through an [`IO`](https://ruby-doc.org/core/IO.html) object. For example, to process the list of all apt keys: +To use a command to read information from the system, `start_read` does not allow input to the process, and its `stderr` is logged at the warning level. The process' `io.stdout` attribute can be used to access the normal output of the command through an [`IO`](https://ruby-doc.org/core/IO.html) object. For example, to process the list of all apt keys: ```ruby class Puppet::Provider::AptKey::AptKey def get(context) - @apt_key_cmd.start_read(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode') do |handle| - handle.stdout.each_line.collect do |line| - # process each line here, and compute a Hash + @apt_key_cmd.start_read(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode') do |process| + process.io.stdout.each_line.collect do |line| + # handle each line here, and compute a Hash end end end @@ -265,8 +265,8 @@ To use a command to write to, `start_write` allows input into the process, but w class Puppet::Provider::AptKey::AptKey def set(context, changes) # ... - @apt_key_cmd.start_write(context, 'add', '-') do |handle| - handle.stdin.puts the_key + @apt_key_cmd.start_write(context, 'add', '-') do |process| + process.io.stdin.puts the_key end end ``` @@ -275,18 +275,19 @@ Like the `run` method, the block forms of `start` will wait after the block has #### Advanced scenarios -For advanced scenarios, the plain `start` method returns a handle with the `stdin`, `stdout`, and `stderr` pipes open, and unhandled. +For advanced scenarios, the plain `start` method returns a `process` handle with the `stdin`, `stdout`, and `stderr` pipes open, and unhandled. This can be particularily useful together with providing your own `IO` objects, by using the `stdin:`, `stdout:`, and `stderr:` keyword arguments. For example redirecting the output of a command to a temporary file: ```ruby +# add a apt key using a file as stdin, capturing the error output in a temporary file error_out = Tempfile.new('err') @apt_key_cmd.start('add', '-', stdin: File.open('/tmp/key_in.gpg'), stdout: nil, stderr: error_out) ``` -> Note that due to buffering on the OS level (or lack thereof), bidirectional communication with that command can randomly hang your process, unless you take extra care only using the non-blocking methods on `IO`. Depending on your needs, you can also go straight to the childprocess gem. +> Note that due to buffering on the OS level (or lack thereof), bidirectional communication with that command can randomly hang your process, unless you take extra care only using the non-blocking methods on `IO`. Depending on your needs, you can also go straight to the childprocess gem, and use its facilities directly. -The handle also can be used to query whether the process is still running with `alive?`, and `exited?`, access the `exit_code` of the command, `wait` for it to finish, or poll for it to exit using `poll_for_exit(seconds)`, and `stop` the process. All those methods correspond to their respective counterparts on [`ChildProcess::AbstractProcess`](http://www.rubydoc.info/gems/childprocess/ChildProcess/AbstractProcess). +The `process` handle also can be used to query whether the process is still running with `alive?`, and `exited?`, access the `exit_code` of the command, `wait` for it to finish, or poll for it to exit using `poll_for_exit(seconds)`, and `stop` the process. See [`ChildProcess::AbstractProcess`](http://www.rubydoc.info/gems/childprocess/ChildProcess/AbstractProcess) for a more in-depth explanation of those methods. > Note: If you don't provide a block to the `start` methods, you will have to take care of exit code handling yourself. From 04a23fa08f67b95685e25b58f895afc2fd6c9a47 Mon Sep 17 00:00:00 2001 From: james-stocks Date: Tue, 12 Sep 2017 15:24:40 +0000 Subject: [PATCH 38/62] (maint) Fix bulletpoints in resource-api/README.md --- language/resource-api/README.md | 46 ++++++++++++++++----------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index e83e001..d66ec3f 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -404,35 +404,35 @@ This example is only for demonstration purposes. In the normal course of operati The following action/context methods are available: * Context functions -** `creating(titles, message: 'Creating', &block)` -** `updating(titles, message: 'Updating', &block)` -** `deleting(titles, message: 'Deleting', &block)` -** `processing(titles, is, should, message: 'Processing', &block)` -** `failing(titles, message: 'Failing', &block)`: unlikely to be used often, but provided for completeness -** `attribute_changed(attribute, is, should, message: nil)`: default to the title from the context + * `creating(titles, message: 'Creating', &block)` + * `updating(titles, message: 'Updating', &block)` + * `deleting(titles, message: 'Deleting', &block)` + * `processing(titles, is, should, message: 'Processing', &block)` + * `failing(titles, message: 'Failing', &block)`: unlikely to be used often, but provided for completeness + * `attribute_changed(attribute, is, should, message: nil)`: default to the title from the context * Action functions -** `created(titles, message: 'Created')` -** `updated(titles, message: 'Updated')` -** `deleted(titles, message: 'Deleted')` -** `unchanged(titles, message: 'Unchanged')`: the resource did not require a change - emit no logging -** `processed(titles, is, should)`: the resource has been processed - emit default logging for the resource and each attribute -** `failed(titles, message:)`: the resource has not been updated successfully -** `attribute_changed(titles, attribute, is, should, message: nil)`: use outside of a context, or in a context with multiple resources + * `created(titles, message: 'Created')` + * `updated(titles, message: 'Updated')` + * `deleted(titles, message: 'Deleted')` + * `unchanged(titles, message: 'Unchanged')`: the resource did not require a change - emit no logging + * `processed(titles, is, should)`: the resource has been processed - emit default logging for the resource and each attribute + * `failed(titles, message:)`: the resource has not been updated successfully + * `attribute_changed(titles, attribute, is, should, message: nil)`: use outside of a context, or in a context with multiple resources * `fail(message)`: abort the current context with an error * Plain messages -** `debug(message)` -** `debug(titles, message:)` -** `info(message)` -** `info(titles, message:)` -** `notice(message)` -** `notice(titles, message:)` -** `warning(message)` -** `warning(titles, message:)` -** `err(message)` -** `err(titles, message:)` + * `debug(message)` + * `debug(titles, message:)` + * `info(message)` + * `info(titles, message:)` + * `notice(message)` + * `notice(titles, message:)` + * `warning(message)` + * `warning(titles, message:)` + * `err(message)` + * `err(titles, message:)` `titles` can be a single identifier for a resource, or an Array of values, if the following block batch-processes multiple resources in one pass. If that processing is not atomic, providers should instead use the non-block forms of logging, and provide accurate status reporting on the individual parts of update operations. From 69454ea8fb169fd2738c619980f5e5e2cdd67b56 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 12 Sep 2017 18:44:00 +0100 Subject: [PATCH 39/62] Add a clarification to composite namevars --- language/resource-api/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index d66ec3f..9835afe 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -492,7 +492,7 @@ Neither of these options is ideal, thus it is documented as a limitation today. ## Composite namevars -The current API does not provide a way to specify composite namevars. [`title_patterns`](https://github.com/puppetlabs/puppet-specifications/blob/master/language/resource_types.md#title-patterns) are already very data driven, and will be easy to add at a later point. +The current API does not provide a way to specify composite namevars (that is, types with multiple namevars). [`title_patterns`](https://github.com/puppetlabs/puppet-specifications/blob/master/language/resource_types.md#title-patterns) are already very data driven, and will be easy to add at a later point. ## Puppet 4 data types From d4f6459dc805516c26eeaeba112b2ddcdfde546a Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Thu, 14 Sep 2017 16:22:52 +0100 Subject: [PATCH 40/62] Rework Commands API to a much simpler interface This gets rid of all the iffy block processing. The change trades off a little bit of memory waste (by storing the output of the process) for a much simpler API with only a single method call, and a bunch of explicit arguments. --- language/resource-api/README.md | 107 ++++++++++++++++++++++---------- 1 file changed, 73 insertions(+), 34 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 9835afe..94c0195 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -201,7 +201,7 @@ To use CLI commands in a safe and comfortable manner, the Resource API provides #### Creating a Command -To create a re-usable command, create a new instance of `Puppet::ResourceApi::Command` passing in the command. You can either specify a full path, or a bare command name. In the latter case the Command will use the system's `PATH` setting to search for the command. +To create a re-usable command, create a new instance of `Puppet::ResourceApi::Command` passing in the command. You can either specify a full path, or a bare command name. In the latter case the Command will use the Runtime Environment's `PATH` setting to search for the command. ```ruby class Puppet::Provider::AptKey::AptKey @@ -217,22 +217,33 @@ You can set default environment variables on the `@cmd.environment` Hash, and a #### Running simple commands -The `run(*args)` method takes any number of arguments, and executes the command using them. For example to call `apt-key` to delete a specific key by id: +The `run(*args)` method takes any number of arguments, and executes the command using them on the command line. For example to call `apt-key` to delete a specific key by id: ```ruby class Puppet::Provider::AptKey::AptKey - def set(context, changes, noop: false) + def set(context, changes) # ... - @apt_key_cmd.run(context, 'del', key_id, noop: noop) + @apt_key_cmd.run(context, 'del', key_id) ``` If the command is not available, a `Puppet::ResourceApi::CommandNotFoundError` will be raised. This can be easily used to fail the resources for a specific run, if the requirements for the provider are not yet met. The call will only return after the command has finished executing. If the command exits with a exitstatus indicating an error condition (that is non-zero), a `Puppet::ResourceApi::CommandExecutionError` is raised, containing the details of the command, and exit status. -The commands take a `noop:` keyword argument, and will signal success while skipping the real execution if necessary. +By default the `stdout` of the command is logged to debug, while the `stderr` is logged to warning. + +#### Implementing `noop` for `noop_handler` + +The `run` method takes a `noop:` keyword argument, and will signal success while skipping the real execution if necessary. Providers implementing the `noop_handler` feature should use this for all commands that are executed in the regular flow of the implementation. + +```ruby +class Puppet::Provider::AptKey::AptKey + def set(context, changes, noo: false) + # ... + @apt_key_cmd.run(context, 'del', key_id, noop: noop) +``` -Using these methods also causes the provider's actions to be logged at the appropriate levels. +#### Passing in specific environment variables To pass additional environment variables through to the command, pass a hash of them as `environment:`: @@ -240,56 +251,84 @@ To pass additional environment variables through to the command, pass a hash of @apt_key_cmd.run(context, 'del', key_id, environment: { 'LC_ALL': 'C' }) ``` -By default the `stdout` of the command is logged to debug, while the `stderr` is logged to warning. +This can also be set on the `@cmd.environment` attribute to run all executions of the command with the same environment. + +#### Running in a specific directory + +To run the command in a specific directory, use the `cwd` keyword argument: + +```ruby +@apt_key_cmd.run(context, 'del', key_id, cwd: '/etc/apt') +``` -#### Processing commands +This can also be set on the `@cmd.cwd` attribute to run all executions of the command with the working directory. -For more involved scenarios, variants of `@cmd.start` take the same arguments as `run`, but will start the command in the background, and return a handle to that process. The different variants have different defaults in how the process is set up. The `process` handle provides functionality to interact with the command, and query its state. +#### Processing command output -To use a command to read information from the system, `start_read` does not allow input to the process, and its `stderr` is logged at the warning level. The process' `io.stdout` attribute can be used to access the normal output of the command through an [`IO`](https://ruby-doc.org/core/IO.html) object. For example, to process the list of all apt keys: +To use a command to read information from the system, `run` can redirect the output from the command to various destinations, using the `stdout_destination:` and `stderr_destination:` keywords: +* `:log`: each line from the specified stream gets logged to the Runtime Environment. Use `stdout_loglevel:`, and `stderr_loglevel:` to specify the intended log-level. +* `:store`: the stream gets captured in a buffer, and will be returned as a string in the `result` object. +* `:discard`: the stream is discarded unprocessed. +* `:io`: the stream is connected to the IO object specified in `stdout_io:`, and `stderr_io:`. +* `:merge_to_stdout`: to get the process' standard error correctly interleaved into its regular output, this option can be specified for `stderr_destination:` only, and will provide the same file descriptor for both stdout, and stderr to the process. + +By default the standard output of the process is logged at the `:debug` level (see below), and the standard error stream is logged at the `:warning` level. To replicate this behaviour: ```ruby -class Puppet::Provider::AptKey::AptKey - def get(context) - @apt_key_cmd.start_read(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode') do |process| - process.io.stdout.each_line.collect do |line| - # handle each line here, and compute a Hash - end - end - end +@apt_key_cmd.run(context, 'del', key_id, stdout_destination: :log, stdout_loglevel: :debug, stderr_destination: :log, stderr_loglevel: :warning) ``` -To use a command to write to, `start_write` allows input into the process, but will only log its output like `run` does. For example, to provide a key on stdin to the apt-key tool: +To store, and process the output from the command, use the `:store` destination, and the `result` object: ```ruby class Puppet::Provider::AptKey::AptKey - def set(context, changes) - # ... - @apt_key_cmd.start_write(context, 'add', '-') do |process| - process.io.stdin.puts the_key - end + def get(context) + run_result = @apt_key_cmd.run(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode', stdout_destination: :store) + run_result.stdout.split('\n').each do |line| + # process/parse stdout_text here end ``` -Like the `run` method, the block forms of `start` will wait after the block has finished processing, to make sure that the command has exited cleanly, and will raise an error if the command returns a non-zero exit code. +To emulate most shell based redirections to files, the `:io` destination lets you (re-)use `File` handles, and temporary files (through `Tempfile` ): -#### Advanced scenarios +```ruby +tmp = Tempfile.new('key_list') +@apt_key_cmd.run(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode', stdout_destination: :io, stdout_io: tmp) +``` -For advanced scenarios, the plain `start` method returns a `process` handle with the `stdin`, `stdout`, and `stderr` pipes open, and unhandled. +#### Providing command input -This can be particularily useful together with providing your own `IO` objects, by using the `stdin:`, `stdout:`, and `stderr:` keyword arguments. For example redirecting the output of a command to a temporary file: +To use a command to write to, `run` allows passing input into the process. For example, to provide a key on stdin to the apt-key tool: ```ruby -# add a apt key using a file as stdin, capturing the error output in a temporary file -error_out = Tempfile.new('err') -@apt_key_cmd.start('add', '-', stdin: File.open('/tmp/key_in.gpg'), stdout: nil, stderr: error_out) +class Puppet::Provider::AptKey::AptKey + def set(context, changes) + # ... + @apt_key_cmd.run(context, 'add', '-', stdin_source: :value, stdin_value: the_key_string) + end ``` -> Note that due to buffering on the OS level (or lack thereof), bidirectional communication with that command can randomly hang your process, unless you take extra care only using the non-blocking methods on `IO`. Depending on your needs, you can also go straight to the childprocess gem, and use its facilities directly. +The `stdin_source:` keyword argument takes the following values: +* `:value`: allows specifying a string to pass on to the process in `stdin_value:`. +* `:io`: the input of the process is connected to the IO object specified in `stdin_io:`. +* `:none`: the input of the process is closed, and the process will receive an error when trying to read input. This is the default. + +#### Summary + +Synopsis of the `run` function: `@cmd.run(context, *args, **kwargs)` with the following keyword arguments: -The `process` handle also can be used to query whether the process is still running with `alive?`, and `exited?`, access the `exit_code` of the command, `wait` for it to finish, or poll for it to exit using `poll_for_exit(seconds)`, and `stop` the process. See [`ChildProcess::AbstractProcess`](http://www.rubydoc.info/gems/childprocess/ChildProcess/AbstractProcess) for a more in-depth explanation of those methods. +* `stdout_destination:` `:log`, `:io`, `:store`, `:discard` +* `stdout_loglevel:` `:debug`, `:info`, `:notice`, `:warning`, `:err` +* `stdout_io:` an `IO` object +* `stderr_destination:` `:log`, `:io`, `:store`, `:discard`, `:merge_to_stdout` +* `stderr_loglevel:` `:debug`, `:info`, `:notice`, `:warning`, `:err` +* `stderr_io:` an `IO` object +* `stdin_source:` `:io`, `:value`, `:none` +* `stdin_io:` an `IO` object +* `stdin_value:` a String +* `ignore_exit_code:` `true` or `false` -> Note: If you don't provide a block to the `start` methods, you will have to take care of exit code handling yourself. +The `run` function returns an object with the attributes `stdout`, `stderr`, and `exit_code`. The first two will only be used, if their respective `destination:` is set to `:store`. The `exit_code` will contain the exit code of the process. ### Logging and Reporting From 5ee58c35f693f10e1da285c4166cf559bd5c4d67 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Thu, 14 Sep 2017 16:23:27 +0100 Subject: [PATCH 41/62] Clarify the logging context examples; update for Commands API --- language/resource-api/README.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 94c0195..c474466 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -376,7 +376,7 @@ Most of those messages are expected to be relative to a specific resource instan ```ruby context.updating(title) do - if key_not_found + if apt_key_not_found(title) context.warning('Original key not found') end @@ -416,14 +416,18 @@ The equivalent long-hand form with manual error handling: ```ruby context.updating(title) begin - if key_not_found - context.warning(title, message: 'Original key not found') + unless title_got_passed_to_set(title) + raise Puppet::DevError, 'Managing resource outside of requested set: %{title}') + end + + if apt_key_not_found(title) + context.warning('Original key not found') end # Update the key by calling CLI tool - try_apt_key(...) + result = @apt_key_cmd.run(...) - if $?.exitstatus != 0 + if result.exitstatus != 0 context.error(title, "Failed executing apt-key #{...}") else context.attribute_changed(title, 'content', nil, content_hash, From 11f753b8428655e47363be749f0a260609e91cce Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Thu, 14 Sep 2017 16:23:48 +0100 Subject: [PATCH 42/62] Replace "foreign" with a proper explanation of what is meant --- language/resource-api/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index c474466..34537f7 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -479,7 +479,7 @@ The following action/context methods are available: `titles` can be a single identifier for a resource, or an Array of values, if the following block batch-processes multiple resources in one pass. If that processing is not atomic, providers should instead use the non-block forms of logging, and provide accurate status reporting on the individual parts of update operations. -A single `set()` execution may only log messages for instances it has been passed as part of the `changes` to process. Logging for foreign instances will cause an exception, as the runtime environment is not prepared for other resources to change. +A single `set()` execution may only log messages for instances it has been passed as part of the `changes` to process. Logging for instances not requested to be changed will cause an exception, as the runtime environment is not prepared for other resources to change. The provider is free to call different logging methods for different resources in any order it needs to. The only ordering restriction is for all calls specifying the same `title`. The `attribute_changed` logging needs to be done before that resource's action logging, and if a context is opened, it needs to be opened before any other logging for this resource. From e447ce0bd5a0ee80aa6ca1f966668a0b3e6f17c3 Mon Sep 17 00:00:00 2001 From: davidmalloncares Date: Fri, 15 Sep 2017 13:20:05 +0100 Subject: [PATCH 43/62] Update README.md --- language/resource-api/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 34537f7..b4ca761 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -1,6 +1,6 @@ # Puppet Resource API -This libarary provides a simple way to write new native resources for [puppet](https://puppet.com). +This library provides a simple way to write new native resources for [puppet](https://puppet.com). A *resource* is the basic thing that is managed by puppet. Each resource has a set of attributes describing its current state. Some of the attributes can be changed throughout the life-time of the resource, some attributes are only reported back, but cannot be changed (see `read_only`) others can only be set once during initial creation (see `init_only`). To gather information about those resources, and to enact changes in the real world, puppet requires a *provider* to implement this interaction. The provider can have parameters that influence its mode of operation (see `parameter`). To describe all these parts to the infrastructure, and the consumers, the resource *type* defines the all the metadata, including the list of the attributes. The *provider* contains the code to *get* and *set* the system state. From 7cbb54b48a3da4fa79a39759aeb9544eafeed3bd Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Mon, 18 Sep 2017 10:55:38 +0100 Subject: [PATCH 44/62] Switch the order of Logging and Commands Logging is the more central concept and requires explanation first. Commands can then reference the loglevels, without requiring forward references. --- language/resource-api/README.md | 270 ++++++++++++++++---------------- 1 file changed, 135 insertions(+), 135 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index b4ca761..18796fe 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -195,141 +195,6 @@ The primary runtime environment for the provider is the puppet agent, a long-run The runtime environment provides some utilities to make the providers's life easier, and provide a uniform experience for its users. -### Commands - -To use CLI commands in a safe and comfortable manner, the Resource API provides a thin wrapper around the excellent [childprocess gem](https://rubygems.org/gems/childprocess) to address the most common use-cases. Through using the library commands and their arguments are never passed through the shell leading to a safer execution environment (no funny parsing), and faster execution times (no extra processes). - -#### Creating a Command - -To create a re-usable command, create a new instance of `Puppet::ResourceApi::Command` passing in the command. You can either specify a full path, or a bare command name. In the latter case the Command will use the Runtime Environment's `PATH` setting to search for the command. - -```ruby -class Puppet::Provider::AptKey::AptKey - def initialize - @apt_key_cmd = Puppet::ResourceApi::Command.new('/usr/bin/apt-key') - @gpg_cmd = Puppet::ResourceApi::Command.new('gpg') - end -``` - -It is recommended to create the command in the `initialize` function of the provider, and store them in a member named after the command, with the `_cmd` suffix. This makes it easy to re-use common settings throughout the provider. - -You can set default environment variables on the `@cmd.environment` Hash, and a default working directory using `@cmd.cwd=`. - -#### Running simple commands - -The `run(*args)` method takes any number of arguments, and executes the command using them on the command line. For example to call `apt-key` to delete a specific key by id: - -```ruby -class Puppet::Provider::AptKey::AptKey - def set(context, changes) - # ... - @apt_key_cmd.run(context, 'del', key_id) -``` - -If the command is not available, a `Puppet::ResourceApi::CommandNotFoundError` will be raised. This can be easily used to fail the resources for a specific run, if the requirements for the provider are not yet met. - -The call will only return after the command has finished executing. If the command exits with a exitstatus indicating an error condition (that is non-zero), a `Puppet::ResourceApi::CommandExecutionError` is raised, containing the details of the command, and exit status. - -By default the `stdout` of the command is logged to debug, while the `stderr` is logged to warning. - -#### Implementing `noop` for `noop_handler` - -The `run` method takes a `noop:` keyword argument, and will signal success while skipping the real execution if necessary. Providers implementing the `noop_handler` feature should use this for all commands that are executed in the regular flow of the implementation. - -```ruby -class Puppet::Provider::AptKey::AptKey - def set(context, changes, noo: false) - # ... - @apt_key_cmd.run(context, 'del', key_id, noop: noop) -``` - -#### Passing in specific environment variables - -To pass additional environment variables through to the command, pass a hash of them as `environment:`: - -```ruby -@apt_key_cmd.run(context, 'del', key_id, environment: { 'LC_ALL': 'C' }) -``` - -This can also be set on the `@cmd.environment` attribute to run all executions of the command with the same environment. - -#### Running in a specific directory - -To run the command in a specific directory, use the `cwd` keyword argument: - -```ruby -@apt_key_cmd.run(context, 'del', key_id, cwd: '/etc/apt') -``` - -This can also be set on the `@cmd.cwd` attribute to run all executions of the command with the working directory. - -#### Processing command output - -To use a command to read information from the system, `run` can redirect the output from the command to various destinations, using the `stdout_destination:` and `stderr_destination:` keywords: -* `:log`: each line from the specified stream gets logged to the Runtime Environment. Use `stdout_loglevel:`, and `stderr_loglevel:` to specify the intended log-level. -* `:store`: the stream gets captured in a buffer, and will be returned as a string in the `result` object. -* `:discard`: the stream is discarded unprocessed. -* `:io`: the stream is connected to the IO object specified in `stdout_io:`, and `stderr_io:`. -* `:merge_to_stdout`: to get the process' standard error correctly interleaved into its regular output, this option can be specified for `stderr_destination:` only, and will provide the same file descriptor for both stdout, and stderr to the process. - -By default the standard output of the process is logged at the `:debug` level (see below), and the standard error stream is logged at the `:warning` level. To replicate this behaviour: - -```ruby -@apt_key_cmd.run(context, 'del', key_id, stdout_destination: :log, stdout_loglevel: :debug, stderr_destination: :log, stderr_loglevel: :warning) -``` - -To store, and process the output from the command, use the `:store` destination, and the `result` object: - -```ruby -class Puppet::Provider::AptKey::AptKey - def get(context) - run_result = @apt_key_cmd.run(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode', stdout_destination: :store) - run_result.stdout.split('\n').each do |line| - # process/parse stdout_text here - end -``` - -To emulate most shell based redirections to files, the `:io` destination lets you (re-)use `File` handles, and temporary files (through `Tempfile` ): - -```ruby -tmp = Tempfile.new('key_list') -@apt_key_cmd.run(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode', stdout_destination: :io, stdout_io: tmp) -``` - -#### Providing command input - -To use a command to write to, `run` allows passing input into the process. For example, to provide a key on stdin to the apt-key tool: - -```ruby -class Puppet::Provider::AptKey::AptKey - def set(context, changes) - # ... - @apt_key_cmd.run(context, 'add', '-', stdin_source: :value, stdin_value: the_key_string) - end -``` - -The `stdin_source:` keyword argument takes the following values: -* `:value`: allows specifying a string to pass on to the process in `stdin_value:`. -* `:io`: the input of the process is connected to the IO object specified in `stdin_io:`. -* `:none`: the input of the process is closed, and the process will receive an error when trying to read input. This is the default. - -#### Summary - -Synopsis of the `run` function: `@cmd.run(context, *args, **kwargs)` with the following keyword arguments: - -* `stdout_destination:` `:log`, `:io`, `:store`, `:discard` -* `stdout_loglevel:` `:debug`, `:info`, `:notice`, `:warning`, `:err` -* `stdout_io:` an `IO` object -* `stderr_destination:` `:log`, `:io`, `:store`, `:discard`, `:merge_to_stdout` -* `stderr_loglevel:` `:debug`, `:info`, `:notice`, `:warning`, `:err` -* `stderr_io:` an `IO` object -* `stdin_source:` `:io`, `:value`, `:none` -* `stdin_io:` an `IO` object -* `stdin_value:` a String -* `ignore_exit_code:` `true` or `false` - -The `run` function returns an object with the attributes `stdout`, `stderr`, and `exit_code`. The first two will only be used, if their respective `destination:` is set to `:store`. The `exit_code` will contain the exit code of the process. - ### Logging and Reporting The provider needs to signal changes, successes and failures to the runtime environment. The `context` is the primary way to do so. It provides a single interface for both the detailed technical information for later automatic processing, as well as human readable progress and status messages for operators. @@ -483,6 +348,141 @@ A single `set()` execution may only log messages for instances it has been passe The provider is free to call different logging methods for different resources in any order it needs to. The only ordering restriction is for all calls specifying the same `title`. The `attribute_changed` logging needs to be done before that resource's action logging, and if a context is opened, it needs to be opened before any other logging for this resource. +### Commands + +To use CLI commands in a safe and comfortable manner, the Resource API provides a thin wrapper around the excellent [childprocess gem](https://rubygems.org/gems/childprocess) to address the most common use-cases. Through using the library commands and their arguments are never passed through the shell leading to a safer execution environment (no funny parsing), and faster execution times (no extra processes). + +#### Creating a Command + +To create a re-usable command, create a new instance of `Puppet::ResourceApi::Command` passing in the command. You can either specify a full path, or a bare command name. In the latter case the Command will use the Runtime Environment's `PATH` setting to search for the command. + +```ruby +class Puppet::Provider::AptKey::AptKey + def initialize + @apt_key_cmd = Puppet::ResourceApi::Command.new('/usr/bin/apt-key') + @gpg_cmd = Puppet::ResourceApi::Command.new('gpg') + end +``` + +> Note: It is recommended to create the command in the `initialize` function of the provider, and store them in a member named after the command, with the `_cmd` suffix. This makes it easy to re-use common settings throughout the provider. + +You can set default environment variables on the `@cmd.environment` Hash, and a default working directory using `@cmd.cwd=`. + +#### Running simple commands + +The `run(*args)` method takes any number of arguments, and executes the command using them on the command line. For example to call `apt-key` to delete a specific key by id: + +```ruby +class Puppet::Provider::AptKey::AptKey + def set(context, changes) + # ... + @apt_key_cmd.run(context, 'del', key_id) +``` + +If the command is not available, a `Puppet::ResourceApi::CommandNotFoundError` will be raised. This can be easily used to fail the resources for a specific run, if the requirements for the provider are not yet met. + +The call will only return after the command has finished executing. If the command exits with a exitstatus indicating an error condition (that is non-zero), a `Puppet::ResourceApi::CommandExecutionError` is raised, containing the details of the command, and exit status. + +By default the `stdout` of the command is logged to debug, while the `stderr` is logged to warning. + +#### Implementing `noop` for `noop_handler` + +The `run` method takes a `noop:` keyword argument, and will signal success while skipping the real execution if necessary. Providers implementing the `noop_handler` feature should use this for all commands that are executed in the regular flow of the implementation. + +```ruby +class Puppet::Provider::AptKey::AptKey + def set(context, changes, noo: false) + # ... + @apt_key_cmd.run(context, 'del', key_id, noop: noop) +``` + +#### Passing in specific environment variables + +To pass additional environment variables through to the command, pass a hash of them as `environment:`: + +```ruby +@apt_key_cmd.run(context, 'del', key_id, environment: { 'LC_ALL': 'C' }) +``` + +This can also be set on the `@cmd.environment` attribute to run all executions of the command with the same environment. + +#### Running in a specific directory + +To run the command in a specific directory, use the `cwd` keyword argument: + +```ruby +@apt_key_cmd.run(context, 'del', key_id, cwd: '/etc/apt') +``` + +This can also be set on the `@cmd.cwd` attribute to run all executions of the command with the working directory. + +#### Processing command output + +To use a command to read information from the system, `run` can redirect the output from the command to various destinations, using the `stdout_destination:` and `stderr_destination:` keywords: +* `:log`: each line from the specified stream gets logged to the Runtime Environment. Use `stdout_loglevel:`, and `stderr_loglevel:` to specify the intended loglevel. +* `:store`: the stream gets captured in a buffer, and will be returned as a string in the `result` object. +* `:discard`: the stream is discarded unprocessed. +* `:io`: the stream is connected to the IO object specified in `stdout_io:`, and `stderr_io:`. +* `:merge_to_stdout`: to get the process' standard error correctly interleaved into its regular output, this option can be specified for `stderr_destination:` only, and will provide the same file descriptor for both stdout, and stderr to the process. + +By default the standard output of the process is logged at the debug level , and the standard error stream is logged at the warning level. To replicate this behaviour: + +```ruby +@apt_key_cmd.run(context, 'del', key_id, stdout_destination: :log, stdout_loglevel: :debug, stderr_destination: :log, stderr_loglevel: :warning) +``` + +To store, and process the output from the command, use the `:store` destination, and the `result` object: + +```ruby +class Puppet::Provider::AptKey::AptKey + def get(context) + run_result = @apt_key_cmd.run(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode', stdout_destination: :store) + run_result.stdout.split('\n').each do |line| + # process/parse stdout_text here + end +``` + +To emulate most shell based redirections to files, the `:io` destination lets you (re-)use `File` handles, and temporary files (through `Tempfile`): + +```ruby +tmp = Tempfile.new('key_list') +@apt_key_cmd.run(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode', stdout_destination: :io, stdout_io: tmp) +``` + +#### Providing command input + +To use a command to write to, `run` allows passing input into the process. For example, to provide a key on stdin to the apt-key tool: + +```ruby +class Puppet::Provider::AptKey::AptKey + def set(context, changes) + # ... + @apt_key_cmd.run(context, 'add', '-', stdin_source: :value, stdin_value: the_key_string) + end +``` + +The `stdin_source:` keyword argument takes the following values: +* `:value`: allows specifying a string to pass on to the process in `stdin_value:`. +* `:io`: the input of the process is connected to the IO object specified in `stdin_io:`. +* `:none`: the input of the process is closed, and the process will receive an error when trying to read input. This is the default. + +#### Summary + +Synopsis of the `run` function: `@cmd.run(context, *args, **kwargs)` with the following keyword arguments: + +* `stdout_destination:` `:log`, `:io`, `:store`, `:discard` +* `stdout_loglevel:` `:debug`, `:info`, `:notice`, `:warning`, `:err` +* `stdout_io:` an `IO` object +* `stderr_destination:` `:log`, `:io`, `:store`, `:discard`, `:merge_to_stdout` +* `stderr_loglevel:` `:debug`, `:info`, `:notice`, `:warning`, `:err` +* `stderr_io:` an `IO` object +* `stdin_source:` `:io`, `:value`, `:none` +* `stdin_io:` an `IO` object +* `stdin_value:` a String +* `ignore_exit_code:` `true` or `false` + +The `run` function returns an object with the attributes `stdout`, `stderr`, and `exit_code`. The first two will only be used, if their respective `destination:` is set to `:store`. The `exit_code` will contain the exit code of the process. + # Known Limitations This API is not a full replacement for the power of 3.x style types and providers. Here is a (incomplete) list of missing pieces and thoughts on how to go about solving these. In the end, the goal of the new Resource API is not to be a complete replacement of prior art, but a cleaner way to get good results for the majority of simple cases. From a07f3ecdf84f45aabea007050dc8344829cf53b1 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 20 Sep 2017 11:55:09 +0100 Subject: [PATCH 45/62] Define character set handling on talking to Commands --- language/resource-api/README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 18796fe..b5da10f 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -466,6 +466,17 @@ The `stdin_source:` keyword argument takes the following values: * `:io`: the input of the process is connected to the IO object specified in `stdin_io:`. * `:none`: the input of the process is closed, and the process will receive an error when trying to read input. This is the default. +#### Character encoding + +To support the widest array of platforms and use-cases, character encoding of a provider's inputs, and outputs needs to be considered. By default, the Commands API follows the ruby model of having all strings tagged with their current [`Encoding`](https://ruby-doc.org/core/Encoding.html), and using the system's current default character set for I/O. That means that strings read from Commands might be tagged with non-UTF-8 character sets on input, and UTF-8 strings will be transcoded on output. + +To influence this behaviour, you can tell the `run` method which encoding to use, and enable transcoding, if required for your use-case. Use the following keyword arguments: + +* `stdout_encoding:`, `stderr_encoding:` The encoding to tag incoming bytes with. +* `stdin_encoding:` ensures that strings are transcoded to this encoding before being written to this command. + +> Note: Use the `ASCII-8BIT` encoding to disable all conversions, and receive the raw bytes. + #### Summary Synopsis of the `run` function: `@cmd.run(context, *args, **kwargs)` with the following keyword arguments: From f71d0f47de1bc9295fb7e3e423c577a319d8b891 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Thu, 21 Sep 2017 10:43:43 +0100 Subject: [PATCH 46/62] Extend character encoding section to expose full underlying capabilities --- language/resource-api/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index b5da10f..fb41b89 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -468,12 +468,13 @@ The `stdin_source:` keyword argument takes the following values: #### Character encoding -To support the widest array of platforms and use-cases, character encoding of a provider's inputs, and outputs needs to be considered. By default, the Commands API follows the ruby model of having all strings tagged with their current [`Encoding`](https://ruby-doc.org/core/Encoding.html), and using the system's current default character set for I/O. That means that strings read from Commands might be tagged with non-UTF-8 character sets on input, and UTF-8 strings will be transcoded on output. +To support the widest array of platforms and use-cases, character encoding of a provider's inputs, and outputs needs to be considered. By default, the Commands API follows the ruby model of having all strings tagged with their current [`Encoding`](https://ruby-doc.org/core/Encoding.html), and using the system's current default character set for I/O. That means that strings read from Commands might be tagged with non-UTF-8 character sets on input, and UTF-8 strings might be transcoded on output. To influence this behaviour, you can tell the `run` method which encoding to use, and enable transcoding, if required for your use-case. Use the following keyword arguments: * `stdout_encoding:`, `stderr_encoding:` The encoding to tag incoming bytes with. * `stdin_encoding:` ensures that strings are transcoded to this encoding before being written to this command. +* `stdout_encoding_opts:`,`stderr_encoding_opts:`,`stdin_encoding_opts:` options for [`String.encode`](https://ruby-doc.org/core-2.4.1/String.html#method-i-encode) for the different streams. > Note: Use the `ASCII-8BIT` encoding to disable all conversions, and receive the raw bytes. From 46129ca70fb70efd0acc5d6b1ac17ae5b1b6cf37 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Tue, 26 Sep 2017 14:19:50 +0100 Subject: [PATCH 47/62] Remove obsolete Known Limitation This has been addressed by having the provider be a plain class. Developers can use regular ruby to build that class in any way they like. --- language/resource-api/README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index fb41b89..bbd0097 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -560,7 +560,3 @@ There is no way to access the catalog from the provider. Several existing types ## Logging for unmanaged instances The provider could provide log messages for resource instances that were not passed into the `set` call. In the current implementation those will cause an error. How this is handeled in the future might change drastically. - -## Sharing code between providers - -Providers in the old API share code through inheritance, using the `:parent` key in the `provide()` call. To reduce entanglement between the business end of code, and the required interactions with the Resource API, it is recommended to put shared code in separate classes, that are used directly, instead of inheriting their contents. This can either happen through normal instantiation and usage, or for small chunks of code through a `Module`, and `include`. From e81ee0076d4df9fcd0335c39c440bbe328d1a7e7 Mon Sep 17 00:00:00 2001 From: james-stocks Date: Thu, 5 Oct 2017 15:20:17 +0000 Subject: [PATCH 48/62] Describe the resource-api processing method --- language/resource-api/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index bbd0097..a3a2df3 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -315,7 +315,7 @@ The following action/context methods are available: * `creating(titles, message: 'Creating', &block)` * `updating(titles, message: 'Updating', &block)` * `deleting(titles, message: 'Deleting', &block)` - * `processing(titles, is, should, message: 'Processing', &block)` + * `processing(title, is, should, message: 'Processing', &block)`: process a resource. `&block` should raise an exception for potential fault cases; if no exception is raised then it is assumed the change from `is` to `should` was succesful. * `failing(titles, message: 'Failing', &block)`: unlikely to be used often, but provided for completeness * `attribute_changed(attribute, is, should, message: nil)`: default to the title from the context @@ -324,7 +324,7 @@ The following action/context methods are available: * `updated(titles, message: 'Updated')` * `deleted(titles, message: 'Deleted')` * `unchanged(titles, message: 'Unchanged')`: the resource did not require a change - emit no logging - * `processed(titles, is, should)`: the resource has been processed - emit default logging for the resource and each attribute + * `processed(title, is, should)`: the resource has been processed - emit default logging for the resource and each attribute * `failed(titles, message:)`: the resource has not been updated successfully * `attribute_changed(titles, attribute, is, should, message: nil)`: use outside of a context, or in a context with multiple resources From 61af448a07cba2c59bc4896fa25636925694bd43 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Mon, 2 Oct 2017 16:55:11 +0100 Subject: [PATCH 49/62] Fix typo --- language/resource-api/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index a3a2df3..133a6ae 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -391,7 +391,7 @@ The `run` method takes a `noop:` keyword argument, and will signal success while ```ruby class Puppet::Provider::AptKey::AptKey - def set(context, changes, noo: false) + def set(context, changes, noop: false) # ... @apt_key_cmd.run(context, 'del', key_id, noop: noop) ``` From ea27a9c1ebbbc3939b38c3a8897783faadf87fa4 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Fri, 6 Oct 2017 11:47:43 +0100 Subject: [PATCH 50/62] Improve explanation of logging methods --- language/resource-api/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 133a6ae..0202e94 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -309,23 +309,24 @@ This example is only for demonstration purposes. In the normal course of operati #### Logging reference -The following action/context methods are available: +The following action/block methods are available: -* Context functions +* Block functions: These functions provide logging, and timing around a provider's core actions. If the the passed `&block` returns, the action is recorded as successful. To signal a failure, the block should raise an exception explaining the problem. * `creating(titles, message: 'Creating', &block)` * `updating(titles, message: 'Updating', &block)` * `deleting(titles, message: 'Deleting', &block)` - * `processing(title, is, should, message: 'Processing', &block)`: process a resource. `&block` should raise an exception for potential fault cases; if no exception is raised then it is assumed the change from `is` to `should` was succesful. - * `failing(titles, message: 'Failing', &block)`: unlikely to be used often, but provided for completeness - * `attribute_changed(attribute, is, should, message: nil)`: default to the title from the context + * `processing(title, is, should, message: 'Processing', &block)`: Generic processing of a resource, emits default change messages for the difference between `is:` and `should:`. + * `failing(titles, message: 'Failing', &block)`: unlikely to be used often, but provided for completeness, always records a failure. * Action functions * `created(titles, message: 'Created')` * `updated(titles, message: 'Updated')` * `deleted(titles, message: 'Deleted')` - * `unchanged(titles, message: 'Unchanged')`: the resource did not require a change - emit no logging * `processed(title, is, should)`: the resource has been processed - emit default logging for the resource and each attribute * `failed(titles, message:)`: the resource has not been updated successfully + +* Attribute Change notifications + * `attribute_changed(attribute, is, should, message: nil)`: Call this from a context, default to the title from the context * `attribute_changed(titles, attribute, is, should, message: nil)`: use outside of a context, or in a context with multiple resources * `fail(message)`: abort the current context with an error From 4b81f1009a459e5a940629ddee8246a6e4181070 Mon Sep 17 00:00:00 2001 From: james-stocks Date: Fri, 6 Oct 2017 14:19:03 +0000 Subject: [PATCH 51/62] Update specification for attribute_changed logging method --- language/resource-api/README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 0202e94..563d6a8 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -326,8 +326,7 @@ The following action/block methods are available: * `failed(titles, message:)`: the resource has not been updated successfully * Attribute Change notifications - * `attribute_changed(attribute, is, should, message: nil)`: Call this from a context, default to the title from the context - * `attribute_changed(titles, attribute, is, should, message: nil)`: use outside of a context, or in a context with multiple resources + * `attribute_changed(title, attribute, is, should, message: nil)`: Notify the runtime environment that a specific attribute for a specific resource has changed. `is` and `should` are the original and the new value of the attribute. Either can be `nil`. * `fail(message)`: abort the current context with an error From 779736d3b142e590da16dedc1a29975c82972f5f Mon Sep 17 00:00:00 2001 From: james-stocks Date: Fri, 6 Oct 2017 14:25:00 +0000 Subject: [PATCH 52/62] Remove specification for fail(message) method --- language/resource-api/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 563d6a8..7333820 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -328,8 +328,6 @@ The following action/block methods are available: * Attribute Change notifications * `attribute_changed(title, attribute, is, should, message: nil)`: Notify the runtime environment that a specific attribute for a specific resource has changed. `is` and `should` are the original and the new value of the attribute. Either can be `nil`. -* `fail(message)`: abort the current context with an error - * Plain messages * `debug(message)` * `debug(titles, message:)` From c3954088bfac74b83bd5814d63c080696f8551d8 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Fri, 17 Nov 2017 13:52:45 +0000 Subject: [PATCH 53/62] (PDK-611) define the `remote_resource` feature --- language/resource-api/README.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 7333820..1fa3117 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -187,6 +187,37 @@ end When a resource is marked with `noop => true`, either locally, or through a global flag, the standard runtime will emit the default change report with a `noop` flag set. In some cases an implementation can provide additional information (e.g. commands that would get executed), or requires additional evaluation before determining the effective changes (e.g. `exec`'s `onlyif` attribute). In those cases, the resource type can specify the `noop_handler` feature to have `set` called for all resources, even those flagged with `noop`. When the `noop` parameter is set to true, the provider must not change the system state, but only report what it would change. The `noop` parameter should default to `false` to allow simple runtimes to ignore this feature. +## Provider Feature: remote_resource + +```ruby +Puppet::ResourceApi.register_type( + name: 'nx9k_vlan', + features: [ 'remote_resource' ], +) + +require 'puppet/util/network_device/simple/device' +module Puppet::Util::NetworkDevice::Nexus + class Device < Puppet::Util::NetworkDevice::Simple::Device + def facts + # access the device and return facts hash + end + end +end + +class Puppet::Provider::Nx9k_vlan::Nx9k_vlan + def set(context, changes, noop: false) + changes.each do |name, change| + is = change.has_key? :is ? change[:is] : get_single(name) + should = change[:should] + # ... + context.device.do_something unless noop + end + end +end +``` + +Declaring this feature restricts the resource from being run "locally". It is now expected to execute all its external interactions through the `context.device` instance. How that instance is set up is runtime specific. In puppet, it is configured through the [`device.conf`](https://puppet.com/docs/puppet/5.3/config_file_device.html) file, and only available when running under [`puppet device`](https://puppet.com/docs/puppet/5.3/man/device.html). It is recommended to use `Puppet::Util::NetworkDevice::Simple::Device` as the base class for all devices, which automatically loads a configuration from the local filesystem of the proxy node where it's running on. + # Runtime Environment The primary runtime environment for the provider is the puppet agent, a long-running daemon process. The provider can also be used in the puppet apply command, a one-shot version of the agent, or the puppet resource command, a short-lived CLI process for listing or managing a single resource type. Other callers who want to access the provider will have to emulate those environments. The primary lifecycle of resource managment in each of those tools is the *transaction*, a single set of changes (e.g. a catalog, or a CLI invocation) to work on. In any case the registered block will be surfaced in a clean class which will be instantiated once for each transaction. The provider can define any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for an provider, and use it immediately, the provider is instantiated as late as possible. A transaction will usually call `get` once, and may call `set` any number of times to effect change. The object instance hosting the `get` and `set` methods can be used to cache ephemeral state during execution. The provider should not try to cache state beyond the transaction, to avoid interfering with the agent daemon. In many other cases caching beyond the transaction won't help anyways, as the hosting process will only manage a single transaction. From b9884eddd93967f3d7fa63068408ebb49e64c2b7 Mon Sep 17 00:00:00 2001 From: claire cadman Date: Tue, 30 Jan 2018 15:47:32 -0800 Subject: [PATCH 54/62] edits to spec resource-api readme --- language/resource-api/README.md | 245 +++++++++++++++++--------------- 1 file changed, 132 insertions(+), 113 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 1fa3117..0373810 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -1,12 +1,12 @@ # Puppet Resource API -This library provides a simple way to write new native resources for [puppet](https://puppet.com). +This library provides a simple way to write new native resources for [Puppet](https://puppet.com). -A *resource* is the basic thing that is managed by puppet. Each resource has a set of attributes describing its current state. Some of the attributes can be changed throughout the life-time of the resource, some attributes are only reported back, but cannot be changed (see `read_only`) others can only be set once during initial creation (see `init_only`). To gather information about those resources, and to enact changes in the real world, puppet requires a *provider* to implement this interaction. The provider can have parameters that influence its mode of operation (see `parameter`). To describe all these parts to the infrastructure, and the consumers, the resource *type* defines the all the metadata, including the list of the attributes. The *provider* contains the code to *get* and *set* the system state. +A *resource* is the basic unit that is managed by Puppet. Each resource has a set of attributes describing its current state. Some attributes can be changed throughout the lifetime of the resource, whereas others are only reported back but cannot be changed (see `read_only`), and some can only be set once during initial creation (see `init_only`). To gather information about those resources and to enact changes, Puppet requires a *provider* to implement this interaction. The provider can have parameters that influence its mode of operation (see `parameter`). To describe all these parts to the infrastructure and the consumers, the resource *type* defines all the metadata, including the list of the attributes. The *provider* contains the code to *get* and *set* the system state. -# Resource Definition ("Type") +## Resource definition ("type") -To make the resource known to the puppet ecosystem, its definition ("type") needs to be registered with puppet: +To make the resource known to the Puppet ecosystem, its definition ("type") needs to be registered with Puppet: ```ruby Puppet::ResourceApi.register_type( @@ -51,28 +51,28 @@ Puppet::ResourceApi.register_type( The `Puppet::ResourceApi.register_type(options)` function takes the following keyword arguments: * `name`: the name of the resource type. -* `desc`: a doc string that describes the overall working of the resource type, gives examples, and explains pre-requisites as well as known issues. -* `attributes`: an hash mapping attribute names to their details. Each attribute is described by a hash containing the puppet 4 data `type`, a `desc` string, a `default` value, and the `behaviour` of the attribute: `namevar`, `read_only`, `init_only`, or a `parameter`. - * `type`: the puppet 4 data type allowed in this attribute. +* `desc`: a doc string that describes the overall working of the resource type, provides examples, and explains prerequisites and known issues. +* `attributes`: a hash mapping attribute names to their details. Each attribute is described by a hash containing the Puppet 4 data `type`, a `desc` string, a `default` value, and the `behaviour` of the attribute: `namevar`, `read_only`, `init_only`, or a `parameter`. + * `type`: the Puppet 4 data type allowed in this attribute. * `desc`: a string describing this attribute. This is used in creating the automated API docs with [puppet-strings](https://github.com/puppetlabs/puppet-strings). - * `default`: a default value that will be used by the runtime environment, whenever the caller doesn't specify a value for this attribute. + * `default`: a default value that will be used by the runtime environment when the caller does not specify a value for this attribute. * `behaviour`/`behavior`: how the attribute behaves. Currently available values: - * `namevar`: marks an attribute as part of the "primary key", or "identity" of the resource. A given set of namevar values needs to distinctively identify a instance. - * `init_only`: this attribute can only be set during creation of the resource. Its value will be reported going forward, but trying to change it later will lead to an error. For example, the base image for a VM, or the UID of a user. - * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example the checksum of a file, or the MAC address of a network interface. - * `parameter`: these attributes influence how the provider behaves, and cannot be read from the target system. For example, the target file on inifile, or credentials to access an API. -* `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a Hash mapping resource types to titles. Currently the titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, puppet will automatically create the relationsships requested here. -* `features`: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently defined: features: `canonicalize`, `simple_get_filter`, and `noop_handler`. See below for details. + * `namevar`: marks an attribute as part of the "primary key" or "identity" of the resource. A given set of `namevar` values needs to distinctively identify an instance. + * `init_only`: this attribute can only be set during the creation of the resource. Its value will be reported going forward, but trying to change it later will lead to an error. For example, the base image for a VM or the UID of a user. + * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example, the checksum of a file or the MAC address of a network interface. + * `parameter`: these attributes influence how the provider behaves and cannot be read from the target system. For example, the target file on inifile or credentials to access an API. +* `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a hash mapping resource types to titles. The titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, Puppet will create the relationsships requested here. +* `features`: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently defined features: `canonicalize`, `simple_get_filter`, and `noop_handler`. See below for details. -For autoloading to work, this code needs to go into `lib/puppet/type/.rb` in your module. +For autoloading work, this code needs to go into `lib/puppet/type/.rb` in your module. -# Resource Implementation ("Provider") +## Resource implementation ("provider") -To effect changes on the real world, a resource also requires an implementation that makes the universe's state available to puppet, and causes the changes to bring reality to whatever state is requested in the catalog. The two fundamental operations to manage resources are reading and writing system state. These operations are implemented as `get` and `set`. The implementation itself is a basic Ruby class in the `Puppet::Provider` namespace, named after the Type using CamelCase. +To affect changes, a resource requires an implementation that makes the universe's state available to Puppet, and causes the changes to bring reality to whatever state is requested in the catalog. The two fundamental operations to manage resources are reading and writing system state. These operations are implemented as `get` and `set`. The implementation itself is a basic Ruby class in the `Puppet::Provider` namespace, named after the type using CamelCase. -> Note: Due to the way puppet autoload works, this has to be in a file called `puppet/provider//.rb` and the class will also have the CamelCased type name twice. +> Note: Due to the way Puppet autoload works, this has to be in a file called `puppet/provider//.rb`. The class will also have the CamelCased type name twice. -At runtime the current and intended system states for a specific resource are always represented as ruby Hashes of the resource's attributes, and applicable operational parameters. +At runtime, the current and intended system states for a specific resource. These are always represented as Ruby hashes of the resource's attributes and applicable operational parameters. ```ruby class Puppet::Provider::AptKey::AptKey @@ -98,17 +98,21 @@ class Puppet::Provider::AptKey::AptKey end ``` -The `get` method reports the current state of the managed resources. It returns an Enumerable of all existing resources. Each resource is a Hash with attribute names as keys, and their respective values as values. It is an error to return values not matching the type specified in the resource type. If a requested resource is not listed in the result, it is considered to not exist on the system. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The exception's message will be reported to the user. +The `get` method reports the current state of the managed resources. It returns an enumerable of all existing resources. Each resource is a hash with attribute names as keys, and their respective values as values. It is an error to return values not matching the type specified in the resource type. If a requested resource is not listed in the result, it is considered to not exist on the system. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The exception message will be reported to the user. -The `set` method updates resources to a new state. The `changes` parameter gets passed an a hash of change requests, keyed by the resource's name. Each value is another hash with the optional `:is` and `:should` keys. At least one of the two has to be specified. The values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. As a special case, a missing `:should` entry indicates that a resource should be removed from the system. Even a type implementing the `ensure => [present, absent]` attribute pattern for its human consumers, still has to react correctly on a missing `:should` entry. For convenience, and performance, `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources was not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. The `set` method should always return `nil`. Any progress signaling should be done through the logging utilities described below. Should the `set` method throw an exception, all resources that should change in this call, and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the `set` method if there are changes to be made. Especially in the case of resources marked with `noop => true` (either locally, or through a global flag), the runtime will not pass them to `set`. See `noop_handler` below for changing this behaviour if required. +The `set` method updates resources to a new state. The `changes` parameter gets passed a hash of change requests, keyed by the resource's name. Each value is another hash with the optional `:is` and `:should` keys. At least one of the two has to be specified. The values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. -Both methods take a `context` parameter which provides utilties from the Runtime Environment, and is decribed in more detail there. +A missing `:should` entry indicates that a resource should be removed from the system. Even a type implementing the `ensure => [present, absent]` attribute pattern still has to react correctly on a missing `:should` entry. `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources were not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. -## Provider Features +The `set` method should always return `nil`. Any progress signaling should be done through the logging utilities described below. If the `set` method throws an exception, all resources that should change in this call and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the `set` method if there are changes to be made, especially in the case of resources marked with `noop => true` (either locally or through a global flag). The runtime will not pass them to `set`. See `noop_handler` below for changing this behaviour if required. -There are some common cases where an implementation might want to provide a better experience in specific usecases than the default runtime environment can provide. To avoid burdening the simplest providers with that additional complexity, these cases are hidden behind feature flags. To enable the special handling, the Resource Definition has a `feature` key to list all features implemented by the provider. +Both methods take a `context` parameter which provides utilties from the runtime environment, and is decribed in more detail there. -## Provider Feature: canonicalize +### Provider features + +There are some use cases where an implementation provides a better experience than the default runtime environment provides. To avoid burdening the simplest providers with that additional complexity, these cases are hidden behind feature flags. To enable the special handling, the resource definition has a `feature` key to list all features implemented by the provider. + +### Provider feature: `canonicalize` Allows the provider to accept a wide range of formats for values without confusing the user. @@ -130,19 +134,19 @@ class Puppet::Provider::AptKey::AptKey end ``` -The runtime environment needs to compare user input from the manifest (the desired state) with values returned from `get` (the actual state) to determine whether or not changes need to be effected. In simple cases, a provider will only accept values from the manifest in the same format as `get` would return. Then no extra work is required, as a trivial value comparison will suffice. In many cases this places a high burden on the user to provide values in an unnaturally constrained format. In the example, the `apt_key` name is a hexadecimal number that can be written with, and without, the `'0x'` prefix, and the casing of the digits is irrelevant. A trivial value comparison on the strings would cause false positives, when the user input format does not match, and there is no Hexadecimal type in the Puppet language. In this case the provider can specify the `canonicalize` feature and implement the `canonicalize` method. +The runtime environment needs to compare user input from the manifest (the desired state) with values returned from `get` (the actual state) to determine whether or not changes need to be affected. In simple cases, a provider will only accept values from the manifest in the same format as `get` returns. No extra work is required, as a value comparison will suffice. This places a high burden on the user to provide values in an unnaturally constrained format. In the example, the `apt_key` name is a hexadecimal number that can be written with, and without, the `'0x'` prefix, and the casing of the digits is irrelevant. A value comparison on the strings would cause false positives if the user input format that does not match. There is no hexadecimal type in the Puppet language. The provider can specify the `canonicalize` feature and implement the `canonicalize` method. -The `canonicalize` method transforms its `resources` argument into the standard format required by the rest of the provider. The `resources` argument to `canonicalize` is an Enumerable of resource hashes matching the structure returned by `get`. It returns all passed values in the same structure, with the required transformations applied. It is free to re-use, or recreate the data structures passed in as arguments. The runtime environment must use `canonicalize` before comparing user input values with values returned from `get`. The runtime environment must always pass canonicalized values into `set`. If the runtime environment must requires the original values for later processing, it must protect itself from modifications to the objects passed into `canonicalize`, for example through creating a deep copy of the objects. +The `canonicalize` method transforms its `resources` argument into the standard format required by the rest of the provider. The `resources` argument to `canonicalize` is an enumerable of resource hashes matching the structure returned by `get`. It returns all passed values in the same structure with the required transformations applied. It is free to reuse or recreate the data structures passed in as arguments. The runtime environment must use `canonicalize` before comparing user input values with values returned from `get`. The runtime environment always passes canonicalized values into `set`. If the runtime environment requires the original values for later processing, it protects itself from modifications to the objects passed into `canonicalize`, for example through creating a deep copy of the objects. -The `context` parameter is the same as passed to `get` and `set` which provides utilties from the Runtime Environment, and is decribed in more detail there. +The `context` parameter is the same passed to `get` and `set`, which provides utilties from the runtime environment, and is decribed in more detail there. -> Note: When the provider implements canonicalisation, it should strive for always logging canonicalized values. By virtue of `get`, and `set` always producing and consuming canonically formatted values, this is not expected to pose extra overhead. +> Note: When the provider implements canonicalization, it always logs canonicalized values. As a result of `get` and `set` producing and consuming canonically formatted values, this is not expected to present extra cost. -> Note: A interesting side-effect of these rules is the fact that the canonicalization of `get`'s return value must not change the processed values. Runtime environments may have strict or development modes that check this property. +> Note: A side effect of these rules is that the canonicalization of `get`'s return value must not change the processed values. Runtime environments may have strict or development modes that check this property. -## Provider Feature: simple_get_filter +### Provider feature: `simple_get_filter` -Allows for more efficient querying of the system state when only specific bits are required. +Allows for more efficient querying of the system state when only specific parts are required. ```ruby Puppet::ResourceApi.register_type( @@ -161,11 +165,11 @@ class Puppet::Provider::AptKey::AptKey end ``` -Some resources are very expensive to enumerate. In this case the provider can implement `simple_get_filter` to signal extended capabilities of the `get` method to address this. The provider's `get` method will be called with an Array of resource names, or `nil`. The `get` method must at least return the resources mentioned in the `names` Array, but may return more than those. As a special case, if the `names` parameter is `nil`, all existing resources should be returned. The `names` parameter should default to `nil` to allow simple runtimes to ignore this feature. +Some resources are very expensive to enumerate. The provider can implement `simple_get_filter` to signal extended capabilities of the `get` method to address this. The provider's `get` method will be called with an array of resource names, or `nil`. The `get` method must at least return the resources mentioned in the `names` array, but may return more than those. If the `names` parameter is `nil`, all existing resources should be returned. The `names` parameter defaults to `nil` to allow simple runtimes to ignore this feature. -The runtime environment should call `get` with a minimal set of names it is interested in, and should keep track of additional instances returned, to avoid double querying. To reap the most benefits from batching implementations, the runtime should minimize the number of calls into `get`. +The runtime environment calls `get` with a minimal set of names, and keeps track of additional instances returned to avoid double querying. To gain the most benefits from batching implementations, the runtime minimizes the number of calls into `get`. -## Provider Feature: noop_handler +### Provider feature: `noop_handler` ```ruby Puppet::ResourceApi.register_type( @@ -185,9 +189,9 @@ class Puppet::Provider::AptKey::AptKey end ``` -When a resource is marked with `noop => true`, either locally, or through a global flag, the standard runtime will emit the default change report with a `noop` flag set. In some cases an implementation can provide additional information (e.g. commands that would get executed), or requires additional evaluation before determining the effective changes (e.g. `exec`'s `onlyif` attribute). In those cases, the resource type can specify the `noop_handler` feature to have `set` called for all resources, even those flagged with `noop`. When the `noop` parameter is set to true, the provider must not change the system state, but only report what it would change. The `noop` parameter should default to `false` to allow simple runtimes to ignore this feature. +When a resource is marked with `noop => true`, either locally or through a global flag, the standard runtime will produce the default change report with a `noop` flag set. In some cases, an implementation provides additional information, for example commands that would get executed, or requires additional evaluation before determining the effective changes, for example the `exec`'s `onlyif` attribute. The resource type specifies the `noop_handler` feature to have `set` called for all resources, even those flagged with `noop`. When the `noop` parameter is set to true, the provider must not change the system state, but only report what it would change. The `noop` parameter should default to `false` to allow simple runtimes to ignore this feature. -## Provider Feature: remote_resource +### Provider feature: `remote_resource` ```ruby Puppet::ResourceApi.register_type( @@ -216,46 +220,52 @@ class Puppet::Provider::Nx9k_vlan::Nx9k_vlan end ``` -Declaring this feature restricts the resource from being run "locally". It is now expected to execute all its external interactions through the `context.device` instance. How that instance is set up is runtime specific. In puppet, it is configured through the [`device.conf`](https://puppet.com/docs/puppet/5.3/config_file_device.html) file, and only available when running under [`puppet device`](https://puppet.com/docs/puppet/5.3/man/device.html). It is recommended to use `Puppet::Util::NetworkDevice::Simple::Device` as the base class for all devices, which automatically loads a configuration from the local filesystem of the proxy node where it's running on. +Declaring this feature restricts the resource from being run "locally". It is expected to execute all external interactions through the `context.device` instance. The way that instance is set up is runtime specific. In Puppet, it is configured through the [`device.conf`](https://puppet.com/docs/puppet/5.3/config_file_device.html) file, and only available when running under [`puppet device`](https://puppet.com/docs/puppet/5.3/man/device.html). It is recommended to use `Puppet::Util::NetworkDevice::Simple::Device` as the base class for all devices, which automatically loads a configuration from the local filesystem of the proxy node where it is running on. + +## Runtime environment -# Runtime Environment +The primary runtime environment for the provider is the Puppet agent, a long-running daemon process. The provider can also be used in the Puppet apply command, a one-shot version of the agent, or the Puppet resource command, a short-lived command line interface (CLI) process for listing or managing a single resource type. Other callers who want to access the provider will have to imitate these environments. -The primary runtime environment for the provider is the puppet agent, a long-running daemon process. The provider can also be used in the puppet apply command, a one-shot version of the agent, or the puppet resource command, a short-lived CLI process for listing or managing a single resource type. Other callers who want to access the provider will have to emulate those environments. The primary lifecycle of resource managment in each of those tools is the *transaction*, a single set of changes (e.g. a catalog, or a CLI invocation) to work on. In any case the registered block will be surfaced in a clean class which will be instantiated once for each transaction. The provider can define any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for an provider, and use it immediately, the provider is instantiated as late as possible. A transaction will usually call `get` once, and may call `set` any number of times to effect change. The object instance hosting the `get` and `set` methods can be used to cache ephemeral state during execution. The provider should not try to cache state beyond the transaction, to avoid interfering with the agent daemon. In many other cases caching beyond the transaction won't help anyways, as the hosting process will only manage a single transaction. +The primary lifecycle of resource managment in each of these tools is the *transaction*, a single set of changes, for example a catalog or a CLI invocation. The registered block will be surfaced in a clean class, and will be instantiated once for each transaction. The provider defines any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for a provider and be used immediately, the provider is instantiated as late as possible. A transaction will usually call `get` once, and may call `set` any number of times to affect change. The object instance hosting the `get` and `set` methods can be used to cache ephemeral state during execution. The provider should not try to cache state beyond the transaction, to avoid interfering with the agent daemon. In some cases, caching beyond the transaction won't help as the hosting process will only manage a single transaction. -## Utilities +### Utilities -The runtime environment provides some utilities to make the providers's life easier, and provide a uniform experience for its users. +The runtime environment has some utilities to provide a uniform experience for its users. -### Logging and Reporting +#### Logging and reporting -The provider needs to signal changes, successes and failures to the runtime environment. The `context` is the primary way to do so. It provides a single interface for both the detailed technical information for later automatic processing, as well as human readable progress and status messages for operators. +The provider needs to signal changes, successes, and failures to the runtime environment. The `context` is the primary way to do this. It provides a single interface for technical information, including automatic processing, human readable progress, and status messages for operators. -#### General messages +[TODO: please check that the sentence above makes sense to you. This is how I understood what you wrote, but I may have changed the meaning] -To provide feedback about the overall operation of the provider, the `context` has the usual set of [loglevel](https://docs.puppet.com/puppet/latest/metaparameter.html#loglevel) methods that take a string, and pass that up to runtime environment's logging infrastructure: +##### General messages + +To provide feedback about the overall operation of the provider, the `context` has the usual set of [loglevel](https://docs.puppet.com/puppet/latest/metaparameter.html#loglevel) methods that take a string, and pass that up to the runtime environments logging infrastructure. ```ruby context.warning("Unexpected state detected, continuing in degraded mode.") ``` -will result in the following message: +results in the following message: ```text Warning: apt_key: Unexpected state detected, continuing in degraded mode. ``` -* debug: detailed messages to understand everything that is happening at runtime; only shown on request -* info: regular progress and status messages; especially useful before long-running operations, or before operations that can fail, to provide context to interactive users -* notice: indicates state changes and other events of notice from the regular operations of the provider -* warning: signals error conditions that do not (yet) prohibit execution of the main part of the provider; for example deprecation warnings, temporary errors -* err: signal error conditions that have caused normal operations to fail -* critical/alert/emerg: should not be used by resource providers +* debug: detailed messages to understand everything that is happening at runtime; shown on request. +* info: regular progress and status messages; especially useful before long-running operations, or before operations that can fail, to provide context to interactive users. +* notice: indicates state changes and other events of notice from the regular operations of the provider. +* warning: signals error conditions that do not (yet) prohibit execution of the main part of the provider; for example, deprecation warnings, temporary errors. +* err: signal error conditions that have caused normal operations to fail. +* critical/alert/emerg: should not be used by resource providers. See [wikipedia](https://en.wikipedia.org/wiki/Syslog#Severity_level) and [RFC424](https://tools.ietf.org/html/rfc5424) for more details. -#### Signalling resource status +[TODO: Could we include the list itself, if needed, and just link to the second link? Or a source wikipedia?] + +##### Signalling resource status -In many simple cases, a provider can pass off the real work to a external tool, detailed logging happens there, and reporting back to puppet only requires acknowledging those changes. In these situations, signalling can be as easy as this: +In some cases, a provider passes off work to an external tool. Detailed logging happens here, and then reports back to Puppet by acknowledging these changes. Signalling can be: ```ruby @apt_key_cmd.run(context, action, key_id) @@ -264,11 +274,11 @@ context.processed(key_id, is, should) This will report all changes from `is` to `should`, using default messages. -Providers that want to have more control over the logging throughout the processing can use the more specific `created(title)`, `updated(title)`, `deleted(title)`, `unchanged(title)` methods for that. To report the change of an attribute, the `context` provides a `attribute_changed(title, attribute, old_value, new_value, message)` method. +Providers that want to have more control over the logging throughout the processing can use the more specific `created(title)`, `updated(title)`, `deleted(title)`, `unchanged(title)` methods. To report the change of an attribute, the `context` provides a `attribute_changed(title, attribute, old_value, new_value, message)` method. -#### Logging contexts +##### Logging contexts -Most of those messages are expected to be relative to a specific resource instance, and a specific operation on that instance. To enable detailed logging without repeating key arguments, and provide consistent error logging, the context provides *logging context* methods that capture the current action and resource instance. +Most of those messages are expected to be relative to a specific resource instance, and a specific operation of that instance. To enable detailed logging without repeating key arguments, and to provide consistent error logging, the context provides *logging context* methods to capture the current action and resource instance: ```ruby context.updating(title) do @@ -305,7 +315,7 @@ Error: Apt_key[F1D2D2F9]: Updating failed: Something went wrong # TODO: update messages to match current log message formats for resource messages ``` -Logging contexts process all exceptions. [`StandardError`s](https://ruby-doc.org/core/StandardError.html) are assumed to be regular failures in handling a resources, and they are swallowed after logging. Everything else is assumed to be a fatal application-level issue, and is passed up the stack, ending execution. See the [ruby documentation](https://ruby-doc.org/core/Exception.html) for details on which exceptions are not `StandardError`s. +Logging contexts process all exceptions. [`StandardError`s](https://ruby-doc.org/core/StandardError.html) are assumed to be regular failures in handling resources, and are consumed after logging. Everything else is assumed to be a fatal application-level issue, and is passed up the stack, ending execution. See the [Ruby documentation](https://ruby-doc.org/core/Exception.html) for details on which exceptions are not a `StandardError`. The equivalent long-hand form with manual error handling: @@ -338,26 +348,26 @@ end This example is only for demonstration purposes. In the normal course of operations, providers should always use the utility functions. -#### Logging reference +##### Logging reference The following action/block methods are available: -* Block functions: These functions provide logging, and timing around a provider's core actions. If the the passed `&block` returns, the action is recorded as successful. To signal a failure, the block should raise an exception explaining the problem. +* Block functions: these functions provide logging and timing around a provider's core actions. If the the passed `&block` returns, the action is recorded as successful. To signal a failure, the block should raise an exception explaining the problem. * `creating(titles, message: 'Creating', &block)` * `updating(titles, message: 'Updating', &block)` * `deleting(titles, message: 'Deleting', &block)` - * `processing(title, is, should, message: 'Processing', &block)`: Generic processing of a resource, emits default change messages for the difference between `is:` and `should:`. - * `failing(titles, message: 'Failing', &block)`: unlikely to be used often, but provided for completeness, always records a failure. + * `processing(title, is, should, message: 'Processing', &block)`: generic processing of a resource, produces default change messages for the difference between `is:` and `should:`. + * `failing(titles, message: 'Failing', &block)`: unlikely to be used often, but provided for completeness - always records a failure. * Action functions * `created(titles, message: 'Created')` * `updated(titles, message: 'Updated')` * `deleted(titles, message: 'Deleted')` - * `processed(title, is, should)`: the resource has been processed - emit default logging for the resource and each attribute + * `processed(title, is, should)`: the resource has been processed - produces default logging for the resource and each attribute * `failed(titles, message:)`: the resource has not been updated successfully * Attribute Change notifications - * `attribute_changed(title, attribute, is, should, message: nil)`: Notify the runtime environment that a specific attribute for a specific resource has changed. `is` and `should` are the original and the new value of the attribute. Either can be `nil`. + * `attribute_changed(title, attribute, is, should, message: nil)`: notify the runtime environment that a specific attribute for a specific resource has changed. `is` and `should` are the original and the new value of the attribute. Either can be `nil`. * Plain messages * `debug(message)` @@ -371,19 +381,19 @@ The following action/block methods are available: * `err(message)` * `err(titles, message:)` -`titles` can be a single identifier for a resource, or an Array of values, if the following block batch-processes multiple resources in one pass. If that processing is not atomic, providers should instead use the non-block forms of logging, and provide accurate status reporting on the individual parts of update operations. +`titles` can be a single identifier for a resource or an array of values, if the following block batch processes multiple resources in one pass. If that processing is not atomic, providers should instead use the non-block forms of logging, and provide accurate status reporting on the individual parts of update operations. -A single `set()` execution may only log messages for instances it has been passed as part of the `changes` to process. Logging for instances not requested to be changed will cause an exception, as the runtime environment is not prepared for other resources to change. +A single `set()` execution may only log messages for instances that have been passed, as part of the `changes` to process. Logging for instances not requested to be changed will cause an exception - the runtime environment is not prepared for other resources to change. -The provider is free to call different logging methods for different resources in any order it needs to. The only ordering restriction is for all calls specifying the same `title`. The `attribute_changed` logging needs to be done before that resource's action logging, and if a context is opened, it needs to be opened before any other logging for this resource. +The provider is free to call different logging methods for different resources in any order it needs to. The only ordering restriction is for all calls specifying the same `title`. The `attribute_changed` logging needs to be done before that resource's action logging, and if a context is opened, needs to be opened before any other logging for this resource. -### Commands +#### Commands -To use CLI commands in a safe and comfortable manner, the Resource API provides a thin wrapper around the excellent [childprocess gem](https://rubygems.org/gems/childprocess) to address the most common use-cases. Through using the library commands and their arguments are never passed through the shell leading to a safer execution environment (no funny parsing), and faster execution times (no extra processes). +To use CLI commands in a safe manner, the Resource API provides a thin wrapper around the [childprocess gem](https://rubygems.org/gems/childprocess) to address the most common use cases. The library commands and arguments are never passed through the shell, leading to a safer execution environment and faster execution times, with no extra processes. -#### Creating a Command +##### Creating a reusable command -To create a re-usable command, create a new instance of `Puppet::ResourceApi::Command` passing in the command. You can either specify a full path, or a bare command name. In the latter case the Command will use the Runtime Environment's `PATH` setting to search for the command. +To create a new instance of `Puppet::ResourceApi::Command` passing in the command, you can either specify a full path or a bare command name. In the latter, the command will use the runtime environment's `PATH` setting to search for the command. ```ruby class Puppet::Provider::AptKey::AptKey @@ -393,13 +403,15 @@ class Puppet::Provider::AptKey::AptKey end ``` -> Note: It is recommended to create the command in the `initialize` function of the provider, and store them in a member named after the command, with the `_cmd` suffix. This makes it easy to re-use common settings throughout the provider. +> Note: It is recommended to create the command in the `initialize` function of the provider, and store them in a member named after the command, with the `_cmd` suffix. This makes it easy to reuse common settings throughout the provider. -You can set default environment variables on the `@cmd.environment` Hash, and a default working directory using `@cmd.cwd=`. +[TODO: it is usually best to avoid saying "It is recommended". Could you be more specific on whether they should do it or not?] -#### Running simple commands +You can set default environment variables on the `@cmd.environment` hash, and a default working directory using `@cmd.cwd=`. -The `run(*args)` method takes any number of arguments, and executes the command using them on the command line. For example to call `apt-key` to delete a specific key by id: +##### Running simple commands + +The `run(*args)` method takes any number of arguments, and executes the command using them on the command line. For example, to call `apt-key` to delete a specific key by id: ```ruby class Puppet::Provider::AptKey::AptKey @@ -408,13 +420,13 @@ class Puppet::Provider::AptKey::AptKey @apt_key_cmd.run(context, 'del', key_id) ``` -If the command is not available, a `Puppet::ResourceApi::CommandNotFoundError` will be raised. This can be easily used to fail the resources for a specific run, if the requirements for the provider are not yet met. +If the command is not available, a `Puppet::ResourceApi::CommandNotFoundError` will appear. This can be used to fail the resources for a specific run if the requirements for the provider are not met. -The call will only return after the command has finished executing. If the command exits with a exitstatus indicating an error condition (that is non-zero), a `Puppet::ResourceApi::CommandExecutionError` is raised, containing the details of the command, and exit status. +The call will only return after the command has finished executing. If the command exits with an exit status indicating an error condition - that is non-zero - a `Puppet::ResourceApi::CommandExecutionError` will be raised, containing the details of the command and exit status. By default the `stdout` of the command is logged to debug, while the `stderr` is logged to warning. -#### Implementing `noop` for `noop_handler` +##### Implementing `noop` for `noop_handler` The `run` method takes a `noop:` keyword argument, and will signal success while skipping the real execution if necessary. Providers implementing the `noop_handler` feature should use this for all commands that are executed in the regular flow of the implementation. @@ -425,7 +437,7 @@ class Puppet::Provider::AptKey::AptKey @apt_key_cmd.run(context, 'del', key_id, noop: noop) ``` -#### Passing in specific environment variables +##### Passing in specific environment variables To pass additional environment variables through to the command, pass a hash of them as `environment:`: @@ -435,7 +447,7 @@ To pass additional environment variables through to the command, pass a hash of This can also be set on the `@cmd.environment` attribute to run all executions of the command with the same environment. -#### Running in a specific directory +##### Running in a specific directory To run the command in a specific directory, use the `cwd` keyword argument: @@ -445,22 +457,22 @@ To run the command in a specific directory, use the `cwd` keyword argument: This can also be set on the `@cmd.cwd` attribute to run all executions of the command with the working directory. -#### Processing command output +##### Processing command output -To use a command to read information from the system, `run` can redirect the output from the command to various destinations, using the `stdout_destination:` and `stderr_destination:` keywords: -* `:log`: each line from the specified stream gets logged to the Runtime Environment. Use `stdout_loglevel:`, and `stderr_loglevel:` to specify the intended loglevel. -* `:store`: the stream gets captured in a buffer, and will be returned as a string in the `result` object. +When using a command to read information from the system, `run` can redirect the output from the command to various destinations, using the `stdout_destination:` and `stderr_destination:` keywords: +* `:log`: each line from the specified stream gets logged to the runtime environment. Use `stdout_loglevel:` and `stderr_loglevel:` to specify the intended loglevel. +* `:store`: the stream gets captured in a buffer and will be returned as a string in the `result` object. * `:discard`: the stream is discarded unprocessed. * `:io`: the stream is connected to the IO object specified in `stdout_io:`, and `stderr_io:`. -* `:merge_to_stdout`: to get the process' standard error correctly interleaved into its regular output, this option can be specified for `stderr_destination:` only, and will provide the same file descriptor for both stdout, and stderr to the process. +* `:merge_to_stdout`: to get the process standard error correctly inserted into its regular output, specify for `stderr_destination:` only. This will provide the same file descriptor for both `stdout` and `stderr` to process. -By default the standard output of the process is logged at the debug level , and the standard error stream is logged at the warning level. To replicate this behaviour: +By default, the standard output of the process is logged at the debug level and the standard error stream is logged at the warning level. To replicate this behaviour: ```ruby @apt_key_cmd.run(context, 'del', key_id, stdout_destination: :log, stdout_loglevel: :debug, stderr_destination: :log, stderr_loglevel: :warning) ``` -To store, and process the output from the command, use the `:store` destination, and the `result` object: +To store and process the output from the command, use the `:store` destination, and the `result` object: ```ruby class Puppet::Provider::AptKey::AptKey @@ -471,16 +483,16 @@ class Puppet::Provider::AptKey::AptKey end ``` -To emulate most shell based redirections to files, the `:io` destination lets you (re-)use `File` handles, and temporary files (through `Tempfile`): +To imitate most shell based redirections to files, the `:io` destination lets you (re)use `File` handles and temporary files through `Tempfile`: ```ruby tmp = Tempfile.new('key_list') @apt_key_cmd.run(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode', stdout_destination: :io, stdout_io: tmp) ``` -#### Providing command input +##### Providing command input -To use a command to write to, `run` allows passing input into the process. For example, to provide a key on stdin to the apt-key tool: +To use a command to write to, `run` allows passing input into the process. For example, to provide a key on `stdin` to the apt-key tool: ```ruby class Puppet::Provider::AptKey::AptKey @@ -495,19 +507,21 @@ The `stdin_source:` keyword argument takes the following values: * `:io`: the input of the process is connected to the IO object specified in `stdin_io:`. * `:none`: the input of the process is closed, and the process will receive an error when trying to read input. This is the default. -#### Character encoding +##### Character encoding + +To support the widest array of platforms and use cases, character encoding of a provider's inputs and outputs need to be considered. By default the commands API follows the Ruby model of having all strings tagged with their current [`Encoding`](https://ruby-doc.org/core/Encoding.html), and uses the system's current default character set for I/O. This means that strings read from commands might be tagged with non-UTF-8 character sets on input and UTF-8 strings transcoded on output. -To support the widest array of platforms and use-cases, character encoding of a provider's inputs, and outputs needs to be considered. By default, the Commands API follows the ruby model of having all strings tagged with their current [`Encoding`](https://ruby-doc.org/core/Encoding.html), and using the system's current default character set for I/O. That means that strings read from Commands might be tagged with non-UTF-8 character sets on input, and UTF-8 strings might be transcoded on output. +[TODO: is it IO or I/O? Or are they different?] -To influence this behaviour, you can tell the `run` method which encoding to use, and enable transcoding, if required for your use-case. Use the following keyword arguments: +To influence this behaviour, tell the `run` method which encoding to use and enable transcoding. Use the following keyword arguments: -* `stdout_encoding:`, `stderr_encoding:` The encoding to tag incoming bytes with. +* `stdout_encoding:`, `stderr_encoding:` the encoding to tag incoming bytes. * `stdin_encoding:` ensures that strings are transcoded to this encoding before being written to this command. * `stdout_encoding_opts:`,`stderr_encoding_opts:`,`stdin_encoding_opts:` options for [`String.encode`](https://ruby-doc.org/core-2.4.1/String.html#method-i-encode) for the different streams. -> Note: Use the `ASCII-8BIT` encoding to disable all conversions, and receive the raw bytes. +> Note: Use the `ASCII-8BIT` encoding to disable all conversions and receive the raw bytes. -#### Summary +##### Summary Synopsis of the `run` function: `@cmd.run(context, *args, **kwargs)` with the following keyword arguments: @@ -522,17 +536,19 @@ Synopsis of the `run` function: `@cmd.run(context, *args, **kwargs)` with the fo * `stdin_value:` a String * `ignore_exit_code:` `true` or `false` -The `run` function returns an object with the attributes `stdout`, `stderr`, and `exit_code`. The first two will only be used, if their respective `destination:` is set to `:store`. The `exit_code` will contain the exit code of the process. +The `run` function returns an object with the attributes `stdout`, `stderr`, and `exit_code`. The first two will only be used if their respective `destination:` is set to `:store`. The `exit_code` will contain the exit code of the process. -# Known Limitations +## Known limitations -This API is not a full replacement for the power of 3.x style types and providers. Here is a (incomplete) list of missing pieces and thoughts on how to go about solving these. In the end, the goal of the new Resource API is not to be a complete replacement of prior art, but a cleaner way to get good results for the majority of simple cases. +This API is not a full replacement for the power of 3.x style types and providers. Here is an (incomplete) list of missing pieces and thoughts on how to solve these. The goal of the new Resource API is not to be a replacement of the prior one, but to be a simplified way to get results for the majority of use cases. -## Multiple providers for the same type +[TODO: where is the list mentioned above?] -The original Puppet Type and Provider API allows multiple providers for the same resource type. This allows the creation of abstract resource types, such as package, which can span multiple operating systems. Automatic selection of an os-appropriate provider means less work for the user, as they don't have to address in their code whether the package needs to be managed using apt, or managed using yum. +### Multiple providers for the same type -Allowing multiple providers doesn't come for free though and in the previous implementation it incurs a number of complexity costs to be shouldered by the type or provider developer. +The original Puppet type and provider API allows multiple providers for the same resource type. This allows the creation of abstract resource types, such as package, which can span multiple operating systems. Automatic selection of an os-appropriate provider means less work for the user, as they don't have to address in their code whether the package needs to be managed using apt or yum. + +Allowing multiple providers does not come for free. The previous implementation incurs a number of complexity costs that are shouldered by the type or provider developer. attribute sprawl disparate feature sets between the different providers for the same abstract type @@ -540,7 +556,10 @@ Allowing multiple providers doesn't come for free though and in the previous imp The Resource API will not implement support for multiple providers at this time. -Today, should support for multiple providers be highly desirable for a given type, the two options are: 1) use the older, more complex API. 2) implement multiple similar types using the Resource API, and select the platform-appropriate type in Puppet code. For example: +Should support for multiple providers be desirable for a given type, the two options are: + +1. Use the older, more complex API. +2. Implement multiple similar types using the Resource API, and select the platform-appropriate type in Puppet code. For example: ```puppet define package ( @@ -572,20 +591,20 @@ define package ( } ``` -Neither of these options is ideal, thus it is documented as a limitation today. Ideas for the future include forward-porting the status quo through enabling multiple Implementations to register for the same Definition, or allowing Definitions to declare (partial) equivalence to other Definitions (ala "apt::package is a package"). +Neither of these options are ideal; and are documented as a limitation. Improvement ideas include forward porting the status quo by enabling multiple implementations to register for the same definition, or allowing definitions to declare (partial) equivalence to other definitions (ala "apt::package is a package"). -## Composite namevars +### Composite namevars -The current API does not provide a way to specify composite namevars (that is, types with multiple namevars). [`title_patterns`](https://github.com/puppetlabs/puppet-specifications/blob/master/language/resource_types.md#title-patterns) are already very data driven, and will be easy to add at a later point. +The current API does not provide a way to specify composite namevars - types with multiple namevars. [`title_patterns`](https://github.com/puppetlabs/puppet-specifications/blob/master/language/resource_types.md#title-patterns) are already very data driven, and will be easier to add at a later point. -## Puppet 4 data types +### Puppet 4 data types -Currently anywhere "puppet 4 data types" are mentioned, only the built-in types are usable. This is because the type information is required on the agent, but puppet doesn't make it available yet. This work is tracked in [PUP-7197](https://tickets.puppetlabs.com/browse/PUP-7197). Even once that is implemented, modules will have to wait until the functionality is widely available, before being able to rely on that. +Currently, only built-in Puppet 4 data types are usable. This is because the type information is required on the agent, but Puppet has not made it available yet. This work is tracked in [PUP-7197](https://tickets.puppetlabs.com/browse/PUP-7197). Even once that is implemented, modules will have to wait until the functionality is widely available before being able to rely on it. -## Catalog access +### Catalog access -There is no way to access the catalog from the provider. Several existing types rely on this to implement advanced functionality. Some of those use-cases would be better suited to be implemented as "external" catalog transformations, instead of munging the catalog from within the compilation process. +There is no way to access the catalog from the provider. Several existing types rely on this to implement advanced functionality. Some of these use cases would be better off being implemented as "external" catalog transformations, instead of munging the catalog from within the compilation process. -## Logging for unmanaged instances +### Logging for unmanaged instances -The provider could provide log messages for resource instances that were not passed into the `set` call. In the current implementation those will cause an error. How this is handeled in the future might change drastically. +Previously, the provider could provide log messages for resource instances that were not passed into the `set` call. In the current implementation, these will cause an error. From 377a05d5a77773576c659d0b58a69de7c0f6da68 Mon Sep 17 00:00:00 2001 From: claire cadman Date: Mon, 19 Feb 2018 17:34:57 -0800 Subject: [PATCH 55/62] more readme updates --- language/resource-api/README.md | 32 +++++++++++--------------------- 1 file changed, 11 insertions(+), 21 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 0373810..46d4ca7 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -108,7 +108,7 @@ The `set` method should always return `nil`. Any progress signaling should be do Both methods take a `context` parameter which provides utilties from the runtime environment, and is decribed in more detail there. -### Provider features +## Provider features There are some use cases where an implementation provides a better experience than the default runtime environment provides. To avoid burdening the simplest providers with that additional complexity, these cases are hidden behind feature flags. To enable the special handling, the resource definition has a `feature` key to list all features implemented by the provider. @@ -134,13 +134,13 @@ class Puppet::Provider::AptKey::AptKey end ``` -The runtime environment needs to compare user input from the manifest (the desired state) with values returned from `get` (the actual state) to determine whether or not changes need to be affected. In simple cases, a provider will only accept values from the manifest in the same format as `get` returns. No extra work is required, as a value comparison will suffice. This places a high burden on the user to provide values in an unnaturally constrained format. In the example, the `apt_key` name is a hexadecimal number that can be written with, and without, the `'0x'` prefix, and the casing of the digits is irrelevant. A value comparison on the strings would cause false positives if the user input format that does not match. There is no hexadecimal type in the Puppet language. The provider can specify the `canonicalize` feature and implement the `canonicalize` method. +The runtime environment needs to compare user input from the manifest (the desired state) with values returned from `get` (the actual state) to determine whether or not changes need to be affected. In simple cases, a provider will only accept values from the manifest in the same format as `get` returns. No extra work is required, as a value comparison will suffice. This places a high burden on the user to provide values in an unnaturally constrained format. In the example, the `apt_key` name is a hexadecimal number that can be written with, and without, the `'0x'` prefix, and the casing of the digits is irrelevant. A value comparison on the strings would cause false positives if the user input format that does not match. There is no hexadecimal type in the Puppet language. To address this, the provider can specify the `canonicalize` feature and implement the `canonicalize` method. The `canonicalize` method transforms its `resources` argument into the standard format required by the rest of the provider. The `resources` argument to `canonicalize` is an enumerable of resource hashes matching the structure returned by `get`. It returns all passed values in the same structure with the required transformations applied. It is free to reuse or recreate the data structures passed in as arguments. The runtime environment must use `canonicalize` before comparing user input values with values returned from `get`. The runtime environment always passes canonicalized values into `set`. If the runtime environment requires the original values for later processing, it protects itself from modifications to the objects passed into `canonicalize`, for example through creating a deep copy of the objects. The `context` parameter is the same passed to `get` and `set`, which provides utilties from the runtime environment, and is decribed in more detail there. -> Note: When the provider implements canonicalization, it always logs canonicalized values. As a result of `get` and `set` producing and consuming canonically formatted values, this is not expected to present extra cost. +> Note: When the provider implements canonicalization, it aims to always log the canonicalized values. As a result of `get` and `set` producing and consuming canonically formatted values, this is not expected to present extra cost. > Note: A side effect of these rules is that the canonicalization of `get`'s return value must not change the processed values. Runtime environments may have strict or development modes that check this property. @@ -226,7 +226,7 @@ Declaring this feature restricts the resource from being run "locally". It is ex The primary runtime environment for the provider is the Puppet agent, a long-running daemon process. The provider can also be used in the Puppet apply command, a one-shot version of the agent, or the Puppet resource command, a short-lived command line interface (CLI) process for listing or managing a single resource type. Other callers who want to access the provider will have to imitate these environments. -The primary lifecycle of resource managment in each of these tools is the *transaction*, a single set of changes, for example a catalog or a CLI invocation. The registered block will be surfaced in a clean class, and will be instantiated once for each transaction. The provider defines any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for a provider and be used immediately, the provider is instantiated as late as possible. A transaction will usually call `get` once, and may call `set` any number of times to affect change. The object instance hosting the `get` and `set` methods can be used to cache ephemeral state during execution. The provider should not try to cache state beyond the transaction, to avoid interfering with the agent daemon. In some cases, caching beyond the transaction won't help as the hosting process will only manage a single transaction. +The primary lifecycle of resource managment in each of these tools is the transaction, a single set of changes, for example a catalog or a CLI invocation. The provider's class will be instantiated once for each transaction. Within that class the provider defines any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for a provider and be used immediately, the provider is instantiated as late as possible. A transaction will usually call `get` once, and may call `set` any number of times to affect change. The object instance hosting the `get` and `set` methods can be used to cache ephemeral state during execution. The provider should not try to cache state outside of its instances. In many cases, such caching won't help as the hosting process will only manage a single transaction. In long-running runtime environments (like the agent) the benefit of the caching needs to be balanced by the cost of the cache at rest, and the lifetime of cache entries, which are only useful when they are longer than the regular `runinterval`. ### Utilities @@ -234,9 +234,7 @@ The runtime environment has some utilities to provide a uniform experience for i #### Logging and reporting -The provider needs to signal changes, successes, and failures to the runtime environment. The `context` is the primary way to do this. It provides a single interface for technical information, including automatic processing, human readable progress, and status messages for operators. - -[TODO: please check that the sentence above makes sense to you. This is how I understood what you wrote, but I may have changed the meaning] +The provider needs to signal changes, successes, and failures to the runtime environment. The `context` is the primary way to do this. It provides a structured logging interface for all provider actions. Using this information the runtime environments can do automatic processing, emit human readable progress information, and provide status messages for operators. ##### General messages @@ -259,13 +257,11 @@ Warning: apt_key: Unexpected state detected, continuing in degraded mode. * err: signal error conditions that have caused normal operations to fail. * critical/alert/emerg: should not be used by resource providers. -See [wikipedia](https://en.wikipedia.org/wiki/Syslog#Severity_level) and [RFC424](https://tools.ietf.org/html/rfc5424) for more details. - -[TODO: Could we include the list itself, if needed, and just link to the second link? Or a source wikipedia?] +See [RFC424](https://tools.ietf.org/html/rfc5424) for more details. ##### Signalling resource status -In some cases, a provider passes off work to an external tool. Detailed logging happens here, and then reports back to Puppet by acknowledging these changes. Signalling can be: +In many simple cases, a provider passes off work to an external tool. Detailed logging happens there, and then reports back to Puppet by acknowledging these changes. Signalling can be: ```ruby @apt_key_cmd.run(context, action, key_id) @@ -278,7 +274,7 @@ Providers that want to have more control over the logging throughout the process ##### Logging contexts -Most of those messages are expected to be relative to a specific resource instance, and a specific operation of that instance. To enable detailed logging without repeating key arguments, and to provide consistent error logging, the context provides *logging context* methods to capture the current action and resource instance: +Most of those messages are expected to be relative to a specific resource instance, and a specific operation on that instance. To enable detailed logging without repeating key arguments, and to provide consistent error logging, the context provides *logging context* methods to capture the current action and resource instance: ```ruby context.updating(title) do @@ -393,7 +389,7 @@ To use CLI commands in a safe manner, the Resource API provides a thin wrapper a ##### Creating a reusable command -To create a new instance of `Puppet::ResourceApi::Command` passing in the command, you can either specify a full path or a bare command name. In the latter, the command will use the runtime environment's `PATH` setting to search for the command. +To create a new instance of `Puppet::ResourceApi::Command` passing in the command, you can either specify a full path or a bare command name. In the latter case the command will use the runtime environment's `PATH` setting to search for the command. ```ruby class Puppet::Provider::AptKey::AptKey @@ -405,8 +401,6 @@ class Puppet::Provider::AptKey::AptKey > Note: It is recommended to create the command in the `initialize` function of the provider, and store them in a member named after the command, with the `_cmd` suffix. This makes it easy to reuse common settings throughout the provider. -[TODO: it is usually best to avoid saying "It is recommended". Could you be more specific on whether they should do it or not?] - You can set default environment variables on the `@cmd.environment` hash, and a default working directory using `@cmd.cwd=`. ##### Running simple commands @@ -420,7 +414,7 @@ class Puppet::Provider::AptKey::AptKey @apt_key_cmd.run(context, 'del', key_id) ``` -If the command is not available, a `Puppet::ResourceApi::CommandNotFoundError` will appear. This can be used to fail the resources for a specific run if the requirements for the provider are not met. +If the command is not available, a `Puppet::ResourceApi::CommandNotFoundError` will be raised. This can be used to fail the resources for a specific run if the requirements for the provider are not met. The call will only return after the command has finished executing. If the command exits with an exit status indicating an error condition - that is non-zero - a `Puppet::ResourceApi::CommandExecutionError` will be raised, containing the details of the command and exit status. @@ -511,8 +505,6 @@ The `stdin_source:` keyword argument takes the following values: To support the widest array of platforms and use cases, character encoding of a provider's inputs and outputs need to be considered. By default the commands API follows the Ruby model of having all strings tagged with their current [`Encoding`](https://ruby-doc.org/core/Encoding.html), and uses the system's current default character set for I/O. This means that strings read from commands might be tagged with non-UTF-8 character sets on input and UTF-8 strings transcoded on output. -[TODO: is it IO or I/O? Or are they different?] - To influence this behaviour, tell the `run` method which encoding to use and enable transcoding. Use the following keyword arguments: * `stdout_encoding:`, `stderr_encoding:` the encoding to tag incoming bytes. @@ -542,8 +534,6 @@ The `run` function returns an object with the attributes `stdout`, `stderr`, and This API is not a full replacement for the power of 3.x style types and providers. Here is an (incomplete) list of missing pieces and thoughts on how to solve these. The goal of the new Resource API is not to be a replacement of the prior one, but to be a simplified way to get results for the majority of use cases. -[TODO: where is the list mentioned above?] - ### Multiple providers for the same type The original Puppet type and provider API allows multiple providers for the same resource type. This allows the creation of abstract resource types, such as package, which can span multiple operating systems. Automatic selection of an os-appropriate provider means less work for the user, as they don't have to address in their code whether the package needs to be managed using apt or yum. @@ -595,7 +585,7 @@ Neither of these options are ideal; and are documented as a limitation. Improvem ### Composite namevars -The current API does not provide a way to specify composite namevars - types with multiple namevars. [`title_patterns`](https://github.com/puppetlabs/puppet-specifications/blob/master/language/resource_types.md#title-patterns) are already very data driven, and will be easier to add at a later point. +The current API does not provide a way to specify composite namevars - types with multiple namevars. [`title_patterns`](https://github.com/puppetlabs/puppet-specifications/blob/master/language/resource_types.md#title-patterns) are already very data driven, and will be easy to add at a later point. ### Puppet 4 data types From c00ae5d7c840126603a62dac66c269d2173cc46d Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 21 Feb 2018 19:20:41 +0000 Subject: [PATCH 56/62] Remove Command API See https://groups.google.com/a/puppet.com/d/msg/discuss-remote-ral/ydCszu_GnTM/lzmKVR8OAgAJ for details --- language/resource-api/README.md | 147 -------------------------------- 1 file changed, 147 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 46d4ca7..1f2fc29 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -383,153 +383,6 @@ A single `set()` execution may only log messages for instances that have been pa The provider is free to call different logging methods for different resources in any order it needs to. The only ordering restriction is for all calls specifying the same `title`. The `attribute_changed` logging needs to be done before that resource's action logging, and if a context is opened, needs to be opened before any other logging for this resource. -#### Commands - -To use CLI commands in a safe manner, the Resource API provides a thin wrapper around the [childprocess gem](https://rubygems.org/gems/childprocess) to address the most common use cases. The library commands and arguments are never passed through the shell, leading to a safer execution environment and faster execution times, with no extra processes. - -##### Creating a reusable command - -To create a new instance of `Puppet::ResourceApi::Command` passing in the command, you can either specify a full path or a bare command name. In the latter case the command will use the runtime environment's `PATH` setting to search for the command. - -```ruby -class Puppet::Provider::AptKey::AptKey - def initialize - @apt_key_cmd = Puppet::ResourceApi::Command.new('/usr/bin/apt-key') - @gpg_cmd = Puppet::ResourceApi::Command.new('gpg') - end -``` - -> Note: It is recommended to create the command in the `initialize` function of the provider, and store them in a member named after the command, with the `_cmd` suffix. This makes it easy to reuse common settings throughout the provider. - -You can set default environment variables on the `@cmd.environment` hash, and a default working directory using `@cmd.cwd=`. - -##### Running simple commands - -The `run(*args)` method takes any number of arguments, and executes the command using them on the command line. For example, to call `apt-key` to delete a specific key by id: - -```ruby -class Puppet::Provider::AptKey::AptKey - def set(context, changes) - # ... - @apt_key_cmd.run(context, 'del', key_id) -``` - -If the command is not available, a `Puppet::ResourceApi::CommandNotFoundError` will be raised. This can be used to fail the resources for a specific run if the requirements for the provider are not met. - -The call will only return after the command has finished executing. If the command exits with an exit status indicating an error condition - that is non-zero - a `Puppet::ResourceApi::CommandExecutionError` will be raised, containing the details of the command and exit status. - -By default the `stdout` of the command is logged to debug, while the `stderr` is logged to warning. - -##### Implementing `noop` for `noop_handler` - -The `run` method takes a `noop:` keyword argument, and will signal success while skipping the real execution if necessary. Providers implementing the `noop_handler` feature should use this for all commands that are executed in the regular flow of the implementation. - -```ruby -class Puppet::Provider::AptKey::AptKey - def set(context, changes, noop: false) - # ... - @apt_key_cmd.run(context, 'del', key_id, noop: noop) -``` - -##### Passing in specific environment variables - -To pass additional environment variables through to the command, pass a hash of them as `environment:`: - -```ruby -@apt_key_cmd.run(context, 'del', key_id, environment: { 'LC_ALL': 'C' }) -``` - -This can also be set on the `@cmd.environment` attribute to run all executions of the command with the same environment. - -##### Running in a specific directory - -To run the command in a specific directory, use the `cwd` keyword argument: - -```ruby -@apt_key_cmd.run(context, 'del', key_id, cwd: '/etc/apt') -``` - -This can also be set on the `@cmd.cwd` attribute to run all executions of the command with the working directory. - -##### Processing command output - -When using a command to read information from the system, `run` can redirect the output from the command to various destinations, using the `stdout_destination:` and `stderr_destination:` keywords: -* `:log`: each line from the specified stream gets logged to the runtime environment. Use `stdout_loglevel:` and `stderr_loglevel:` to specify the intended loglevel. -* `:store`: the stream gets captured in a buffer and will be returned as a string in the `result` object. -* `:discard`: the stream is discarded unprocessed. -* `:io`: the stream is connected to the IO object specified in `stdout_io:`, and `stderr_io:`. -* `:merge_to_stdout`: to get the process standard error correctly inserted into its regular output, specify for `stderr_destination:` only. This will provide the same file descriptor for both `stdout` and `stderr` to process. - -By default, the standard output of the process is logged at the debug level and the standard error stream is logged at the warning level. To replicate this behaviour: - -```ruby -@apt_key_cmd.run(context, 'del', key_id, stdout_destination: :log, stdout_loglevel: :debug, stderr_destination: :log, stderr_loglevel: :warning) -``` - -To store and process the output from the command, use the `:store` destination, and the `result` object: - -```ruby -class Puppet::Provider::AptKey::AptKey - def get(context) - run_result = @apt_key_cmd.run(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode', stdout_destination: :store) - run_result.stdout.split('\n').each do |line| - # process/parse stdout_text here - end -``` - -To imitate most shell based redirections to files, the `:io` destination lets you (re)use `File` handles and temporary files through `Tempfile`: - -```ruby -tmp = Tempfile.new('key_list') -@apt_key_cmd.run(context, 'adv', '--list-keys', '--with-colons', '--fingerprint', '--fixed-list-mode', stdout_destination: :io, stdout_io: tmp) -``` - -##### Providing command input - -To use a command to write to, `run` allows passing input into the process. For example, to provide a key on `stdin` to the apt-key tool: - -```ruby -class Puppet::Provider::AptKey::AptKey - def set(context, changes) - # ... - @apt_key_cmd.run(context, 'add', '-', stdin_source: :value, stdin_value: the_key_string) - end -``` - -The `stdin_source:` keyword argument takes the following values: -* `:value`: allows specifying a string to pass on to the process in `stdin_value:`. -* `:io`: the input of the process is connected to the IO object specified in `stdin_io:`. -* `:none`: the input of the process is closed, and the process will receive an error when trying to read input. This is the default. - -##### Character encoding - -To support the widest array of platforms and use cases, character encoding of a provider's inputs and outputs need to be considered. By default the commands API follows the Ruby model of having all strings tagged with their current [`Encoding`](https://ruby-doc.org/core/Encoding.html), and uses the system's current default character set for I/O. This means that strings read from commands might be tagged with non-UTF-8 character sets on input and UTF-8 strings transcoded on output. - -To influence this behaviour, tell the `run` method which encoding to use and enable transcoding. Use the following keyword arguments: - -* `stdout_encoding:`, `stderr_encoding:` the encoding to tag incoming bytes. -* `stdin_encoding:` ensures that strings are transcoded to this encoding before being written to this command. -* `stdout_encoding_opts:`,`stderr_encoding_opts:`,`stdin_encoding_opts:` options for [`String.encode`](https://ruby-doc.org/core-2.4.1/String.html#method-i-encode) for the different streams. - -> Note: Use the `ASCII-8BIT` encoding to disable all conversions and receive the raw bytes. - -##### Summary - -Synopsis of the `run` function: `@cmd.run(context, *args, **kwargs)` with the following keyword arguments: - -* `stdout_destination:` `:log`, `:io`, `:store`, `:discard` -* `stdout_loglevel:` `:debug`, `:info`, `:notice`, `:warning`, `:err` -* `stdout_io:` an `IO` object -* `stderr_destination:` `:log`, `:io`, `:store`, `:discard`, `:merge_to_stdout` -* `stderr_loglevel:` `:debug`, `:info`, `:notice`, `:warning`, `:err` -* `stderr_io:` an `IO` object -* `stdin_source:` `:io`, `:value`, `:none` -* `stdin_io:` an `IO` object -* `stdin_value:` a String -* `ignore_exit_code:` `true` or `false` - -The `run` function returns an object with the attributes `stdout`, `stderr`, and `exit_code`. The first two will only be used if their respective `destination:` is set to `:store`. The `exit_code` will contain the exit code of the process. - ## Known limitations This API is not a full replacement for the power of 3.x style types and providers. Here is an (incomplete) list of missing pieces and thoughts on how to solve these. The goal of the new Resource API is not to be a replacement of the prior one, but to be a simplified way to get results for the majority of use cases. From e84a07fe23830593e0793dafb31277fb67568c9b Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 21 Feb 2018 19:20:48 +0000 Subject: [PATCH 57/62] Cleanup whitespace --- language/resource-api/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 1f2fc29..f3ca5d5 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -68,7 +68,7 @@ For autoloading work, this code needs to go into `lib/puppet/type/.rb` in ## Resource implementation ("provider") -To affect changes, a resource requires an implementation that makes the universe's state available to Puppet, and causes the changes to bring reality to whatever state is requested in the catalog. The two fundamental operations to manage resources are reading and writing system state. These operations are implemented as `get` and `set`. The implementation itself is a basic Ruby class in the `Puppet::Provider` namespace, named after the type using CamelCase. +To affect changes, a resource requires an implementation that makes the universe's state available to Puppet, and causes the changes to bring reality to whatever state is requested in the catalog. The two fundamental operations to manage resources are reading and writing system state. These operations are implemented as `get` and `set`. The implementation itself is a basic Ruby class in the `Puppet::Provider` namespace, named after the type using CamelCase. > Note: Due to the way Puppet autoload works, this has to be in a file called `puppet/provider//.rb`. The class will also have the CamelCased type name twice. @@ -100,9 +100,9 @@ end The `get` method reports the current state of the managed resources. It returns an enumerable of all existing resources. Each resource is a hash with attribute names as keys, and their respective values as values. It is an error to return values not matching the type specified in the resource type. If a requested resource is not listed in the result, it is considered to not exist on the system. If the `get` method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The exception message will be reported to the user. -The `set` method updates resources to a new state. The `changes` parameter gets passed a hash of change requests, keyed by the resource's name. Each value is another hash with the optional `:is` and `:should` keys. At least one of the two has to be specified. The values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. +The `set` method updates resources to a new state. The `changes` parameter gets passed a hash of change requests, keyed by the resource's name. Each value is another hash with the optional `:is` and `:should` keys. At least one of the two has to be specified. The values will be of the same shape as those returned by `get`. After the `set`, all resources should be in the state defined by the `:should` values. -A missing `:should` entry indicates that a resource should be removed from the system. Even a type implementing the `ensure => [present, absent]` attribute pattern still has to react correctly on a missing `:should` entry. `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources were not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. +A missing `:should` entry indicates that a resource should be removed from the system. Even a type implementing the `ensure => [present, absent]` attribute pattern still has to react correctly on a missing `:should` entry. `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources were not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. The `set` method should always return `nil`. Any progress signaling should be done through the logging utilities described below. If the `set` method throws an exception, all resources that should change in this call and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the `set` method if there are changes to be made, especially in the case of resources marked with `noop => true` (either locally or through a global flag). The runtime will not pass them to `set`. See `noop_handler` below for changing this behaviour if required. @@ -224,7 +224,7 @@ Declaring this feature restricts the resource from being run "locally". It is ex ## Runtime environment -The primary runtime environment for the provider is the Puppet agent, a long-running daemon process. The provider can also be used in the Puppet apply command, a one-shot version of the agent, or the Puppet resource command, a short-lived command line interface (CLI) process for listing or managing a single resource type. Other callers who want to access the provider will have to imitate these environments. +The primary runtime environment for the provider is the Puppet agent, a long-running daemon process. The provider can also be used in the Puppet apply command, a one-shot version of the agent, or the Puppet resource command, a short-lived command line interface (CLI) process for listing or managing a single resource type. Other callers who want to access the provider will have to imitate these environments. The primary lifecycle of resource managment in each of these tools is the transaction, a single set of changes, for example a catalog or a CLI invocation. The provider's class will be instantiated once for each transaction. Within that class the provider defines any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for a provider and be used immediately, the provider is instantiated as late as possible. A transaction will usually call `get` once, and may call `set` any number of times to affect change. The object instance hosting the `get` and `set` methods can be used to cache ephemeral state during execution. The provider should not try to cache state outside of its instances. In many cases, such caching won't help as the hosting process will only manage a single transaction. In long-running runtime environments (like the agent) the benefit of the caching needs to be balanced by the cost of the cache at rest, and the lifetime of cache entries, which are only useful when they are longer than the regular `runinterval`. @@ -363,7 +363,7 @@ The following action/block methods are available: * `failed(titles, message:)`: the resource has not been updated successfully * Attribute Change notifications - * `attribute_changed(title, attribute, is, should, message: nil)`: notify the runtime environment that a specific attribute for a specific resource has changed. `is` and `should` are the original and the new value of the attribute. Either can be `nil`. + * `attribute_changed(title, attribute, is, should, message: nil)`: notify the runtime environment that a specific attribute for a specific resource has changed. `is` and `should` are the original and the new value of the attribute. Either can be `nil`. * Plain messages * `debug(message)` @@ -399,9 +399,9 @@ Allowing multiple providers does not come for free. The previous implementation The Resource API will not implement support for multiple providers at this time. -Should support for multiple providers be desirable for a given type, the two options are: +Should support for multiple providers be desirable for a given type, the two options are: -1. Use the older, more complex API. +1. Use the older, more complex API. 2. Implement multiple similar types using the Resource API, and select the platform-appropriate type in Puppet code. For example: ```puppet From ace536801590f5348d3bf008ef570b8d5071054f Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Fri, 23 Feb 2018 11:06:47 +0000 Subject: [PATCH 58/62] Remove obsolete examples See `pdk new provider` and https://github.com/puppetlabs/pdk-templates/tree/master/object_templates provider*.erb for live examples. --- language/resource-api/apt_key.rb | 377 ------------------------ language/resource-api/apt_key_get.rb | 120 -------- language/resource-api/apt_key_set.rb | 256 ---------------- language/resource-api/examples.rb | 29 -- language/resource-api/package.pp | 31 -- language/resource-api/simple_apt.rb | 234 --------------- language/resource-api/simpleresource.rb | 51 ---- language/resource-api/tests.rb | 41 --- 8 files changed, 1139 deletions(-) delete mode 100644 language/resource-api/apt_key.rb delete mode 100755 language/resource-api/apt_key_get.rb delete mode 100644 language/resource-api/apt_key_set.rb delete mode 100644 language/resource-api/examples.rb delete mode 100644 language/resource-api/package.pp delete mode 100644 language/resource-api/simple_apt.rb delete mode 100644 language/resource-api/simpleresource.rb delete mode 100644 language/resource-api/tests.rb diff --git a/language/resource-api/apt_key.rb b/language/resource-api/apt_key.rb deleted file mode 100644 index f67060a..0000000 --- a/language/resource-api/apt_key.rb +++ /dev/null @@ -1,377 +0,0 @@ - -# This is a experimental hardcoded implementation of what will be come the Resource API's runtime -# environment. This code is used as test-bed to see that the proposal is technically feasible. - -require 'puppet/pops/patterns' -require 'puppet/pops/utils' - -DEFINITION = { - name: 'apt_key', - docs: <<-EOS, - This type provides Puppet with the capabilities to manage GPG keys needed - by apt to perform package validation. Apt has it's own GPG keyring that can - be manipulated through the `apt-key` command. - - apt_key { '6F6B15509CF8E59E6E469F327F438280EF8D349F': - source => 'http://apt.puppetlabs.com/pubkey.gpg' - } - - **Autorequires**: - If Puppet is given the location of a key file which looks like an absolute - path this type will autorequire that file. - EOS - attributes: { - ensure: { - type: 'Enum[present, absent]', - docs: 'Whether this apt key should be present or absent on the target system.' - }, - id: { - type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]', - docs: 'The ID of the key you want to manage.', - namevar: true, - }, - content: { - type: 'Optional[String]', - docs: 'The content of, or string representing, a GPG key.', - }, - source: { - type: 'Variant[Stdlib::Absolutepath, Pattern[/\A(https?|ftp):\/\//]]', - docs: 'Location of a GPG key file, /path/to/file, ftp://, http:// or https://', - }, - server: { - type: 'Pattern[/\A((hkp|http|https):\/\/)?([a-z\d])([a-z\d-]{0,61}\.)+[a-z\d]+(:\d{2,5})?$/]', - docs: 'The key server to fetch the key from based on the ID. It can either be a domain name or url.', - default: 'keyserver.ubuntu.com' - }, - options: { - type: 'Optional[String]', - docs: 'Additional options to pass to apt-key\'s --keyserver-options.', - }, - fingerprint: { - type: 'Pattern[/[a-f]{40}/]', - docs: 'The 40-digit hexadecimal fingerprint of the specified GPG key.', - read_only: true, - }, - long: { - type: 'Pattern[/[a-f]{16}/]', - docs: 'The 16-digit hexadecimal id of the specified GPG key.', - read_only: true, - }, - short: { - type: 'Pattern[/[a-f]{8}/]', - docs: 'The 8-digit hexadecimal id of the specified GPG key.', - read_only: true, - }, - expired: { - type: 'Boolean', - docs: 'Indicates if the key has expired.', - read_only: true, - }, - expiry: { - # TODO: should be DateTime - type: 'String', - docs: 'The date the key will expire, or nil if it has no expiry date, in ISO format.', - read_only: true, - }, - size: { - type: 'Integer', - docs: 'The key size, usually a multiple of 1024.', - read_only: true, - }, - type: { - type: 'String', - docs: 'The key type, one of: rsa, dsa, ecc, ecdsa.', - read_only: true, - }, - created: { - type: 'String', - docs: 'Date the key was created, in ISO format.', - read_only: true, - }, - }, - autorequires: { - file: '$source', # will evaluate to the value of the `source` attribute - package: 'apt', - }, -} - -module Puppet::SimpleResource - class TypeShim - attr_reader :values - - def initialize(title, resource_hash) - # internalize and protect - needs to go deeper - @values = resource_hash.dup - # "name" is a privileged key - @values[:name] = title - @values.freeze - end - - def to_resource - ResourceShim.new(@values) - end - - def name - values[:name] - end - end - - class ResourceShim - attr_reader :values - - def initialize(resource_hash) - @values = resource_hash.dup.freeze # whatevs - end - - def title - values[:name] - end - - def prune_parameters(*args) - puts "not pruning #{args.inspect}" if args.length > 0 - self - end - - def to_manifest - [ - "apt_key { #{values[:name].inspect}: ", - ] + values.keys.select { |k| k != :name }.collect { |k| " #{k} => #{values[k].inspect}," } + ['}'] - end - end -end - -Puppet::Type.newtype(DEFINITION[:name].to_sym) do - @doc = DEFINITION[:docs] - - has_namevar = false - - DEFINITION[:attributes].each do |name, options| - puts "#{name}: #{options.inspect}" - - # TODO: using newparam everywhere would suppress change reporting - # that would allow more fine-grained reporting through logger, - # but require more invest in hooking up the infrastructure to emulate existing data - param_or_property = if options[:read_only] || options[:namevar] - :newparam - else - :newproperty - end - send(param_or_property, name.to_sym) do - unless options[:type] - fail("#{DEFINITION[:name]}.#{name} has no type") - end - - if options[:docs] - desc "#{options[:docs]} (a #{options[:type]}" - else - warn("#{DEFINITION[:name]}.#{name} has no docs") - end - - if options[:namevar] - puts 'setting namevar' - isnamevar - has_namevar = true - end - - # read-only values do not need type checking - if not options[:read_only] - # TODO: this should use Pops infrastructure to avoid hardcoding stuff, and enhance type fidelity - # validate do |v| - # type = Puppet::Pops::Types::TypeParser.singleton.parse(options[:type]).normalize - # if type.instance?(v) - # return true - # else - # inferred_type = Puppet::Pops::Types::TypeCalculator.infer_set(value) - # error_msg = Puppet::Pops::Types::TypeMismatchDescriber.new.describe_mismatch("#{DEFINITION[:name]}.#{name}", type, inferred_type) - # raise Puppet::ResourceError, error_msg - # end - # end - - case options[:type] - when 'String' - # require any string value - newvalue // do - end - when 'Boolean' - ['true', 'false', :true, :false, true, false].each do |v| - newvalue v do - end - end - - munge do |v| - case v - when 'true', :true - true - when 'false', :false - false - else - v - end - end - when 'Integer' - newvalue /^\d+$/ do - end - munge do |v| - Puppet::Pops::Utils.to_n(v) - end - when 'Float', 'Numeric' - newvalue Puppet::Pops::Patterns::NUMERIC do - end - munge do |v| - Puppet::Pops::Utils.to_n(v) - end - when 'Enum[present, absent]' - newvalue :absent do - end - newvalue :present do - end - when 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]' - # the namevar needs to be a Parameter, which only has newvalue*s* - newvalues(/\A(0x)?[0-9a-fA-F]{8}\Z/, /\A(0x)?[0-9a-fA-F]{16}\Z/, /\A(0x)?[0-9a-fA-F]{40}\Z/) - when 'Optional[String]' - newvalue :undef do - end - newvalue // do - end - when 'Variant[Stdlib::Absolutepath, Pattern[/\A(https?|ftp):\/\//]]' - # TODO: this is wrong, but matches original implementation - [/^\//, /\A(https?|ftp):\/\//].each do |v| - newvalue v do - end - end - when /^(Enum|Optional|Variant)/ - fail("#{$1} is not currently supported") - end - end - end - end - - unless has_namevar - fail("#{DEFINITION[:name]} has no namevar") - end - - def self.fake_system_state - @fake_system_state ||= { - 'BBCB188AD7B3228BCF05BD554C0BE21B5FF054BD' => { - ensure: :present, - fingerprint: 'BBCB188AD7B3228BCF05BD554C0BE21B5FF054BD', - long: '4C0BE21B5FF054BD', - short: '5FF054BD', - size: 2048, - type: :rsa, - created: '2013-06-07 23:55:31 +0100', - expiry: nil, - expired: false, - }, - 'B71ACDE6B52658D12C3106F44AB781597254279C' => { - ensure: :present, - fingerprint: 'B71ACDE6B52658D12C3106F44AB781597254279C', - long: '4AB781597254279C', - short: '7254279C', - size: 1024, - type: :dsa, - created: '2007-03-08 20:17:10 +0000', - expiry: nil, - expired: false - }, - '9534C9C4130B4DC9927992BF4F30B6B4C07CB649' => { - ensure: :present, - fingerprint: '9534C9C4130B4DC9927992BF4F30B6B4C07CB649', - long: '4F30B6B4C07CB649', - short: 'C07CB649', - size: 4096, - type: :rsa, - created: '2014-11-21 21:01:13 +0000', - expiry: '2022-11-19 21:01:13 +0000', - expired: false - }, - '126C0D24BD8A2942CC7DF8AC7638D0442B90D010' => { - ensure: :present, - fingerprint: '126C0D24BD8A2942CC7DF8AC7638D0442B90D010', - long: '7638D0442B90D010', - short: '2B90D010', - size: 4096, - type: :rsa, - created: '2014-11-21 21:13:37 +0000', - expiry: '2022-11-19 21:13:37 +0000', - expired: false - }, - 'ED6D65271AACF0FF15D123036FB2A1C265FFB764' => { - ensure: :present, - fingerprint: 'ED6D65271AACF0FF15D123036FB2A1C265FFB764', - long: '6FB2A1C265FFB764', - short: '65FFB764', - size: 4096, - type: :rsa, - created: '2010-07-10 01:13:52 +0100', - expiry: '2017-01-05 00:06:37 +0000', - expired: true - }, - } - end - - def self.get - puts 'get' - fake_system_state - end - - def self.set(current_state, target_state, noop = false) - puts "enforcing change from #{current_state} to #{target_state} (noop=#{noop})" - target_state.each do |title, resource| - # additional validation for this resource goes here - - # set default value - resource[:ensure] ||= :present - - current = current_state[title] - if current && resource[:ensure].to_s == 'absent' - # delete the resource - puts "deleting #{title}" - fake_system_state.delete_if { |k, _| k==title } - elsif current && resource[:ensure].to_s == 'present' - # update the resource - puts "updating #{title}" - resource = current.merge(resource) - fake_system_state[title] = resource.dup - elsif !current && resource[:ensure].to_s == 'present' - # create the resource - puts "creating #{title}" - fake_system_state[title] = resource.dup - end - # TODO: update Type's notion of reality to ensure correct puppet resource output with all available attributes - end - end - - def self.instances - puts 'instances' - # klass = Puppet::Type.type(:api) - get.collect do |title, resource_hash| - Puppet::SimpleResource::TypeShim.new(title, resource_hash) - end - end - - def retrieve - puts 'retrieve' - result = Puppet::Resource.new(self.class, title) - current_state = self.class.get[title] - - if current_state - current_state.each do |k, v| - result[k]=v - end - else - result[:ensure] = :absent - end - - @rapi_current_state = current_state - result - end - - def flush - puts 'flush' - # binding.pry - target_state = Hash[@parameters.collect { |k, v| [k, v.value] }] - self.class.set({title => @rapi_current_state}, {title => target_state}, false) - end - -end diff --git a/language/resource-api/apt_key_get.rb b/language/resource-api/apt_key_get.rb deleted file mode 100755 index 2613a0e..0000000 --- a/language/resource-api/apt_key_get.rb +++ /dev/null @@ -1,120 +0,0 @@ -#!/usr/bin/ruby - -require 'json' - - -def key_line_to_hash(pub_line, fpr_line) - pub_split = pub_line.split(':') - fpr_split = fpr_line.split(':') - - # set key type based on types defined in /usr/share/doc/gnupg/DETAILS.gz - key_type = case pub_split[3] - when '1' - :rsa - when '17' - :dsa - when '18' - :ecc - when '19' - :ecdsa - else - :unrecognized - end - - fingerprint = fpr_split.last - expiry = pub_split[6].empty? ? nil : Time.at(pub_split[6].to_i) - - { - name: fingerprint, - ensure: 'present', - fingerprint: fingerprint, - long: fingerprint[-16..-1], # last 16 characters of fingerprint - short: fingerprint[-8..-1], # last 8 characters of fingerprint - size: pub_split[2], - type: key_type, - created: Time.at(pub_split[5].to_i), - expiry: expiry, - expired: !!(expiry && Time.now >= expiry), - } -end - -key_output = <: -sub:-:2048:1:4AB781597254279C:1370645731::::::e:::::: -fpr:::::::::B71ACDE6B52658D12C3106F44AB781597254279C: -pub:-:1024:17:A040830F7FAC5991:1173385030:::-:::scESC::::::: -fpr:::::::::4CCA1EAF950CEE4AB83976DCA040830F7FAC5991: -uid:-::::1175811711::0F5F08408BC3D293942A5E5A2D1AE1BD277FF5DB::Google, Inc. Linux Package Signing Key : -sub:-:2048:16:4F30B6B4C07CB649:1173385035::::::e:::::: -fpr:::::::::9534C9C4130B4DC9927992BF4F30B6B4C07CB649: -pub:-:4096:1:7638D0442B90D010:1416603673:1668891673::-:::scSC::::::: -rvk:::1::::::309911BEA966D0613053045711B4E5FF15B0FD82:80: -rvk:::1::::::FBFABDB541B5DC955BD9BA6EDB16CF5BB12525C4:80: -rvk:::1::::::80E976F14A508A48E9CA3FE9BC372252CA1CF964:80: -fpr:::::::::126C0D24BD8A2942CC7DF8AC7638D0442B90D010: -uid:-::::1416603673::15C761B84F0C9C293316B30F007E34BE74546B48::Debian Archive Automatic Signing Key (8/jessie) : -pub:-:4096:1:9D6D8F6BC857C906:1416604417:1668892417::-:::scSC::::::: -rvk:::1::::::FBFABDB541B5DC955BD9BA6EDB16CF5BB12525C4:80: -rvk:::1::::::309911BEA966D0613053045711B4E5FF15B0FD82:80: -rvk:::1::::::80E976F14A508A48E9CA3FE9BC372252CA1CF964:80: -fpr:::::::::D21169141CECD440F2EB8DDA9D6D8F6BC857C906: -uid:-::::1416604417::088FA6B00E33BCC6F6EB4DFEFAC591F9940E06F0::Debian Security Archive Automatic Signing Key (8/jessie) : -pub:-:4096:1:CBF8D6FD518E17E1:1376739416:1629027416::-:::scSC::::::: -fpr:::::::::75DDC3C4A499F1A18CB5F3C8CBF8D6FD518E17E1: -uid:-::::1376739416::2D9AEBB80FC7D1724686A20DC5712C7D0DC07AF6::Jessie Stable Release Key : -pub:-:4096:1:AED4B06F473041FA:1282940623:1520281423::-:::scSC::::::: -fpr:::::::::9FED2BCBDCD29CDF762678CBAED4B06F473041FA: -uid:-::::1282940896::CED55047A1889F383B10CE9D04346A5CA12E2445::Debian Archive Automatic Signing Key (6.0/squeeze) : -pub:-:4096:1:64481591B98321F9:1281140461:1501892461::-:::scSC::::::: -fpr:::::::::0E4EDE2C7F3E1FC0D033800E64481591B98321F9: -uid:-::::1281140461::BB638CC58BB7B36929C2C6DEBE580CC46FC94B36::Squeeze Stable Release Key : -pub:-:4096:1:8B48AD6246925553:1335553717:1587841717::-:::scSC::::::: -fpr:::::::::A1BD8E9D78F7FE5C3E65D8AF8B48AD6246925553: -uid:-::::1335553717::BCBD552DFB543AADFE3812AF631B17F5EDEF820E::Debian Archive Automatic Signing Key (7.0/wheezy) : -pub:-:4096:1:6FB2A1C265FFB764:1336489909:1557241909::-:::scSC::::::: -fpr:::::::::ED6D65271AACF0FF15D123036FB2A1C265FFB764: -uid:-::::1336489909::0BB8E4C85595D59CE65881DDD593ECBAE583607B::Wheezy Stable Release Key : -pub:e:4096:1:1054B7A24BD6EC30:1278720832:1483574797::-:::sc::::::: -fpr:::::::::47B320EB4C7C375AA9DAE1A01054B7A24BD6EC30: -uid:e::::1460074501::BA4BCA138CEBDF8444241CE928DEE1AD79612E6C::Puppet Labs Release Key (Puppet Labs Release Key) : -pub:-:4096:1:B8F999C007BB6C57:1360109177:1549910347::-:::scESC::::::: -fpr:::::::::8735F5AF62A99A628EC13377B8F999C007BB6C57: -uid:-::::1455302347::A8FC88656336852AD4301DF059CEE6134FD37C21::Puppet Labs Nightly Build Key (Puppet Labs Nightly Build Key) : -uid:-::::1455302347::4EF2A82F1FF355343885012A832C628E1A4F73A8::Puppet Labs Nightly Build Key (Puppet Labs Nightly Build Key) : -sub:-:4096:1:AE8282E5A5FC3E74:1360109177:1549910293:::::e:::::: -fpr:::::::::F838D657CCAF0E4A6375B0E9AE8282E5A5FC3E74: -pub:-:4096:1:7F438280EF8D349F:1471554366:1629234366::-:::scESC::::::: -fpr:::::::::6F6B15509CF8E59E6E469F327F438280EF8D349F: -uid:-::::1471554366::B648B946D1E13EEA5F4081D8FE5CF4D001200BC7::Puppet, Inc. Release Key (Puppet, Inc. Release Key) : -sub:-:4096:1:A2D80E04656674AE:1471554366:1629234366:::::e:::::: -fpr:::::::::07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE: -EOM - - - -pub_line = nil -fpr_line = nil - -instances = key_output.split("\n").collect do |line| - if line.start_with?('pub') - pub_line = line - elsif line.start_with?('fpr') - fpr_line = line - end - - next unless (pub_line and fpr_line) - - result = key_line_to_hash(pub_line, fpr_line) - - # reset everything - pub_line = nil - fpr_line = nil - - result -end.compact! - -puts JSON.generate(instances) diff --git a/language/resource-api/apt_key_set.rb b/language/resource-api/apt_key_set.rb deleted file mode 100644 index 6f1f760..0000000 --- a/language/resource-api/apt_key_set.rb +++ /dev/null @@ -1,256 +0,0 @@ -#!/usr/bin/ruby - -require 'json' - -current_state_json = < false) - end while r.exitstatus == 0 - end - elsif current && key[:ensure].to_s == 'present' - # No updating implemented - # update(key, noop: noop) - elsif !current && key[:ensure].to_s == 'present' - create(key, noop: noop) - end - end -end - -def set(current_state, target_state, noop = false) - existing_keys = Hash[current_state.collect { |k| [k[:name], k] }] - target_state.each do |resource| - # additional validation for this resource goes here - - current = existing_keys[resource[:name]] - if current && resource[:ensure].to_s == 'absent' - # delete the resource - elsif current && resource[:ensure].to_s == 'present' - # update the resource - elsif !current && resource[:ensure].to_s == 'present' - # create the resource - end - end -end - - -def create(key, noop = false) - logger.creating(key[:name]) do |logger| - if key[:source].nil? and key[:content].nil? - # Breaking up the command like this is needed because it blows up - # if --recv-keys isn't the last argument. - args = ['adv', '--keyserver', key[:server]] - if key[:options] - args.push('--keyserver-options', key[:options]) - end - args.push('--recv-keys', key[:id]) - apt_key(*args, noop: noop) - elsif key[:content] - temp_key_file(key[:content], logger) do |key_file| - apt_key('add', key_file, noop: noop) - end - elsif key[:source] - key_file = source_to_file(key[:source]) - apt_key('add', key_file.path, noop: noop) - # In case we really screwed up, better safe than sorry. - else - logger.fail("an unexpected condition occurred while trying to add the key: #{key[:id]} (content: #{key[:content].inspect}, source: #{key[:source].inspect})") - end - end -end - -# This method writes out the specified contents to a temporary file and -# confirms that the fingerprint from the file, matches the long key that is in the manifest -def temp_key_file(key, logger) - file = Tempfile.new('apt_key') - begin - file.write key[:content] - file.close - if name.size == 40 - if File.executable? command(:gpg) - extracted_key = execute(["#{command(:gpg)} --with-fingerprint --with-colons #{file.path} | awk -F: '/^fpr:/ { print $10 }'"], :failonfail => false) - extracted_key = extracted_key.chomp - - unless extracted_key.match(/^#{name}$/) - logger.fail("The id in your manifest #{key[:name]} and the fingerprint from content/source do not match. Please check there is not an error in the id or check the content/source is legitimate.") - end - else - logger.warning('/usr/bin/gpg cannot be found for verification of the id.') - end - end - yield file.path - ensure - file.close - file.unlink - end -end - - -set(current_state, target_state) diff --git a/language/resource-api/examples.rb b/language/resource-api/examples.rb deleted file mode 100644 index 8767a2d..0000000 --- a/language/resource-api/examples.rb +++ /dev/null @@ -1,29 +0,0 @@ -SHARED_PACKAGE_ATTRIBUTES = { - name: { type: 'String' }, - ENSURE_ATTRIBUTE, - reinstall_on_refresh: { type: 'Boolean'}, -}.freeze - -LOCAL_PACKAGE_ATTRIBUTES = { - source: { type: 'String' }, -}.freeze - -VERSIONABLE_PACKAGE_ATTRIBUTES = { - version: { type: 'String' }, -}.freeze - -APT_PACKAGE_ATTRIBUTES = { - install_options: { type: 'String' }, - responsefile: { type: 'String' }, -}.freeze - -Puppet::SimpleResource.define( - name: 'package_rpm', - attributes: {}.merge(SHARED_PACKAGE_ATTRIBUTES).merge(LOCAL_PACKAGE_ATTRIBUTES), -) - -Puppet::SimpleResource.define( - name: 'package_apt', - attributes: {}.merge(SHARED_PACKAGE_ATTRIBUTES).merge(LOCAL_PACKAGE_ATTRIBUTES).merge(VERSIONABLE_PACKAGE_ATTRIBUTES).merge(APT_PACKAGE_ATTRIBUTES), -) - diff --git a/language/resource-api/package.pp b/language/resource-api/package.pp deleted file mode 100644 index d91e7f4..0000000 --- a/language/resource-api/package.pp +++ /dev/null @@ -1,31 +0,0 @@ -define package ( - Ensure $ensure, - Enum[apt, rpm] $provider, - Optional[String] $source = undef, - Optional[String] $version = undef, - Optional[String] $install_options = undef, - Optional[String] $responsefile = undef, - Optional[Hash] $options = { }, -) { - case $provider { - apt: { - package_apt { $title: - ensure => $ensure, - source => $source, - version => $version, - install_options => $install_options, - responsefile => $responsefile, - * => $options, - } - } - rpm: { - package_rpm { $title: - ensure => $ensure, - source => $source, - * => $options, - } - if defined($version) { fail("RPM doesn't support \$version") } - # ... - } - } -} diff --git a/language/resource-api/simple_apt.rb b/language/resource-api/simple_apt.rb deleted file mode 100644 index 1346444..0000000 --- a/language/resource-api/simple_apt.rb +++ /dev/null @@ -1,234 +0,0 @@ -Puppet::SimpleResource.define( - name: 'apt_key', - docs: <<-EOS, - This type provides Puppet with the capabilities to manage GPG keys needed - by apt to perform package validation. Apt has it's own GPG keyring that can - be manipulated through the `apt-key` command. - - apt_key { '6F6B15509CF8E59E6E469F327F438280EF8D349F': - source => 'http://apt.puppetlabs.com/pubkey.gpg' - } - - **Autorequires**: - If Puppet is given the location of a key file which looks like an absolute - path this type will autorequire that file. - EOS - attributes: { - ensure: { - type: 'Enum[present, absent]', - docs: 'Whether this apt key should be present or absent on the target system.' - }, - id: { - type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]', - docs: 'The ID of the key you want to manage.', - namevar: true, - }, - content: { - type: 'Optional[String]', - docs: 'The content of, or string representing, a GPG key.', - }, - source: { - type: 'Variant[Stdlib::Absolutepath, Pattern[/\A(https?|ftp):\/\//]]', - docs: 'Location of a GPG key file, /path/to/file, ftp://, http:// or https://', - }, - server: { - type: 'Pattern[/\A((hkp|http|https):\/\/)?([a-z\d])([a-z\d-]{0,61}\.)+[a-z\d]+(:\d{2,5})?$/]', - docs: 'The key server to fetch the key from based on the ID. It can either be a domain name or url.', - default: 'keyserver.ubuntu.com' - }, - options: { - type: 'Optional[String]', - docs: 'Additional options to pass to apt-key\'s --keyserver-options.', - }, - fingerprint: { - type: 'String[40, 40]', - docs: 'The 40-digit hexadecimal fingerprint of the specified GPG key.', - read_only: true, - }, - long: { - type: 'String[16, 16]', - docs: 'The 16-digit hexadecimal id of the specified GPG key.', - read_only: true, - }, - short: { - type: 'String[8, 8]', - docs: 'The 8-digit hexadecimal id of the specified GPG key.', - read_only: true, - }, - expired: { - type: 'Boolean', - docs: 'Indicates if the key has expired.', - read_only: true, - }, - expiry: { - type: 'String', - docs: 'The date the key will expire, or nil if it has no expiry date, in ISO format.', - read_only: true, - }, - size: { - type: 'Integer', - docs: 'The key size, usually a multiple of 1024.', - read_only: true, - }, - type: { - type: 'String', - docs: 'The key type, one of: rsa, dsa, ecc, ecdsa.', - read_only: true, - }, - created: { - type: 'String', - docs: 'Date the key was created, in ISO format.', - read_only: true, - }, - }, - autorequires: { - file: '$source', # will evaluate to the value of the `source` attribute - package: 'apt', - }, -) - -Puppet::SimpleResource.implement('apt_key') do - commands apt_key: 'apt-key' - commands gpg: '/usr/bin/gpg' - - def get - cli_args = %w(adv --list-keys --with-colons --fingerprint --fixed-list-mode) - key_output = apt_key(cli_args).encode('UTF-8', 'binary', :invalid => :replace, :undef => :replace, :replace => '') - pub_line = nil - fpr_line = nil - - kv_pairs = key_output.split("\n").collect do |line| - if line.start_with?('pub') - pub_line = line - elsif line.start_with?('fpr') - fpr_line = line - end - - next unless (pub_line and fpr_line) - - result = key_line_to_kv_pair(pub_line, fpr_line) - - # reset everything - pub_line = nil - fpr_line = nil - - result - end.compact! - - Hash[kv_pairs] - end - - def self.key_line_to_kv_pair(pub_line, fpr_line) - pub_split = pub_line.split(':') - fpr_split = fpr_line.split(':') - - # set key type based on types defined in /usr/share/doc/gnupg/DETAILS.gz - key_type = case pub_split[3] - when '1' - :rsa - when '17' - :dsa - when '18' - :ecc - when '19' - :ecdsa - else - :unrecognized - end - - fingerprint = fpr_split.last - expiry = pub_split[6].empty? ? nil : Time.at(pub_split[6].to_i) - - [ - fingerprint, - { - ensure: 'present', - id: fingerprint, - fingerprint: fingerprint, - long: fingerprint[-16..-1], # last 16 characters of fingerprint - short: fingerprint[-8..-1], # last 8 characters of fingerprint - size: pub_split[2].to_i, - type: key_type, - created: Time.at(pub_split[5].to_i), - expiry: expiry, - expired: !!(expiry && Time.now >= expiry), - } - ] - end - - def set(current_state, target_state, noop = false) - target_state.each do |title, resource| - logger.warning(title, 'The id should be a full fingerprint (40 characters) to avoid collision attacks, see the README for details.') if title.length < 40 - if resource[:source] and resource[:content] - logger.fail(title, 'The properties content and source are mutually exclusive') - next - end - - current = current_state[title] - if current && resource[:ensure].to_s == 'absent' - logger.deleting(title) do - begin - apt_key('del', resource[:short], noop: noop) - r = execute(["#{command(:apt_key)} list | grep '/#{resource[:short]}\s'"], :failonfail => false) - end while r.exitstatus == 0 - end - elsif current && resource[:ensure].to_s == 'present' - # No updating implemented - # update(key, noop: noop) - elsif !current && resource[:ensure].to_s == 'present' - create(title, resource, noop: noop) - end - end - end - - def create(title, resource, noop = false) - logger.creating(title) do |logger| - if resource[:source].nil? and resource[:content].nil? - # Breaking up the command like this is needed because it blows up - # if --recv-keys isn't the last argument. - args = ['adv', '--keyserver', resource[:server]] - if resource[:options] - args.push('--keyserver-options', resource[:options]) - end - args.push('--recv-keys', resource[:id]) - apt_key(*args, noop: noop) - elsif resource[:content] - temp_key_file(resource[:content], logger) do |key_file| - apt_key('add', key_file, noop: noop) - end - elsif resource[:source] - key_file = source_to_file(resource[:source]) - apt_key('add', key_file.path, noop: noop) - # In case we really screwed up, better safe than sorry. - else - logger.fail("an unexpected condition occurred while trying to add the key: #{title} (content: #{resource[:content].inspect}, source: #{resource[:source].inspect})") - end - end - end - - # This method writes out the specified contents to a temporary file and - # confirms that the fingerprint from the file, matches the long key that is in the manifest - def temp_key_file(resource, logger) - file = Tempfile.new('apt_key') - begin - file.write resource[:content] - file.close - if name.size == 40 - if File.executable? command(:gpg) - extracted_key = execute(["#{command(:gpg)} --with-fingerprint --with-colons #{file.path} | awk -F: '/^fpr:/ { print $10 }'"], :failonfail => false) - extracted_key = extracted_key.chomp - - unless extracted_key.match(/^#{name}$/) - logger.fail("The id in your manifest #{resource[:id]} and the fingerprint from content/source do not match. Please check there is not an error in the id or check the content/source is legitimate.") - end - else - logger.warning('/usr/bin/gpg cannot be found for verification of the id.') - end - end - yield file.path - ensure - file.close - file.unlink - end - end -end diff --git a/language/resource-api/simpleresource.rb b/language/resource-api/simpleresource.rb deleted file mode 100644 index 7ded072..0000000 --- a/language/resource-api/simpleresource.rb +++ /dev/null @@ -1,51 +0,0 @@ -Puppet::SimpleResource.define( - name: 'iis_application_pool', - docs: 'Manage an IIS application pool through a powershell proxy.', - attributes: { - ensure: { - type: 'Enum[present, absent]', - docs: 'Whether this ApplicationPool should be present or absent on the target system.' - }, - name: { - type: 'String', - docs: 'The name of the ApplicationPool.', - namevar: true, - }, - state: { - type: 'Enum[running, stopped]', - docs: 'The state of the ApplicationPool.', - default: 'running', - }, - managedpipelinemode: { - type: 'String', - docs: 'The managedPipelineMode of the ApplicationPool.', - }, - managedruntimeversion: { - type: 'String', - docs: 'The managedRuntimeVersion of the ApplicationPool.', - }, - } -) - -Puppet::SimpleResource.implement('iis_application_pool') do - # hiding all the nasty bits - require 'puppet/provider/iis_powershell' - include Puppet::Provider::IIS_PowerShell - - def get - # call out to PowerShell to talk to the API - result = run('fetch_application_pools.ps1', logger) - - # returns an array of hashes with data according to the schema above - JSON.parse(result) - end - - def set(current_state, target_state, noop = false) - # call out to PowerShell to talk to the API - result = run('enforce_application_pools.ps1', JSON.generate(current_state), JSON.generate(target_state), logger, noop) - - # returns an array of hashes with status data from the changes - JSON.parse(result) - end - -end diff --git a/language/resource-api/tests.rb b/language/resource-api/tests.rb deleted file mode 100644 index 60a9420..0000000 --- a/language/resource-api/tests.rb +++ /dev/null @@ -1,41 +0,0 @@ -import 'facts_db.pp' - -example 'mongodb::db' { - example 'default' { - $facts_db.each { |$loop_facts| - given 'default' ( - $facts = $loop_facts, - $modules = $default_modules, - ) { - mongodb::db { 'testdb' : - user => 'testuser', - password => 'testpass', - } - } - assert 'it contains mongodb_database with mongodb::server requirement' { - mongodb_database { 'testdb' : } - } - assert 'it contains mongodb_user with mongodb_database requirement' { - mongodb_user { 'User testuser on db testdb' : - username => 'testuser', - database => 'testdb', - require => Mongodb_database['testdb'], - } - } - } - } - example 'old modules' { - given 'default' ( - $facts = {} - $modules = {'puppetlabs-stdlib' => '4.4.0', }.merge($default_modules), - ) { - mongodb::db { 'testdb' : - user => 'testuser', - password => 'testpass', - } - } - assert { - # it compiles, at least - } - } -} From b56ed4caa4490b65f69ad319d72ad9adebe5eda3 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Fri, 23 Feb 2018 14:25:54 +0000 Subject: [PATCH 59/62] Fix a typo: autorequires -> autorequire --- language/resource-api/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index f3ca5d5..58fafe3 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -41,7 +41,7 @@ Puppet::ResourceApi.register_type( desc: 'Date the key was created, in ISO format.', }, }, - autorequires: { + autorequire: { file: '$source', # will evaluate to the value of the `source` attribute package: 'apt', }, @@ -61,7 +61,7 @@ The `Puppet::ResourceApi.register_type(options)` function takes the following ke * `init_only`: this attribute can only be set during the creation of the resource. Its value will be reported going forward, but trying to change it later will lead to an error. For example, the base image for a VM or the UID of a user. * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example, the checksum of a file or the MAC address of a network interface. * `parameter`: these attributes influence how the provider behaves and cannot be read from the target system. For example, the target file on inifile or credentials to access an API. -* `autorequires`, `autobefore`, `autosubscribe`, and `autonotify`: a hash mapping resource types to titles. The titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, Puppet will create the relationsships requested here. +* `autorequire`, `autobefore`, `autosubscribe`, and `autonotify`: a hash mapping resource types to titles. The titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, Puppet will create the relationsships requested here. * `features`: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently defined features: `canonicalize`, `simple_get_filter`, and `noop_handler`. See below for details. For autoloading work, this code needs to go into `lib/puppet/type/.rb` in your module. From e72cc6170a18d46e3a93f0e5d5d10599de7fe6e2 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 28 Feb 2018 13:52:20 +0000 Subject: [PATCH 60/62] (PDK-513) updated noop_handler to supports_noop to match Task's language --- language/resource-api/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 58fafe3..9128f4c 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -62,7 +62,7 @@ The `Puppet::ResourceApi.register_type(options)` function takes the following ke * `read_only`: values for this attribute will be returned by `get()`, but `set()` is not able to change them. Values for this should never be specified in a manifest. For example, the checksum of a file or the MAC address of a network interface. * `parameter`: these attributes influence how the provider behaves and cannot be read from the target system. For example, the target file on inifile or credentials to access an API. * `autorequire`, `autobefore`, `autosubscribe`, and `autonotify`: a hash mapping resource types to titles. The titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, Puppet will create the relationsships requested here. -* `features`: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently defined features: `canonicalize`, `simple_get_filter`, and `noop_handler`. See below for details. +* `features`: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently defined features: `canonicalize`, `simple_get_filter`, and `supports_noop`. See below for details. For autoloading work, this code needs to go into `lib/puppet/type/.rb` in your module. @@ -104,7 +104,7 @@ The `set` method updates resources to a new state. The `changes` parameter gets A missing `:should` entry indicates that a resource should be removed from the system. Even a type implementing the `ensure => [present, absent]` attribute pattern still has to react correctly on a missing `:should` entry. `:is` may contain the last available system state from a prior `get` call. If the `:is` value is `nil`, the resources were not found by `get`. If there is no `:is` key, the runtime did not have a cached state available. -The `set` method should always return `nil`. Any progress signaling should be done through the logging utilities described below. If the `set` method throws an exception, all resources that should change in this call and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the `set` method if there are changes to be made, especially in the case of resources marked with `noop => true` (either locally or through a global flag). The runtime will not pass them to `set`. See `noop_handler` below for changing this behaviour if required. +The `set` method should always return `nil`. Any progress signaling should be done through the logging utilities described below. If the `set` method throws an exception, all resources that should change in this call and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the `set` method if there are changes to be made, especially in the case of resources marked with `noop => true` (either locally or through a global flag). The runtime will not pass them to `set`. See `supports_noop` below for changing this behaviour if required. Both methods take a `context` parameter which provides utilties from the runtime environment, and is decribed in more detail there. @@ -169,12 +169,12 @@ Some resources are very expensive to enumerate. The provider can implement `simp The runtime environment calls `get` with a minimal set of names, and keeps track of additional instances returned to avoid double querying. To gain the most benefits from batching implementations, the runtime minimizes the number of calls into `get`. -### Provider feature: `noop_handler` +### Provider feature: `supports_noop` ```ruby Puppet::ResourceApi.register_type( name: 'apt_key', - features: [ 'noop_handler' ], + features: [ 'supports_noop' ], ) class Puppet::Provider::AptKey::AptKey @@ -189,7 +189,7 @@ class Puppet::Provider::AptKey::AptKey end ``` -When a resource is marked with `noop => true`, either locally or through a global flag, the standard runtime will produce the default change report with a `noop` flag set. In some cases, an implementation provides additional information, for example commands that would get executed, or requires additional evaluation before determining the effective changes, for example the `exec`'s `onlyif` attribute. The resource type specifies the `noop_handler` feature to have `set` called for all resources, even those flagged with `noop`. When the `noop` parameter is set to true, the provider must not change the system state, but only report what it would change. The `noop` parameter should default to `false` to allow simple runtimes to ignore this feature. +When a resource is marked with `noop => true`, either locally or through a global flag, the standard runtime will produce the default change report with a `noop` flag set. In some cases, an implementation provides additional information, for example commands that would get executed, or requires additional evaluation before determining the effective changes, for example the `exec`'s `onlyif` attribute. The resource type specifies the `supports_noop` feature to have `set` called for all resources, even those flagged with `noop`. When the `noop` parameter is set to true, the provider must not change the system state, but only report what it would change. The `noop` parameter should default to `false` to allow simple runtimes to ignore this feature. ### Provider feature: `remote_resource` From 086bd8e210eaca9400a2916367191fc8dac0cca3 Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 7 Mar 2018 14:32:05 +0000 Subject: [PATCH 61/62] Improve the type example to show all parts for the autorequire --- language/resource-api/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 9128f4c..5999faa 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -34,6 +34,10 @@ Puppet::ResourceApi.register_type( behaviour: :namevar, desc: 'The ID of the key you want to manage.', }, + source: { + type: 'String', + desc: 'Where to retrieve the key from, can be a HTTP(s) URL, or a local file. Files get automatically required.', + }, # ... created: { type: 'String', From 5b9a4bf2be1b0be47c52dca18b58022a09623ebf Mon Sep 17 00:00:00 2001 From: David Schmitt Date: Wed, 7 Mar 2018 17:36:38 +0000 Subject: [PATCH 62/62] Add a note on the restrictions of autorequire and friends --- language/resource-api/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/language/resource-api/README.md b/language/resource-api/README.md index 5999faa..2f11a1c 100644 --- a/language/resource-api/README.md +++ b/language/resource-api/README.md @@ -455,3 +455,7 @@ There is no way to access the catalog from the provider. Several existing types ### Logging for unmanaged instances Previously, the provider could provide log messages for resource instances that were not passed into the `set` call. In the current implementation, these will cause an error. + +### Automatic relationships constrained to consts and attribute values + +The puppet3 type API allows arbitrary code execution for calculating automatic relationship targets. The current approach in the Resource API is more restrained, but allowes understanding the type's needs by inspecting the metadata. This is a tradeoff that can be easily revisited by adding a feature later to allow the provider to implement a custom transformation.