Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resource API v1.0 #93

Merged
merged 64 commits into from Mar 28, 2018
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
683010b
snapshot
DavidS Jan 25, 2017
7c694fa
more design drafting
DavidS Jan 27, 2017
154d753
Final touches
DavidS Jan 27, 2017
7ac03ba
Finalise Resource API draft for community review
DavidS Jan 31, 2017
5ee1e22
More touchups
DavidS Jan 31, 2017
ecc3b4e
Add proper README for resource API; update test implementations
DavidS Feb 19, 2017
c05f1f2
apt_key prototype shim
DavidS Feb 19, 2017
4fcc524
Update readme with feedback from puppet-dev
DavidS Mar 3, 2017
b076462
Updates from internal feedback
DavidS Mar 6, 2017
c491dac
Sketch of pops based validation
DavidS Mar 6, 2017
2e88825
Add `processed` logging for very simple implementations
DavidS Mar 7, 2017
f60df1d
More logging updates
DavidS Mar 7, 2017
9de46f4
Update language to be closer to what experienced developers would expect
DavidS Mar 8, 2017
4d97447
Allow `get()` to provide filtering capabilities
DavidS Mar 8, 2017
2956b71
Change get return type to an array
DavidS Mar 8, 2017
faf46c4
Explicitly call out missing resources from get() return value
DavidS Mar 8, 2017
d394c29
Update experimental apt_key implementation
DavidS Apr 4, 2017
959411e
Switch to unified attribute `kind`
DavidS Apr 4, 2017
952e313
Add provider features and pull out the first two optional parts
DavidS Apr 4, 2017
36af9fa
Fix typo in `features` description
DavidS Apr 12, 2017
d7a5d61
Add the canonicalization feature
DavidS Apr 13, 2017
0b05cc9
Clarify the `canonicalize` feature
DavidS Jun 21, 2017
bd772fe
Fix canonicalization example
DavidS Jun 21, 2017
7e8bbf2
fixup
DavidS Jun 21, 2017
5526216
Fix syntax
DavidS Jul 26, 2017
5968441
Change `kind` to `behaviour` to avoid confusion
DavidS Jul 26, 2017
bbf7292
Improve wording around "Multiple providers for the same type"
DavidS Aug 2, 2017
30de9f1
Copy-edit multi-provider description
reidmv Aug 4, 2017
b3fcdda
Another editing pass over the specification
DavidS Aug 29, 2017
7cf9ff6
Add a note on code sharing between providers
DavidS Sep 4, 2017
81b6abd
Improve description of the spec around attribute definition
DavidS Sep 5, 2017
f8d4dde
Define a `default` value for attributes
DavidS Sep 5, 2017
66ec5bd
Change the Implementation to a plain class
DavidS Sep 7, 2017
57fca35
Add a note on puppet's requirements around autoloading
DavidS Sep 7, 2017
e974464
Remove the automatic noop handling of Commands
DavidS Sep 8, 2017
027aa8e
Add missing `context` argument in example
DavidS Sep 11, 2017
4e91983
Update the process handle to be a straight up childprocess process
DavidS Sep 12, 2017
04a23fa
(maint) Fix bulletpoints in resource-api/README.md
Sep 12, 2017
42e89aa
Merge pull request #1 from james-stocks/update_readme
DavidS Sep 12, 2017
69454ea
Add a clarification to composite namevars
DavidS Sep 12, 2017
d4f6459
Rework Commands API to a much simpler interface
DavidS Sep 14, 2017
5ee58c3
Clarify the logging context examples; update for Commands API
DavidS Sep 14, 2017
11f753b
Replace "foreign" with a proper explanation of what is meant
DavidS Sep 14, 2017
e447ce0
Update README.md
davidmalloncares Sep 15, 2017
7cbb54b
Switch the order of Logging and Commands
DavidS Sep 18, 2017
a07f3ec
Define character set handling on talking to Commands
DavidS Sep 20, 2017
f71d0f4
Extend character encoding section to expose full underlying capabilities
DavidS Sep 21, 2017
46129ca
Remove obsolete Known Limitation
DavidS Sep 26, 2017
e81ee00
Describe the resource-api processing method
Oct 5, 2017
61af448
Fix typo
DavidS Oct 2, 2017
ea27a9c
Improve explanation of logging methods
DavidS Oct 6, 2017
4b81f10
Update specification for attribute_changed logging method
Oct 6, 2017
779736d
Remove specification for fail(message) method
Oct 6, 2017
c395408
(PDK-611) define the `remote_resource` feature
DavidS Nov 17, 2017
b9884ed
edits to spec resource-api readme
clairecadman Jan 30, 2018
377a05d
more readme updates
clairecadman Feb 20, 2018
8aa82c8
Merge pull request #3 from clairecadman/resourceapi
DavidS Feb 20, 2018
c00ae5d
Remove Command API
DavidS Feb 21, 2018
e84a07f
Cleanup whitespace
DavidS Feb 21, 2018
ace5368
Remove obsolete examples
DavidS Feb 23, 2018
b56ed4c
Fix a typo: autorequires -> autorequire
DavidS Feb 23, 2018
e72cc61
(PDK-513) updated noop_handler to supports_noop to match Task's language
DavidS Feb 28, 2018
086bd8e
Improve the type example to show all parts for the autorequire
DavidS Mar 7, 2018
5b9a4bf
Add a note on the restrictions of autorequire and friends
DavidS Mar 7, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
437 changes: 437 additions & 0 deletions language/resource-api/README.md

Large diffs are not rendered by default.

377 changes: 377 additions & 0 deletions language/resource-api/apt_key.rb
@@ -0,0 +1,377 @@

# This is a experimental hardcoded implementation of what will be come the Resource API's runtime
# environment. This code is used as test-bed to see that the proposal is technically feasible.

require 'puppet/pops/patterns'
require 'puppet/pops/utils'

DEFINITION = {
name: 'apt_key',
docs: <<-EOS,
This type provides Puppet with the capabilities to manage GPG keys needed
by apt to perform package validation. Apt has it's own GPG keyring that can
be manipulated through the `apt-key` command.

apt_key { '6F6B15509CF8E59E6E469F327F438280EF8D349F':
source => 'http://apt.puppetlabs.com/pubkey.gpg'
}

**Autorequires**:
If Puppet is given the location of a key file which looks like an absolute
path this type will autorequire that file.
EOS
attributes: {
ensure: {
type: 'Enum[present, absent]',
docs: 'Whether this apt key should be present or absent on the target system.'
},
id: {
type: 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]',
docs: 'The ID of the key you want to manage.',
namevar: true,
},
content: {
type: 'Optional[String]',
docs: 'The content of, or string representing, a GPG key.',
},
source: {
type: 'Variant[Stdlib::Absolutepath, Pattern[/\A(https?|ftp):\/\//]]',
docs: 'Location of a GPG key file, /path/to/file, ftp://, http:// or https://',
},
server: {
type: 'Pattern[/\A((hkp|http|https):\/\/)?([a-z\d])([a-z\d-]{0,61}\.)+[a-z\d]+(:\d{2,5})?$/]',
docs: 'The key server to fetch the key from based on the ID. It can either be a domain name or url.',
default: 'keyserver.ubuntu.com'
},
options: {
type: 'Optional[String]',
docs: 'Additional options to pass to apt-key\'s --keyserver-options.',
},
fingerprint: {
type: 'Pattern[/[a-f]{40}/]',
docs: 'The 40-digit hexadecimal fingerprint of the specified GPG key.',
read_only: true,
},
long: {
type: 'Pattern[/[a-f]{16}/]',
docs: 'The 16-digit hexadecimal id of the specified GPG key.',
read_only: true,
},
short: {
type: 'Pattern[/[a-f]{8}/]',
docs: 'The 8-digit hexadecimal id of the specified GPG key.',
read_only: true,
},
expired: {
type: 'Boolean',
docs: 'Indicates if the key has expired.',
read_only: true,
},
expiry: {
# TODO: should be DateTime
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

type: 'String',
docs: 'The date the key will expire, or nil if it has no expiry date, in ISO format.',
read_only: true,
},
size: {
type: 'Integer',
docs: 'The key size, usually a multiple of 1024.',
read_only: true,
},
type: {
type: 'String',
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enum[rsa, dsa, ecc, ecdsa]

docs: 'The key type, one of: rsa, dsa, ecc, ecdsa.',
read_only: true,
},
created: {
type: 'String',
docs: 'Date the key was created, in ISO format.',
read_only: true,
},
},
autorequires: {
file: '$source', # will evaluate to the value of the `source` attribute
package: 'apt',
},
}

module Puppet::SimpleResource
class TypeShim
attr_reader :values

def initialize(title, resource_hash)
# internalize and protect - needs to go deeper
@values = resource_hash.dup
# "name" is a privileged key
@values[:name] = title
@values.freeze
end

def to_resource
ResourceShim.new(@values)
end

def name
values[:name]
end
end

class ResourceShim
attr_reader :values

def initialize(resource_hash)
@values = resource_hash.dup.freeze # whatevs
end

def title
values[:name]
end

def prune_parameters(*args)
puts "not pruning #{args.inspect}" if args.length > 0
self
end

def to_manifest
[
"apt_key { #{values[:name].inspect}: ",
] + values.keys.select { |k| k != :name }.collect { |k| " #{k} => #{values[k].inspect}," } + ['}']
end
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this feels very clear and logical, but the rest up there just reads like boilerplate…
what am i missing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole file is a mixture of example and prototype library code, I've added an explanation at the top:

# This is a experimental hardcoded implementation of what will be come the Resource API's runtime
# environment. This code is used as test-bed to see that the proposal is technically feasible.

the simpleapt.rb is closer to what a real provider would look like.

end
end

Puppet::Type.newtype(DEFINITION[:name].to_sym) do
@doc = DEFINITION[:docs]

has_namevar = false

DEFINITION[:attributes].each do |name, options|
puts "#{name}: #{options.inspect}"

# TODO: using newparam everywhere would suppress change reporting
# that would allow more fine-grained reporting through logger,
# but require more invest in hooking up the infrastructure to emulate existing data
param_or_property = if options[:read_only] || options[:namevar]
:newparam
else
:newproperty
end
send(param_or_property, name.to_sym) do
unless options[:type]
fail("#{DEFINITION[:name]}.#{name} has no type")
end

if options[:docs]
desc "#{options[:docs]} (a #{options[:type]}"
else
warn("#{DEFINITION[:name]}.#{name} has no docs")
end

if options[:namevar]
puts 'setting namevar'
isnamevar
has_namevar = true
end

# read-only values do not need type checking
if not options[:read_only]
# TODO: this should use Pops infrastructure to avoid hardcoding stuff, and enhance type fidelity
# validate do |v|
# type = Puppet::Pops::Types::TypeParser.singleton.parse(options[:type]).normalize
# if type.instance?(v)
# return true
# else
# inferred_type = Puppet::Pops::Types::TypeCalculator.infer_set(value)
# error_msg = Puppet::Pops::Types::TypeMismatchDescriber.new.describe_mismatch("#{DEFINITION[:name]}.#{name}", type, inferred_type)
# raise Puppet::ResourceError, error_msg
# end
# end

case options[:type]
when 'String'
# require any string value
newvalue // do
end
when 'Boolean'
['true', 'false', :true, :false, true, false].each do |v|
newvalue v do
end
end

munge do |v|
case v
when 'true', :true
true
when 'false', :false
false
else
v
end
end
when 'Integer'
newvalue /^\d+$/ do
end
munge do |v|
Puppet::Pops::Utils.to_n(v)
end
when 'Float', 'Numeric'
newvalue Puppet::Pops::Patterns::NUMERIC do
end
munge do |v|
Puppet::Pops::Utils.to_n(v)
end
when 'Enum[present, absent]'
newvalue :absent do
end
newvalue :present do
end
when 'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]'
# the namevar needs to be a Parameter, which only has newvalue*s*
newvalues(/\A(0x)?[0-9a-fA-F]{8}\Z/, /\A(0x)?[0-9a-fA-F]{16}\Z/, /\A(0x)?[0-9a-fA-F]{40}\Z/)
when 'Optional[String]'
newvalue :undef do
end
newvalue // do
end
when 'Variant[Stdlib::Absolutepath, Pattern[/\A(https?|ftp):\/\//]]'
# TODO: this is wrong, but matches original implementation
[/^\//, /\A(https?|ftp):\/\//].each do |v|
newvalue v do
end
end
when /^(Enum|Optional|Variant)/
fail("#{$1} is not currently supported")
end
end
end
end

unless has_namevar
fail("#{DEFINITION[:name]} has no namevar")
end

def self.fake_system_state
@fake_system_state ||= {
'BBCB188AD7B3228BCF05BD554C0BE21B5FF054BD' => {
ensure: :present,
fingerprint: 'BBCB188AD7B3228BCF05BD554C0BE21B5FF054BD',
long: '4C0BE21B5FF054BD',
short: '5FF054BD',
size: 2048,
type: :rsa,
created: '2013-06-07 23:55:31 +0100',
expiry: nil,
expired: false,
},
'B71ACDE6B52658D12C3106F44AB781597254279C' => {
ensure: :present,
fingerprint: 'B71ACDE6B52658D12C3106F44AB781597254279C',
long: '4AB781597254279C',
short: '7254279C',
size: 1024,
type: :dsa,
created: '2007-03-08 20:17:10 +0000',
expiry: nil,
expired: false
},
'9534C9C4130B4DC9927992BF4F30B6B4C07CB649' => {
ensure: :present,
fingerprint: '9534C9C4130B4DC9927992BF4F30B6B4C07CB649',
long: '4F30B6B4C07CB649',
short: 'C07CB649',
size: 4096,
type: :rsa,
created: '2014-11-21 21:01:13 +0000',
expiry: '2022-11-19 21:01:13 +0000',
expired: false
},
'126C0D24BD8A2942CC7DF8AC7638D0442B90D010' => {
ensure: :present,
fingerprint: '126C0D24BD8A2942CC7DF8AC7638D0442B90D010',
long: '7638D0442B90D010',
short: '2B90D010',
size: 4096,
type: :rsa,
created: '2014-11-21 21:13:37 +0000',
expiry: '2022-11-19 21:13:37 +0000',
expired: false
},
'ED6D65271AACF0FF15D123036FB2A1C265FFB764' => {
ensure: :present,
fingerprint: 'ED6D65271AACF0FF15D123036FB2A1C265FFB764',
long: '6FB2A1C265FFB764',
short: '65FFB764',
size: 4096,
type: :rsa,
created: '2010-07-10 01:13:52 +0100',
expiry: '2017-01-05 00:06:37 +0000',
expired: true
},
}
end

def self.get
puts 'get'
fake_system_state
end

def self.set(current_state, target_state, noop = false)
puts "enforcing change from #{current_state} to #{target_state} (noop=#{noop})"
target_state.each do |title, resource|
# additional validation for this resource goes here

# set default value
resource[:ensure] ||= :present

current = current_state[title]
if current && resource[:ensure].to_s == 'absent'
# delete the resource
puts "deleting #{title}"
fake_system_state.delete_if { |k, _| k==title }
elsif current && resource[:ensure].to_s == 'present'
# update the resource
puts "updating #{title}"
resource = current.merge(resource)
fake_system_state[title] = resource.dup
elsif !current && resource[:ensure].to_s == 'present'
# create the resource
puts "creating #{title}"
fake_system_state[title] = resource.dup
end
# TODO: update Type's notion of reality to ensure correct puppet resource output with all available attributes
end
end

def self.instances
puts 'instances'
# klass = Puppet::Type.type(:api)
get.collect do |title, resource_hash|
Puppet::SimpleResource::TypeShim.new(title, resource_hash)
end
end

def retrieve
puts 'retrieve'
result = Puppet::Resource.new(self.class, title)
current_state = self.class.get[title]

if current_state
current_state.each do |k, v|
result[k]=v
end
else
result[:ensure] = :absent
end

@rapi_current_state = current_state
result
end

def flush
puts 'flush'
# binding.pry
target_state = Hash[@parameters.collect { |k, v| [k, v.value] }]
self.class.set({title => @rapi_current_state}, {title => target_state}, false)
end

end