Skip to content

(GH-137) Load Puppet Custom Types, Defined Types and Classes via Puppet API v4 #138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 6, 2019
Merged

(GH-137) Load Puppet Custom Types, Defined Types and Classes via Puppet API v4 #138

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ad31264
(GH-137) Load Puppet Custom Types via Puppet Strings
glennsarti May 16, 2019
c0c7431
(GH-137) Load Puppet Classes and Defined Types via Puppet Strings
glennsarti May 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
233 changes: 23 additions & 210 deletions lib/puppet-languageserver-sidecar/puppet_helper_puppetstrings.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
module PuppetLanguageServerSidecar
module PuppetHelper
SIDECAR_PUPPET_ENVIRONMENT = 'sidecarenvironment'
DISCOVERER_LOADER = 'path-discoverer-null-loader'

def self.path_has_child?(path, child)
# Doesn't matter what the child is, if the path is nil it's true.
Expand Down Expand Up @@ -36,7 +37,7 @@ def self.get_puppet_resource(typename, title = nil)

# Puppet Strings loading
def self.available_documentation_types
[:function]
%I[class function type]
end

# Retrieve objects via the Puppet 4 API loaders
Expand All @@ -49,23 +50,40 @@ def self.retrieve_via_puppet_strings(cache, options = {})
result = {}
return result if object_types.empty?

result[:classes] = PuppetLanguageServer::Sidecar::Protocol::PuppetClassList.new if object_types.include?(:class)
result[:functions] = PuppetLanguageServer::Sidecar::Protocol::PuppetFunctionList.new if object_types.include?(:function)
result[:types] = PuppetLanguageServer::Sidecar::Protocol::PuppetTypeList.new if object_types.include?(:type)

current_env = current_environment
for_agent = options[:for_agent].nil? ? true : options[:for_agent]
loaders = Puppet::Pops::Loaders.new(current_env, for_agent)
# Add any custom loaders
path_discoverer_loader = Puppet::Pops::Loader::PathDiscoveryNullLoader.new(nil, DISCOVERER_LOADER)
loaders.add_loader_by_name(path_discoverer_loader)

paths = []
# :sidecar_manifest isn't technically a Loadable thing. This is useful because we know that any type
# of loader will just ignore it.
paths.concat(discover_type_paths(:sidecar_manifest, loaders)) if object_types.include?(:class)
paths.concat(discover_type_paths(:function, loaders)) if object_types.include?(:function)
paths.concat(discover_type_paths(:type, loaders)) if object_types.include?(:type)

paths.each do |path|
next unless path_has_child?(options[:root_path], path)
file_doc = PuppetLanguageServerSidecar::PuppetStringsHelper.file_documentation(path, cache)
next if file_doc.nil?

if object_types.include?(:class) # rubocop:disable Style/IfUnlessModifier This reads better
file_doc.classes.each { |item| result[:classes] << item }
end
if object_types.include?(:function) # rubocop:disable Style/IfUnlessModifier This reads better
file_doc.functions.each { |item| result[:functions] << item }
end
if object_types.include?(:type)
file_doc.types.each do |item|
result[:types] << item unless name == 'whit' || name == 'component' # rubocop:disable Style/MultipleComparison
end
end
end

# Remove Puppet3 functions which have a Puppet4 function already loaded
Expand All @@ -79,71 +97,10 @@ def self.retrieve_via_puppet_strings(cache, options = {})
end

def self.discover_type_paths(type, loaders)
loaders.private_environment_loader.discover_paths(type)
end

# Class and Defined Type loading
def self.retrieve_classes(cache, options = {})
PuppetLanguageServerSidecar.log_message(:debug, '[PuppetHelper::retrieve_classes] Starting')

# TODO: Can probably do this better, but this works.
current_env = current_environment
module_path_list = current_env
.modules
.select { |mod| Dir.exist?(File.join(mod.path, 'manifests')) }
.map { |mod| mod.path }
manifest_path_list = module_path_list.map { |mod_path| File.join(mod_path, 'manifests') }
PuppetLanguageServerSidecar.log_message(:debug, "[PuppetHelper::retrieve_classes] Loading classes from #{module_path_list}")

# Find and parse all manifests in the manifest paths
classes = PuppetLanguageServer::Sidecar::Protocol::PuppetClassList.new
manifest_path_list.each do |manifest_path|
Dir.glob("#{manifest_path}/**/*.pp").each do |manifest_file|
begin
if path_has_child?(options[:root_path], manifest_file) # rubocop:disable Style/IfUnlessModifier Nicer to read like this
classes.concat(load_classes_from_manifest(cache, manifest_file))
end
rescue StandardError => e
PuppetLanguageServerSidecar.log_message(:error, "[PuppetHelper::retrieve_classes] Error loading manifest #{manifest_file}: #{e} #{e.backtrace}")
end
end
end

PuppetLanguageServerSidecar.log_message(:debug, "[PuppetHelper::retrieve_classes] Finished loading #{classes.count} classes")
classes
end

def self.retrieve_types(cache, options = {})
PuppetLanguageServerSidecar.log_message(:debug, '[PuppetHelper::retrieve_types] Starting')

# From https://github.com/puppetlabs/puppet/blob/ebd96213cab43bb2a8071b7ac0206c3ed0be8e58/lib/puppet/metatype/manager.rb#L182-L189
autoloader = Puppet::Util::Autoload.new(self, 'puppet/type')
current_env = current_environment
types = PuppetLanguageServer::Sidecar::Protocol::PuppetTypeList.new

# This is an expensive call
if autoloader.method(:files_to_load).arity.zero?
params = []
else
params = [current_env]
end
autoloader.files_to_load(*params).each do |file|
name = file.gsub(autoloader.path + '/', '')
begin
expanded_name = autoloader.expand(name)
absolute_name = Puppet::Util::Autoload.get_file(expanded_name, current_env)
raise("Could not find absolute path of type #{name}") if absolute_name.nil?
if path_has_child?(options[:root_path], absolute_name) # rubocop:disable Style/IfUnlessModifier Nicer to read like this
types.concat(load_type_file(cache, name, absolute_name, autoloader, current_env))
end
rescue StandardError => e
PuppetLanguageServerSidecar.log_message(:error, "[PuppetHelper::retrieve_types] Error loading type #{file}: #{e} #{e.backtrace}")
end
end

PuppetLanguageServerSidecar.log_message(:debug, "[PuppetHelper::retrieve_types] Finished loading #{types.count} type/s")

types
[].concat(
loaders.private_environment_loader.discover_paths(type),
loaders[DISCOVERER_LOADER].discover_paths(type)
)
end

# Private functions
Expand All @@ -170,149 +127,5 @@ def self.current_environment
Puppet.lookup(:current_environment)
end
private_class_method :current_environment

def self.load_classes_from_manifest(cache, manifest_file)
class_info = PuppetLanguageServer::Sidecar::Protocol::PuppetClassList.new

if cache.active?
cached_result = cache.load(manifest_file, PuppetLanguageServerSidecar::Cache::CLASSES_SECTION)
unless cached_result.nil?
begin
class_info.from_json!(cached_result)
return class_info
rescue StandardError => e
PuppetLanguageServerSidecar.log_message(:warn, "[PuppetHelper::load_classes_from_manifest] Error while deserializing #{manifest_file} from cache: #{e}")
class_info = PuppetLanguageServer::Sidecar::Protocol::PuppetClassList.new
end
end
end

file_content = File.open(manifest_file, 'r:UTF-8') { |f| f.read }

parser = Puppet::Pops::Parser::Parser.new
result = nil
begin
result = parser.parse_string(file_content, '')
rescue Puppet::ParseErrorWithIssue
# Any parsing errors means we can't inspect the document
return class_info
end

# Enumerate the entire AST looking for classes and defined types
# TODO: Need to learn how to read the help/docs for hover support
if result.model.respond_to? :eAllContents
# Puppet 4 AST
result.model.eAllContents.select do |item|
puppet_class = {}
case item.class.to_s
when 'Puppet::Pops::Model::HostClassDefinition'
puppet_class['type'] = :class
when 'Puppet::Pops::Model::ResourceTypeDefinition'
puppet_class['type'] = :typedefinition
else
next
end
puppet_class['name'] = item.name
puppet_class['doc'] = nil
puppet_class['parameters'] = item.parameters
puppet_class['source'] = manifest_file
puppet_class['line'] = result.locator.line_for_offset(item.offset) - 1
puppet_class['char'] = result.locator.offset_on_line(item.offset)

obj = PuppetLanguageServerSidecar::Protocol::PuppetClass.from_puppet(item.name, puppet_class, result.locator)
class_info << obj
end
else
result.model._pcore_all_contents([]) do |item|
puppet_class = {}
case item.class.to_s
when 'Puppet::Pops::Model::HostClassDefinition'
puppet_class['type'] = :class
when 'Puppet::Pops::Model::ResourceTypeDefinition'
puppet_class['type'] = :typedefinition
else
next
end
puppet_class['name'] = item.name
puppet_class['doc'] = nil
puppet_class['parameters'] = item.parameters
puppet_class['source'] = manifest_file
puppet_class['line'] = item.line
puppet_class['char'] = item.pos
obj = PuppetLanguageServerSidecar::Protocol::PuppetClass.from_puppet(item.name, puppet_class, item.locator)
class_info << obj
end
end
cache.save(manifest_file, PuppetLanguageServerSidecar::Cache::CLASSES_SECTION, class_info.to_json) if cache.active?

class_info
end
private_class_method :load_classes_from_manifest

def self.load_type_file(cache, name, absolute_name, autoloader, env)
types = PuppetLanguageServer::Sidecar::Protocol::PuppetTypeList.new
if cache.active?
cached_result = cache.load(absolute_name, PuppetLanguageServerSidecar::Cache::TYPES_SECTION)
unless cached_result.nil?
begin
types.from_json!(cached_result)
return types
rescue StandardError => e
PuppetLanguageServerSidecar.log_message(:warn, "[PuppetHelper::load_type_file] Error while deserializing #{absolute_name} from cache: #{e}")
types = PuppetLanguageServer::Sidecar::Protocol::PuppetFunctionList.new
end
end
end

# Get the list of currently loaded types
loaded_types = []
# Due to PUP-8301, if no types have been loaded yet then Puppet::Type.eachtype
# will throw instead of not yielding.
begin
Puppet::Type.eachtype { |item| loaded_types << item.name }
rescue NoMethodError => e
# Detect PUP-8301
if e.respond_to?(:receiver)
raise unless e.name == :each && e.receiver.nil?
else
raise unless e.name == :each && e.message =~ /nil:NilClass/
end
end

unless autoloader.loaded?(name)
# This is an expensive call
unless autoloader.load(name, env) # rubocop:disable Style/IfUnlessModifier Nicer to read like this
PuppetLanguageServerSidecar.log_message(:error, "[PuppetHelper::load_type_file] type #{absolute_name} did not load")
end
end

# Find the types that were loaded
# Due to PUP-8301, if no types have been loaded yet then Puppet::Type.eachtype
# will throw instead of not yielding.
begin
Puppet::Type.eachtype do |item|
next if loaded_types.include?(item.name)
# Ignore the internal only Puppet Types
next if item.name == :component || item.name == :whit
obj = PuppetLanguageServerSidecar::Protocol::PuppetType.from_puppet(item.name, item)
# TODO: Need to use calling_source in the cache backing store
# Perhaps I should be incrementally adding items to the cache instead of batch mode?
obj.calling_source = absolute_name
types << obj
end
rescue NoMethodError => e
# Detect PUP-8301
if e.respond_to?(:receiver)
raise unless e.name == :each && e.receiver.nil?
else
raise unless e.name == :each && e.message =~ /nil:NilClass/
end
end
PuppetLanguageServerSidecar.log_message(:warn, "[PuppetHelper::load_type_file] type #{absolute_name} did not load any types") if types.empty?
cache.save(absolute_name, PuppetLanguageServerSidecar::Cache::TYPES_SECTION, types.to_json) if cache.active?

types
end
private_class_method :load_type_file
end
end
57 changes: 57 additions & 0 deletions lib/puppet-languageserver-sidecar/puppet_monkey_patches_puppetstrings.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,63 @@ def discover_paths(type, name_authority = Pcore::RUNTIME_NAME_AUTHORITY)
end
end

# While this is not a monkey patch, but a new class, this class is used purely to
# enumerate the paths of puppet "things" that aren't already covered as part of the
# usual loaders. It is implemented as a null loader as it can't actually _load_
# anything.
module Puppet
module Pops
module Loader
class PathDiscoveryNullLoader < Puppet::Pops::Loader::NullLoader
def discover_paths(type, name_authority = Pcore::RUNTIME_NAME_AUTHORITY)
result = []

if type == :type
autoloader = Puppet::Util::Autoload.new(self, 'puppet/type')
current_env = current_environment

# This is an expensive call
Copy link
Contributor Author

@glennsarti glennsarti Jun 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: this is in the wrong spot.

if autoloader.method(:files_to_load).arity.zero?
params = []
else
params = [current_env]
end
autoloader.files_to_load(*params).each do |file|
name = file.gsub(autoloader.path + '/', '')
expanded_name = autoloader.expand(name)
absolute_name = Puppet::Util::Autoload.get_file(expanded_name, current_env)
result << absolute_name unless absolute_name.nil?
end
end

if type == :sidecar_manifest
current_environment.modules.each do |mod|
result.concat(mod.all_manifests)
end
end

result.concat(super)
result.uniq
end

private

def current_environment
begin
env = Puppet.lookup(:environments).get!(Puppet.settings[:environment])
return env unless env.nil?
rescue Puppet::Environments::EnvironmentNotFound
PuppetLanguageServerSidecar.log_message(:warning, "[Puppet::Pops::Loader::PathDiscoveryNullLoader::current_environment] Unable to load environment #{Puppet.settings[:environment]}")
rescue StandardError => e
PuppetLanguageServerSidecar.log_message(:warning, "[Puppet::Pops::Loader::PathDiscoveryNullLoader::current_environment] Error loading environment #{Puppet.settings[:environment]}: #{e}")
end
Puppet.lookup(:current_environment)
end
end
end
end
end

module Puppet
module Pops
module Loader
Expand Down
Loading