From c454c3addb5c31942fd8e8b5824065f1da53df12 Mon Sep 17 00:00:00 2001 From: hestonhoffman Date: Thu, 6 May 2021 14:08:14 -0700 Subject: [PATCH] Get generated docs working with DITA In the future we'll generate our reference docs in this repo, but publish them along with the rest of our documentation so that the reference docs work on the new docs site, and are included in our PDFs. To this end, I've had to adjust the code to get the markdown output to play nicely with the DITA-OT validation. Also: - Adds a workaround for the Facter 3 docs, so the build doesn't attempt to generate the CLI docs for Facter 3. - Removes RONN and dumps the man pages into a codeblock instead. The man pages were causing issues, and I don't have time to mess around with this gem. Also, the docs should be just as (or more) readable if they keep their original man formatting. --- Gemfile | 1 - Gemfile.lock | 11 +++------- lib/puppet_references.rb | 11 ++++++++-- .../facter/core_facts_preamble.md | 4 ++-- .../facter/facter_cli_preamble.md | 1 - lib/puppet_references/puppet/functions.rb | 7 ++++--- .../puppet/functions_preamble.md | 8 +++---- .../puppet/functions_template.erb | 4 ++-- lib/puppet_references/puppet/man.rb | 21 +++++++------------ lib/puppet_references/puppet/type.rb | 10 ++++----- .../puppet/type_template.erb | 18 ++++++++-------- lib/puppet_references/util.rb | 3 ++- 12 files changed, 46 insertions(+), 53 deletions(-) diff --git a/Gemfile b/Gemfile index dc908ca67a..71d477a779 100644 --- a/Gemfile +++ b/Gemfile @@ -16,7 +16,6 @@ group(:build_site) do end group(:generate_references) do - gem 'ronn', '~> 0.7' gem 'yard', '~> 0.9' gem 'rdoc', '~> 6.2' gem 'rgen', '~> 0.8' diff --git a/Gemfile.lock b/Gemfile.lock index 0c17c75978..c4c58cf1be 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -38,7 +38,6 @@ GEM rchardet (~> 1.8) hiera (3.7.0) hocon (1.3.1) - hpricot (0.8.6) http_parser.rb (0.6.0) httpclient (2.8.3) i18n (1.8.10) @@ -77,7 +76,6 @@ GEM mini_portile2 (2.5.0) minitest (5.14.4) multi_json (1.15.0) - mustache (1.1.1) nokogiri (1.11.3) mini_portile2 (~> 2.5.0) racc (~> 1.4) @@ -129,14 +127,9 @@ GEM rb-inotify (0.10.1) ffi (~> 1.0) rchardet (1.8.0) - rdiscount (2.2.0.2) rdoc (6.3.0) rexml (3.2.5) rgen (0.8.2) - ronn (0.7.3) - hpricot (>= 0.8.2) - mustache (>= 0.7.0) - rdiscount (>= 1.5.8) rouge (3.26.0) safe_yaml (1.0.5) sassc (2.4.0) @@ -182,8 +175,10 @@ DEPENDENCIES rake (~> 13.0, >= 13.0.1) rdoc (~> 6.2) rgen (~> 0.8) - ronn (~> 0.7) versionomy (~> 0.5.0) vlad (~> 2.7) vlad-git (~> 2.1) yard (~> 0.9) + +BUNDLED WITH + 2.2.14 diff --git a/lib/puppet_references.rb b/lib/puppet_references.rb index 9e8d578641..89b94cd7aa 100644 --- a/lib/puppet_references.rb +++ b/lib/puppet_references.rb @@ -43,11 +43,18 @@ def self.build_puppet_references(commit) build_from_list_of_classes(references, real_commit) end + def self.is_semantic?(string) + Gem::Version.correct?(string) + end + def self.build_facter_references(commit) references = [ - PuppetReferences::Facter::CoreFacts, - PuppetReferences::Facter::FacterCli + PuppetReferences::Facter::CoreFacts ] + # Adding this workaround so the build doesn't fail for 3.y. Check with Claire to see if + # we need the CLI docs for 3.y. We can remove this when we stop building 3.y. + version_4 = Gem::Version.create('4.0.0') + references << PuppetReferences::Facter::FacterCli if !is_semantic?(commit) || !is_semantic?(commit) && Gem::Version.create(commit) >= version_4 repo = PuppetReferences::Repo.new('facter', FACTER_DIR) real_commit = repo.checkout(commit) build_from_list_of_classes(references, real_commit) diff --git a/lib/puppet_references/facter/core_facts_preamble.md b/lib/puppet_references/facter/core_facts_preamble.md index f82ef33848..51ee1ac017 100644 --- a/lib/puppet_references/facter/core_facts_preamble.md +++ b/lib/puppet_references/facter/core_facts_preamble.md @@ -1,8 +1,8 @@ This is a list of all of the built-in facts that ship with Facter, which includes both legacy facts and newer structured facts. -Not all of them apply to every system, and your site might also use [custom facts](./custom_facts.html) delivered via Puppet modules. To see the full list of structured facts and values on a given system (including plugin facts), run `puppet facts` at the command line. If you are using Puppet Enterprise, you can view all of the facts for any node on the node's page in the console. +Not all of them apply to every system, and your site might also use [custom facts](custom_facts.dita) delivered via Puppet modules. To see the full list of structured facts and values on a given system (including plugin facts), run `puppet facts` at the command line. If you are using Puppet Enterprise, you can view all of the facts for any node on the node's page in the console. -You can access facts in your Puppet manifests as `$fact_name` or `$facts[fact_name]`. For more information, see [the Puppet docs on facts and built-in variables.]({{puppet}}/lang_facts_and_builtin_vars.html) +You can access facts in your Puppet manifests as `$fact_name` or `$facts[fact_name]`. For more information, see [the Puppet docs on facts and built-in variables.](lang_facts_and_builtin_vars.dita) > **Legacy Facts Note:** As of Facter 3, legacy facts such as `architecture` are hidden by default to reduce noise in Facter's default command-line output. These older facts are now part of more useful structured facts; for example, `architecture` is now part of the `os` fact and accessible as `os.architecture`. You can still use these legacy facts in Puppet manifests (`$architecture`), request them on the command line (`facter architecture`), and view them alongside structured facts (`facter --show-legacy`). diff --git a/lib/puppet_references/facter/facter_cli_preamble.md b/lib/puppet_references/facter/facter_cli_preamble.md index c4201b4280..88a0cbf018 100644 --- a/lib/puppet_references/facter/facter_cli_preamble.md +++ b/lib/puppet_references/facter/facter_cli_preamble.md @@ -1,3 +1,2 @@ -# Facter CLI --- diff --git a/lib/puppet_references/puppet/functions.rb b/lib/puppet_references/puppet/functions.rb index 1d1ffdab19..90c61f7c5f 100644 --- a/lib/puppet_references/puppet/functions.rb +++ b/lib/puppet_references/puppet/functions.rb @@ -26,7 +26,6 @@ def build_variant(filename, preferred_version = 'ruby4x') puts "Functions ref (#{filename}): Building" strings_data = PuppetReferences::Puppet::Strings.new functions = strings_data['puppet_functions'] - generated_at = "> **NOTE:** This page was generated from the Puppet source code on #{Time.now.to_s}" header_data = {title: 'Built-in function reference', canonical: "#{@latest}/function.html", toc_levels: 2, @@ -46,7 +45,10 @@ def build_variant(filename, preferred_version = 'ruby4x') template_binding = OpenStruct.new({ functions: functions }).instance_eval {binding} body = ERB.new(File.read(TEMPLATE_FILE), nil, '-').result(template_binding) - content = make_header(header_data) + generated_at + "\n\n" + PREAMBLE + "\n\n" + body + generated_at + # This substitution could potentially make things a bit brittle, but it has to be done because the jump + # From H2s to H4s is causing issues with the DITA-OT, which sees this as a rule violation. If it + # Does become an issue, we should return to this and figure out a better way to generate the functions doc. + content = make_header(header_data) + "\n\n" + PREAMBLE + "\n\n" + body.gsub(/#####\s(.*?:)/,'**\1**').gsub(/####\s/,'###\s') output_path = OUTPUT_DIR + filename output_path.open('w') {|f| f.write(content)} puts "Functions ref (#{filename}): Done!" @@ -54,4 +56,3 @@ def build_variant(filename, preferred_version = 'ruby4x') end end end - diff --git a/lib/puppet_references/puppet/functions_preamble.md b/lib/puppet_references/puppet/functions_preamble.md index fa460a7604..dfbbf874b6 100644 --- a/lib/puppet_references/puppet/functions_preamble.md +++ b/lib/puppet_references/puppet/functions_preamble.md @@ -1,18 +1,18 @@ This page is a list of Puppet's built-in functions, with descriptions of what they do and how to use them. -Functions are plugins you can call during catalog compilation. A call to any function is an expression that resolves to a value. For more information on how to call functions, see [the language reference page about function calls.](./lang_functions.html) +Functions are plugins you can call during catalog compilation. A call to any function is an expression that resolves to a value. For more information on how to call functions, see [the language reference page about function calls.](lang_functions.dita) -Many of these function descriptions include auto-detected _signatures,_ which are short reminders of the function's allowed arguments. These signatures aren't identical to the syntax you use to call the function; instead, they resemble a parameter list from a Puppet [class](./lang_classes.html), [defined resource type](./lang_defined_types.html), [function](./lang_write_functions_in_puppet.html), or [lambda](./lang_lambdas.html). The syntax of a signature is: +Many of these function descriptions include auto-detected _signatures,_ which are short reminders of the function's allowed arguments. These signatures aren't identical to the syntax you use to call the function; instead, they resemble a parameter list from a Puppet [class](lang_classes.dita), [defined resource type](lang_defined_types.dita), [function](lang_write_functions_in_puppet.dita), or [lambda](lang_lambdas.dita). The syntax of a signature is: ``` ( , ...) ``` -The `` is a [Puppet data type value](./lang_data_type.html), like `String` or `Optional[Array[String]]`. The `` is a descriptive name chosen by the function's author to indicate what the argument is used for. +The `` is a [Puppet data type value](lang_data_type.dita), like `String` or `Optional[Array[String]]`. The `` is a descriptive name chosen by the function's author to indicate what the argument is used for. * Any arguments with an `Optional` data type can be omitted from the function call. * Arguments that start with an asterisk (like `*$values`) can be repeated any number of times. -* Arguments that start with an ampersand (like `&$block`) aren't normal arguments; they represent a code block, provided with [Puppet's lambda syntax.](./lang_lambdas.html) +* Arguments that start with an ampersand (like `&$block`) aren't normal arguments; they represent a code block, provided with [Puppet's lambda syntax.](lang_lambdas.dita) ## `undef` values in Puppet 6 diff --git a/lib/puppet_references/puppet/functions_template.erb b/lib/puppet_references/puppet/functions_template.erb index 08827ea249..7812434b2c 100644 --- a/lib/puppet_references/puppet/functions_template.erb +++ b/lib/puppet_references/puppet/functions_template.erb @@ -16,7 +16,7 @@ Signature <%= index+1 %> <% has_parameters = signature['docstring']['tags'].detect {|tag| tag['tag_name'] == 'param' && tag['text'] != '' && tag['text'] != nil } if has_parameters -%> -#### Parameters +### Parameters <% signature['docstring']['tags'].select {|tag| tag['tag_name'] == 'param' && tag['text'] != '' && tag['text'] != nil}.each do |param| -%> @@ -31,7 +31,7 @@ Return type(s): <%= return_types['types'].map {|t| "`#{t}`"}.join(', ') %>. <%= has_examples = signature['docstring']['tags'].detect {|tag| tag['tag_name'] == 'example' && tag['text'] != '' && tag['text'] != nil } if has_examples %> -#### Examples +### Examples <% signature['docstring']['tags'].select {|tag| tag['tag_name'] == 'example' && tag['text'] != '' && tag['text'] != nil}.each do |example| -%> <%= example['name'] %> diff --git a/lib/puppet_references/puppet/man.rb b/lib/puppet_references/puppet/man.rb index d2d764b340..4a67c56b67 100644 --- a/lib/puppet_references/puppet/man.rb +++ b/lib/puppet_references/puppet/man.rb @@ -1,4 +1,5 @@ require 'puppet_references' + module PuppetReferences module Puppet class Man < PuppetReferences::Reference @@ -82,7 +83,7 @@ def build_index(commands) These subcommands form the core of Puppet's tool set, and every user should understand what they do. -#{ categories[:core].reduce('') {|memo, item| memo << "- [puppet #{item}](./#{item}.html)\n"} } +#{ categories[:core].reduce('') {|memo, item| memo << "- [puppet #{item}](#{item}.md)\n"} } > Note: The `puppet cert` command is available only in Puppet versions prior to 6.0. For 6.0 and later, use the [`puppetserver cert`command](https://puppet.com/docs/puppet/6.0/puppet_server_ca_cli.html). @@ -91,14 +92,14 @@ def build_index(commands) Many or most users need to use these subcommands at some point, but they aren't needed for daily use the way the core tools are. -#{ categories[:occasional].reduce('') {|memo, item| memo << "- [puppet #{item}](./#{item}.html)\n"} } +#{ categories[:occasional].reduce('') {|memo, item| memo << "- [puppet #{item}](#{item}.md)\n"} } Niche subcommands ----- Most users can ignore these subcommands. They're only useful for certain niche workflows, and most of them are interfaces to Puppet's internal subsystems. -#{ categories[:weird].reduce('') {|memo, item| memo << "- [puppet #{item}](./#{item}.html)\n"} } +#{ categories[:weird].reduce('') {|memo, item| memo << "- [puppet #{item}](#{item}.md)\n"} } EOT # write index @@ -114,16 +115,8 @@ def get_subcommands applications end - def render_with_ronn(raw_text) - rendered_html = '' - Dir.chdir(PuppetReferences::BASE_DIR) do - ronn = IO.popen("bundle exec ronn --pipe -f", "r+") - ronn.write(raw_text) - ronn.close_write - rendered_html = ronn.read - ronn.close - end - rendered_html + def code_wrap(raw_text) + "
" + raw_text.gsub('<','❮').gsub('>','❯') + "
" end def build_manpage(subcommand) @@ -131,7 +124,7 @@ def build_manpage(subcommand) header_data = {title: "Man Page: puppet #{subcommand}", canonical: "#{@latest}/#{subcommand}.html"} raw_text = PuppetReferences::ManCommand.new(subcommand).get - content = make_header(header_data) + render_with_ronn(raw_text) + content = make_header(header_data) + code_wrap(raw_text) filename = OUTPUT_DIR + "#{subcommand}.md" filename.open('w') {|f| f.write(content)} end diff --git a/lib/puppet_references/puppet/type.rb b/lib/puppet_references/puppet/type.rb index ab6ddde7e8..fe6a835cda 100644 --- a/lib/puppet_references/puppet/type.rb +++ b/lib/puppet_references/puppet/type.rb @@ -38,9 +38,9 @@ def build_all def build_index(names) header_data = {title: 'Resource Types: Index', - canonical: "#{@latest}/types/index.html"} + canonical: "#{@latest}/types/index.md"} links = names.map {|name| - "* [#{name}](./#{name}.html)" + "* [#{name}](./#{name}.md)" unless name == 'component' || name == 'whit' } content = make_header(header_data) + "## List of Resource Types\n\n" + links.join("\n") + "\n\n" + PREAMBLE filename = @output_dir_individual + 'index.md' @@ -58,14 +58,13 @@ def build_unified_page(typedocs) canonical: "#{@latest}/type.html", toc_levels: 2, toc: 'columns'} - generated_at = "> **NOTE:** This page was generated from the Puppet source code on #{Time.now.to_s}" sorted_type_list = typedocs.keys.sort all_type_docs = sorted_type_list.collect{|name| text_for_type(name, typedocs[name]) }.join("\n\n---------\n\n") - content = make_header(header_data) + generated_at + "\n\n" + PREAMBLE + all_type_docs + "\n\n" + generated_at + content = make_header(header_data) + "\n\n" + PREAMBLE + all_type_docs + "\n\n" filename = @output_dir_unified + "#{@base_filename}.md" filename.open('w') {|f| f.write(content)} end @@ -80,8 +79,7 @@ def build_page(name, data) puts "Type ref: Building #{name}" header_data = {title: "Resource Type: #{name}", canonical: "#{@latest}/types/#{name}.html"} - generated_at = "> **NOTE:** This page was generated from the Puppet source code on #{Time.now.to_s}" - content = make_header(header_data) + generated_at + "\n\n" + text_for_type(name, data) + "\n\n" + generated_at + content = make_header(header_data) + "\n\n" + text_for_type(name, data) + "\n\n" filename = @output_dir_individual + "#{name}.md" filename.open('w') {|f| f.write(content)} end diff --git a/lib/puppet_references/puppet/type_template.erb b/lib/puppet_references/puppet/type_template.erb index 1c90869683..fa8b8cd919 100644 --- a/lib/puppet_references/puppet/type_template.erb +++ b/lib/puppet_references/puppet/type_template.erb @@ -1,5 +1,4 @@ -<%= name %> ------ +## <%= name %> * [Attributes](#<%= name %>-attributes) <% if !this_type['providers'].empty? -%> @@ -9,21 +8,22 @@ * [Provider Features](#<%= name %>-provider-features) <% end -%> -

Description

+### [Description](#<%= name %>-description) <%= this_type['description'] %> -

Attributes

+### [Attributes](#<%= name %>-attributes) 
<%= name %> { 'resource title':
 <% sorted_attribute_list.each do |attribute_name| -%>
   <%= attribute_name %><%= ' ' * (longest_attribute_name - attribute_name.length) %> => # <% if this_type['attributes'][attribute_name]['namevar'] %>(namevar) <% end %><%= this_type['attributes'][attribute_name]['description'][0,49].gsub("\n", ' ').gsub('<', '<').sub(/\W? \S+$/, '...') %>
 <% end -%>
-  # ...plus any applicable metaparameters.
+  # ...plus any applicable metaparameters.
 }
<% sorted_attribute_list.each do |attribute_name| -%> -

<%= attribute_name %>

+ +#### [<%= attribute_name %>](#<%= name %>-attribute-<%= attribute_name %>) <% if this_type['attributes'][attribute_name]['namevar'] -%> <% if attribute_name != 'provider' %>_(**Namevar:** If omitted, this attribute's value defaults to the resource's title.)_<%= "\n\n" -%> @@ -44,17 +44,17 @@ Requires features <%= this_type['attributes'][attribute_name]['required_features -%> <% if !this_type['providers'].empty? -%> -

Providers

+### [Providers](#<%= name %>-providers) <% end -%> <% this_type['providers'].keys.sort.each do |provider_name| -%> -

<%= provider_name %>

+#### [<%= provider_name %>](#<%= name %>-provider-<%= provider_name %>) <%= this_type['providers'][provider_name]['description'] %> <% end -%> <% if !this_type['features'].empty? -%> -

Provider Features

+### [Provider Features](#<%= name %>-provider-features) Available features: diff --git a/lib/puppet_references/util.rb b/lib/puppet_references/util.rb index f01f14e3b0..671803ea26 100644 --- a/lib/puppet_references/util.rb +++ b/lib/puppet_references/util.rb @@ -6,11 +6,12 @@ module Util # Given a hash of data, return YAML frontmatter suitable for the docs site. def self.make_header(data) # clean out any symbols: + generated_at = "> **NOTE:** This page was generated from the Puppet source code on #{Time.now.to_s}" clean_data = data.reduce( {} ) do |result, (key,val)| result[key.to_s]=val result end - YAML.dump(clean_data) + "---\n\n" + YAML.dump(clean_data) + "---\n\n" + "# #{clean_data['title']}" + "\n\n" + generated_at + "\n\n" end # Run a command that can't cope with a contaminated shell environment.