Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Multiple updates to the concat codebase.

- append_newline parameter added to concat_build.
- Default file delimiter is now and empty file delimiter which is affected by
  append_newline.
- Sort and uniq work on all lines, not whole file fragments.
- Normal concats work on whole file fragments so be sure to match up your
  delimiters and newline settings in sub-builds if that's what you want.
- insync? etc... now works properly for both fragments and build objects.
- Fragments auto-create an associated build object if one does not exist.
- Build now separates the overwriting of a target from the creation of the
  output file.
- Documentation updated to describe multi-level builds.
- Code updated for compatiblity with Ruby 1.9.
- All instances of a hard coded 'vardir' have been replaced with calls to
  Puppet[:vardir]

Modifications performed by:

- Trevor Vaughan (trevor.vaughan@onyxpoint.com)
- Morgan Haskel (morgan.haskel@onyxpoint.com)
  • Loading branch information...
commit f438b6b42c6e59f2be64d1ba0a376398d5895fa8 1 parent b379478
@trevor-vaughan trevor-vaughan authored
View
57 README.md
@@ -38,20 +38,63 @@ See the comments in the code for the definition of all options.
content => "Some other random stuff"
}
-Notes
------
+Appending without newlines
+--------------------------
+
+If, for example, you wanted your fragments to join together to be a
+comma-separated list, you can achieve this by doing:
+
+ concat_build { "identifier":
+ order => ['*.tmp'],
+ target => '/tmp/test',
+ file_delimiter => ",",
+ append_newline => false
+ }
-Concat fragments are stored under /var/lib/puppet/concat/fragments.
+Chained Builds
+--------------
-TODO
-----
+There are times where you're going to want to chain multiple concat fragment
+builds into a single entity. Multi-part or INI-style files is generally where
+this comes into play.
+
+An example of this type of build is as follows:
+
+ concat_build { "identifier":
+ order => ['*.tmp'],
+ target => '/tmp/test'
+ }
+
+ concat_fragment { "identifier+01.tmp":
+ content => "Some random stuff"
+ }
+
+ concat_fragment { "identifier+02.tmp":
+ content => "Some other random stuff"
+ }
+
+ concat_build { "subtest":
+ parent_build => "identifier",
+ target => "/var/lib/puppet/concat/fragments/identifier/subtest.tmp"
+ }
+
+ concat_fragment { "subtest+sub1":
+ content => "Sub-build stuff"
+ }
+
+ concat_fragment { "subtest+sub2":
+ content => "More sub-build stuff"
+ }
+
+Notes
+-----
-* Don't hardcode /var/lib/puppet. Instead use the value of $vardir.
+Concat fragments are stored under Puppet[:vardir]/concat/fragments.
Copyright
---------
-Copyright (C) 2011 Onyx Point, Inc. <http://onyxpoint.com/>
+Copyright (C) 2012 Onyx Point, Inc. <http://onyxpoint.com/>
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
View
4 lib/puppet/parser/functions/concat_output.rb
@@ -1,5 +1,5 @@
#
-# Copyright (C) 2011 Onyx Point, Inc. <http://onyxpoint.com/>
+# Copyright (C) 2012 Onyx Point, Inc. <http://onyxpoint.com/>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -15,6 +15,6 @@
#
module Puppet::Parser::Functions
newfunction(:concat_output, :type => :rvalue, :doc => "Returns the output file for a given concat build.") do |args|
- "/var/lib/puppet/concat/output/#{args}.out"
+ "#{Puppet[:vardir]}/concat/output/#{args}.out"
end
end
View
4 lib/puppet/parser/functions/fragmentdir.rb
@@ -1,5 +1,5 @@
#
-# Copyright (C) 2011 Onyx Point, Inc. <http://onyxpoint.com/>
+# Copyright (C) 2012 Onyx Point, Inc. <http://onyxpoint.com/>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -15,6 +15,6 @@
#
module Puppet::Parser::Functions
newfunction(:fragmentdir, :type => :rvalue, :doc => "Returns the fragment directory for a given concat build.") do |args|
- "/var/lib/puppet/concat/fragments/#{args}"
+ "#{Puppet[:vardir]}/concat/fragments/#{args}"
end
end
View
127 lib/puppet/provider/concat_build/build.rb
@@ -1,5 +1,5 @@
#
-# Copyright (C) 2011 Onyx Point, Inc. <http://onyxpoint.com/>
+# Copyright (C) 2012 Onyx Point, Inc. <http://onyxpoint.com/>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -18,20 +18,60 @@
desc "concat_build provider"
+ def insync?(orig_file,new_file)
+ sync=false
+
+ !orig_file and return false
+
+ if File.size?(new_file) != File.size?(orig_file) then
+ return false
+ end
+
+ buf_size = 1024
+ offset = 0
+ found_diff = false
+
+ begin
+ ofh = File.open(orig_file,'r')
+ nfh = File.open(new_file,'r')
+
+ while !ofh.eof? do
+ if ofh.read(buf_size) != nfh.read(buf_size) then
+ found_diff = true
+ break
+ end
+ end
+
+ ofh.close
+ nfh.close
+
+ sync = !found_diff
+ rescue
+ debug "Issue diffing files #{orig_file} and #{new_file}, syncing...."
+ end
+
+ if sync then
+ FileUtils.rm(new_file)
+ end
+
+ return sync
+ end
+
+ # Build a consistent temp file.
def build_file
- if File.directory?("/var/lib/puppet/concat/fragments/#{@resource[:name]}") then
+ if File.directory?("#{Puppet[:vardir]}/concat/fragments/#{@resource[:name]}") then
begin
- FileUtils.mkdir_p("/var/lib/puppet/concat/output")
+ FileUtils.mkdir_p("#{Puppet[:vardir]}/concat/output")
- f = File.open("/var/lib/puppet/concat/output/#{@resource[:name]}.out", "w+")
+ ofh = File.open("#{Puppet[:vardir]}/concat/output/#{@resource[:name]}.new", "w+")
input_lines = Array.new
- Dir.chdir("/var/lib/puppet/concat/fragments/#{@resource[:name]}") do
+ Dir.chdir("#{Puppet[:vardir]}/concat/fragments/#{@resource[:name]}") do
Array(@resource[:order]).flatten.each do |pattern|
Dir.glob(pattern).sort_by{ |k| human_sort(k) }.each do |file|
prev_line = nil
File.open(file).each do |line|
-
+ line.chomp!
if @resource.squeeze_blank? and line =~ /^\s*$/ then
if prev_line == :whitespace then
next
@@ -42,23 +82,27 @@ def build_file
out = clean_line(line)
if not out.nil? then
- # This is a bit hackish, but it would be nice to keep as much
- # of the file out of memory as possible in the general case.
+ # This is a bit hackish, but it would be nice to keep as much
+ # of the file out of memory as possible in the general case.
if @resource.sort? or not @resource[:unique].eql?(:false) then
input_lines.push(line)
else
- f.puts(line)
+ ofh.puts(line)
end
end
end
- if not @resource.sort? and @resource[:unique].eql?(:false) then
+ if input_lines.empty? then
# Separate the files by the specified delimiter.
- f.seek(-1, IO::SEEK_END)
- if f.getc.chr.eql?("\n") then
- f.seek(-1, IO::SEEK_END)
- f.print(String(@resource[:file_delimiter]))
+ ofh.seek(-1, IO::SEEK_END)
+ if ofh.getc.chr.eql?("\n") then
+ ofh.seek(-1, IO::SEEK_END)
+ end
+ if @resource[:append_newline].eql?(:false) then
+ ofh.print(String(@resource[:file_delimiter]))
+ else
+ ofh.puts(String(@resource[:file_delimiter]))
end
end
end
@@ -72,7 +116,7 @@ def build_file
if not @resource[:unique].eql?(:false) then
if @resource[:unique].eql?(:uniq) then
require 'enumerator'
- input_lines = input_lines.enum_with_index.map { |x,i|
+ input_lines = input_lines.each_with_index.map { |x,i|
if x.eql?(input_lines[i+1]) then
nil
else
@@ -84,34 +128,51 @@ def build_file
end
end
- f.puts(input_lines.join(@resource[:file_delimiter]))
+ delimiter = @resource[:file_delimiter]
+ if @resource[:append_newline].eql?(:true) then
+ ofh.puts(input_lines.join("#{delimiter}\n"))
+ else
+ ofh.puts(input_lines.join(delimiter))
+ end
else
# Ensure that the end of the file is a '\n'
- f.seek(-(String(@resource[:file_delimiter]).length), IO::SEEK_END)
- curpos = f.pos
- if not f.getc.chr.eql?("\n") then
- f.seek(curpos)
- f.print("\n")
+ if String(@resource[:file_delimiter]).length > 0 then
+ ofh.seek(-(String(@resource[:file_delimiter]).length), IO::SEEK_END)
+ curpos = ofh.pos
+ if not ofh.getc.chr.eql?("\n") then
+ ofh.seek(curpos)
+ ofh.print("\n")
+ end
+ ofh.truncate(ofh.pos)
end
- f.truncate(f.pos)
end
- f.close
+ ofh_name = ofh.path
+ ofh.close
- FileUtils.touch("/var/lib/puppet/concat/fragments/#{@resource[:name]}/.~concat_fragments")
- if @resource[:target] and check_onlyif then
- debug "Copying /var/lib/puppet/concat/output/#{@resource[:name]}.out to #{@resource[:target]}"
- FileUtils.cp("/var/lib/puppet/concat/output/#{@resource[:name]}.out", @resource[:target])
- elsif @resource[:target] then
- debug "Not copying to #{@resource[:target]}, 'onlyif' check failed"
- elsif @resource[:onlyif] then
- debug "Specified 'onlyif' without 'target', ignoring."
- end
rescue Exception => e
fail Puppet::Error, e
end
elsif not @resource.quiet? then
- fail Puppet::Error, "The fragments directory at '/var/lib/puppet/concat/fragments/#{@resource[:name]}' does not exist!"
+ fail Puppet::Error, "The fragments directory at '#{Puppet[:vardir]}/concat/fragments/#{@resource[:name]}' does not exist!"
+ end
+ return ofh_name
+
+ end
+
+ def sync
+ if @resource[:target] and check_onlyif then
+ orig_file = "#{Puppet[:vardir]}/concat/output/#{@resource[:name]}.new"
+ out_file = "#{File.dirname(orig_file)}/#{File.basename(orig_file,'.new')}.out"
+
+ debug "Moving #{orig_file} to #{@resource[:target]}"
+
+ FileUtils.cp(orig_file, out_file)
+ FileUtils.mv(orig_file, @resource[:target])
+ elsif @resource[:target] then
+ debug "Not copying to #{@resource[:target]}, 'onlyif' check failed"
+ elsif @resource[:onlyif] then
+ debug "Specified 'onlyif' without 'target', ignoring."
end
end
View
31 lib/puppet/provider/concat_fragment/fragment.rb
@@ -1,5 +1,5 @@
#
-# Copyright (C) 2011 Onyx Point, Inc. <http://onyxpoint.com/>
+# Copyright (C) 2012 Onyx Point, Inc. <http://onyxpoint.com/>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -18,22 +18,31 @@
desc "concat_fragment provider"
- def create
- begin
- group = @resource[:name].split('+').first
- fragment = @resource[:name].split('+')[1..-1].join('+')
+ def retrieve
+ cur_file = "#{Puppet[:vardir]}/concat/fragments/#{@resource[:frag_group]}/#{@resource[:frag_name]}"
- if File.file?("/var/lib/puppet/concat/fragments/#{group}/.~concat_fragments") then
- debug "Purging /var/lib/puppet/concat/fragments/#{group}!"
- FileUtils.rm_rf("/var/lib/puppet/concat/fragments/#{group}")
+ cur_val = nil
+ begin
+ cur_val = File.read(cur_file)
+ rescue Exception => e
+ debug "Could not find file fragment #{cur_file}"
end
- FileUtils.mkdir_p("/var/lib/puppet/concat/fragments/#{group}")
- f = File.new("/var/lib/puppet/concat/fragments/#{group}/#{fragment}", "w")
- f.puts @resource[:content]
+ return cur_val
+ end
+
+ def create
+ begin
+ group = @resource[:frag_group]
+ fragment = @resource[:frag_name]
+
+ FileUtils.mkdir_p("#{Puppet[:vardir]}/concat/fragments/#{group}")
+ f = File.new("#{Puppet[:vardir]}/concat/fragments/#{group}/#{fragment}", "w")
+ f.print @resource[:content]
f.close
rescue Exception => e
fail Puppet::Error, e
end
end
+
end
View
80 lib/puppet/type/concat_build.rb
@@ -1,5 +1,5 @@
#
-# Copyright (C) 2011 Onyx Point, Inc. <http://onyxpoint.com/>
+# Copyright (C) 2012 Onyx Point, Inc. <http://onyxpoint.com/>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -59,10 +59,19 @@ def validatecmd(cmd)
newparam(:file_delimiter) do
desc "Acts as the delimiter between concatenated file fragments. For
- instance, if you have two files with contents 'foo' and 'bar', the
- result with a file_delimiter of ':' will be a file containing
+ instance, if you have two files with contents 'foo' and 'bar', the
+ result with a file_delimiter of ':' will be a file containing
'foo:bar'."
- defaultto "\n"
+ defaultto ""
+ end
+
+ newparam(:append_newline, :boolean => true) do
+ desc "Whether or not to automatically append a newline to the file
+ delimiter. For example, with no file_delimiter and
+ 'append_newline => false' the fragments will all wind up on the same
+ line."
+ newvalues(:true, :false)
+ defaultto :true
end
newparam(:name) do
@@ -89,7 +98,7 @@ def validatecmd(cmd)
newparam(:sort, :boolean => true) do
desc "Sort the built file. This tries to sort in a human fashion with
- 1 < 2 < 10 < 20 < a, etc.. sort. Note that this will need to read
+ 1 < 2 < 10 < 20 < a, etc.. sort. Note that this will need to read
the entire file into memory
Example Sort:
@@ -111,13 +120,42 @@ def validatecmd(cmd)
defaultto :false
end
- newparam(:target) do
+ newproperty(:target) do
desc "Fully qualified path to copy output file to"
+ defaultto 'unknown'
+
validate do |path|
- unless path =~ /^\/$/ or path =~ /^\/[^\/]/
+ unless path == 'unknown' or path =~ /^\/$/ or path =~ /^\/[^\/]/
fail Puppet::Error, "File paths must be fully qualified, not '#{path}'"
end
end
+
+ munge do |value|
+ if value == 'unknown' then
+ value = "#{Puppet[:vardir]}/concat/output/#{resource[:name]}.out"
+ end
+ value
+ end
+
+ def retrieve
+ if File.exist?(@resource[:target]) then
+ return @resource[:target]
+ end
+
+ return nil
+ end
+
+ def insync?(is)
+ return provider.insync?(is,provider.build_file)
+ end
+
+ def sync
+ provider.sync
+ end
+
+ def change_to_s(currentvalue, newvalue)
+ "#{@resource[:order]} used for ordering"
+ end
end
newparam(:parent_build) do
@@ -135,36 +173,20 @@ def validatecmd(cmd)
desc "Only print unique lines to the output file. Sort takes precedence.
This does not affect file delimiters.
- true: Uses Ruby's Array.uniq function. It will remove all duplicates
+ true: Uses Ruby's Array.uniq function. It will remove all duplicates
regardless of where they are in the file.
- uniq: Acts like the uniq command found in GNU coreutils and only
+ uniq: Acts like the uniq command found in GNU coreutils and only
removes consecutive duplicates."
newvalues(:true, :false, :uniq)
defaultto :false
end
- newproperty(:order, :array_matching => :all) do
+ newparam(:order, :array_matching => :all) do
desc "Array containing ordering info for build"
defaultto ["*"]
-
- def retrieve
- return resource[:order].join(',')
- end
-
- def insync?(is)
- return false
- end
-
- def sync
- provider.build_file
- end
-
- def change_to_s(currentvalue, newvalue)
- "#{[newvalue].flatten.join(',')} used for ordering"
- end
end
autorequire(:concat_build) do
@@ -192,14 +214,14 @@ def change_to_s(currentvalue, newvalue)
end
# clean up the fragments directory for this build if there are no fragments
# in the catalog
- if resource.empty? and File.directory?("/var/lib/puppet/concat/fragments/#{self[:name]}") then
- FileUtils.rm_rf("/var/lib/puppet/concat/fragments/#{self[:name]}")
+ if resource.empty? and File.directory?("#{Puppet[:vardir]}/concat/fragments/#{self[:name]}") then
+ FileUtils.rm_rf("#{Puppet[:vardir]}/concat/fragments/#{self[:name]}")
end
if self[:parent_build] then
found_parent = false
Array(self[:parent_build]).flatten.each do |parent_build|
# Checks to see if there is a concat_build for each parent_build specified
- if not catalog.resources.find { |r| r.is_a?(Puppet::Type.type(:concat_build)) and r[:name].eql?(parent_build)}.nil? then
+ if catalog.resource("Concat_build[#{parent_build}]") then
found_parent = true
elsif not self.quiet? then
warning "No concat_build found for parent_build #{parent_build}"
View
98 lib/puppet/type/concat_fragment.rb
@@ -1,5 +1,5 @@
#
-# Copyright (C) 2011 Onyx Point, Inc. <http://onyxpoint.com/>
+# Copyright (C) 2012 Onyx Point, Inc. <http://onyxpoint.com/>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -14,41 +14,76 @@
# limitations under the License.
#
Puppet::Type.newtype(:concat_fragment) do
- @doc = "Create a concat fragment"
+ @doc = "Create a concat fragment. If you do not create an associated
+ concat_build object, then one will be created for you and the
+ defaults will be used."
+
+ def initialize(args)
+ super
+
+ # If the user did not specify a concat_build object in their manifest,
+ # assume that they want the defaults and create one for them.
+ if not @catalog.resource("Concat_build[#{self[:frag_group]}]") then
+ debug "Auto-adding 'concat_build' resource Concat_build['#{self[:frag_group]}'] to the catalog"
+ @catalog.add_resource(Puppet::Type.type(:concat_build).new(
+ :name => "#{self[:frag_group]}"
+ ))
+ end
+ end
+
+ newparam(:name) do
+ isnamevar
+ validate do |value|
+ fail Puppet::Error, "name is missing group or name. Name format must be 'group+fragment_name'" if value !~ /.+\+.+/
+ fail Puppet::Error, "name cannot include '../'!" if value =~ /\.\.\//
+ end
+ end
+
+ newparam(:frag_group) do
+ desc "Ignore me, I'm a convienience stub"
+ defaultto 'fake'
+
+ munge do |value|
+ @resource[:name].split('+').first
+ end
+ end
+
+ newparam(:frag_name) do
+ desc "Ignore me, I'm a convienience stub"
+ defaultto 'fake'
+
+ munge do |value|
+ @resource[:name].split('+')[1..-1].join('+')
+ end
+ end
newproperty(:content) do
def retrieve
- return resource[:content]
+
+ !@frags_to_delete and @frags_to_delete = []
+ return provider.retrieve
end
def insync?(is)
- return false
+ is and @should or return false
+
+ # @should is an Array through Puppet magic.
+ is.strip == @should.first.strip
end
def sync
provider.create
end
- def change_to_s(currentvalue, newvalue)
- "executed successfully"
- end
- end
-
- newparam(:name) do
- isnamevar
- validate do |value|
- fail Puppet::Error, "name is missing group or name. Name format must be 'group+fragment_name'" if value !~ /.+\+.+/
- fail Puppet::Error, "name cannot include '../'!" if value =~ /\.\.\//
- end
end
# This is only here because, at this point, we can be sure that the catalog
# has been compiled. This checks to see if we have a concat_build specified
# for our particular concat_fragment group.
autorequire(:file) do
- if catalog.resources.find_all { |r| r.is_a?(Puppet::Type.type(:concat_build)) and r[:name] == self[:name].split('+').first }.empty? then
- err "No 'concat_build' specified for group #{self[:name].split('+').first}!"
+ if not catalog.resource("Concat_build[#{self[:frag_group]}]") then
+ err "No 'concat_build' specified for group '#{self[:frag_group]}'"
end
""
end
@@ -56,4 +91,33 @@ def change_to_s(currentvalue, newvalue)
validate do
fail Puppet::Error, "You must specify content" unless self[:content]
end
+
+ def purge_unknown_fragments
+ # Kill all unmanaged fragments for this group
+ known_resources = []
+
+ # Find everything that we shouldn't delete.
+ catalog.resources.find_all { |r|
+ (r.is_a?(Puppet::Type.type(:concat_fragment)) and r[:frag_group] == self[:frag_group] ) or
+ (r.is_a?(Puppet::Type.type(:concat_build)) and r[:target] and File.dirname(r[:target]) == "#{Puppet[:vardir]}/concat/fragments/#{self[:frag_group]}")
+ }.each do |frag_res|
+ if frag_res.is_a?(Puppet::Type.type(:concat_fragment)) then
+ known_resources << "#{Puppet[:vardir]}/concat/fragments/#{frag_res[:frag_group]}/#{frag_res[:frag_name]}"
+ elsif frag_res.is_a?(Puppet::Type.type(:concat_build)) then
+ known_resources << frag_res[:target]
+ else
+ debug "Woops, found an unknown fragment of type #{frag_res.class}"
+ end
+ end
+
+ (Dir.glob("#{Puppet[:vardir]}/concat/fragments/#{self[:frag_group]}/*") - known_resources).each do |to_del|
+ debug "Deleting Unused Fragment: #{to_del}"
+ FileUtils.rm(to_del)
+ end
+ end
+
+ def finish
+ purge_unknown_fragments
+ super
+ end
end
View
24 pkg/pupmod-concat.spec
@@ -1,5 +1,5 @@
#
-# Copyright (C) 2011 Onyx Point, Inc. <http://onyxpoint.com/>
+# Copyright (C) 2012 Onyx Point, Inc. <http://onyxpoint.com/>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -15,8 +15,8 @@
#
Summary: Concat Puppet Module
Name: pupmod-concat
-Version: 1.0
-Release: 1
+Version: 2.0
+Release: 0
License: Apache 2.0
Group: Applications/System
Source: %{name}-%{version}-%{release}.tar.gz
@@ -50,6 +50,24 @@ chmod -R u=rwX,g=rX,o-rwx %{buildroot}/etc/puppet/modules/concat
/etc/puppet/modules/concat
%changelog
+* Sun Jul 22 2012 Trevor Vaughan <tvaughan@onyxpoint.com> - 2.0-0
+ - Multiple updates to the concat codebase.
+
+ - append_newline parameter added to concat_build.
+ - Default file delimiter is now and empty file delimiter which is affected by
+ append_newline.
+ - Sort and uniq work on all lines, not whole file fragments.
+ - Normal concats work on whole file fragments so be sure to match up your
+ delimiters and newline settings in sub-builds if that's what you want.
+ - insync? etc... now works properly for both fragments and build objects.
+ - Fragments auto-create an associated build object if one does not exist.
+ - Build now separates the overwriting of a target from the creation of the
+ output file.
+ - Documentation updated to describe multi-level builds.
+ - Code updated for compatiblity with Ruby 1.9.
+ - All instances of a hard coded 'vardir' have been replaced with calls to
+ Puppet[:vardir]
+
* Sun Jun 26 2011 Trevor Vaughan <tvaughan@onyxpoint.com> - 1.0-1
- Modified to use the Apache 2.0 license instead of the GPLv3
Please sign in to comment.
Something went wrong with that request. Please try again.