(#15841,#13542) PMT tarball install & embed Minitar gem #1508

Closed
wants to merge 3 commits into
from

Conversation

Projects
None yet
9 participants
@mruzicka
Contributor

mruzicka commented Feb 27, 2013

This pull request should really be two separate ones but because of it its
gradual changes it ended up as a single large one. It introduces
two principal changes:

  • Adds support to PMT for installing/upgrading puppet modules
    from local tarballs. This support depends on a new forge API
    introduced in:
    https://github.com/puppetlabs/puppet-forge-api/pull/32
    In addition to the posibility of installing modules from tarballs a new
    algorithm for module dependency resolution is also introduced.
    The algorithm is capable of backtracking so is able to find resolutions
    even in situations in which the original algorithm would have failed.
    Many of the data structures used by the depedecy resolution code have
    been changed but care has been taken to preserve backward compatiblity
    of any externally visible data structures.
    Also the installer and upgrader applications have been greatly unified,
    so they now share the majority of their code.
  • Embeds the minitar gem into Puppet while changing its namespace to:
    Puppet::Util::Archive::Tar::Minitar
    and makes use of it at various places in the PMT code instead of
    calling external tar commands.

The two additional commits in this pull request update and add acceptance
& unit tests to keep up with the code changes, they are deliberately kept
separate to make the review of this pull request somewhat easier.

@domcleal

This comment has been minimized.

Show comment
Hide comment
@domcleal

domcleal Feb 28, 2013

Member

The new rubygems dependency is probably OK I think, but it will have packaging implications - the spec file will need updating too for instance. Today Puppet will function with or without rubygems (see Puppet::Util::Rubygems for instance).

Member

domcleal commented Feb 28, 2013

The new rubygems dependency is probably OK I think, but it will have packaging implications - the spec file will need updating too for instance. Today Puppet will function with or without rubygems (see Puppet::Util::Rubygems for instance).

results[:result] = :success
- results[:installed_modules] = @graph

This comment has been minimized.

@domcleal

domcleal Feb 28, 2013

Member

This will be an incompatible change in the JSON/YAML output of these commands, so should probably be left in. I haven't delved into @graph, is it unchanged?

@domcleal

domcleal Feb 28, 2013

Member

This will be an incompatible change in the JSON/YAML output of these commands, so should probably be left in. I haven't delved into @graph, is it unchanged?

This comment has been minimized.

@mruzicka

mruzicka Feb 28, 2013

Contributor

Could we perhaps, include both? Like so:
results[:installed_modules] = results[:affected_modules] = @graph
the first for backwards compatibility and the second as its name better describes the content of the graph and to unify it with what the upgrade does

@mruzicka

mruzicka Feb 28, 2013

Contributor

Could we perhaps, include both? Like so:
results[:installed_modules] = results[:affected_modules] = @graph
the first for backwards compatibility and the second as its name better describes the content of the graph and to unify it with what the upgrade does

This comment has been minimized.

@mruzicka

mruzicka Feb 28, 2013

Contributor

The basic structure of the graph is unchanged, (the method to print it dind't need any adjustments for instance) but there are changes. The same (or more) information is present in each graph node, the structure of the nodes is different though.

@mruzicka

mruzicka Feb 28, 2013

Contributor

The basic structure of the graph is unchanged, (the method to print it dind't need any adjustments for instance) but there are changes. The same (or more) information is present in each graph node, the structure of the nodes is different though.

This comment has been minimized.

@domcleal

domcleal Feb 28, 2013

Member

@mruzicka yes, I think including both would be ideal, thanks. If you're feeling keen, it could be listed for removal in 4.x.

@domcleal

domcleal Feb 28, 2013

Member

@mruzicka yes, I think including both would be ideal, thanks. If you're feeling keen, it could be listed for removal in 4.x.

end
end
-
- results[:result] = :success
- results[:base_dir] = @graph.first[:path]

This comment has been minimized.

@domcleal

domcleal Feb 28, 2013

Member

I like that install_dir here is now consistent with the install action, but would suggest leaving base_dir in place too for compatibility.

@domcleal

domcleal Feb 28, 2013

Member

I like that install_dir here is now consistent with the install action, but would suggest leaving base_dir in place too for compatibility.

This comment has been minimized.

@mruzicka

mruzicka Feb 28, 2013

Contributor

again how about setting both?

@mruzicka

mruzicka Feb 28, 2013

Contributor

again how about setting both?

This comment has been minimized.

@domcleal

domcleal Feb 28, 2013

Member

ditto, that'd be great.

@domcleal

domcleal Feb 28, 2013

Member

ditto, that'd be great.

@domcleal

View changes

lib/puppet/forge.rb
@@ -89,6 +89,21 @@ def remote_dependency_info(author, mod_name, version)
end
end
+ def multiple_remote_dependency_info(modules)

This comment has been minimized.

@domcleal

domcleal Feb 28, 2013

Member

Some YARD on the expected response would be useful here, looks like it's the same as remote_dependency_info but combining results for multiple modules?

@domcleal

domcleal Feb 28, 2013

Member

Some YARD on the expected response would be useful here, looks like it's the same as remote_dependency_info but combining results for multiple modules?

This comment has been minimized.

@mruzicka

mruzicka Feb 28, 2013

Contributor

yes, it is the same as in the simple query scenario, I'll add the doc

@mruzicka

mruzicka Feb 28, 2013

Contributor

yes, it is the same as in the simple query scenario, I'll add the doc

+ modules.each do |mod|
+ query_string << "&module[]=#{mod.first}&version[]=#{mod.last}"
+ end
+ response = repository.make_http_request('/api/v1/releases.json?' << query_string[1..-1])

This comment has been minimized.

@domcleal

domcleal Feb 28, 2013

Member

@mhrivnak FYI, it looks like support for this may need adding to Pulp.

@domcleal

domcleal Feb 28, 2013

Member

@mhrivnak FYI, it looks like support for this may need adding to Pulp.

@adrienthebo

View changes

lib/puppet/forge/errors.rb
+ class VerboseForgeError < ForgeError
+ def initialize(message, mutliline_message, original=nil)
+ super(message, original)
+ @mutliline_message = mutliline_message

This comment has been minimized.

@adrienthebo

adrienthebo Feb 28, 2013

Contributor

Typo, mutliline vs multiline

@adrienthebo

adrienthebo Feb 28, 2013

Contributor

Typo, mutliline vs multiline

@adrienthebo

This comment has been minimized.

Show comment
Hide comment
@adrienthebo

adrienthebo Mar 1, 2013

Contributor

This fails pretty hard on Centos 5.8:

[root@localhost ~]# puppet module install branan/eight_hundred
/usr/lib/ruby/site_ruby/1.8/puppet/module_tool/applications/builder.rb:1:in `require': no such file to load -- rubygems (LoadError)
    from /usr/lib/ruby/site_ruby/1.8/puppet/module_tool/applications/builder.rb:1
    from /usr/lib/ruby/site_ruby/1.8/puppet/module_tool/applications.rb:6:in `require'
    from /usr/lib/ruby/site_ruby/1.8/puppet/module_tool/applications.rb:6
    from /usr/lib/ruby/site_ruby/1.8/puppet/module.rb:3:in `require'
    from /usr/lib/ruby/site_ruby/1.8/puppet/module.rb:3
    from /usr/lib/ruby/site_ruby/1.8/puppet/parser/files.rb:1:in `require'
    from /usr/lib/ruby/site_ruby/1.8/puppet/parser/files.rb:1
    from /usr/lib/ruby/site_ruby/1.8/puppet/parser/templatewrapper.rb:1:in `require'
     ... 13 levels...
    from /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:12:in `require'
    from /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:12
    from /usr/bin/puppet:3:in `require'
    from /usr/bin/puppet:3
[root@localhost ~]# cat /etc/redhat-release 
CentOS release 5.8 (Final)
[root@localhost ~]# ruby --version
ruby 1.8.5 (2006-08-25) [i386-linux]
[root@localhost ~]# which gem
/usr/bin/which: no gem in (/usr/kerberos/sbin:/usr/local/sbin:/usr/sbin:/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/vagrant/bin:/root/bin)
Contributor

adrienthebo commented Mar 1, 2013

This fails pretty hard on Centos 5.8:

[root@localhost ~]# puppet module install branan/eight_hundred
/usr/lib/ruby/site_ruby/1.8/puppet/module_tool/applications/builder.rb:1:in `require': no such file to load -- rubygems (LoadError)
    from /usr/lib/ruby/site_ruby/1.8/puppet/module_tool/applications/builder.rb:1
    from /usr/lib/ruby/site_ruby/1.8/puppet/module_tool/applications.rb:6:in `require'
    from /usr/lib/ruby/site_ruby/1.8/puppet/module_tool/applications.rb:6
    from /usr/lib/ruby/site_ruby/1.8/puppet/module.rb:3:in `require'
    from /usr/lib/ruby/site_ruby/1.8/puppet/module.rb:3
    from /usr/lib/ruby/site_ruby/1.8/puppet/parser/files.rb:1:in `require'
    from /usr/lib/ruby/site_ruby/1.8/puppet/parser/files.rb:1
    from /usr/lib/ruby/site_ruby/1.8/puppet/parser/templatewrapper.rb:1:in `require'
     ... 13 levels...
    from /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:12:in `require'
    from /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:12
    from /usr/bin/puppet:3:in `require'
    from /usr/bin/puppet:3
[root@localhost ~]# cat /etc/redhat-release 
CentOS release 5.8 (Final)
[root@localhost ~]# ruby --version
ruby 1.8.5 (2006-08-25) [i386-linux]
[root@localhost ~]# which gem
/usr/bin/which: no gem in (/usr/kerberos/sbin:/usr/local/sbin:/usr/sbin:/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/vagrant/bin:/root/bin)
@@ -56,10 +56,10 @@
on master, puppet("module install pmtacceptance-nginx --force") do
assert_output <<-OUTPUT
\e[mNotice: Preparing to install into /etc/puppet/modules ...\e[0m
- \e[mNotice: Downloading from https://forge.puppetlabs.com ...\e[0m

This comment has been minimized.

@adrienthebo

adrienthebo Mar 1, 2013

Contributor

This line is duplicated a ton of times across these tests; does it make sense to extract this information to some sort of shared fixture data?

@adrienthebo

adrienthebo Mar 1, 2013

Contributor

This line is duplicated a ton of times across these tests; does it make sense to extract this information to some sort of shared fixture data?

This comment has been minimized.

@mruzicka

mruzicka Mar 7, 2013

Contributor

The tests would need to manage the puppet.conf file (or pass the --module_repository option to all commands) to control which repository is used. We could stop stubbing the forge.puppetlabs.com at that point too.

@mruzicka

mruzicka Mar 7, 2013

Contributor

The tests would need to manage the puppet.conf file (or pass the --module_repository option to all commands) to control which repository is used. We could stop stubbing the forge.puppetlabs.com at that point too.

@adrienthebo

This comment has been minimized.

Show comment
Hide comment
@adrienthebo

adrienthebo Mar 1, 2013

Contributor

It looks like Github ate my comment on the commit, but 753d7fb has a lot of changes (866 additions, 353 deletions) but doesn't have any unit tests for new behavior.

In addition the travis-ci build has 24 failed tests, as demonstrated in (https://gist.github.com/adrienthebo/5061785).

Contributor

adrienthebo commented Mar 1, 2013

It looks like Github ate my comment on the commit, but 753d7fb has a lot of changes (866 additions, 353 deletions) but doesn't have any unit tests for new behavior.

In addition the travis-ci build has 24 failed tests, as demonstrated in (https://gist.github.com/adrienthebo/5061785).

@joshcooper

This comment has been minimized.

Show comment
Hide comment
@mruzicka

This comment has been minimized.

Show comment
Hide comment
@mruzicka

mruzicka Mar 7, 2013

Contributor

The code has a hard dependency on the rubygems tar code. So how do I ensure the rubygems library is loaded in the bundle environment? Is there a way to have bundler load it in a way that doesn't interfere with it?
Note that it would be possible to use the minitar implementation instead but again it would be a hard dependency, i.e. the code would not run without it.

I completely forgot about the unit test, I'll update them and add new ones.

Contributor

mruzicka commented Mar 7, 2013

The code has a hard dependency on the rubygems tar code. So how do I ensure the rubygems library is loaded in the bundle environment? Is there a way to have bundler load it in a way that doesn't interfere with it?
Note that it would be possible to use the minitar implementation instead but again it would be a hard dependency, i.e. the code would not run without it.

I completely forgot about the unit test, I'll update them and add new ones.

@mruzicka

This comment has been minimized.

Show comment
Hide comment
@mruzicka

mruzicka Mar 7, 2013

Contributor

There is another alternative to using minitar or rubygems tar code and that is including the Tar handling code directly in puppet - at least the rubygems tar code is very simple only a handful of classes really.

Contributor

mruzicka commented Mar 7, 2013

There is another alternative to using minitar or rubygems tar code and that is including the Tar handling code directly in puppet - at least the rubygems tar code is very simple only a handful of classes really.

@joshcooper

This comment has been minimized.

Show comment
Hide comment
@joshcooper

joshcooper Mar 7, 2013

Member

@mruzicka I recently discovered rubygems is not present on rhel6 that we use in acceptance testing, i.e. require 'rubygems' will fail. Also using minitar is not a hard-dependency provided you wrap access to it by first checking whether the Puppet.features.minitar? is available or not, then fall back to shelling out to tar/gtar.

Member

joshcooper commented Mar 7, 2013

@mruzicka I recently discovered rubygems is not present on rhel6 that we use in acceptance testing, i.e. require 'rubygems' will fail. Also using minitar is not a hard-dependency provided you wrap access to it by first checking whether the Puppet.features.minitar? is available or not, then fall back to shelling out to tar/gtar.

@jeffmccune

This comment has been minimized.

Show comment
Hide comment
@jeffmccune

jeffmccune Mar 7, 2013

Contributor

There is another alternative to using minitar or rubygems tar code and that is including the Tar handling code directly in puppet - at least the rubygems tar code is very simple only a handful of classes really.

I think this would work well, provided the classes are properly namespaced to avoid collisions and monkey patching and the classes work on all of our platforms including windows.

@mruzicka Would you mind amending this change set and re-pushing?

-Jeff

Contributor

jeffmccune commented Mar 7, 2013

There is another alternative to using minitar or rubygems tar code and that is including the Tar handling code directly in puppet - at least the rubygems tar code is very simple only a handful of classes really.

I think this would work well, provided the classes are properly namespaced to avoid collisions and monkey patching and the classes work on all of our platforms including windows.

@mruzicka Would you mind amending this change set and re-pushing?

-Jeff

@haus

This comment has been minimized.

Show comment
Hide comment
@haus

haus Mar 7, 2013

Contributor

@mruzicka if going down the road of including the tar code in a utility namespace of puppet, I would want to know the source of the tar code and other things like how often does upstream get updated (do they see a lot of security events), and if our plan is to fork it or to keep it up to date with upstream.

Contributor

haus commented Mar 7, 2013

@mruzicka if going down the road of including the tar code in a utility namespace of puppet, I would want to know the source of the tar code and other things like how often does upstream get updated (do they see a lot of security events), and if our plan is to fork it or to keep it up to date with upstream.

@mruzicka

This comment has been minimized.

Show comment
Hide comment
@mruzicka

mruzicka Mar 8, 2013

Contributor

@haus the code we are talking about can be found here: https://github.com/rubygems/rubygems/tree/master/lib/rubygems/package
It hasn't received any significant updates in 2 years, IMO such a frequency of updates will probably be the norm for the code, given the tar format itself is fairly stable.
My thinking is that we should just copy the files we need:
tar_reader/entry.rb
tar_header.rb
tar_reader.rb
tar_writer.rb
and possibly:
tar_test_case.rb
and adjust them to suit our needs (change the namespace, drop requires on the rest of the rubygems code, add support for symliks), which would probably make them difficult to update. I.e. I don't see us updating them from the upstream.

Contributor

mruzicka commented Mar 8, 2013

@haus the code we are talking about can be found here: https://github.com/rubygems/rubygems/tree/master/lib/rubygems/package
It hasn't received any significant updates in 2 years, IMO such a frequency of updates will probably be the norm for the code, given the tar format itself is fairly stable.
My thinking is that we should just copy the files we need:
tar_reader/entry.rb
tar_header.rb
tar_reader.rb
tar_writer.rb
and possibly:
tar_test_case.rb
and adjust them to suit our needs (change the namespace, drop requires on the rest of the rubygems code, add support for symliks), which would probably make them difficult to update. I.e. I don't see us updating them from the upstream.

@mruzicka

This comment has been minimized.

Show comment
Hide comment
@mruzicka

mruzicka Mar 8, 2013

Contributor

@jeffmccune I'm ready to start looking into that, which namespace should I use for the tar code?

Contributor

mruzicka commented Mar 8, 2013

@jeffmccune I'm ready to start looking into that, which namespace should I use for the tar code?

@joshcooper

This comment has been minimized.

Show comment
Hide comment
@joshcooper

joshcooper Mar 8, 2013

Member

@mruzicka I think you'll find the minitar code would be easier to vendor, especially as it already provides library like methods for extracting, e.g. you call a method and pass it a block that gets called once for each extracted file. The tar code in rubygems was a bit more entangled with rubygems specific implementation.

Member

joshcooper commented Mar 8, 2013

@mruzicka I think you'll find the minitar code would be easier to vendor, especially as it already provides library like methods for extracting, e.g. you call a method and pass it a block that gets called once for each extracted file. The tar code in rubygems was a bit more entangled with rubygems specific implementation.

@mruzicka

This comment has been minimized.

Show comment
Hide comment
@mruzicka

mruzicka Mar 12, 2013

Contributor

@joshcooper @jeffmccune @zaphod42 I was able to replace the rubygems tar code with the minitar implementation, it turns out the minitar is not that much different and moreover better suited for embedding as it has no dependencies (although it contains more code we probably would not use).
So we could embed it with minimal changes which means it should be possible to update the code easily from the upstream.

The question is what namespace should I embed it under?
(It's current namespace is Archive::Tar::Minitar)

@haus the minitar code can be found here: https://github.com/atoulme/minitar/tree/master/lib/archive/tar
It appears to be similarly stable as the rubygems tar code, IMO again due the stability of the tar format itself.

Contributor

mruzicka commented Mar 12, 2013

@joshcooper @jeffmccune @zaphod42 I was able to replace the rubygems tar code with the minitar implementation, it turns out the minitar is not that much different and moreover better suited for embedding as it has no dependencies (although it contains more code we probably would not use).
So we could embed it with minimal changes which means it should be possible to update the code easily from the upstream.

The question is what namespace should I embed it under?
(It's current namespace is Archive::Tar::Minitar)

@haus the minitar code can be found here: https://github.com/atoulme/minitar/tree/master/lib/archive/tar
It appears to be similarly stable as the rubygems tar code, IMO again due the stability of the tar format itself.

@jeffmccune

This comment has been minimized.

Show comment
Hide comment
@jeffmccune

jeffmccune Mar 12, 2013

Contributor

On Tue, Mar 12, 2013 at 11:35 AM, mruzicka notifications@github.com wrote:

@joshcooper https://github.com/joshcooper @jeffmccunehttps://github.com/jeffmccune
@zaphod42 https://github.com/zaphod42 I was able to replace the
rubygems tar code with the minitar implementation, it turns out the minitar
is not that much different and moreover better suited for embedding as it
has no dependencies (although it contains more code we probably would not
use).
So we could embed it with minimal changes which means it should be
possible to update the code easily from the upstream.

The question is what namespace should I embed it under?
(It's current namespace is Archive::Tar::Minitar)

As long as the convention of writing the class names in the CamelCase form
of the relative file path is followed, I'm fine putting it anywhere that
makes sense.

Inside of Puppet::Util is customary. For example, Trollop is in
puppet/util/command_line/trollop with a class name of
Puppet::Util::CommandLine::Trollop

We could similarly use Puppet::Util::Archive::Tar::Minitar with a path of
lib/puppet/util/archive/tar/minitar

-Jeff

Contributor

jeffmccune commented Mar 12, 2013

On Tue, Mar 12, 2013 at 11:35 AM, mruzicka notifications@github.com wrote:

@joshcooper https://github.com/joshcooper @jeffmccunehttps://github.com/jeffmccune
@zaphod42 https://github.com/zaphod42 I was able to replace the
rubygems tar code with the minitar implementation, it turns out the minitar
is not that much different and moreover better suited for embedding as it
has no dependencies (although it contains more code we probably would not
use).
So we could embed it with minimal changes which means it should be
possible to update the code easily from the upstream.

The question is what namespace should I embed it under?
(It's current namespace is Archive::Tar::Minitar)

As long as the convention of writing the class names in the CamelCase form
of the relative file path is followed, I'm fine putting it anywhere that
makes sense.

Inside of Puppet::Util is customary. For example, Trollop is in
puppet/util/command_line/trollop with a class name of
Puppet::Util::CommandLine::Trollop

We could similarly use Puppet::Util::Archive::Tar::Minitar with a path of
lib/puppet/util/archive/tar/minitar

-Jeff

@pvande

This comment has been minimized.

Show comment
Hide comment
@pvande

pvande Mar 13, 2013

Contributor

Inside of Puppet::Util is customary. For example, Trollop is in
puppet/util/command_line/trollop with a class name of
Puppet::Util::CommandLine::Trollop

We could similarly use Puppet::Util::Archive::Tar::Minitar with a path of
lib/puppet/util/archive/tar/minitar

How will this change if and when we extract the module tool into an independent module?

Contributor

pvande commented Mar 13, 2013

Inside of Puppet::Util is customary. For example, Trollop is in
puppet/util/command_line/trollop with a class name of
Puppet::Util::CommandLine::Trollop

We could similarly use Puppet::Util::Archive::Tar::Minitar with a path of
lib/puppet/util/archive/tar/minitar

How will this change if and when we extract the module tool into an independent module?

@jeffmccune

This comment has been minimized.

Show comment
Hide comment
@jeffmccune

jeffmccune Mar 13, 2013

Contributor

How will this change if and when we extract the module tool into an independent module?

I think in this situation the module should vendor the Minitar using the puppet_x and PuppetX conventions we've established in 14149. An example of migrating existing code to this convention can be found at https://github.com/puppetlabs/puppetlabs-registry/pull/18/commits

Please let me know if this doesn't fully answer your question or there's more information I can provide.

Contributor

jeffmccune commented Mar 13, 2013

How will this change if and when we extract the module tool into an independent module?

I think in this situation the module should vendor the Minitar using the puppet_x and PuppetX conventions we've established in 14149. An example of migrating existing code to this convention can be found at https://github.com/puppetlabs/puppetlabs-registry/pull/18/commits

Please let me know if this doesn't fully answer your question or there's more information I can provide.

@mruzicka

This comment has been minimized.

Show comment
Hide comment
@mruzicka

mruzicka Mar 15, 2013

Contributor

I've added 3 new commits which bring the following changes:

  • add the requested YARD and fix the typo mentioned in one of the previous comments
  • replace the rubygems tar code with minitar implementation, which in turn is embedded into puppet at the Puppet::Util::Archive::Tar::Minitar namespace.
  • improve backward compatibility of the data structure returned by the PMT install/upgrade commands
  • fix the unit tests broken by the PMT install/upgrade code changes

AFAIKT that should address all the comments on this pull request, except for the request to add new unit tests,
which I'll add in another commit.

Contributor

mruzicka commented Mar 15, 2013

I've added 3 new commits which bring the following changes:

  • add the requested YARD and fix the typo mentioned in one of the previous comments
  • replace the rubygems tar code with minitar implementation, which in turn is embedded into puppet at the Puppet::Util::Archive::Tar::Minitar namespace.
  • improve backward compatibility of the data structure returned by the PMT install/upgrade commands
  • fix the unit tests broken by the PMT install/upgrade code changes

AFAIKT that should address all the comments on this pull request, except for the request to add new unit tests,
which I'll add in another commit.

@adrienthebo

This comment has been minimized.

Show comment
Hide comment
@adrienthebo

adrienthebo Mar 18, 2013

Contributor

Tests are passing and the use of minitar looks good by me, 👍 when we get unit test coverage.

Contributor

adrienthebo commented Mar 18, 2013

Tests are passing and the use of minitar looks good by me, 👍 when we get unit test coverage.

+ destination_file.chmod(mode)
+ when entry.file?
+ destination_file.unlink() if (destination_file.exist? || destination_file.symlink?)
+ destination_file.open(File::WRONLY|File::CREAT|File::EXCL, mode) do |f|

This comment has been minimized.

@joshcooper

joshcooper Mar 19, 2013

Member

@mruzicka I think we want to use puppet's built-in Puppet::Util.replace_file method to ensure we write files securely and atomically (we write to a tempfile and then rename)

@joshcooper

joshcooper Mar 19, 2013

Member

@mruzicka I think we want to use puppet's built-in Puppet::Util.replace_file method to ensure we write files securely and atomically (we write to a tempfile and then rename)

This comment has been minimized.

@mruzicka

mruzicka Apr 2, 2013

Contributor

This is in fact a bit paranoid as the entire module is unpacked to an empty temporary directory so we should never be overwriting any files unless the tar contains duplicate entries, in which case I just make sure the last one wins.

@mruzicka

mruzicka Apr 2, 2013

Contributor

This is in fact a bit paranoid as the entire module is unpacked to an empty temporary directory so we should never be overwriting any files unless the tar contains duplicate entries, in which case I just make sure the last one wins.

+ tar.each do |entry|
+ destination_file = Pathname.new(entry.full_name).cleanpath
+ if destination_file.absolute? ||
+ (destination_file_path = destination_file.to_s).start_with?('..') &&

This comment has been minimized.

@joshcooper

joshcooper Mar 19, 2013

Member

@mruzicka I'm not sure this is sufficient, what about a malicious path of ./../../? The rubygems' tar code has similar logic for checking that we don't escape the module directory, how do they handle it?

@joshcooper

joshcooper Mar 19, 2013

Member

@mruzicka I'm not sure this is sufficient, what about a malicious path of ./../../? The rubygems' tar code has similar logic for checking that we don't escape the module directory, how do they handle it?

This comment has been minimized.

@mruzicka

mruzicka Apr 2, 2013

Contributor

The Pathanme.cleanpath is supposed to canonicalize such malicious paths so the test for .. should be sufficient.
The rubygems' tar code uses similar approach utilizing the File.expand_path to canonicalize the path and subsequently making sure the start of the path matches the expected install directory, see here:
https://github.com/rubygems/rubygems/blob/master/lib/rubygems/package.rb#L368-L378

@mruzicka

mruzicka Apr 2, 2013

Contributor

The Pathanme.cleanpath is supposed to canonicalize such malicious paths so the test for .. should be sufficient.
The rubygems' tar code uses similar approach utilizing the File.expand_path to canonicalize the path and subsequently making sure the start of the path matches the expected install directory, see here:
https://github.com/rubygems/rubygems/blob/master/lib/rubygems/package.rb#L368-L378

+# This file contains an adaptation of Ruby/ProgressBar by Satoru
+# Takabayashi <satoru@namazu.org>, copyright 2001 - 2004.
+#
+# It is licensed under the GNU General Public Licence or Ruby's licence.

This comment has been minimized.

@joshcooper

joshcooper Mar 19, 2013

Member

@mruzicka can you check with the release team (@haus or @stahnma) to ensure the license is compatible with puppet's?

@joshcooper

joshcooper Mar 19, 2013

Member

@mruzicka can you check with the release team (@haus or @stahnma) to ensure the license is compatible with puppet's?

This comment has been minimized.

@haus

haus Mar 25, 2013

Contributor

The ruby license is Apache compatible, it should be fine.

@haus

haus Mar 25, 2013

Contributor

The ruby license is Apache compatible, it should be fine.

@haus

This comment has been minimized.

Show comment
Hide comment
@haus

haus Mar 25, 2013

Contributor

We're working on a document to explain the preferred/accepted method and workflow of vendoring code in our projects. We'll have it ready for review in the next couple of days. Please hold on merging this until we have that solidified. Thanks!

Contributor

haus commented Mar 25, 2013

We're working on a document to explain the preferred/accepted method and workflow of vendoring code in our projects. We'll have it ready for review in the next couple of days. Please hold on merging this until we have that solidified. Thanks!

mruzicka added some commits Jan 15, 2013

mruzicka
(#15841,#13542) PMT install/upgrade from a local tarball & embed Minitar
This patch should really be two separate ones but because of it its
gradual changes it ended up as a single large one. It introduces
two principal changes:
* Adds support to PMT for installing/upgrading puppet modules
  from local tarballs. This support depends on a new forge API
  introduced in:
    https://github.com/puppetlabs/puppet-forge-api/pull/32
  In addition to the posibility of installing modules from tarballs a new
  algorithm for module dependency resolution is also introduced.
  The algorithm is capable of backtracking so is able to find resolutions
  even in situations in which the original algorithm would have failed.
  Many of the data structures used by the depedecy resolution code have
  been changed but care has been taken to preserve backward compatiblity
  of any externally visible data structures.
  Also the installer and upgrader applications have been greatly unified,
  so they now share the majority of their code.

* Embeds the minitar gem into Puppet while changing its namespace to:
    Puppet::Util::Archive::Tar::Minitar
  and makes use of it at various places in the PMT code instead of
  calling external tar utilities.
mruzicka
(#13542) New and updated acceptance tests for the PMT tarball install
This patch adds new and updates original acceptance tests to keep up
with the changes in PMT which now uses the ebeded minitar to work with
the module packages instead of calling external tar commands and has
gained the capability to install modules from tarballs.
mruzicka
(#13542) New and updated unit tests for the PMT tarball install
This patch adds new and updates the original unit tests for the PMT
install/upgrade code which now uses the ebeded minitar to work with
the module packages instead of calling external tar commands and has
gained the capability to install modules from tarballs.
@mruzicka

This comment has been minimized.

Show comment
Hide comment
@mruzicka

mruzicka Apr 4, 2013

Contributor

I've rebased this pull request on current master and while at it I have removed all references to the use of the rubygems' tar code as the pull request ended up embedding Minitar instead.

Contributor

mruzicka commented Apr 4, 2013

I've rebased this pull request on current master and while at it I have removed all references to the use of the rubygems' tar code as the pull request ended up embedding Minitar instead.

@joshcooper

This comment has been minimized.

Show comment
Hide comment
@joshcooper

joshcooper Apr 15, 2013

Member

@mruzicka (cc @khussey @adreyer @zaphod42 @RColeman) Sorry for the back and forth on this. We have some concerns about the changes made to the vendored minitar to handle symbolic links and permissions. At the same time, we need to get puppet module tool on windows into Puppet 3.2 RC, which will go into PE3. So we've decided to continue using native commands (tar, gzip) for most platforms, but fall back to minitar on windows. This way we are not blocked on making minitar work on all platforms.

I am working on a branch to add minitar support on windows will be submitting a PR today, so I am going to close this PR. Can you split out your changes to install from local tarball changes(#13542) and submit a new PR?

Member

joshcooper commented Apr 15, 2013

@mruzicka (cc @khussey @adreyer @zaphod42 @RColeman) Sorry for the back and forth on this. We have some concerns about the changes made to the vendored minitar to handle symbolic links and permissions. At the same time, we need to get puppet module tool on windows into Puppet 3.2 RC, which will go into PE3. So we've decided to continue using native commands (tar, gzip) for most platforms, but fall back to minitar on windows. This way we are not blocked on making minitar work on all platforms.

I am working on a branch to add minitar support on windows will be submitting a PR today, so I am going to close this PR. Can you split out your changes to install from local tarball changes(#13542) and submit a new PR?

@joshcooper joshcooper closed this Apr 15, 2013

@ryanycoleman

This comment has been minimized.

Show comment
Hide comment
@ryanycoleman

ryanycoleman May 6, 2013

Member

Hi @joshcooper, I'm trying to figure out where this work stands. Did you end up making a new PR?

Member

ryanycoleman commented May 6, 2013

Hi @joshcooper, I'm trying to figure out where this work stands. Did you end up making a new PR?

@ryanycoleman

This comment has been minimized.

Show comment
Hide comment
@ryanycoleman

ryanycoleman May 10, 2013

Member

@joshcooper, @ahpook, Could someone point me at the new PR for this work? I'm specifically interested in whether it's still a candidate for Puppet destined for PE 3.

Member

ryanycoleman commented May 10, 2013

@joshcooper, @ahpook, Could someone point me at the new PR for this work? I'm specifically interested in whether it's still a candidate for Puppet destined for PE 3.

@ahpook

This comment has been minimized.

Show comment
Hide comment
@ahpook

ahpook May 10, 2013

Contributor

@RColeman the minitar on windows support is in as a targeted fix , see pull #1603 via redmine 11276

Contributor

ahpook commented May 10, 2013

@RColeman the minitar on windows support is in as a targeted fix , see pull #1603 via redmine 11276

@ryanycoleman

This comment has been minimized.

Show comment
Hide comment
@ryanycoleman

ryanycoleman May 10, 2013

Member

On Fri, May 10, 2013 at 11:21 AM, Eric Sorenson notifications@github.comwrote:

@RColeman https://github.com/rcoleman the minitar on windows support is
in as a targeted fix , see pull #1603https://github.com/puppetlabs/puppet/issues/1603via redmine
11276 http://projects.puppetlabs.com/issues/11276

Fantastic. Thank you both!

Member

ryanycoleman commented May 10, 2013

On Fri, May 10, 2013 at 11:21 AM, Eric Sorenson notifications@github.comwrote:

@RColeman https://github.com/rcoleman the minitar on windows support is
in as a targeted fix , see pull #1603https://github.com/puppetlabs/puppet/issues/1603via redmine
11276 http://projects.puppetlabs.com/issues/11276

Fantastic. Thank you both!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment