Missing gem with ruby 3.1

[jas01]

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 7.23
• Ruby: 3.1
• Distribution: FreeBSD
• Module version: All

How to reproduce (e.g Puppet code you use)

Just run the puppet agent, if the module puppet-archive are installed the agent would failed

What are you seeing

When the puppet agent run (well start) it failed with this message

Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not autoload puppet/provider/archive/ruby: Unable to find archive module in modulepath /usr/local/etc/puppet/modules:/usr/local/etc/puppet/modules
Error: Could not autoload puppet/provider/archive/curl: Could not autoload puppet/provider/archive/ruby: Unable to find archive module in modulepath /usr/local/etc/puppet/modules:/usr/local/etc/puppet/modules
Error: Could not autoload puppet/type/archive: Could not autoload puppet/provider/archive/curl: Could not autoload puppet/provider/archive/ruby: Unable to find archive module in modulepath /usr/local/etc/puppet/modules:/usr/local/etc/puppet/modules
Error: Failed to apply catalog: Could not autoload puppet/type/archive: Could not autoload puppet/provider/archive/curl: Could not autoload puppet/provider/archive/ruby: Unable to find archive module in modulepath /usr/local/etc/puppet/modules:/usr/local/etc/puppet/modules

The reason was find by @smortex (big thanks to him) is with ruby 3.1 the net-ftp become a gem (check https://www.ruby-lang.org/en/news/2021/12/25/ruby-3-1-0-released/) the module puppet-archive need this gem. So without this gem installed the agent cannot run

What behaviour did you expect instead

The module puppet-archive should installed the gem so don't need to installed manually on each client (and because puppet agent without puppet-archive don't need net-ftp)

[alexjfisher]

@bastelfreak Which ruby is Puppet 8 going to ship with? If ruby 3.1 or higher, maybe the AIO packages should vendor this gem??

[bastelfreak]

Ruby 3.2

[bastelfreak]

https://stdgems.org/ net-ftp is a stdlib gem. The ruby package in freebsd is supposed to vendor this (or at least have a dependency on it, but tha causes other issues. I'm fighting with this on Arch Linux). If Puppet creates the AIO packages properly, the gem will be vendored. I recommend raising a bug on the freebsd site.

[smortex]

👋 @bastelfreak good catch! I saw there was a net-ftp gem on Rubygem and though this was removed from the gems shipped with Ruby. I was not aware of stdgems which shows this should still be bundled into the default ruby.

So this is a FreeBSD specific issue I think due to the way we package Ruby:
https://cgit.freebsd.org/ports/tree/lang/ruby31/pkg-message
There is a chicken and egg problem (using this specific net/ftp as an example, but the issue is the same for each gem listed in the link above):

• ftp/rubygem-net-ftp has a dependency on lang/ruby31;
• lang/ruby31 should have a dependency on ftp/rubygem-net-ftp (to match stdgems).

I wonder if we can try to lazy-load net-ftp only when trying to use it, and only cause a resource application failure if prerequisites are not found, instead of having a catalog compilation failure when net-ftp is not available as we have today? If we can do this, we can even have the archive module ensure net-ftp is installed, and in the worst case scenario, the configuration should be applied successfully after 2 puppet run.

[bastelfreak]

We tried really long on arch linux to ship each stdlib gem as own package. it's painful, it didn't work in the end and the ruby developers recommend vendoring the gems. I think that's the current goal for arch linux.

[david22swan]

Hey just checking if there's any movement on this?
Got hit by it on the ci while adding support for Puppet 8 to Java:
https://github.com/puppetlabs/puppetlabs-java/actions/runs/4598161712/jobs/8121839822?pr=548#step:6:33

This is on Ubuntu

[bastelfreak]

@david22swan are you sure that's related? The error message is a different one.

[david22swan]

hmmm, maybe not I guess, had zeroed in on the Could not autoload puppet/type/archive: Could not autoload puppet/provider/archive/curl: Could not autoload puppet/provider/archive/ruby so thought they had to be at least related, though you are right, the final error does differ.

[smortex]

Yet, the backtrace ends with:

     # --- Caused by: ---
     # LoadError:
     #   cannot load such file -- net/ftp
     #   ./spec/fixtures/modules/archive/lib/puppet_x/bodeco/util.rb:148:in `<class:FTP>'

🙄

@david22swan just to be sure, what happen when you add net-ftp to the Gemfile?

[david22swan]

@smortex Adding the gem got my test's running green

[smortex]

So as I understand it, Ruby 3.2 on GitHub Action does not include net-ftp, so when installing puppet as a gem it does not pull net-ftp and the archive code does not load properly. I'll try to investigate lazy loading net-ftp / not failing hard when the gem is not found to see if it can help and if it make sense to integrate it into the archive module.

[smortex]

Fun: when net-ftp is not installed but you prevent this failure, the archive module use curl if it is available to download the file over FTP. I'll open a PR to not fail hard when the net-ftp gem is not found so that it can rely on another system to download, and only if no download solution is found fail at catalog application time.

[smortex]

#491 should fix the issue