(PUP-12031) Consolidate gem depenendencies & metadata into puppet.gem…

[AriaXLi]

…spec

This PR:

• Moves puppet gem metadata from ext/project_yaml.data and .gemspec into puppet.gemspec so puppet gems can be built using gem build puppet.gemspec
• Adds conditional logic in puppet.gemspec for platform specific dependencies like ffi and CFPropertyList when building platform specific gems (using --platform)
• Adds Rake tasks that builds these puppet gems

[AriaXLi]

Not sure if enhancement or maintenance label is better?

[AriaXLi]

Successfully built gem using 'gem build puppet.gemspec'

❯ gem build puppet.gemspec
  Successfully built RubyGem
  Name: puppet
  Version: 8.7.0
  File: puppet-8.7.0.gem

❯ gem build puppet.gemspec --platform universal-darwin
  Successfully built RubyGem
  Name: puppet
  Version: 8.7.0
  File: puppet-8.7.0-universal-darwin.gem

❯ gem build puppet.gemspec --platform x64-mingw32
  Successfully built RubyGem
  Name: puppet
  Version: 8.7.0
  File: puppet-8.7.0-x64-mingw32.gem

  ❯ gem build puppet.gemspec --platform x86-mingw32
  Successfully built RubyGem
  Name: puppet
  Version: 8.7.0
  File: puppet-8.7.0-x86-mingw32.gem

[joshcooper]

Somehow bundle install on Linux used to install the CFPropertyList gem:

https://github.com/puppetlabs/puppet/actions/runs/8713544694/job/23902159199#step:4:90

I'm guessing it's due to the all platform in

puppet/ext/project_data.yaml

Line 49 in fb44fd9

universal-darwin: all

and

puppet/.gemspec

Line 67 in fb44fd9

if bundle_platform == "all"

but not 100% sure.

To resolve this, I think we want to guard the directoryservice_spec tests so they only run when the CFPropertyList feature is present like we do in

puppet/spec/unit/util/plist_spec.rb

Line 7 in fb44fd9

describe Puppet::Util::Plist, :if => Puppet.features.cfpropertylist? do

Also add the CFPropertyList gem in the features group in the Gemfile like we do for diff-lcs

puppet/Gemfile

Line 22 in fb44fd9

gem 'diff-lcs', '~> 1.3', require: false

You might also need to append platforms: [:ruby] or possibly platforms: [:ruby, :darwin] to the gem 'CFPropertyList', so that the we don't bother installing the gem or running those tests on Windows or JRuby.

[AriaXLi]

:if => Puppet.features.cfpropertylist? do

Thanks for the advice, quick q: what does adding CFPropertyList gem to the features group do specifically? Are those gems only installed on specific platforms?

[AriaXLi]

$ bundle exec rake -T
...
rake pl_ci:gem_build                                       # Build puppet gems
rake rubocop                                               # run static analysis with rubocop
rake warnings                                              # verify that changed files are clean of Ruby warnings

[AriaXLi]

❯ bundle exec rake pl_ci:gem_build
gem build puppet.gemspec --platform x86-mingw32
  Successfully built RubyGem
  Name: puppet
  Version: 8.7.0
  File: puppet-8.7.0-x86-mingw32.gem
gem build puppet.gemspec --platform x64-mingw32
  Successfully built RubyGem
  Name: puppet
  Version: 8.7.0
  File: puppet-8.7.0-x64-mingw32.gem
gem build puppet.gemspec --platform universal-darwin
  Successfully built RubyGem
  Name: puppet
  Version: 8.7.0
  File: puppet-8.7.0-universal-darwin.gem
gem build puppet.gemspec
  Successfully built RubyGem
  Name: puppet
  Version: 8.7.0
  File: puppet-8.7.0.gem

[AriaXLi]

Diffs to verify that the gems are being built correctly:

❯ diff <(gem specification puppet-8.6.0.gem) <(gem specification puppet-8.7.0.gem)
4c4
<   version: 8.6.0
---
>   version: 8.7.0
11c11
< date: 2024-04-09 00:00:00.000000000 Z
---
> date: 2012-08-17 00:00:00.000000000 Z
151c151,153
< description: Puppet, an automated configuration management tool
---
> description: |
>   Puppet, an automated administrative engine for your Linux, Unix, and Windows systems, performs administrative tasks
>   (such as adding users, installing packages, and updating server configurations) based on a centralized specification.

❯ diff <(gem specification puppet-8.6.0-x86-mingw32.gem) <(gem specification puppet-8.7.0-x86-mingw32.gem)
4c4
<   version: 8.6.0
---
>   version: 8.7.0
11c11
< date: 2024-04-09 00:00:00.000000000 Z
---
> date: 2012-08-17 00:00:00.000000000 Z
179c179,181
< description: Puppet, an automated configuration management tool
---
> description: |
>   Puppet, an automated administrative engine for your Linux, Unix, and Windows systems, performs administrative tasks
>   (such as adding users, installing packages, and updating server configurations) based on a centralized specification.

❯ diff <(gem specification puppet-8.6.0-x64-mingw32.gem) <(gem specification puppet-8.7.0-x64-mingw32.gem)
4c4
<   version: 8.6.0
---
>   version: 8.7.0
11c11
< date: 2024-04-09 00:00:00.000000000 Z
---
> date: 2012-08-17 00:00:00.000000000 Z
179c179,181
< description: Puppet, an automated configuration management tool
---
> description: |
>   Puppet, an automated administrative engine for your Linux, Unix, and Windows systems, performs administrative tasks
>   (such as adding users, installing packages, and updating server configurations) based on a centralized specification.

❯ diff <(gem specification puppet-8.6.0-universal-darwin.gem) <(gem specification puppet-8.7.0-universal-darwin.gem)
4c4
<   version: 8.6.0
---
>   version: 8.7.0
11c11
< date: 2024-04-09 00:00:00.000000000 Z
---
> date: 2012-08-17 00:00:00.000000000 Z
155c155
<     - - "~>"
---
>     - - ">="
157c157,160
<         version: '2.2'
---
>         version: 3.0.6
>     - - "<"
>       - !ruby/object:Gem::Version
>         version: '4'
162c165
<     - - "~>"
---
>     - - ">="
164,165c167,173
<         version: '2.2'
< description: Puppet, an automated configuration management tool
---
>         version: 3.0.6
>     - - "<"
>       - !ruby/object:Gem::Version
>         version: '4'
> description: |
>   Puppet, an automated administrative engine for your Linux, Unix, and Windows systems, performs administrative tasks
>   (such as adding users, installing packages, and updating server configurations) based on a centralized specification.

The difference in the universal-darwin gem is because in this PR, the puppet.gemspec was updated to use a newer version of CFPropertyList for universal-darwin: gem 'CFPropertyList', ['>= 3.0.6', '< 4'], require: false

❯ gem specification puppet-8.6.0-universal-darwin.gem
- !ruby/object:Gem::Dependency
  name: CFPropertyList
  requirement: !ruby/object:Gem::Requirement
    requirements:
    - - "~>"
      - !ruby/object:Gem::Version
        version: '2.2'

❯ gem specification puppet-8.7.0-universal-darwin.gem
  name: CFPropertyList
 requirement: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: 3.0.6
   - - "<"
     - !ruby/object:Gem::Version
       version: '4'

[joshcooper]

what does adding CFPropertyList gem to the features group do specifically? Are those gems only installed on specific platforms?

We generally add gems to the features group for optional dependencies. For example, puppet has an optional dependency on the rdoc gem, which is used by the puppet doc application. If the gem is present, then the puppet doc functionality is enabled and we want to the rspec tests to exercise those behaviors.

Usually the need for features occurs because the gem we want to add contains native extensions (or one of its transitive dependencies does). If a gem contains native extensions and is added as a (hard) runtime dependency, but the author doesn't ship precompiled binaries, then gem install puppet will only work if you have a compiler and ruby headers and libraries (from ruby-dev/ruby-devel) installed. So it won't generally work on Windows.

FFI is an example of a gem that does provide precompiled binaries, which we rely on for Windows platforms. But many other gems aren't as thorough.

[AriaXLi]

When rake command fails:

❯ bundle exec rake pl_ci:gem_build
bundler: failed to load command: rake (/Users/aria.li/.rbenv/versions/3.1.1/bin/rake)
/Users/aria.li/.rbenv/versions/3.1.1/lib/ruby/gems/3.1.0/gems/bundler-2.3.13/lib/bundler/rubygems_integration.rb:72:in `rescue in validate': The gemspec at /Users/aria.li/gh_repos/puppet/puppet.gemspec is not valid. Please fix this gemspec. (Gem::InvalidSpecificationException)
The validation error was 'puppet-8.7.0 contains itself (puppet-8.7.0.gem), check your files list'

[github-actions]

Backport failed for 7.x, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git fetch origin 7.x
git worktree add -d .worktree/backport-9325-to-7.x origin/7.x
cd .worktree/backport-9325-to-7.x
git checkout -b backport-9325-to-7.x
ancref=$(git merge-base fb44fd90c64ee11f0a29fb8924adbab5b0695555 e58bc11b60c6781deb5b172e674cac405d4b4d6d)
git cherry-pick -x $ancref..e58bc11b60c6781deb5b172e674cac405d4b4d6d