Gems not found in pre-release #254

dan-wittenberg · 2017-08-14T21:45:24Z

$ pdk --version
1.0.0.pre

$ pdk validate
pdk (DEBUG): PDK package installation not found, trying 'bundle' from the system PATH instead
pdk (DEBUG): Executing 'bundle binstubs rubocop --force'
pdk (DEBUG): Execution of 'bundle binstubs rubocop --force' complete (duration: 0.39786395s; exit code: 7)
pdk (FATAL): Failed to generate binstubs for 'rubocop':
Could not find gem 'rubocop'.

$ gem list rubocop

*** LOCAL GEMS ***

rubocop (0.49.1)

DavidS · 2017-08-15T13:44:45Z

Dan, was this addressed, or caused by #249? Can you retest with current master, or the released 1.0.0 gem?

dan-wittenberg · 2017-08-15T13:50:35Z

$ gem list pdk

*** LOCAL GEMS ***

pdk (1.0.0)

$ pdk validate
pdk (INFO): Running all available validators...
[✔] Checking metadata syntax (metadata.json)
[✖] Checking metadata style (metadata.json)
[✖] Checking Puppet manifest syntax (/.pp)
pdk (FATAL): Failed to generate binstubs for 'rubocop':
Could not find gem 'rubocop'.

pdk (FATAL): Unable to install requested binstubs.

$ gem list rubocop

*** LOCAL GEMS ***

rubocop (0.49.1)

DavidS · 2017-08-15T14:15:51Z

Errr... apologies, I should have noticed that earlier. The rubocop gem doesn't need to be installed in the system gems. All requirements will be pulled in through the module's Gemfile, which should be correctly set up by pdk new module.

If you're available, I'd love to talk to you for a few minutes about your use-case, so that we can ensure future planning catches up with your requirements.

Local repro:

david@davids:~/tmp/testmod$ gem list pdk

*** LOCAL GEMS ***

pdk (1.0.0)
david@davids:~/tmp/testmod$ /usr/local/bin/pdk validate --debug
pdk (INFO): Running all available validators...
pdk (DEBUG): Checking for missing Gemfile dependencies.
pdk (DEBUG): PDK package installation not found, trying 'bundle' from the system PATH instead
pdk (DEBUG): Executing 'bundle check --gemfile=/home/david/tmp/testmod/Gemfile --path=/home/david/.pdk/cache'
pdk (DEBUG): Execution of 'bundle check --gemfile=/home/david/tmp/testmod/Gemfile --path=/home/david/.pdk/cache' complete (duration: 0.14162319s; exit code: 0)
pdk (DEBUG): Executing '/home/david/tmp/testmod/bin/metadata-json-lint --format json --strict-dependencies /home/david/tmp/testmod/metadata.json'
pdk (DEBUG): Execution of '/home/david/tmp/testmod/bin/metadata-json-lint --format json --strict-dependencies /home/david/tmp/testmod/metadata.json' complete (duration: 0.221624836s; exit code: 0)
pdk (DEBUG): Executing '/home/david/tmp/testmod/bin/puppet parser validate /home/david/tmp/testmod/manifests/init.pp /home/david/tmp/testmod/manifests/testclass.pp'
pdk (DEBUG): Execution of '/home/david/tmp/testmod/bin/puppet parser validate /home/david/tmp/testmod/manifests/init.pp /home/david/tmp/testmod/manifests/testclass.pp' complete (duration: 0.918874039s; exit code: 0)
pdk (DEBUG): Executing '/home/david/tmp/testmod/bin/puppet-lint --json /home/david/tmp/testmod/manifests/init.pp /home/david/tmp/testmod/manifests/testclass.pp'
pdk (DEBUG): Execution of '/home/david/tmp/testmod/bin/puppet-lint --json /home/david/tmp/testmod/manifests/init.pp /home/david/tmp/testmod/manifests/testclass.pp' complete (duration: 0.172973057s; exit code: 0)
pdk (DEBUG): Executing '/home/david/tmp/testmod/bin/rubocop --format json /home/david/tmp/testmod/spec/spec_helper.rb /home/david/tmp/testmod/spec/classes/testmod_spec.rb /home/david/tmp/testmod/spec/classes/testclass_spec.rb'
pdk (DEBUG): Execution of '/home/david/tmp/testmod/bin/rubocop --format json /home/david/tmp/testmod/spec/spec_helper.rb /home/david/tmp/testmod/spec/classes/testmod_spec.rb /home/david/tmp/testmod/spec/classes/testclass_spec.rb' complete (duration: 0.605709233s; exit code: 0)
david@davids:~/tmp/testmod$ gem list rubocop

*** LOCAL GEMS ***


david@davids:~/tmp/testmod$

bittner · 2017-09-09T08:33:50Z

I have a similar issue, albeit with the new version 1.0.1 of pdk:

$ pdk validate
pdk (INFO): Running all available validators...
[✖] Installing missing Gemfile dependencies
pdk (FATAL): Fetching gem metadata from https://rubygems.org/...........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
You have requested:
  puppet ~> 3.8

The bundle currently has puppet locked at 5.0.1.
Try running `bundle update puppet`

If you are updating multiple gems in your Gemfile at once,
try passing them all to `bundle update`


pdk (FATAL): Unable to install missing Gemfile dependencies.
$ pdk --version
1.0.1

bundle command "missing" too

There is no command bundle available for execution in my path. I can see that it's present in some way in the pdk installation folder, though:

$ find /opt/puppetlabs/pdk/ -name bundle
/opt/puppetlabs/pdk/private/ruby/2.1.9/bin/bundle
/opt/puppetlabs/pdk/private/ruby/2.1.9/lib/ruby/gems/2.1.0/gems/bundler-1.15.1/exe/bundle
/opt/puppetlabs/pdk/share/cache/ruby/2.1.0/bin/bundle
/opt/puppetlabs/pdk/share/cache/ruby/2.1.0/gems/bundler-1.15.1/exe/bundle

When I run the suggested bundle command from above (as an unprivileged user) I do get a dependency error:

$ /opt/puppetlabs/pdk/private/ruby/2.1.9/bin/bundle update puppet
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies...
Bundler could not find compatible versions for gem "puppet-lint":
  In snapshot (Gemfile.lock):
    puppet-lint (= 2.3.0)

  In Gemfile:
    puppet-lint-appends-check was resolved to 0.1.0, which depends on
      puppet-lint (~> 1.0)

    puppetlabs_spec_helper was resolved to 2.3.2, which depends on
      puppet-lint (~> 2.0)

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

Looks like there are a few issues to fix to make this work seamlessly.

(Sorry that I can't be more specific in suggesting a solution as I'm not too practical with Ruby!)

My use case

I'm really just trying to lint a puppet module that I'm modifying (without installing puppet-lint via apt beforehand).

The system I'm working on is a rather freshly installed Ubuntu 17.04 Zesty box (with Bash as a default shell and a Cinnamon desktop).

bittner · 2017-09-09T08:43:49Z

More trouble-shooting on the above issue with bundle:

Running bundle update (which asks for root permissions in the course of its execution) leads to the next error:

$ /opt/puppetlabs/pdk/private/ruby/2.1.9/bin/bundle update
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies.......................................................................................................
Fetching rake 12.0.0
Fetching json 1.8.6 (was 1.8.1)
Fetching addressable 2.4.0 (was 2.5.1)


Your user account isn't allowed to install to the system RubyGems.
  You can cancel this installation and run:

      bundle install --path vendor/bundle

  to install the gems into ./vendor/bundle/, or you can enter your password
  and install the bundled gems to RubyGems using sudo.

  Password: 

Installing ... (COMMENT: several packages are being listed)

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    /opt/puppetlabs/pdk/private/ruby/2.1.9/bin/ruby extconf.rb
checking if the C compiler accepts ... yes
Building nokogiri using packaged libraries.
Using mini_portile version 2.2.0
checking for gzdopen() in -lz... no
zlib is missing; necessary for building libxml2
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/opt/puppetlabs/pdk/private/ruby/2.1.9/bin/ruby
	--help
	--clean
	--use-system-libraries
	--enable-static
	--disable-static
	--with-zlib-dir
	--without-zlib-dir
	--with-zlib-include
	--without-zlib-include=${zlib-dir}/include
	--with-zlib-lib
	--without-zlib-lib=${zlib-dir}/lib
	--enable-cross-build
	--disable-cross-build

extconf failed, exit code 1

Gem files will remain installed in /tmp/bundler20170909-9501-6hh7qunokogiri-1.8.0/gems/nokogiri-1.8.0 for inspection.
Results logged to /tmp/bundler20170909-9501-6hh7qunokogiri-1.8.0/extensions/x86_64-linux/2.1.0/nokogiri-1.8.0/gem_make.out

An error occurred while installing nokogiri (1.8.0), and Bundler cannot continue.
Make sure that `gem install nokogiri -v '1.8.0'` succeeds before bundling.

In Gemfile:
  beaker-rspec was resolved to 5.6.0, which depends on
    beaker was resolved to 2.52.0, which depends on
      aws-sdk-v1 was resolved to 1.67.0, which depends on
        nokogiri

DavidS · 2017-09-11T08:07:29Z

Please use pdk bundle to run arbitrary commands in the PDK's ruby in the context of a module.
The PDK ships a version of nokogiri that you can use. See 1. for how to execute in an environment where it's available.
To unconfuse bundler, please remove .bundle/config and the Gemfile.lock from your module before running pdk.

bittner · 2017-09-11T09:09:11Z

@DavidS Was this addressed to me?

I'm just reporting what doesn't work from the perspective of an end-user. I'm neither a PDK nor a Ruby developer, remember.

Bottom-line: PDK is useless for me and my team in the current state (apart from being able to run pdk new, which worked for us). We have to install more packages in addition and run a workflow which is probably different than envisioned by you guys (i.e. using puppet-lint & friends locally and on the CI server).

IIUC and you implicitly suggest a PDK-only workflow, that deserves being fixed. (Call it "a usability issue", but a serious one.)

DavidS · 2017-09-11T09:21:02Z

@bittner yes, that was adressed to you.

The pdk has a bundle subcommand that allows you to run arbitrarty commands, including puppet-lint with PDK's ruby in the context of your module:

david@davids:~/tmp/foo$ pdk --debug bundle exec puppet-lint manifests/init.pp 
pdk (DEBUG): Using 'private/ruby/2.1.9/bin/bundle' from PDK package
pdk (DEBUG): Executing '/opt/puppetlabs/pdk/private/ruby/2.1.9/bin/bundle exec puppet-lint manifests/init.pp'
pdk (DEBUG): Execution of '/opt/puppetlabs/pdk/private/ruby/2.1.9/bin/bundle exec puppet-lint manifests/init.pp' complete (duration: 0.353587791s; exit code: 1)
ERROR: two-space soft tabs not used on line 12

(Using --debug only for demonstration purposes here)

I've checked nokogiri, and we do not ship a copy of that anymore, my apologies for confusing that.

I also feel compelled to point out that neither puppet 3.8, nor beaker 2 will receive any official updates, testing, or support from our side.

bittner · 2017-09-11T15:13:19Z

I also feel compelled to point out that neither puppet 3.8, nor beaker 2 will receive any official updates, testing, or support from our side.

I have no idea what this means. I have not chosen to install those components. Those versions get installed automatically with either the base system (we're running a fresh Foreman installation) or the PDK.

Our own Puppet control repository is actually targeted at somewhat the latest Puppet version that corresponds to stable PE (I thought that was 4.x).

DavidS · 2017-09-11T15:16:20Z

That is very curious, since the logs you've posted show both versions being requested by your module. Can you (privately, even) hook me up with a copy of the module you are working on?

bittner · 2017-09-11T21:11:15Z

You may be right. I think I produced the output above with https://github.com/edestecd/puppet-software

Though, that sounds like a different issue. Originally, I had the same or a very similar problem with our control repository.

DavidS · 2017-09-12T09:44:48Z

I looked at that repo, and its Gemfile does things the PDK is not prepared to handle. In this particular case:

hardcoding a default puppet version pin
carrying a puppet-lint plugin that is not compatible with puppet-lint 2 (puppet-lint-resource_outside_class-check).

We're planning to provide an update feature to make converting existing modules to the PDK easier than manually copying over files (https://tickets.puppetlabs.com/browse/PDK-428), as well as support testing against multiple puppet versions as a top-level feature (https://tickets.puppetlabs.com/browse/PDK-414).

bmjen · 2018-03-21T23:17:06Z

Going to close this issue. The convert and update features added to PDK should help transition older modules to be used with PDK. Those that can't be changed in that manner with the default template, can fork and customize the pdk-templates at https://github.com/puppetlabs/pdk-templates

bittner mentioned this issue Sep 12, 2017

Fix or silence linter error "not in autoload module layout" edestecd/puppet-software#5

Open

bmjen closed this as completed Mar 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gems not found in pre-release #254

Gems not found in pre-release #254

dan-wittenberg commented Aug 14, 2017

DavidS commented Aug 15, 2017

dan-wittenberg commented Aug 15, 2017

DavidS commented Aug 15, 2017 •

edited

Loading

bittner commented Sep 9, 2017

bittner commented Sep 9, 2017

DavidS commented Sep 11, 2017

bittner commented Sep 11, 2017

DavidS commented Sep 11, 2017

bittner commented Sep 11, 2017

DavidS commented Sep 11, 2017

bittner commented Sep 11, 2017

DavidS commented Sep 12, 2017

bmjen commented Mar 21, 2018

Gems not found in pre-release #254

Gems not found in pre-release #254

Comments

dan-wittenberg commented Aug 14, 2017

DavidS commented Aug 15, 2017

dan-wittenberg commented Aug 15, 2017

DavidS commented Aug 15, 2017 • edited Loading

bittner commented Sep 9, 2017

bundle command "missing" too

My use case

bittner commented Sep 9, 2017

DavidS commented Sep 11, 2017

bittner commented Sep 11, 2017

DavidS commented Sep 11, 2017

bittner commented Sep 11, 2017

DavidS commented Sep 11, 2017

bittner commented Sep 11, 2017

DavidS commented Sep 12, 2017

bmjen commented Mar 21, 2018

DavidS commented Aug 15, 2017 •

edited

Loading