run fails complaining that a puppet:/// file resource cannot be found

[systeemkabouter]

When running the gem or git version of the tool, it fails when it encounters a file resource with a source => puppet:/// construct.

My setup had 2 module dirs : modules/ and site/ site/ comes from git and modules/ is setup during bootstrap downloading modules using r10k.

The puppetcode is working nicely in production, the file is actually there in git, I doublechecked.

You probably need more info, but I'm not sure what, so please advise.

Command used and debugging output

vagrant@ptester:/vagrant/environments/octocatalog-diff$ /vagrant/octocatalog-diff/bin/octocatalog-diff -f production -t fea-puppetv4 -n [REDACTED] --fact-file /vagrant/puppetdb_facts/[REDACTED].yaml --to-fact-override vagrant_puppetrole=nagiosserver --from-fact-override vagrant_puppetrole=nagiosserver --bootstrap-script repo-bootstrap.sh -d --suppress-absent-file-details
D, [2016-11-03T03:48:52.976966 #28239] DEBUG -- : Running octocatalog-diff 0.5.3 with ruby 2.3.1
D, [2016-11-03T03:48:52.977280 #28239] DEBUG -- : Command line arguments: ["-f", "production", "-t", "fea-puppetv4", "-n", "[REDACTED]", "--fact-file", "/vagrant/puppetdb_facts/[REDACTED].yaml", "--to-fact-override", "vagrant_puppetrole=nagiosserver", "--from-fact-override", "vagrant_puppetrole=nagiosserver", "--bootstrap-script", "repo-bootstrap.sh", "-d", "--suppress-absent-file-details"]
D, [2016-11-03T03:48:52.977567 #28239] DEBUG -- : Running on host ptester.localdomain (x86_64-linux-gnu)
D, [2016-11-03T03:48:52.978009 #28239] DEBUG -- : Initialized OctocatalogDiff::Catalog::Computed for from-catalog
D, [2016-11-03T03:48:52.978284 #28239] DEBUG -- : Initialized OctocatalogDiff::Catalog::Computed for to-catalog
D, [2016-11-03T03:48:52.978495 #28239] DEBUG -- : Initialized parallel task result array: size=2
D, [2016-11-03T03:48:52.980954 #28241] DEBUG -- : Begin build_catalog for production
D, [2016-11-03T03:48:52.981330 #28241] DEBUG -- : Setting up Puppet catalog build for production
D, [2016-11-03T03:48:52.981614 #28241] DEBUG -- : Catalog for production will be built with OctocatalogDiff::Catalog::Computed
D, [2016-11-03T03:48:52.981999 #28241] DEBUG -- : Calling build for object OctocatalogDiff::Catalog::Computed
D, [2016-11-03T03:48:52.982377 #28241] DEBUG -- : Start retrieving facts for [REDACTED] from OctocatalogDiff::Catalog::Computed
D, [2016-11-03T03:48:52.982664 #28241] DEBUG -- : Success retrieving facts for [REDACTED] from OctocatalogDiff::Catalog::Computed
D, [2016-11-03T03:48:52.983370 #28241] DEBUG -- : Begin git checkout /vagrant/environments/octocatalog-diff:production -> /tmp/d20161103-28241-1tett3x
D, [2016-11-03T03:48:52.984352 #28241] DEBUG -- : Begin git archive /vagrant/environments/octocatalog-diff:production -> /tmp/d20161103-28241-1tett3x
D, [2016-11-03T03:48:53.076234 #28241] DEBUG -- : Success git archive /vagrant/environments/octocatalog-diff:production
D, [2016-11-03T03:48:53.076551 #28241] DEBUG -- : Success git checkout /vagrant/environments/octocatalog-diff:production -> /tmp/d20161103-28241-1tett3x
D, [2016-11-03T03:48:53.076953 #28241] DEBUG -- : Begin install bootstrap script in target directory
D, [2016-11-03T03:48:53.080222 #28241] DEBUG -- : Success: copied /vagrant/environments/octocatalog-diff/repo-bootstrap.sh to /tmp/d20161103-28241-1tett3x/repo-bootstrap.sh
D, [2016-11-03T03:48:53.080379 #28241] DEBUG -- : Begin bootstrap with 'repo-bootstrap.sh' in /tmp/d20161103-28241-1tett3x
D, [2016-11-03T03:50:10.720551 #28241] DEBUG -- : Success bootstrap in /tmp/d20161103-28241-1tett3x
D, [2016-11-03T03:50:10.721492 #28241] DEBUG -- : Symlinked /tmp/d20161103-28241-148qf2a/environments/production -> /tmp/d20161103-28241-1tett3x
D, [2016-11-03T03:50:10.723933 #28241] DEBUG -- : Installed hiera.yaml from /tmp/d20161103-28241-148qf2a/environments/production/hiera.yaml to /tmp/d20161103-28241-148qf2a/hiera.yaml
D, [2016-11-03T03:50:10.724408 #28241] DEBUG -- : Override vagrant_puppetrole from nil to "nagiosserver"
D, [2016-11-03T03:50:10.729859 #28241] DEBUG -- : Installed fact file at /tmp/d20161103-28241-148qf2a/var/yaml/facts/[REDACTED]l.yaml
D, [2016-11-03T03:50:10.730572 #28241] DEBUG -- : Installed SSL client certificate in /tmp/d20161103-28241-148qf2a/var/ssl/certs/ptester.localdomain.pem
D, [2016-11-03T03:50:10.730932 #28241] DEBUG -- : Installed SSL client key in /tmp/d20161103-28241-148qf2a/var/ssl/private_keys/ptester.localdomain.pem
D, [2016-11-03T03:50:10.733681 #28241] DEBUG -- : Installed CA certificate in /tmp/d20161103-28241-148qf2a/var/ssl/certs/ca.pem
D, [2016-11-03T03:50:11.208200 #28241] DEBUG -- : (from) Try 1 executing Puppet 3.8.5: /usr/bin/puppet master --compile [REDACTED] --no-storeconfigs --factpath=/tmp/d20161103-28241-148qf2a/var/yaml/facts --facts_terminus=yaml --no-daemonize --no-ca --color=false --config_version="/bin/echo catalogscript" --environment=production --hiera_config=/tmp/d20161103-28241-148qf2a/hiera.yaml --environmentpath=/tmp/d20161103-28241-148qf2a/environments --vardir=/tmp/d20161103-28241-148qf2a/var --logdir=/tmp/d20161103-28241-148qf2a/var --ssldir=/tmp/d20161103-28241-148qf2a/var/ssl --confdir=/tmp/d20161103-28241-148qf2a
D, [2016-11-03T03:50:12.796125 #28241] DEBUG -- : (from) Catalog succeeded on try 1 in 2.062129317 seconds
D, [2016-11-03T03:50:12.796739 #28241] DEBUG -- : Catalog for production successfully built with OctocatalogDiff::Catalog::Computed in 79.814747402 seconds retries = 0
D, [2016-11-03T03:50:12.796928 #28241] DEBUG -- : Success build_catalog for production
D, [2016-11-03T03:50:12.798665 #28239] DEBUG -- : Success build_catalog for production
D, [2016-11-03T03:50:12.799050 #28241] DEBUG -- : Begin build_catalog for fea-puppetv4
D, [2016-11-03T03:50:12.799277 #28241] DEBUG -- : Setting up Puppet catalog build for fea-puppetv4
D, [2016-11-03T03:50:12.799466 #28241] DEBUG -- : Catalog for fea-puppetv4 will be built with OctocatalogDiff::Catalog::Computed
D, [2016-11-03T03:50:12.799704 #28241] DEBUG -- : Calling build for object OctocatalogDiff::Catalog::Computed
D, [2016-11-03T03:50:12.799913 #28241] DEBUG -- : Start retrieving facts for [REDACTED] from OctocatalogDiff::Catalog::Computed
D, [2016-11-03T03:50:12.800138 #28241] DEBUG -- : Success retrieving facts for [REDACTED] from OctocatalogDiff::Catalog::Computed
D, [2016-11-03T03:50:12.800470 #28241] DEBUG -- : Begin git checkout /vagrant/environments/octocatalog-diff:fea-puppetv4 -> /tmp/d20161103-28241-722vfw
D, [2016-11-03T03:50:12.801073 #28241] DEBUG -- : Begin git archive /vagrant/environments/octocatalog-diff:fea-puppetv4 -> /tmp/d20161103-28241-722vfw
D, [2016-11-03T03:50:12.984256 #28241] DEBUG -- : Success git archive /vagrant/environments/octocatalog-diff:fea-puppetv4
D, [2016-11-03T03:50:12.984651 #28241] DEBUG -- : Success git checkout /vagrant/environments/octocatalog-diff:fea-puppetv4 -> /tmp/d20161103-28241-722vfw
D, [2016-11-03T03:50:12.985055 #28241] DEBUG -- : Begin install bootstrap script in target directory
D, [2016-11-03T03:50:12.987851 #28241] DEBUG -- : Success: copied /vagrant/environments/octocatalog-diff/repo-bootstrap.sh to /tmp/d20161103-28241-722vfw/repo-bootstrap.sh
D, [2016-11-03T03:50:12.988135 #28241] DEBUG -- : Begin bootstrap with 'repo-bootstrap.sh' in /tmp/d20161103-28241-722vfw
D, [2016-11-03T03:51:30.402657 #28241] DEBUG -- : Success bootstrap in /tmp/d20161103-28241-722vfw
D, [2016-11-03T03:51:30.403468 #28241] DEBUG -- : Symlinked /tmp/d20161103-28241-12v50sb/environments/production -> /tmp/d20161103-28241-722vfw
D, [2016-11-03T03:51:30.406119 #28241] DEBUG -- : Installed hiera.yaml from /tmp/d20161103-28241-12v50sb/environments/production/hiera.yaml to /tmp/d20161103-28241-12v50sb/hiera.yaml
D, [2016-11-03T03:51:30.406909 #28241] DEBUG -- : Override vagrant_puppetrole from nil to "nagiosserver"
D, [2016-11-03T03:51:30.413732 #28241] DEBUG -- : Installed fact file at /tmp/d20161103-28241-12v50sb/var/yaml/facts/[REDACTED].yaml
D, [2016-11-03T03:51:30.416047 #28241] DEBUG -- : Installed SSL client certificate in /tmp/d20161103-28241-12v50sb/var/ssl/certs/ptester.localdomain.pem
D, [2016-11-03T03:51:30.416411 #28241] DEBUG -- : Installed SSL client key in /tmp/d20161103-28241-12v50sb/var/ssl/private_keys/ptester.localdomain.pem
D, [2016-11-03T03:51:30.418888 #28241] DEBUG -- : Installed CA certificate in /tmp/d20161103-28241-12v50sb/var/ssl/certs/ca.pem
D, [2016-11-03T03:51:30.892639 #28241] DEBUG -- : (to) Try 1 executing Puppet 3.8.5: /usr/bin/puppet master --compile [REDACTED] --no-storeconfigs --factpath=/tmp/d20161103-28241-12v50sb/var/yaml/facts --facts_terminus=yaml --no-daemonize --no-ca --color=false --config_version="/bin/echo catalogscript" --environment=production --hiera_config=/tmp/d20161103-28241-12v50sb/hiera.yaml --environmentpath=/tmp/d20161103-28241-12v50sb/environments --vardir=/tmp/d20161103-28241-12v50sb/var --logdir=/tmp/d20161103-28241-12v50sb/var --ssldir=/tmp/d20161103-28241-12v50sb/var/ssl --confdir=/tmp/d20161103-28241-12v50sb
D, [2016-11-03T03:51:34.123775 #28241] DEBUG -- : (to) Catalog succeeded on try 1 in 3.704408344 seconds
D, [2016-11-03T03:51:34.136377 #28241] DEBUG -- : Failed build_catalog for fea-puppetv4: Errno::ENOENT No such file or directory - Unable to find 'puppet:///modules/mitbackup/duplicity_0.7.10.deb' at /tmp/d20161103-28241-12v50sb/environments/production/modules/mitbackup/files/duplicity_0.7.10.deb!
D, [2016-11-03T03:51:34.163328 #28241] DEBUG -- : Cleaning up temporary directory /tmp/d20161103-28241-722vfw
D, [2016-11-03T03:51:34.399600 #28241] DEBUG -- : Cleaning up temporary directory /tmp/d20161103-28241-1tett3x
/vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:209:in add_parallel_result': Catalog for 'to' (fea-puppetv4) failed to compile with Errno::ENOENT: No such file or directory - Unable to find 'puppet:///modules/mitbackup/duplicity_0.7.10.deb' at /tmp/d20161103-28241-12v50sb/environments/production/modules/mitbackup/files/duplicity_0.7.10.deb! (OctocatalogDiff::CatalogDiff::Cli::Catalogs::CatalogError) /vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-util/fileresources.rb:63:in block in _convert_file_resources'
/vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-util/fileresources.rb:43:in map!' /vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-util/fileresources.rb:43:in _convert_file_resources'
/vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-util/fileresources.rb:16:in convert_file_resources' /vagrant/octocatalog-diff/lib/octocatalog-diff/catalog.rb:74:in build'
/vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:221:in build_catalog' /vagrant/octocatalog-diff/lib/octocatalog-diff/util/parallel.rb:26:in call'
/vagrant/octocatalog-diff/lib/octocatalog-diff/util/parallel.rb:26:in execute' /vagrant/octocatalog-diff/lib/octocatalog-diff/util/parallel.rb:114:in block in run_tasks_parallel'
/var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:451:in call_with_index' /var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:422:in process_incoming_jobs'
/var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:404:in block in worker' /var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:395:in fork'
/var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:395:in worker' /var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:386:in block in create_workers'
/var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:385:in each' /var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:385:in each_with_index'
/var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:385:in create_workers' /var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:327:in work_in_processes'
/var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:241:in map' /var/lib/gems/2.3.0/gems/parallel-1.9.0/lib/parallel.rb:203:in each'
/vagrant/octocatalog-diff/lib/octocatalog-diff/util/parallel.rb:90:in run_tasks_parallel' /vagrant/octocatalog-diff/lib/octocatalog-diff/util/parallel.rb:72:in run_tasks'
/vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:98:in build_catalog_parallelizer' /vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:32:in catalogs'
/vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-diff/cli.rb:104:in cli' /vagrant/octocatalog-diff/bin/octocatalog-diff:74:in

'
from /vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:118:in block in build_catalog_parallelizer' from /vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:112:in each'
from /vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:112:in build_catalog_parallelizer' from /vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-diff/cli/catalogs.rb:32:in catalogs'
from /vagrant/octocatalog-diff/lib/octocatalog-diff/catalog-diff/cli.rb:104:in cli' from /vagrant/octocatalog-diff/bin/octocatalog-diff:74:in '

Platform and version information

• Your OS: Ubuntu 16.04
• Your Ruby version: 2.3.1
• Your version of Puppet: OS 3.8.7
• Your version of octocatalog-diff: 0.5.3

Do the tests pass from a clean checkout?
No, see #11

Anything else to add that you think will be helpful?

[systeemkabouter]

As a workaround I symlink away in my bootstrap :

r10k puppetfile install Puppetfile
for i in site/*; do BLA=echo $i |sed -e 's#site/##'; ln -s ../site/$BLA modules/$BLA; done

[systeemkabouter]

At some point thinks worked OK when I applied the fix above and the parameter --suppress-absent-file-details. Without the (unrelated?) parameter the run would fail.

Now that I've just merged my working branch in to 'production/master' branch the issue seems to have disappeared. So it may have been some missing config in the ' master / production' branch that I recently added to my 'current' branch to get this tool to work. I've added a hiera.yaml, bootstrap script and .octocatalog-diff.cfg.rb to the git repo.

[kpaulisse]

You could use --no-compare-file-text as well, to stop the comparison of the text of files.

Could you let me know what the directory structure of your repository looks like -- do you have all of your modules under site or do you have them split between site and modules, etc.? We could probably get this working without the symlink by allowing you to specify your own module directory for octocatalog-diff, because right now this is hard-coded to "modules".

[systeemkabouter]

Our puppet uses the environment.conf file to set the modulepath for the puppet server. I guess it would make sense to let octocatalog-diff read the file too.

$ more environment.conf
modulepath = site:modules:$basemodulepath

[systeemkabouter]

We use a 'site/' or 'site_modules/' modules directory for site specific modules that are all contained in the puppet control repository together with hieradata. r10k managed modules end up in the 'modules/' dir.

[kpaulisse]

I guess it would make sense to let octocatalog-diff read the file too.

Seems like a reasonable approach. I'll see what I can do... 😸