Could not autoload puppet/provider/elasticsearch_user_file/oss_xpack

[Celedhrim]

• Module version: 6.3.3
• Puppet version: 4.10.12
• OS and version:Debian GNU/Linux 9.5 (stretch)

Bug description

Hello,

I have install elasticsearch 6.3.2 with puppet module version 6.2.2 but I have the scripts symlink bug.

So I have update elasticsearch module to 6.3.3.
I have made change on my manifest to take care of change ( oss + repository)

But now when I try to apply manifest :

 ~# puppet agent -t
Info: Using configured environment 'my_build_branch'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Virtual Query, Could not autoload puppet/type/elasticsearch_user_file: Could not autoload puppet/provider/elasticsearch_user_file/oss_xpack: undefined method `oss_xpack_config' for Puppet::Type::Elasticsearch_user_file::ProviderOss_xpack:Class at /etc/puppetlabs/code/environments/my_build_branch/modules/elasticsearch/manifests/init.pp:555:6 on node my.server
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Have you an idea ? this was introduce in 6.3.0.

[tylerjl]

Hi @Celedhrim, there's a possibility that this came up when you updated the module on the puppetmaster and the native types and providers changed. Please try restarting the puppetmaster (and potentially any effected puppet agents) and try re-running puppet again.

[aadiwarden]

We are seeing similar issues or our side.

Could not autoload puppet/type/elasticsearch_user_file: Could not autoload puppet/provider/elasticsearch_user_file/oss_xpack: undefined method `oss_xpack_config' for Puppet::Type::Elasticsearch_user_file::ProviderOss_xpack:Class at /etc/puppetlabs/code/environments/elastic6x_cfg/modules/elasticsearch/manifests/init.pp:545:6

[Celedhrim]

Hi @Celedhrim, there's a possibility that this came up when you updated the module on the puppetmaster and the native types and providers changed. Please try restarting the puppetmaster (and potentially any effected puppet agents) and try re-running puppet again.

I've tried it before opening the issue , both puppet server and puppet agent, but same error

[Noles]

I can confirm this with module version 6.3.3 and 6.3.2.
6.3.1 does not show this error.

[tylerjl]

Thanks everyone, this is important to get feedback to help figure out whether this is a bad bug or not ✅

@Celedhrim you mention this behavior was introduced in 6.3.0, and you're correct that things changed with the 6.3.0 release: this diff shows that the provider was introduced in 6.3.0, but it requires the library that brings in that method like all the others.

@Noles, your case is even more perplexing because the diff between the 6.3.1 release and 6.3.2 doesn't even touch any of the affected files, so I don't know how moving from a version like 6.3.1 to 6.3.2 could have possibly changed anything 🤔

One thing to keep in mind with regards to these custom types and providers is that when they're changed or updated, pluginsync needs to be working correctly so that those ruby files are updated. I'm still trying to think of whether something on the module side could have caused this, though my current best guess is that something on the Puppet server or agent may be caching something incorrectly.

[ynnt]

Seems like it's related to SERVER-84 Puppet bug.

The issue has gone after I merge updated module to all environments I have.

[scamfield]

Puppet Agent version: 5.5.6
Puppet Server version: 5.3.5
Module version: 6.3.3
OS and version:Debian GNU/Linux 9.5 (stretch)

We are seeing similar issues but only when the puppet server is restarted. And starts working on the 3rd re-try (with no changes).

Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Virtual Query, Could not autoload puppet/type/elasticsearch_user: Could not autoload puppet/provider/elasticsearch_user/esusers: no such file to load -- puppet/provider/elastic_user_command (file: /etc/puppetlabs/code/environments/production/modules/elasticsearch/manifests/init.pp, line: 554, column: 3)

[tylerjl]

@scamfield thanks for the info. Is this behavior (succeeding after multiple puppet runs) similar for others? I'm wondering if this is emerging from the Puppet server's caching mechanisms.

[ghost]

I can +1 what @scamfield reports - Success after 3 failed puppet runs following a server restart

edit:
Puppet Agent version: 5.5.8
Puppet Server version: 5.3.6
Module version: 6.3.0
OS and version:Ubuntu 16.04 (Xenial)

[scamfield]

@tylerjl I can confirm that after the 3rd run it continues to work until the next puppetserver restart. I've also done some testing by downgrading the module to see if its 6.3.x related after the big changes :

6.1.0 - works no errors
6.2.2 - works no errors
6.3.0 - fails with 500 error but works on 3rd run
6.3.1 - fails with 500 error but works on 3rd run
6.3.2 - fails with 500 error but works on 3rd run
6.3.3 - fails with 500 error but works on 3rd run

Reviewing the bugs open for 6.3.x, I think issue #990 may be related to this one.

[bakaie]

Same issue here. I have not made a change/upgrades to puppet or the module in several months.

module version: 6.3.1

This was working with no issues until today. I had all of my config in the pp role file and decided to move everything into hiera. I started to have this issue after moving all the config into the yaml file. I get the same 3 failed runs then one success.
I have restarted the clients but not the puppet master. If i just run the agent on the client over and over i get the same pattern, 3 failed one success.

[joernott]

Our puppet guru and I tried to track down that issue today. It looks like these issues are related to the way you use confines in your providers. The selection of the provider is tied to the presence of /etc/elasticsearch/x-pack/roles.yml for xpack, respective other files for x-pack-oss and shield. Sadly, the error happens when compiling the catalog. This is long before these files are created.

As a result, the puppet run fails once for the role provider, once for the user provider (due to lines 554 / 555 in init.pp), then runs, which actually installs the package (offering the chance to have that file created).

I guess, a way to select the provider via some parameter instead of existence of a file which should be created by the module could solve that problem.

[tylerjl]

Thanks for digging into it @joernott. Given that the module already accepts a parameter for security_plugin, I suppose we could use that, although we would need to differentiate between the plugin version of x-pack (< 6.3) and bundled x-pack (>= 6.3) since the config file paths are different between the two.

[Infraded]

Any update on this? We're still seeing failures on these providers in our logs after any puppetserver restart.

[tylerjl]

Given the complexities of trying to support multiple instances of ES and very old versions, I think the cleanest solution will likely be making some backwards-incompatible changes in a major version release alleviate the need for the module to auto-detect these file paths.

[tobias-urdin]

@tylerjl I assume you mean changing the require statement in the elasticsearch_user provider [1] to something like:

require File.join(File.dirname(__FILE__), '..','..','..', 'puppet/provider/elastic_user_command')

(And in all other places where it's required)

Would that really be disruptive? We've been using this in other modules and I dont think this would cause any issues, maybe I'm missing something

Best regards

[1] https://github.com/elastic/puppet-elasticsearch/blob/master/lib/puppet/provider/elasticsearch_user/esusers.rb

[tylerjl]

@tobias-urdin can you confirm whether using that version of require resolves your issue? I'm still not sure there's a definitive reason that we've tracked down in this issue, but ruby failing to find a changed path in the puppet master's autoloaded environment sounds like the most plausible explanation. (my previous comment wasn't related to require statements in provider code, but it's what seems to make the most sense to me currently)

[tobias-urdin]

@tylerjl I have not tested this approach on the puppet-elasticsearch module, I can maybe test this in a while if I have some time over.

[h0tw1r3]

Patched as suggested., running in production.
https://github.com/btolab/puppet-elasticsearch/tree/fix-autoload-provider

[zez3]

I tested and it works
There are 3 files that need to be touched

#egrep -r "elastic_user_command" /etc/puppetlabs/code/environments/development/modules/elasticsearch/lib/puppet/provider/

/etc/puppetlabs/code/environments/development/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/elasticsearch_users.rb:require 'puppet/provider/elastic_user_command'
/etc/puppetlabs/code/environments/development/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/users.rb:require 'puppet/provider/elastic_user_command'
/etc/puppetlabs/code/environments/development/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/esusers.rb:#require 'puppet/provider/elastic_user_command'

I ended up with some ugly sed

sed -i 's/require '''puppet/provider/elastic_user_command'''/require File.join(File.dirname(__FILE__), '''..''','''..''','''..''', '''puppet/provider/elastic_user_command'''/' /etc/puppetlabs/code/environments/development/modules/elasticsearch/lib/puppet/provider/elasticsearch_user/*

Why is this fix not yet merged ?

[juliantaylor]

I too can confirm that this fixes the failures (using puppet 6.0) and I have created a PR with the change.

[smasa90]

After some research, I found a ticket which seems relating this issue. -> https://tickets.puppetlabs.com/browse/PUP-3513
It seems to be due that puppetserver doesn't generate metadata from freshly pulled modules for a specific environment. It is recommended to use Environment Isolation to avoid this problem.
Issuing a puppet generate types --environment {env} directly after a puppet-code deploy or a r10k puppetfile install fixes this issue and with --force flag after each puppetserver updates.
Cf. https://puppet.com/docs/puppet/6.5/environment_isolation.html#reference-6554

[webzakimbo]

Issuing a puppet generate types --environment {env} directly after a puppet-code deploy

Double thumbs up on this!

[mvilain]

6.4.0 still throws this error unless you do

puppet generate types --environment {env}

after r10k deploy or a puppet-code deploy