Multiple default providers found after upgrade to ES 6.3.2

[JohnLyman]

• Module version: 6.3.1
• Puppet version: 5.5.1
• OS and version: RHEL 7.3

Bug description

After upgrading ES 6.2.4 -> 6.3.2, I'm getting the following:

Warning: Found multiple default providers for elasticsearch_role_mapping: oss_xpack, xpack; using oss_xpack
Warning: Found multiple default providers for elasticsearch_user_file: oss_xpack, xpack; using oss_xpack
Warning: Found multiple default providers for elasticsearch_user_roles: oss_xpack, xpack; using oss_xpack

Note: I am not using the OSS package.

As far as I can tell, the provider code is looking for the existence of (for example) /etc/elasticsearch/role_mapping.yml or /etc/elasticsearch/x-pack/role_mapping.yml to determine which provider to use. I have both (as well as /etc/elasticsearch/<instance>/role_mapping.yml and /etc/elasticsearch/<instance>/x-pack/role_mapping.yml!)

Because this was an upgrade, it makes sense that I have these files in /etc/elasticsearch/x-pack. The problem is that the rpm also places default versions of the files in /etc/elasticsearch:

$ rpm -qf /etc/elasticsearch/role_mapping.yml
elasticsearch-6.3.2-1.noarch

If I remove the files in /etc/elasticsearch and keep the /etc/elasticsearch/x-pack versions, Puppet does not complain about multiple providers and works fine. If I do the opposite (remove x-pack subdirectory), Puppet does not complain about multiple providers, but fails after creating the x-pack directory and x-pack/log4j2.properties due to Elasticsearch_role dependency failures.

Although I could have Puppet ensure the /etc/elasticsearch versions of the file do not exist, I'm afraid I'd have the same issue after any future upgrade. It's also not ideal because the provider code would detect the existence of those files before Puppet would have a chance to remove them.

[JohnLyman]

I forgot to note: I also tried to set a default provider with Elasticsearch_role_mapping { provider => 'xpack' } as a work around. This didn't work. Puppet failed with:

Error: Could not retrieve catalog from remote server:
Error 500 on SERVER: Server Error: no parameter named 'provider'
(file: (...)/modules/elasticsearch/manifests/role.pp, line: 56) on Elasticsearch_role_mapping ...)

[JohnLyman]

It looks like the proper work around is to remove role_mapping.yml, users, and users_roles (and maybe roles.yml?) from /etc/elasticsearch/x-pack. I was attempting to remove the entire x-pack sub directory, but it still needs to exist for log4j2.properties.

[tylerjl]

Thanks for doing so much debugging legwork here @JohnLyman, I really appreciate it ❤️ . When you say that the log4j2 file needs to be there, is ES failing to startup without it? I don't think Puppet manages that file in the x-pack directory (as far as I know).

The whole situation is certainly kind of tricky in the upgrade scenario, but I do think the module shouldn't just outright break in that case.

[marcomusso]

In the end what is the correct precedence for user/role files?
We just upgraded to 6.3.2 (using module v6.3.1, puppet masterless 4.10.12) and this setup will not allow users to login correctly (single instance on a kibana node):

# cd /etc/elasticsearch
# ll instance-1/{role*,users*}
-rw-r--r--. 1 elasticsearch elasticsearch 473 Aug 10 07:13 instance-1/role_mapping.yml
-rw-r--r--. 1 elasticsearch elasticsearch 197 Aug 10 07:13 instance-1/roles.yml
-rw-r--r--. 1 elasticsearch elasticsearch   0 Aug 10 07:13 instance-1/users
-rw-r--r--. 1 elasticsearch elasticsearch   0 Aug 10 07:13 instance-1/users_roles
# ll instance-1/x-pack/{role*,users*}
-rwxr-x---. 1 root elasticsearch  539 Aug 10 07:15 instance-1/x-pack/role_mapping.yml
-rwxr-x---. 1 root elasticsearch 1759 Aug 10 07:15 instance-1/x-pack/roles.yml
-rwxr-x---. 1 root elasticsearch  797 Aug 10 07:15 instance-1/x-pack/users
-rwxr-x---. 1 root elasticsearch  215 Aug 10 07:15 instance-1/x-pack/users_roles

The correct files are in the x-pack dir (with strange perms) but it seems also the default (empty) ones are created by the module and take precedence.
Removing the empty ones and symlinking to the x-pack one seem to work but I wonder what's going on...

[JohnLyman]

@tylerjl log4j2.properties is managed in the x-pack directory: https://github.com/elastic/puppet-elasticsearch/blob/master/manifests/config.pp#L150-L158

@marcomusso I'm not completely sure of the precedence. I'm pretty sure ES 6.3.0 and above wants those files in /etc/elasticsearch/<instance>. I know that if it only finds them in /etc/elasticsearch/<instance>/x-pack, it will still read and use them, but log a deprecation warning. I don't know what the behavior is if it finds them in both locations. From what you said, it sounds like ES is only reading the files in /etc/elasticsearch/<instance> though.

[marcomusso]

Indeed, it seems that the puppet modules creates default empty files and another part of the module (x-pack) creates the correct file without the instance knowing which are the effective ones and probably acting on a default that reads those empty ones.

[dannygoulder]

Hi,

I can confirm that it only reads the files in /etc/elasticsearch/instance. I found out when all my role mappings were broken.

First fix I tried was to symlink the files but I figured that'd be quite fragile and might interfere with upgrades. on the next node I tried to clean up the files that aren't inside the x-pack directories (and haven't been managed by the Puppet module), but this didn't allow the ES node to pick up the configurations.

Reading here: https://www.elastic.co/guide/en/elastic-stack-overview/6.3/mapping-roles.html#mapping-roles-file seems to indicate that the default location isn't inside the x-pack directory but in the ES_PATH_CONF directory, so the module should be creating them there in a 6.3+ system.

[dannygoulder]

Just to follow up on my comment. If you remove the files in /etc/elasticsearch/<instance> then you need to restart ES afterwards (at least in my case for role mappings).

[msvechla]

We have the exact same issue as @marcomusso. Do you have any eta for a fix here?
x-pack security is unusable like this currently

[marcomusso]

I have a weird workaround in place: symlinking to the correct files.

[dannygoulder]

Hi, just checking in here. @marcomusso - did your workaround persist when you were upgrading elasticsearch? In which direction did you symlink the files? Was the symlink in the /etc/elasticsearch/<instance> directory or in the x-pack subdirectory?

[marcomusso]

I symlink from the instance config directory to the xpack directory (output slightly redacted):

# pwd
/etc/elasticsearch/instance-1
# ll
lrwxrwxrwx. 1 root          root            53 Aug 27 09:49 role_mapping.yml -> /etc/elasticsearch/instance-1/x-pack/role_mapping.yml
lrwxrwxrwx. 1 root          root            46 Aug 27 09:49 roles.yml -> /etc/elasticsearch/instance-1/x-pack/roles.yml
lrwxrwxrwx. 1 root          root            42 Aug 27 09:49 users -> /etc/elasticsearch/instance-1/x-pack/users
lrwxrwxrwx. 1 root          root            48 Aug 27 09:49 users_roles -> /etc/elasticsearch/instance-1/x-pack/users_roles
drwxr-x---. 2 root          elasticsearch  130 Nov  7 11:33 x-pack

The link persists as I enforce it with Puppet.

[TJM]

We also see this for elasticsearch_license, even on 5.6.14...

Warning: Found multiple default providers for elasticsearch_license: shield, xpack; using shield

Probably shouldn't have multiple "default" providers :)