Plugin Installation Permissions Issue

[juan131]

Elasticsearch is unable to install plugins when the plugin requires to install certain configuration (e.g. repository-s3) in the config directory.

It complains of lacking the corresponding permissions to write in the directory even when the permissions are correct.

Elasticsearch version: 5.6.15

Plugins installed: repository-s3

JVM version: 1.8.201

OS version: Debian 9

Description of the problem including expected versus actual behavior:

I'm using Bitnami ElasticSearch Docker Image which allows to install plugins (using elasticsearch-plugin install command) during the bootstrap of the container by listing them in the env. variable "ELASTICSEARCH_PLUGINS".

This image is a non-root container (it runs as user 1001 by default) and it uses the directory /opt/bitnami/elasticsearch/config as "config dir". By default, this directory has the permissions configuration below:

• Permissions: 775
• Ownership: root:root

Please note that the user running the "install plugin command" (the user 1001) belongs to the "root" group and, therefore, it has permissions to write on that directory. You can check it running the command below:

$ docker run --rm bitnami/elasticsearch:6 /bin/bash -c 'mkdir /opt/bitnami/elasticsearch/config/new-directory && ls -la /opt/bitnami/elasticsearch/config'

However, Elasticsearch complains of lacking permissions when trying to install the plugin (see provided logs).

I think the reason might be related to the code below:

https://github.com/elastic/elasticsearch/blob/master/distribution/tools/plugin-cli/src/main/java/org/elasticsearch/plugins/InstallPluginCommand.java#L915

For some reason I don't understand, the elasticsearch-plugin install command tries to modify the ownership of the "config dir" and then, it complains about being unable to create the repository-s3 directory even having the proper permissions!

• Why is this necessary?
• Shouldn't it check whether the user has writing permissions and create the directory if so?

Steps to reproduce:

Run the command below:

$ docker run --rm -e "ELASTICSEARCH_PLUGINS=repository-s3" -e "BITNAMI_DEBUG=true" bitnami/elasticsearch:6

Provide logs (if relevant):

Checking if url exists: https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-linux-x86_64-6.7.1.zip
Downloading repository-s3 from elastic
Retrieving zip from https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-6.7.1.zip
- Plugin information:
Name: repository-s3
Description: The S3 repository plugin adds S3 repositories
Version: 6.7.1
Elasticsearch Version: 6.7.1
Java Version: 1.8
Native Controller: false
Extended Plugins: []
* Classname: org.elasticsearch.repositories.s3.S3RepositoryPlugin
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission es.allow_insecure_settings read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
Exception in thread "main" java.nio.file.FileSystemException: /opt/bitnami/elasticsearch/config/repository-s3: Operation not permitted
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileAttributeViews$Posix.setOwners(UnixFileAttributeViews.java:252)
 	at sun.nio.fs.UnixFileAttributeViews$Posix.setOwner(UnixFileAttributeViews.java:274)
	at org.elasticsearch.plugins.InstallPluginCommand.setOwnerGroup(InstallPluginCommand.java:919)
	at org.elasticsearch.plugins.InstallPluginCommand.installConfig(InstallPluginCommand.java:892)
	at org.elasticsearch.plugins.InstallPluginCommand.installPluginSupportFiles(InstallPluginCommand.java:826)
	at org.elasticsearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:807)
	at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:776)
	at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:231)
	at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:216)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)

[elasticmachine]

Pinging @elastic/es-core-infra

[juan131]

Any news?

[jaymode]

the elasticsearch-plugin install command tries to modify the ownership of the "config dir"

We do this so that we can ensure proper permissions when these directories are created. There may be an improvement here where we set the attributes when creating the directory but that is not going to resolve the issue here.

The issue in this setup is that the bitnami image uses root:root for owner and group of the config directory. The plugin installation creates the repository-s3 directory within the configuration directory and attempts to set the owner and permissions of the repository-s3 directory to be the same as those on the config directory. Changing the owner from 1001 to root is the failing step. You may be able to workaround this by getting bitnami to change the owner of the config directory to 1001, so that way the owner doesn't actually need to change.

[juan131]

Hi @jaymode

Thanks so much for your response!

I am a Bitnami developer and I am trying to make this image compatible with OpenShift.

OpenShift launches containers with a random non-root user (see https://docs.openshift.com/container-platform/3.3/creating_images/guidelines.html#openshift-container-platform-specific-guidelines, Support Arbitrary User IDs section). Therefore, I cannot assume the user running the container is 1001 an I cannot set the owner of the config directory to 1001.

That's the reason why we add the "non-root" user to the "root" group so we can provide the required permissions to the group.

I do not really understand why the owner of each plugin directory must be the same as the one used in the parent directory. I mean, from a "Unix permissions point of view", the user running ElasticSearch has the required permissions even if the owner of the directory is different.

[juan131]

Hi @jaymode, did you see my last comment?

[juan131]

Ping @jaymode

[jaymode]

I looked into fixing this and ran into some issues that make a proper fix a much larger problem than just whether or not we change permissions when installing the plugin. The issue is that our default packages do not allow for group writes to the bin, plugin, and config directories. While we could make a smaller change that would enable this use case, I have concerns regarding how such a change would impact different installations and whether allowing a group to write to these directories is the right way forward. There are security implications if the Java Security Manager is bypassed for file system access that could allow a compromised elasticsearch process itself to write a plugin or modify configuration.

[juan131]

Sorry for the late response @jaymode

So, you don't see other alternative rather than configuring the ownership of the folder(s) so the user running ElasticSearch is the owner, do you?

[barkbay]

We have the same issue with ECK.
It is not specific to Openshift but to any secured Kubernetes cluster where containers are running with a "random" ID and not with a privileged user: elastic/cloud-on-k8s#1801

[juan131]

Exactly @barkbay

Any platform with those requirements (random and not privileged user) will face this issue.

[williamrandolph]

The original reproduction line provided no longer works:

docker run --rm -e "ELASTICSEARCH_PLUGINS=repository-s3" -e "BITNAMI_DEBUG=true" bitnami/elasticsearch:6

Furthermore, I believe our more-or-less official advice on plugins for docker containers is:

We recommend that users of our Docker containers instead derive an image that installs the necessary plugins (to create a new image), and then deploy that.

Since this ticket no longer has a working reproduction line and we don't have any plans to change our security settings, I am going to close this ticket. Please feel free to re-open it or file a new ticket if there is another way of reproducing the problem and the custom-image approach doesn't work for your use case.