Missing base64 library for secured HDFS repository initialization :Plugin Repository HDFS

[ggiessl]

Elasticsearch version: 5.4.1

Plugins installed: [ "repository-hdfs" ]

JVM version (java -version):
openjdk version "1.8.0_101"
OpenJDK Runtime Environment (build 1.8.0_101-b13)
OpenJDK 64-Bit Server VM (build 25.101-b13, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux elasticsearch 3.10.0-327.10.1.el7.x86_64 #1 SMP Sat Jan 23 04:54:55 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I am trying to initialize a snapshot repository on secured HDFS (krb, transfer.protection, rpc.protection). That only works if I add the missing base64 library to the repository-hdfs plugin directory.

Solution: copy commons-codec-1.10.jar to /usr/share/elasticsearch/plugins/repository-hdfs/

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. Install the repository-hdfs plugin
2. Place a valid keytab file in /etc/elasticsearch/repository-hdfs/krb5.keytab
3. Initialize the repository:
   curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_snapshot/hdfs_repository?pretty -d '
   {
   "type": "hdfs",
   "settings": {
   "uri": "hdfs://hdfs-namenode:8020/",
   "path": "/elastic-backup",
   "security.principal": "HDFS",
   "conf.dfs.data.transfer.protection": "privacy",
   "conf.hadoop.rpc.protection": "privacy"
   }
   }'

Provide logs (if relevant):
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Self-suppression not permitted"
}
],
"type" : "illegal_argument_exception",
"reason" : "Self-suppression not permitted",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "DataStreamer Exception: ",
"caused_by" : {
"type" : "no_class_def_found_error",
"reason" : "org/apache/commons/codec/binary/Base64"
}
}
},
"status" : 400
}

Proposed solution

Include commons-codec-1.10.jar

[jbaiera]

@ggiessl Could you include the logs from the Elasticsearch Node when this error occurs?

[ggiessl]

Hi!
please find the logs attached.
25022_logs.txt

[jbaiera]

This looks more like a general problem when enabling hadoop RPC protection. Seems this was not caught in the tests, so we'll need to make sure to add reasonable testing to ensure that this is covered.

[ggiessl]

Hi!
@jbaiera did you (or are you planning to) include commons-codec-1.10.jar?

[jbaiera]

@ggiessl - I've been working on reproducing the error to make sure that we are actually testing with SSL enabled on secure HDFS, but there are quite a few things that have gotten in the way. I'm hoping to have an update on this soon.

[ggiessl]

Thanx for the update @jbaiera!

[ywelsch]

This should be fixed by #27196
@jbaiera can you confirm?

[jbaiera]

This has been fixed for a while now. Seems this issue was missed. Closing.