Password Protected Keystore (Feature Branch) (cleaned up) (#49268)

* Reload secure settings with password (#43197)

If a password is not set, we assume an empty string to be
compatible with previous behavior.
Only allow the reload to be broadcast to other nodes if TLS is
enabled for the transport layer.

* Add passphrase support to elasticsearch-keystore (#38498)

This change adds support for keystore passphrases to all subcommands
of the elasticsearch-keystore cli tool and adds a subcommand for
changing the passphrase of an existing keystore.
The work to read the passphrase in Elasticsearch when
loading, which will be addressed in a different PR.

Subcommands of elasticsearch-keystore can handle (open and create)
passphrase protected keystores

When reading a keystore, a user is only prompted for a passphrase
only if the keystore is passphrase protected.

When creating a keystore, a user is allowed (default behavior) to create one with an
empty passphrase

Passphrase can be set to be empty when changing/setting it for an
existing keystore

Relates to: #32691
Supersedes: #37472

* Restore behavior for force parameter (#44847)

Turns out that the behavior of `-f` for the add and add-file sub
commands where it would also forcibly create the keystore if it
didn't exist, was by design - although undocumented.
This change restores that behavior auto-creating a keystore that
is not password protected if the force flag is used. The force
OptionSpec is moved to the BaseKeyStoreCommand as we will presumably
want to maintain the same behavior in any other command that takes
a force option.

*  Handle pwd protected keystores in all CLI tools  (#45289)

This change ensures that `elasticsearch-setup-passwords` and
`elasticsearch-saml-metadata` can handle a password protected
elasticsearch.keystore.
For setup passwords the user would be prompted to add the
elasticsearch keystore password upon running the tool. There is no
option to pass the password as a parameter as we assume the user is
present in order to enter the desired passwords for the built-in
users.
For saml-metadata, we prompt for the keystore password at all times
even though we'd only need to read something from the keystore when
there is a signing or encryption configuration.

* Modify docs for setup passwords and saml metadata cli (#45797)

Adds a sentence in the documentation of `elasticsearch-setup-passwords`
and `elasticsearch-saml-metadata` to describe that users would be
prompted for the keystore's password when running these CLI tools,
when the keystore is password protected.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

* Elasticsearch keystore passphrase for startup scripts (#44775)

This commit allows a user to provide a keystore password on Elasticsearch
startup, but only prompts when the keystore exists and is encrypted.

The entrypoint in Java code is standard input. When the Bootstrap class is
checking for secure keystore settings, it checks whether or not the keystore
is encrypted. If so, we read one line from standard input and use this as the
password. For simplicity's sake, we allow a maximum passphrase length of 128
characters. (This is an arbitrary limit and could be increased or eliminated.
It is also enforced in the keystore tools, so that a user can't create a
password that's too long to enter at startup.)

In order to provide a password on standard input, we have to account for four
different ways of starting Elasticsearch: the bash startup script, the Windows
batch startup script, systemd startup, and docker startup. We use wrapper
scripts to reduce systemd and docker to the bash case: in both cases, a
wrapper script can read a passphrase from the filesystem and pass it to the
bash script.

In order to simplify testing the need for a passphrase, I have added a
has-passwd command to the keystore tool. This command can run silently, and
exit with status 0 when the keystore has a password. It exits with status 1 if
the keystore doesn't exist or exists and is unencrypted.

A good deal of the code-change in this commit has to do with refactoring
packaging tests to cleanly use the same tests for both the "archive" and the
"package" cases. This required not only moving tests around, but also adding
some convenience methods for an abstraction layer over distribution-specific
commands.

I will write some user-facing documentation for these changes in a follow-up
commit.

* Adjust docs for password protected keystore (#45054)

This commit adds relevant parts in the elasticsearch-keystore
sub-commands reference docs and in the reload secure settings API
doc.

* Cleanup after feature branch reconstruction

The feature branch for the password-protected keystore, due to an
accident, contains a large number of unrelated commits. In order to get
a cleaner merge, I've cherry-picked the main commits that went into the
feature branch against a branch derived from master — essentially, a
rebase onto master. We've ignored some tests that will addressed in
follow-up PRs to the feature branch.

Vagrantfile

@@ -475,6 +475,7 @@ JAVA
     ensure curl
     ensure unzip
     ensure rsync
+    ensure expect
 
     installed bats || {
       # Bats lives in a git repository....

build.gradle

@@ -205,8 +205,8 @@ task verifyVersions {
  * after the backport of the backcompat code is complete.
  */
 
-boolean bwc_tests_enabled = true
-final String bwc_tests_disabled_issue = "" /* place a PR link here when committing bwc changes */
+boolean bwc_tests_enabled = false
+final String bwc_tests_disabled_issue = "https://github.com/elastic/elasticsearch/pull/43197"
 if (bwc_tests_enabled == false) {
   if (bwc_tests_disabled_issue.isEmpty()) {
     throw new GradleException("bwc_tests_disabled_issue must be set when bwc_tests_enabled == false")

distribution/docker/docker-test-entrypoint.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 cd /usr/share/elasticsearch/bin/
-./elasticsearch-users useradd x_pack_rest_user -p x-pack-test-password -r superuser || true 
+./elasticsearch-users useradd x_pack_rest_user -p x-pack-test-password -r superuser || true
 echo "testnode" > /tmp/password
 cat /tmp/password  | ./elasticsearch-keystore add -x -f -v 'xpack.security.transport.ssl.keystore.secure_password'
 cat /tmp/password  | ./elasticsearch-keystore add -x -f -v 'xpack.security.http.ssl.keystore.secure_password'

distribution/docker/src/docker/bin/docker-entrypoint.sh

@@ -117,8 +117,18 @@ if [[ -f bin/elasticsearch-users ]]; then
   # honor the variable if it's present.
   if [[ -n "$ELASTIC_PASSWORD" ]]; then
     [[ -f /usr/share/elasticsearch/config/elasticsearch.keystore ]] || (run_as_other_user_if_needed elasticsearch-keystore create)
-    if ! (run_as_other_user_if_needed elasticsearch-keystore list | grep -q '^bootstrap.password$'); then
-      (run_as_other_user_if_needed echo "$ELASTIC_PASSWORD" | elasticsearch-keystore add -x 'bootstrap.password')
+    if ! (run_as_other_user_if_needed elasticsearch-keystore has-passwd --silent) ; then
+      # keystore is unencrypted
+      if ! (run_as_other_user_if_needed elasticsearch-keystore list | grep -q '^bootstrap.password$'); then
+        (run_as_other_user_if_needed echo "$ELASTIC_PASSWORD" | elasticsearch-keystore add -x 'bootstrap.password')
+      fi
+    else
+      # keystore requires password
+      if ! (run_as_other_user_if_needed echo "$KEYSTORE_PASSWORD" \
+          | elasticsearch-keystore list | grep -q '^bootstrap.password$') ; then
+        COMMANDS="$(printf "%s\n%s" "$KEYSTORE_PASSWORD" "$ELASTIC_PASSWORD")"
+        (run_as_other_user_if_needed echo "$COMMANDS" | elasticsearch-keystore add -x 'bootstrap.password')
+      fi
     fi
   fi
 fi
@@ -130,4 +140,4 @@ if [[ "$(id -u)" == "0" ]]; then
   fi
 fi
 
-run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch "${es_opts[@]}"
+run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch "${es_opts[@]}" <<<"$KEYSTORE_PASSWORD"

distribution/packages/build.gradle

@@ -231,6 +231,10 @@ Closure commonPackageConfig(String type, boolean oss, boolean jdk) {
       from "${packagingFiles}/systemd/sysctl/elasticsearch.conf"
       fileMode 0644
     }
+    into('/usr/share/elasticsearch/bin') {
+      from "${packagingFiles}/systemd/systemd-entrypoint"
+      fileMode 0755
+    }
 
     // ========= sysV init =========
     configurationFile '/etc/init.d/elasticsearch'

distribution/packages/src/common/scripts/posttrans

@@ -4,7 +4,12 @@ if [ ! -f /etc/elasticsearch/elasticsearch.keystore ]; then
     chmod 660 /etc/elasticsearch/elasticsearch.keystore
     md5sum /etc/elasticsearch/elasticsearch.keystore > /etc/elasticsearch/.elasticsearch.keystore.initial_md5sum
 else
-    /usr/share/elasticsearch/bin/elasticsearch-keystore upgrade
+    if /usr/share/elasticsearch/bin/elasticsearch-keystore has-passwd --silent ; then
+      echo "### Warning: unable to upgrade encrypted keystore" 1>&2
+      echo " Please run elasticsearch-keystore upgrade and enter password" 1>&2
+    else
+      /usr/share/elasticsearch/bin/elasticsearch-keystore upgrade
+    fi
 fi
 
 ${scripts.footer}

distribution/packages/src/common/systemd/elasticsearch.service

@@ -19,7 +19,7 @@ WorkingDirectory=/usr/share/elasticsearch
 User=elasticsearch
 Group=elasticsearch
 
-ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet
+ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet
 
 # StandardOutput is configured to redirect to journalctl since
 # some error messages may be logged in standard output before

distribution/packages/src/common/systemd/systemd-entrypoint

@@ -0,0 +1,10 @@
+#!/bin/sh
+
+# This wrapper script allows SystemD to feed a file containing a passphrase into
+# the main Elasticsearch startup script
+
+if [ -n "$ES_KEYSTORE_PASSPHRASE_FILE" ] ; then
+  exec /usr/share/elasticsearch/bin/elasticsearch "$@" < "$ES_KEYSTORE_PASSPHRASE_FILE"
+else
+  exec /usr/share/elasticsearch/bin/elasticsearch "$@"
+fi

distribution/src/bin/elasticsearch

@@ -20,6 +20,19 @@ if [ -z "$ES_TMPDIR" ]; then
   ES_TMPDIR=`"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.TempDirectory`
 fi
 
+# get keystore password before setting java options to avoid
+# conflicting GC configurations for the keystore tools
+unset KEYSTORE_PASSWORD
+KEYSTORE_PASSWORD=
+if ! echo $* | grep -E -q '(^-h |-h$| -h |--help$|--help |^-V |-V$| -V |--version$|--version )' \
+    && "`dirname "$0"`"/elasticsearch-keystore has-passwd --silent
+then
+  if ! read -s -r -p "Elasticsearch keystore password: " KEYSTORE_PASSWORD ; then
+    echo "Failed to read keystore password on console" 1>&2
+    exit 1
+  fi
+fi
+
 ES_JVM_OPTIONS="$ES_PATH_CONF"/jvm.options
 ES_JAVA_OPTS=`export ES_TMPDIR; "$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JvmOptionsParser "$ES_JVM_OPTIONS"`
 
@@ -35,7 +48,7 @@ if ! echo $* | grep -E '(^-d |-d$| -d |--daemonize$|--daemonize )' > /dev/null;
     -Des.bundled_jdk="$ES_BUNDLED_JDK" \
     -cp "$ES_CLASSPATH" \
     org.elasticsearch.bootstrap.Elasticsearch \
-    "$@"
+    "$@" <<<"$KEYSTORE_PASSWORD"
 else
   exec \
     "$JAVA" \
@@ -48,7 +61,7 @@ else
     -cp "$ES_CLASSPATH" \
     org.elasticsearch.bootstrap.Elasticsearch \
     "$@" \
-    <&- &
+    <<<"$KEYSTORE_PASSWORD" &
   retval=$?
   pid=$!
   [ $retval -eq 0 ] || exit $retval

distribution/src/bin/elasticsearch-cli.bat

@@ -25,5 +25,5 @@ set ES_JAVA_OPTS=-Xms4m -Xmx64m -XX:+UseSerialGC %ES_JAVA_OPTS%
   -cp "%ES_CLASSPATH%" ^
   "%ES_MAIN_CLASS%" ^
   %*
-  
+
 exit /b %ERRORLEVEL%

distribution/src/bin/elasticsearch.bat

@@ -4,6 +4,7 @@ setlocal enabledelayedexpansion
 setlocal enableextensions
 
 SET params='%*'
+SET checkpassword=Y
 
 :loop
 FOR /F "usebackq tokens=1* delims= " %%A IN (!params!) DO (
@@ -18,6 +19,20 @@ FOR /F "usebackq tokens=1* delims= " %%A IN (!params!) DO (
 		SET silent=Y
 	)
 
+	IF "!current!" == "-h" (
+		SET checkpassword=N
+	)
+	IF "!current!" == "--help" (
+		SET checkpassword=N
+	)
+
+	IF "!current!" == "-V" (
+		SET checkpassword=N
+	)
+	IF "!current!" == "--version" (
+		SET checkpassword=N
+	)
+
 	IF "!silent!" == "Y" (
 		SET nopauseonerror=Y
 	) ELSE (
@@ -41,6 +56,18 @@ IF ERRORLEVEL 1 (
 	EXIT /B %ERRORLEVEL%
 )
 
+SET KEYSTORE_PASSWORD=
+IF "%checkpassword%"=="Y" (
+  CALL "%~dp0elasticsearch-keystore.bat" has-passwd --silent
+  IF !ERRORLEVEL! EQU 0 (
+    SET /P KEYSTORE_PASSWORD=Elasticsearch keystore password:
+    IF !ERRORLEVEL! NEQ 0 (
+      ECHO Failed to read keystore password on standard input
+      EXIT /B !ERRORLEVEL!
+    )
+  )
+)
+
 if not defined ES_TMPDIR (
   for /f "tokens=* usebackq" %%a in (`CALL %JAVA% -cp "!ES_CLASSPATH!" "org.elasticsearch.tools.launchers.TempDirectory"`) do set  ES_TMPDIR=%%a
 )
@@ -54,7 +81,20 @@ if "%MAYBE_JVM_OPTIONS_PARSER_FAILED%" == "jvm_options_parser_failed" (
   exit /b 1
 )
 
-%JAVA% %ES_JAVA_OPTS% -Delasticsearch -Des.path.home="%ES_HOME%" -Des.path.conf="%ES_PATH_CONF%" -Des.distribution.flavor="%ES_DISTRIBUTION_FLAVOR%" -Des.distribution.type="%ES_DISTRIBUTION_TYPE%" -Des.bundled_jdk="%ES_BUNDLED_JDK%" -cp "%ES_CLASSPATH%" "org.elasticsearch.bootstrap.Elasticsearch" !newparams!
+rem windows batch pipe will choke on special characters in strings
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^^=^^^^!
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^&=^^^&!
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^|=^^^|!
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^<=^^^<!
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^>=^^^>!
+SET KEYSTORE_PASSWORD=!KEYSTORE_PASSWORD:^\=^^^\!
+
+ECHO.!KEYSTORE_PASSWORD!| %JAVA% %ES_JAVA_OPTS% -Delasticsearch ^
+  -Des.path.home="%ES_HOME%" -Des.path.conf="%ES_PATH_CONF%" ^
+  -Des.distribution.flavor="%ES_DISTRIBUTION_FLAVOR%" ^
+  -Des.distribution.type="%ES_DISTRIBUTION_TYPE%" ^
+  -Des.bundled_jdk="%ES_BUNDLED_JDK%" ^
+  -cp "%ES_CLASSPATH%" "org.elasticsearch.bootstrap.Elasticsearch" !newparams!
 
 endlocal
 endlocal

distribution/tools/keystore-cli/src/main/java/org/elasticsearch/common/settings/AddFileKeyStoreCommand.java

@@ -26,7 +26,6 @@
 
 import joptsimple.OptionSet;
 import joptsimple.OptionSpec;
-import org.elasticsearch.cli.EnvironmentAwareCommand;
 import org.elasticsearch.cli.ExitCodes;
 import org.elasticsearch.cli.Terminal;
 import org.elasticsearch.cli.UserException;
@@ -37,42 +36,29 @@
 /**
  * A subcommand for the keystore cli which adds a file setting.
  */
-class AddFileKeyStoreCommand extends EnvironmentAwareCommand {
+class AddFileKeyStoreCommand extends BaseKeyStoreCommand {
 
-    private final OptionSpec<Void> forceOption;
     private final OptionSpec<String> arguments;
 
     AddFileKeyStoreCommand() {
-        super("Add a file setting to the keystore");
-        this.forceOption = parser.acceptsAll(Arrays.asList("f", "force"), "Overwrite existing setting without prompting");
+        super("Add a file setting to the keystore", false);
+        this.forceOption = parser.acceptsAll(Arrays.asList("f", "force"),
+            "Overwrite existing setting without prompting, creating keystore if necessary");
         // jopt simple has issue with multiple non options, so we just get one set of them here
         // and convert to File when necessary
         // see https://github.com/jopt-simple/jopt-simple/issues/103
         this.arguments = parser.nonOptions("setting [filepath]");
     }
 
     @Override
-    protected void execute(Terminal terminal, OptionSet options, Environment env) throws Exception {
-        KeyStoreWrapper keystore = KeyStoreWrapper.load(env.configFile());
-        if (keystore == null) {
-            if (options.has(forceOption) == false &&
-                terminal.promptYesNo("The elasticsearch keystore does not exist. Do you want to create it?", false) == false) {
-                terminal.println("Exiting without creating keystore.");
-                return;
-            }
-            keystore = KeyStoreWrapper.create();
-            keystore.save(env.configFile(), new char[0] /* always use empty passphrase for auto created keystore */);
-            terminal.println("Created elasticsearch keystore in " + env.configFile());
-        } else {
-            keystore.decrypt(new char[0] /* TODO: prompt for password when they are supported */);
-        }
-
+    protected void executeCommand(Terminal terminal, OptionSet options, Environment env) throws Exception {
         List<String> argumentValues = arguments.values(options);
         if (argumentValues.size() == 0) {
             throw new UserException(ExitCodes.USAGE, "Missing setting name");
         }
         String setting = argumentValues.get(0);
-        if (keystore.getSettingNames().contains(setting) && options.has(forceOption) == false) {
+        final KeyStoreWrapper keyStore = getKeyStore();
+        if (keyStore.getSettingNames().contains(setting) && options.has(forceOption) == false) {
             if (terminal.promptYesNo("Setting " + setting + " already exists. Overwrite?", false) == false) {
                 terminal.println("Exiting without modifying keystore.");
                 return;
@@ -90,11 +76,11 @@ protected void execute(Terminal terminal, OptionSet options, Environment env) th
             throw new UserException(ExitCodes.USAGE, "Unrecognized extra arguments [" +
                 String.join(", ", argumentValues.subList(2, argumentValues.size())) + "] after filepath");
         }
-        keystore.setFile(setting, Files.readAllBytes(file));
-        keystore.save(env.configFile(), new char[0]);
+        keyStore.setFile(setting, Files.readAllBytes(file));
+        keyStore.save(env.configFile(), getKeyStorePassword().getChars());
     }
 
-    @SuppressForbidden(reason="file arg for cli")
+    @SuppressForbidden(reason = "file arg for cli")
     private Path getPath(String file) {
         return PathUtils.get(file);
     }

distribution/tools/keystore-cli/src/main/java/org/elasticsearch/common/settings/AddStringKeyStoreCommand.java

@@ -20,15 +20,13 @@
 package org.elasticsearch.common.settings;
 
 import java.io.BufferedReader;
-import java.io.CharArrayWriter;
 import java.io.InputStream;
 import java.io.InputStreamReader;
 import java.nio.charset.StandardCharsets;
 import java.util.Arrays;
 
 import joptsimple.OptionSet;
 import joptsimple.OptionSpec;
-import org.elasticsearch.cli.EnvironmentAwareCommand;
 import org.elasticsearch.cli.ExitCodes;
 import org.elasticsearch.cli.Terminal;
 import org.elasticsearch.cli.UserException;
@@ -37,16 +35,16 @@
 /**
  * A subcommand for the keystore cli which adds a string setting.
  */
-class AddStringKeyStoreCommand extends EnvironmentAwareCommand {
+class AddStringKeyStoreCommand extends BaseKeyStoreCommand {
 
     private final OptionSpec<Void> stdinOption;
-    private final OptionSpec<Void> forceOption;
     private final OptionSpec<String> arguments;
 
     AddStringKeyStoreCommand() {
-        super("Add a string setting to the keystore");
+        super("Add a string setting to the keystore", false);
         this.stdinOption = parser.acceptsAll(Arrays.asList("x", "stdin"), "Read setting value from stdin");
-        this.forceOption = parser.acceptsAll(Arrays.asList("f", "force"), "Overwrite existing setting without prompting");
+        this.forceOption = parser.acceptsAll(Arrays.asList("f", "force"),
+            "Overwrite existing setting without prompting, creating keystore if necessary");
         this.arguments = parser.nonOptions("setting name");
     }
 
@@ -56,26 +54,13 @@ InputStream getStdin() {
     }
 
     @Override
-    protected void execute(Terminal terminal, OptionSet options, Environment env) throws Exception {
-        KeyStoreWrapper keystore = KeyStoreWrapper.load(env.configFile());
-        if (keystore == null) {
-            if (options.has(forceOption) == false &&
-                terminal.promptYesNo("The elasticsearch keystore does not exist. Do you want to create it?", false) == false) {
-                terminal.println("Exiting without creating keystore.");
-                return;
-            }
-            keystore = KeyStoreWrapper.create();
-            keystore.save(env.configFile(), new char[0] /* always use empty passphrase for auto created keystore */);
-            terminal.println("Created elasticsearch keystore in " + env.configFile());
-        } else {
-            keystore.decrypt(new char[0] /* TODO: prompt for password when they are supported */);
-        }
-
+    protected void executeCommand(Terminal terminal, OptionSet options, Environment env) throws Exception {
         String setting = arguments.value(options);
         if (setting == null) {
             throw new UserException(ExitCodes.USAGE, "The setting name can not be null");
         }
-        if (keystore.getSettingNames().contains(setting) && options.has(forceOption) == false) {
+        final KeyStoreWrapper keyStore = getKeyStore();
+        if (keyStore.getSettingNames().contains(setting) && options.has(forceOption) == false) {
             if (terminal.promptYesNo("Setting " + setting + " already exists. Overwrite?", false) == false) {
                 terminal.println("Exiting without modifying keystore.");
                 return;
@@ -84,26 +69,18 @@ protected void execute(Terminal terminal, OptionSet options, Environment env) th
 
         final char[] value;
         if (options.has(stdinOption)) {
-            try (BufferedReader stdinReader = new BufferedReader(new InputStreamReader(getStdin(), StandardCharsets.UTF_8));
-                 CharArrayWriter writer = new CharArrayWriter()) {
-                int charInt;
-                while ((charInt = stdinReader.read()) != -1) {
-                    if ((char) charInt == '\r' || (char) charInt == '\n') {
-                        break;
-                    }
-                    writer.write((char) charInt);
-                }
-                value = writer.toCharArray();
-            }
+            BufferedReader stdinReader = new BufferedReader(new InputStreamReader(getStdin(), StandardCharsets.UTF_8));
+            value = stdinReader.readLine().toCharArray();
         } else {
             value = terminal.readSecret("Enter value for " + setting + ": ");
         }
 
         try {
-            keystore.setString(setting, value);
-        } catch (final IllegalArgumentException e) {
+            keyStore.setString(setting, value);
+        } catch (IllegalArgumentException e) {
             throw new UserException(ExitCodes.DATA_ERROR, e.getMessage());
         }
-        keystore.save(env.configFile(), new char[0]);
+        keyStore.save(env.configFile(), getKeyStorePassword().getChars());
+
     }
 }