"elasticsearch-plugin.bat remove" removes plugin descriptor file even if the plugin jar is in use

[bittusarkar]

Elasticsearch version (bin/elasticsearch --version): 6.2.4

Plugins installed: [Any plugin]

JVM version (java -version): 1.8.0_212

OS version (uname -a if on a Unix-like system): Windows Server 2012 R2

Description of the problem including expected versus actual behavior:
elasticsearch-plugin.bat remove <plugin name> performs partial deletion which leaves Elasticsearch in a bad state.

Steps to reproduce:

1. Have any plugin called say CustomPlugin installed in Elasticsearch. It just needs to have file plugin-descriptor.properties along with the CustomPlugin.jar file in \elasticsearch\plugins\CustomPlugin directory.
2. Make sure Elasticsearch is running.
3. Execute bin\elasticsearch-plugin.bat remove CustomPlugin
   This command fails saying that the jar file is being used. This is expected because Elasticsearch is running. But it deletes the plugin-descriptor.properties file.
4. Once you figure out that it failed due to Elasticsearch being in running state, stop Elasticsearch.
5. Execute bin\elasticsearch-plugin.bat remove CustomPlugin again.
   This time it fails with the following error:

Exception in thread "main" java.lang.IllegalStateException: Could not load plugin descriptor for existing plugin [CustomPlugin]. Was the plugin built before 2.0?
        at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:352)
        at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:89)
        at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:67)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:75)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:48)
Caused by: java.nio.file.NoSuchFileException: C:\elasticsearch\plugins\CustomPlugin\plugin-descriptor.properties
        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
        at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230)
        at java.nio.file.Files.newByteChannel(Files.java:361)
        at java.nio.file.Files.newByteChannel(Files.java:407)
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
        at java.nio.file.Files.newInputStream(Files.java:152)
        at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:202)
        at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:349)
        ... 8 more

The bug is that elasticsearch-plugin.bat remove command can leave Elasticsearch in an unexpected state. There can be two ways to fix this:

1. Either it does not delete any file if Elasticsearch is running, or
2. It deletes all files if Elasticsearch is not running without checking for the existence of plugin-descriptor.properties file.

Personally I prefer the first option as that always leaves the system in a good state.

You may ask why not stop Elasticsearch before removing the plugin. We actually do that. I will explain our scenario below:
We have automated scripts to install security updates and then upgrade our custom plugins if required. Installing security updates can sometime result in restarting the computer. After the computer restarts, we attempt to upgrade the plugin. We first check if Elasticsearch is in stopped state. We happen to pass this check sometimes because Elasticsearch service took too much time to start after computer restart. Then we go ahead and execute the remove command. Just before we do that, Elasticsearch service starts. And this is how we hit this bug. Of course, we can add more checks to bypass this issue but IMO, this is a bug in the plugin removal code.

[elasticmachine]

Pinging @elastic/es-core-infra