Windows setup fails on upgrading to 7.7.0

[v-echo]

Environment

Windows Server 2016
ElasticSearch 7.6.2 installed.

Expected Behaviour

Installer upgrades version 7.6.2 to version 7.7.0.

Actual Behaviour

Installer is stuck for a long time at the final step, "Starting Elastic service", then fails with an error and rolls back to 7.6.2.

Steps to reproduce the behaviour

Upgrade install from a 7.6.2 to a 7.7.0 using the .msi package.
All settings were left as default, on both original installation and upgrade.
Reproduced on multiple VMs, all using Windows Server 2016.

installation_log.txt

[ghost]

Hello @v-echo , logs say that service was not able to start. One potential problem I see in the logs is value of USERNAME has a space in it: USERNAME=Windows User

[v-echo]

Hello @ygel , as far as I know no such user was explicitly configured within the Elastic setup process. The standard users were configured, eg. "elastic", "kibana", etc. through the .bat files.
Perhaps it's related to the service handle?

Also, worth mentioning that yesterday I upgraded from 7.4.0 to 7.6.2 without any issues, also using the .msi installer. And all settings left to default.

[GrzegorzLyp]

The problem is with an error from Windows Event Viewer. Probably the name "jvm.options\jvm.options" is the problem.

Service cannot be started. Elastic.ProcessHosts.Process.StartupException: Could not evaluate jvm.options file: C:\ProgramData\Elastic\Elasticsearch\config\jvm.options result: Exception in thread "main" java.nio.file.NoSuchFileException: C:\ProgramData\Elastic\Elasticsearch\config\jvm.options\jvm.options
at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:85)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at java.base/sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:235)
at java.base/java.nio.file.Files.newByteChannel(Files.java:375)
at java.base/java.nio.file.Files.newByteChannel(Files.java:426)
at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
at java.base/java.nio.file.Files.newInputStream(Files.java:160)
at org.elasticsearch.tools.launchers.JvmOptionsParser.readJvmOptionsFi...

[maciz84]

Anyone solved this yet?

[GrzegorzLyp]

Tried to build the whole project to debug, but build process fails. We need to wait for Yuriy.

[AndyHunt66]

Reproduced on a clean 7.5.0 -> 7.7.0 upgrade.

Full error from Event Viewer:

Service cannot be started. 
Elastic.ProcessHosts.Process.StartupException: 
  Could not evaluate jvm.options file: C:\ProgramData\Elastic\Elasticsearch\config\jvm.options 
    result: Exception in thread "main" java.nio.file.NoSuchFileException: 
      C:\ProgramData\Elastic\Elasticsearch\config\jvm.options\jvm.options

Workaround:
1 - Move the jvm.options file to the C:\ProgramData\Elastic\Elasticsearch\config\jvm.options.d directory
2 - rename C:\ProgramData\Elastic\Elasticsearch\config\jvm.options.d to C:\ProgramData\Elastic\Elasticsearch\config\jvm.options

[russcam]

A change in how jvm.options file is located, checked and parsed means that an MSI installation that tries to start the windows service as part of the installation will fail. This is fixed in #368 and will be in the next release.

As a workaround for 7.7.0 MSI, please follow these steps:

1. For installation, if install as a service is selected, uncheck "Start the service after this installation is complete"

For upgrade, if the previous version installed as a service, uncheck "Start the service after this installation is complete" in the upgrade UI

2. Select any other options in the installer that you require, and run installation.

3. After installation has completed successfully, exit the installer UI.

4. Open a new cmd or PowerShell, and navigate to the selected config directory. In a terminal launched after installation, the ES_PATH_CONF environment variable will point to the config directory

   In cmd

   cd "%ES_PATH_CONF%"

   In PowerShell

   cd "$env:ES_PATH_CONF"

5. Rename jvm.options file to something else temporarily, so that a jvm.options directory can be created, move jvm.options file into this new directory and rename back to jvm.options

   In cmd

   rename jvm.options jvm.options.temp
   mkdir jvm.options
   move jvm.options.temp jvm.options
   rename jvm.options\jvm.options.temp jvm.options

   In PowerShell

   Rename-Item jvm.options jvm.options.temp
   New-Item jvm.options -Type Directory
   Move-Item jvm.options.temp jvm.options
   Rename-Item jvm.options\jvm.options.temp jvm.options

6. Now you can run elasticsearch.exe from the command line, or start the windows service if installing as a windows service was selected. Starting Windows service from command line

   In cmd

   sc start Elasticsearch

   In PowerShell

   Start-Service Elasticsearch

[david-at-fiziico]

@russcam follow your steps can make msi installation finish successfully. However, the service still won't start. Checking the event viewer it shows:
[.NET Runtime]
Exception Info: Elastic.ProcessHosts.Process.StartupException
Stacks:
at Elastic.ProcessHosts.Process.ProcessBase.HandleException(System.Exception)
at System.Reactive.Observer`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral...
[Application Error]
Faulting application name: elasticsearch.exe, version: 7.7.0.0, timestamp:...
Faulting module name: KERNELBASE.dll, version: 6.3.9600.17031, timestamp:...

My environment is Windows Server 2012, using Administrator to launch the service.

[ghost]

@david-at-fiziico There should be one or more messages before this one. They should provide the true cause. Please also look for elasticsearch.log file.

[david-at-fiziico]

Logs is pasted here, what's the minimum requirement of RAM?

Java HotSpot(TM) 64-Bit Server VM (25.212-b10) for windows-amd64 JRE (1.8.0_212-b10), built on Apr 1 2019 22:50:23 by "java_re" with MS VC++ 10.0 (VS2010)
Memory: 4k page, physical 2096696k(1023132k free), swap 3033920k(1456520k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=75 -XX:ErrorFile=logs/hs_err_pid%p.log -XX:GCLogFileSize=67108864 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 -XX:MaxNewSize=87244800 -XX:MaxTenuringThreshold=6 -XX:NewSize=87244800 -XX:NumberOfGCLogFiles=32 -XX:OldPLABSize=16 -XX:OldSize=174489600 -XX:+PrintFlagsFinal -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:-RequireSharedSpaces -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:-UseLargePagesIndividualAllocation -XX:+UseParNewGC -XX:-UseSharedSpaces
Heap
par new generation total 76672K, used 2726K [0x00000000e0000000, 0x00000000e5330000, 0x00000000e5330000)
eden space 68160K, 4% used [0x00000000e0000000, 0x00000000e02a9b18, 0x00000000e4290000)
from space 8512K, 0% used [0x00000000e4290000, 0x00000000e4290000, 0x00000000e4ae0000)
to space 8512K, 0% used [0x00000000e4ae0000, 0x00000000e4ae0000, 0x00000000e5330000)
concurrent mark-sweep generation total 439104K, used 0K [0x00000000e5330000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 2246K, capacity 4480K, committed 4480K, reserved 1056768K
class space used 246K, capacity 384K, committed 384K, reserved 1048576K

[russcam]

I think it would be good to keep this issue focused on installation failure with 7.7.0. Since the installation succeeds for you, @david-at-fiziico, but there is some subsequent failure, it might be better if we continue this conversation over on your original issue, #370, and reopen that.

I'm going to close this issue.