If /var/lib/elasticsearch already exists and is empty, nothing is installed

[rightaway]

When you install elasticsearch but /var/lib/elasticsearch already exists and is empty, nothing actually gets installed to that directory. I'm installing on debian 10 but it applies generally.

If I do this with other database apps it works as expected. A common way to install is to create /var/lib/elasticsearch as a symlink to an external mounted volume so that your data files get saved to that volume. Then when you run the installation elasticsearch should put everything there.

It requires workarounds to do a basic install to a path that's symlinked to another volume. This should be supported as it's a very common use case especially when installing to cloud providers that offer storage on external volumes.

[elasticmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[mark-vieira]

It requires workarounds to do a basic install to a path that's symlinked to another volume. This should be supported as it's a very common use case especially when installing to cloud providers that offer storage on external volumes.

I think in this scenario the recommended approach would simply be to set the path.data configuration setting to point to the mounted volume location you desire.

[rightaway]

@mark-vieira Is there any reason why /var/lib/elasticsearch is treated differently? If path.data is not changed from the default then it doesn't work if that directory exists.

[mark-vieira]

The /var/lib/elasticsearch directory is simply the root directly in which the packages place their data. If something exists here we bail out to avoid potentially overwriting existing data. We might consider changing this behavior, but as for you specific use case, just using the existing settings options is preferable to relying on symlinks.

[rightaway]

If the path in the settings option exists then you don't try to overwrite anything. So why overwrite if the default /var/lib/elasticsearch exists? They can be treated the same way.

[mark-vieira]

If the path in the settings option exists then you don't try to overwrite anything. So why overwrite if the default /var/lib/elasticsearch exists? They can be treated the same way.

Because we don't want to overwrite stuff based on "implicit" defaults. I mean we could check to see if the folder is empty, and then proceed in that scenario, but I'm just not sure how common that is. As I said, if folks don't want to install to that location they should configure the setting, not create symlinks and I can't think of another scenario in which that location would exist and the user would still want us to put stuff there.

To circle back to your use case, which I'd rather focus on than hypotheticals, is configuring the path.data setting a sufficient solution for you or is the symlink solution still preferable for some reason?

[rightaway]

It's preferable for consistency with other apps including database apps. They support installing to empty symlinks, so you can look in /var/lib and see all the data directories you are using in one place (the ones that are symlinked to mounted volumes and the ones that aren't). With path.data /var/lib would be missing the elasticsearch directory when you would expect to see it next to all the other database dirs.

[mark-vieira]

That all makes sense. We'll look at investigating this, although at this time I can't give any specific priority to this work.

[pugnascotia]

@rightaway I haven't been able to replicate this on Debian 11 using ES 8.3.2. I noted that installing the .deb doesn't write anything to /var/lib/elasticsearch, it just creates the directory, but pre-creating the directory didn't break installation. The only problem I ran into was being unable to start ES because /var/lib/elasticsearch had the wrong ownership. Any details you can offer would be helpful.