Include templated systemd service file in packages

[tylerjl]

Describe the feature:

Although running more than one Elasticsearch process per machine is more of the exception than the rule, it's a common enough use case that both the ansible and puppet modules support this by manually defining service files for instances of Elasticsearch rather than using the service files that ship with the upstream service files. It isn't the prettiest way to run Elasticsearch, but we run into it a lot.

The primary problem here is that any upstream changes to the service files get lost once Puppet/Ansible/etc. suffer drift in the service file contents. Even if the config management service templates are kept in lock-step sync with upstream, there's the potential for different versions of service files to clash with the specific version of Elasticsearch being used (we haven't seen this yet, but it's more of a lucky streak than something we can rely on).

Prior to systemd, supporting this with the upstream Elasticsearch packages' SysV service files wasn't really an option; the desired instances at runtime can't be anticipated with service files like /etc/init.d/elasticsearch-es-01, /etc/init.d/elasticsearch-es-02, and so on. Older distros that don't support systemd will probably have to keep with this fragile method the config management solutions use and keep the service files committed as templates in their respective modules.

With all that being said, systemd specifiers permit "template" units that allow multiple instances of a service to be instantiated based on a single service file. If the Elasticsearch packages were to ship both elasticsearch.service and elasticsearch@%i.service, users could continue to use the elasticsearch.service in a backwards-incompatible manner while all users could have the additional option of using discrete services of the form elasticsearch@master.service to manage >1 instances per host if they desire to.

Pros:

• Multiple-instance use cases are natively supported by the OS packages without any reliance on downstream config management modules
• Upstream changes to service files are handled appropriately as the service file contents are no longer duplicated in config manage templates and subject to drift between projects
• This change should be backwards-compatible; the packages can still ship a generic Elasticsearch service and also the templated service instance file

Cons:

• This is only feasible for systemd-based distributions. Though we could ostensibly do this with special-named SysV files (like dropping a /etc/init.d/elasticsearch@ and symlinking instances to it), it's not supported and very hacky.
• The actual implementation for this would need to be "opinionated" in a way that we'd need to agree on, for example, that an instance's CONF_DIR resides in /etc/elasticsearch/$instance. Fortunately most (all?) of the CLI tools associated with Elasticsearch can handle CONF_DIR values outside the default, as long as it's set appropriately at the time they're invoked.

As a real-world example of what this could look like, consider the Arch Linux service template file that does exactly this (though we'd obviously need to base it on our elasticsearch.service file instead).

[jasontedor]

Thanks for opening this @tylerjl, this is something that I had been thinking about too and I very much appreciate you laid out everything so clearly here. I don't think we need to worry about Sys V init at all. I for one have come to accept our new init system overlord.

[ppf2]

+1 Here is an example of something I came across while trying to put together an install using RPM (with an additional systemd script for the 2nd instance).

2nd instance systemd script:

[Service]
RuntimeDirectory=elasticsearch
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch1
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/sysconfig/elasticsearch1

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch1.pid --quiet

The problem here is that if the 2nd instance uses the same ES_HOME, the /usr/share/elasticsearch/bin/elasticsearch script will source /usr/share/elasticsearch/bin/elasticsearch-env, and when it gets to the if [ -f /etc/sysconfig/elasticsearch ]; then source /etc/sysconfig/elasticsearch; fi line of elasticsearch-env, it will override the ES_PATH_CONF set within the service for the 2nd instance with /etc/elasticsearch again, and the 2nd instance will end up picking up /etc/elasticsearch (the 1st instance's config directory). To workaround this, I have commented out the line in the elasticsearch-env so that it will pick up the right ES_PATH_CONF (since the service script already has both ES_PATH_CONF and EnvironmentFile specified) - probably not the ideal solution here for it will break on upgrades (alternatively, use a different ES_HOME for the 2nd instance). But it does show that having native support for multiple instances will be a nice feature to have, thx!

[tylerjl]

@ppf2 indeed; the elasticsearch-env behavior that overrides any environment variables with those found in /etc/sysconfig/elasticsearch is something that the Puppet module has had to compensate for by removing it from sysconfig as you did and manually injecting it into any operations that require it for non-instance related tasks like plugin operations. FWIW it feels like elasticsearch-env overriding any existing environment variables is a bug.

[ppf2]

Linking #28159 to see if this is a bug we can address until there is formal support for multiple instances.

[jasontedor]

FWIW it feels like elasticsearch-env overriding any existing environment variables is a bug.

This is not a bug, this is behaving as intended. Reasonable people can disagree on whether or not that behavior is desirable, but it is not behaving differently than I intended it to.

[tylerjl]

@jasontedor fair enough, my main concern was whether the work around (i.e., removing any env vars in question from /etc/sysconfig/elasticsearch) would break down the road/was an unsupported way to do it, but if that's the canonical way of doing so then it works for me. @ppf2, I haven't yet run into a case wherein removing it from sysconfig and then setting it as needed for other commands (invoking ES, running plugin commands, running x-pack utilities, etc.) hasn't worked as expected.

[jasontedor]

That is completely supported. Note that is the only way to supply custom environment variables in the archive distributions (there’s no environment variable file to source, the script doesn’t even include such a line and we default ES_PATH_CONF differently). One thing I do not like about this design is different behavior for archive distributions and package distributions but there are other fundamental differences anyway so I came to terms with it. I am open to alternatives but to reiterate this is deliberate so any changes here would be breaking.

[siddharthgoel88]

Hey @jasontedor . May be I missed the point in this whole change of conversations, I still have the question.

How is one expected to extend custom environment variables in RPMs?

From the product point of view, I believe all the deliverable (zip, rpm, msi, ...) should behave the same way. We had always been able to configure the environment variables previously and it gave us good flexibility. And now with this change it is only supported in archive it becomes a problem for us.

I would vote to add this flexibility in the script for this extension in RPMs also. What do you think about it?

[martinb3]

FWIW, this also broke the elasticsearch Chef cookbook on v6.x+. It's pretty weird that the init scripts respect existing variables but elasticsearch-env doesn't. (Is that change documented? I couldn't find it when I reviewed the guidance on upgrading and breaking changes.)

It sounds like we should always remove /etc/sysconfig/elasticsearch or /etc/default/elasticsearch, and always write an elasticsearch-env file instead? Is that the way forward?

How do we write multiple elasticsearch-env files for multiple instances? Or do we also never share ES_HOME either?

[rjernst]

We discussed this today in FixItThursday, and agreed that we either need to document examples of how to run multiple nodes, or fully support it within our systemd service file(s). Given that we would want testing of such documentation, actually adding support makes that testing easier, thus we have agreed to move forward with adding templated service files.

[zez3]

Any progress or hope that this will be included in the upcoming 7.14 or 8 release?

[nemphys]

+1 on the template file (although changes to either elasticsearch-env or /etc/default/elasticsearch will be needed, see #28159 (comment))

[dennisse]

FWIW, just using the command line argument -E key=value works well in a systemd-template. At least if you don't need the actual config to be complete (like for using the cli).

elasticsearch@.service example:

[Unit]
Description=Elasticsearch %i
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
RuntimeDirectory=elasticsearch
PrivateTmp=true
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch
Environment=PID_DIR=/run/elasticsearch
Environment=ES_SD_NOTIFY=true
EnvironmentFile=-/etc/default/elasticsearch
PIDFile=/run/elasticsearch/%i.pid

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

ExecStartPre=/usr/bin/mkdir -p /var/lib/elasticsearch/%i /var/log/elasticsearch/%i

ExecStart=/usr/share/elasticsearch/bin/elasticsearch --quiet \
    -p ${PID_DIR}/%i.pid \
    -Epath.data=/var/lib/elasticsearch/%i \
    -Epath.logs=/var/log/elasticsearch/%i \
    -Ehttp.port=920%i \
    -Enode.name=%H-%i \
    -Enode.attr.host=%H

[...]

[Install]
WantedBy=multi-user.target

Any shared config between the instances goes in the config file. Any instance-specific settings goes in the systemd-template, as shown above.

With this, you can spin up instances with systemctl start elasticsearch@1.service, and it'll start up an instance that listens to http port 9201 (and so on).