How to use environment variable in 30-output.conf file

[mominmaaz]

I am running logstash(sebp/elk) in docker. The below mentioned configuration is unable to use environmental variable 'MYVAL'. I am adding environment parameter in docker-compose.yml.

30-output.conf

input {
elasticsearch{
...
}
}

filter {
if ![my_field] {
mutate {
replace => { "my_field" => "${MYVAL:default-value}" }
}
}
}

output {
elasticsearch {
...
}
stdout {
codec => rubydebug {metadata => true}
}
}

[spujadas]

tl;dr - Due to the way Logstash is started in the image, I’m not sure this is feasible in an easy way.

In the image, Logstash is started as a service, and AFAIK env vars are unset by systemd before the service is started, so indeed it can’t see MYVAL even though you’ve set it in your docker-compose.yml file (and if you bash into the container, you’ll see it).

For a number of predefined env vars (see image documentation), the start.sh script "manually" sets the env vars in the service start-up files (see e.g. https://github.com/spujadas/elk-docker/blob/master/start.sh#L187-L191 for the LS_OPTS env var).
But other ad hoc env vars such as your MYVAL would be discarded when the service is launched.

In terms of solutions:

• If you know the names of the env vars ahead of time, you could do something similar to what I did in the start.sh script to make them visible to the Logstash service.
• The accepted solutions at https://stackoverflow.com/questions/52571577/issues-in-accessing-docker-environment-variables-in-systemd-service-files (for Docker specifically) and https://serverfault.com/questions/413397/how-to-set-environment-variable-in-systemd-service (for systemd-managed services in general) might help you design a custom solution.

[mominmaaz]

Thanks this helps.