'include functionality' for logstash config files

[ris-ppum]

We are currently using a lot of different logstash config files for several sources, and keeping tracking of the configs in a modular manner is not easily possible. Are there any plans on adding some sort of 'include functionality' to logstash.

For example:

A.conf :

filter {
  .... # do stuff A
}

B.conf :

filter {
  .... # do stuff B
}

In the actual config file We would use something like:

run.conf :

input { ... }

include('B')
include('C')

output { ... }

---

We are currently using symlinks and whole directories as config source as a workaround, but running into different problems here.

[hauluk]

(Y)
We also use a lot of different logstash config files atm, some parts are mostly the same on different projects, so a include of configs would be a nice idea to keep readability of our files

[jordansissel]

We are currently using symlinks and whole directories as config source as a workaround, but running into different problems here.

what problems?

[riosdavi]

An "include" directive that allow to reference specific config files is a "must have" feature.

[Mormaii]

+1 Would come in really handy for me since we do filters based on APIs and there's around 80 of them

[mitchcapper]

Pleae do not +1 a thread, you can emoji thumbs up or similar but +1'ing doesn't further the bug and emails everyone subscribed to it.

[mostolog]

@jordansissel @ph @suyograo Some feedback about why we upvote this proposal.

In our infrastructure, developers are able to write their own filter/grok/conf, hence we are very interested in loading "all configuration files within a directory" and being able to reload it including new files it may be created, any changes...

As stated in documentation:

 -f, --path.config CONFIG_PATH
    Load the Logstash config from a specific file or directory. If a directory is given, all files in that directory will be concatenated in lexicographical order and then parsed as a single config file. Specifying this flag multiple times is not supported. If you specify this flag multiple times, Logstash uses the last occurrence (for example, -f foo -f bar is the same as -f bar).
    You can specify wildcards (globs) and any matched files will be loaded in the order described above. For example, you can use the wildcard feature to load specific files by name:
    bin/logstash --debug -f '/tmp/{one,two,three}'
    With this command, Logstash concatenates three config files, /tmp/one, /tmp/two, and /tmp/three, and parses them into a single config.

Sadly, this multiple directory configuration doesn't seem to work with directories, but files.
ie:
bin/logstash --debug -f '/tmp/{firstFile,secondFile}' <---works
bin/logstash --debug -f '/tmp/{firstDir,secondDir}' <--- doesn't work

apart from filing an issue for the above, having an include would be very useful, in order to do something like:
include('firstDir')
include('secondDir')

Hence, any file (or new directory containing a file) added into secondDir would be detected by logstash and reloaded into current config.

Hope explanation is clear enough.

[jordansissel]

I have deleted all the "+1" comments. Please refrain from making high-noise-low-signal comments like this.

[jordansissel]

Sadly, this multiple directory configuration doesn't seem to work with directories, but files.

In your example, you use /tmp/{firstDir,secondDir} and state that it does not work. This is a bug. Let's fix that.

As a workaround until the bug you found is fixed, you can do /tmp/{firstDir,secondDir}/* which should have the same effect .