Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic lookup processor #2186

Closed
urso opened this issue Aug 8, 2016 · 0 comments

Comments

Projects
None yet
1 participant
@urso
Copy link
Collaborator

commented Aug 8, 2016

Sometimes it might be only possible to collect additional meta-information for events on host collecting data only.

Proposal: lookup processor with running executable scripts

Syntax. Lookup forms a namespace for potentially different lookup backends like:

processors:
  - lookup:
      [condition]
      [query-runner]

With lookup processor proposed here will be named exec. Using partially collapsed naming filter can be used more compact like:

processors:
- lookup.exec:
    [options]
    [condition]

The exec lookup processor will execute some configurable query, which has to return an JSON-object. The JSON object will be treated like fields config, adding/merging the returned fields into the fields field of the active event.

The scripts stdout is read and parsed as JSON object to be merged with event fields-field. The stderr is partially collected to construct error messages on failure.

[condition]: Conditions being optional enable lookup only for selected events. Examples:

  • filter by document type due to lookup only needed for one or two prospectors (file types)
  • enable lookup only for system/cpu metricset in metricbeat

Conditions support are automatically provided to all processors by libbeat.

[options]:

  • run: format string for passing arguemnts to script based on the active event. If event fields are missing, the script will not be executed
  • fields_under_root: merge returned fields right into root object
  • expire: remove entries from lookup cache if not used for configured time interval (default 1m)
  • key: list of field names used to compute the compound cache key from the active event. If not present, the key is determined by the run options field accessors. Keys computed from events must be unique in regard to key values and order of keys being extracted to eliminate the chance of collisions.
  • timeout: kill process after (must be >0)
  • user: run script as configured user (required)
  • group: run script with group id (optional?)
  • chroot: run script within chroot environment
  • working_directory: run script from this path
  • ...: more security related options as required?

Example configuration filebeat:

processors:
- lookup.exec:
    run: log-meta.sh %{[filename]}
    when.equals:
      type: logs  # document_type in filebeat prospector

Example configuration for metricbeat:

processors:
- lookup.exec:
    run: cgroups-meta.sh %{[pid]}
    timeout: 100ms
    user: mbinfo
    key: ['start_time', 'pid']
    expire: 180s
    when.equals:
      metric: cgroups
      metricset: process

Security:

Proposed behavior to reduce the surface area of potential vulnerabilities:

(list partially inspired from here: https://httpd.apache.org/docs/2.4/suexec.html)

  • The lookup functionality is disabled by default and can only be enabled via a CLI flag, e.g. -enableLookup or something similar. The goal is that if central configuration management is used, if the lookup functionality is not needed, it wouldn't be enough for the attacker to enable the script via configuration to execute arbitrary programs from the servers.
  • A user id and group id can be configured in the lookup functionality. We need names, or can we just ask the user to configure IDs?
  • If uid and gid are not provided, the uid/gid of the current process are used, but all the following checks are still made, including that uid/gid cannot be root.
  • On startup the Beat makes the following checks, and exit with errors if any of these are not met:
    • the user id and group id exist and are valid on the system
    • the user id must not be root
    • the group id must not be root
    • the user id must not be lower than 500, to avoid system users
    • the script must be configured by an absolute path. Env $PATH must not be used.
    • the script path must not be under /bin or /usr/bin to avoid executing things like /bin/bash or /usr/bin/python
  • On startup and on each script execution the Beat makes the following checks:
    • the script path must not be a symlink
    • the script must be owned by the user id
    • the script must be writable only by the owner (the configured user id)
    • the script is executable
    • the script doesn't have the setuid or setgid flags
  • When executing the script:
    • should make sure not to use any shell wrapping, to avoid potential use of redirections, sub-shells, a.s.o
    • should close stdin (stdout, stderr used for collecting results)
    • should cleanup the environment

Limitations:

  • fields returned/computed by script MUST NOT CHANGE for a given compound key during data collection. While meta-data will be cached with epxiration (removing unused entries), there is a very high chance of races between meta-data updates and event-generation (e.g. back-pressure in filebeat). Races will result in having the wrong meta-data being added to old events.
    => lookup is for adding dynamic, but time-invariant meta-data to events.
  • some run settings require setting some syscall attributes, which might not be supported will by all OSes. Sometimes not all settings required are exposed, limiting security requirements on some systems (consider adding customized fork-exec support + add back to golang stdlib?).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.