Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the ability to set custom fields #1092

Merged
merged 1 commit into from Mar 3, 2016

Conversation

Projects
None yet
6 participants
@andrewkroh
Copy link
Member

commented Mar 2, 2016

This pull request implements the feature requested in #726.

Changes Affecting All Beats
  • Added a fields and fields_under_root as options available
    under the shipper configuration. These settings are analogous to settings
    currently available in Filebeat. fields defined in the prospector
    configuration take precedence over fields define in the shipper
    configuration.
Changes Affecting Filebeat
  • Added the ability to set a list of tags for each prospector. These tags will
    be appended to the list of tags specified in the shipper configuration.
  • Scalar values under fields are not longer automatically converted to strings.
Changes Affecting Winlogbeat
  • Added the ability to set tags, fields, and fields_under_root as options
    for each event log.

Examples

Topbeat

shipper:
  fields_under_root: true
  fields:
    aws:
      instance_id: i-33458498
      region: us-east-1
output:
  console:
    pretty: true
{
  "@timestamp": "2016-03-02T03:53:49.793Z",
  "aws": {
    "instance_id": "i-33458498",
    "region": "us-east-1"
  },
  "beat": {
    "hostname": "x",
    "name": "x"
  },
  "count": 1,
  "proc": {
    "cmdline": "/usr/bin/agentX",
    "cpu": {
      "start_time": "Feb05",
      "system": 7178,
      "total": 10789,
      "total_p": 0,
      "user": 3611
    },
    "mem": {
      "rss": 2162688,
      "rss_p": 0,
      "share": 0,
      "size": 2593021952
    },
    "name": "agentX",
    "pid": 395,
    "ppid": 1,
    "state": "running",
    "username": "someuser"
  },
  "type": "process"
}

Filebeat

filebeat:
  prospectors:
    - paths: ["/var/log/myapp/log.json"]
      fields: {app_id: 456789}
      tags: [json]
output:
  logstash:
    hosts: ["localhost:5044"]
shipper:
  tags: ['digital ocean', centos7]
  fields:
    customer_id: 1234567
{
  "@timestamp": "2016-03-02T04:12:32.491Z",
  "beat": {
    "hostname": "x",
    "name": "x"
  },
  "count": 1,
  "fields": {
    "app_id": 456789,
    "customer_id": 1234567
  },
  "input_type": "log",
  "message": "{ \"threadName\": \"MainThread\", \"name\": \"root\", \"thread\": 140735202359648, \"created\": 1336281068.506248, \"process\": 41937, \"processName\": \"MainProcess\", \"relativeCreated\": 9.100914001464844, \"module\": \"tests\", \"funcName\": \"testFormatKeys\", \"levelno\": 20, \"msecs\": 506.24799728393555, \"pathname\": \"tests/tests.py\", \"lineno\": 60, \"asctime\": [\"12-05-05 22:11:08,506248\"], \"message\": \"testing logging format\", \"filename\": \"tests.py\", \"levelname\": \"INFO\", \"special\": \"value\", \"run\": 12 }",
  "offset": 0,
  "source": "/var/log/myapp/log.json",
  "tags": [
    "digital ocean",
    "centos7",
    "json"
  ],
  "type": "log"
}

Winlogbeat

winlogbeat:
  event_logs:
    - name: MyCustomEventLog
      tags: [web]
      fields_under_root: true
      fields: {service_id: web01}
shipper:
  fields_under_root: true
  fields:
    aws:
      instance_id: i-33458498
      region: us-east-1
output:
  console:
    pretty: true

See the test cases for more examples.

Open Questions

  • Are you happy with keeping this called fields? (same as it's called in Filebeat)

@andrewkroh andrewkroh force-pushed the andrewkroh:feature/fields-and-tags branch from 66b1dd0 to 87e1371 Mar 2, 2016

@andrewkroh

This comment has been minimized.

Copy link
Member Author

commented Mar 2, 2016

The Jenkins failure is caused by #1083.

@@ -57,11 +57,11 @@ type ProspectorConfig struct {
}

type HarvesterConfig struct {
common.EventMetadata `config:",inline"` // Fields and tags to add to events.

This comment has been minimized.

Copy link
@ruflin

ruflin Mar 2, 2016

Collaborator

+1 on approaching this directly with an eventMetadata object. That makes it easy to extend it later with other fields.

}
}

// Add fields and override.

This comment has been minimized.

Copy link
@ruflin

ruflin Mar 2, 2016

Collaborator

Should we log a warning when fields are overwritten, or some debug info? Overwriting can also be on purpose.

This comment has been minimized.

Copy link
@urso

urso Mar 2, 2016

Collaborator

some fields are special (e.g. @timestamp).

This comment has been minimized.

Copy link
@andrewkroh

andrewkroh Mar 2, 2016

Author Member

Filebeat logged a warning when overwriting. I think warning would be too verbose if overwriting were done on purpose (either to overwrite a field defined in the shipper config or to overwrite one of the events other fields). The debug level would be ok if it were isolated to its own selector. But I'm thinking to omit it to keep the publishing path as fast a possible. If you feel strongly about it, I'm happy to add it.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Mar 2, 2016

LGTM.

Even though fields has its issue, as far as a I know we haven't found a better option yet and people got used to call it fields, so I would stay with it.

What happens if inside a prospector fields_under_root:true is set and in the shipper fields_under_root:false?

@monicasarbu

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2016

Great work! LGTM

@urso

This comment has been minimized.

Copy link
Collaborator

commented Mar 2, 2016

we still need tags, if we got fields?

Optional fields that you can specify to add additional information to the
output. For example, you might add fields that you can use for filtering log
data. Fields can be scalar values, arrays, dictionaries, or any nested
combination of these. All scalar values will be interpreted as strings. By

This comment has been minimized.

Copy link
@urso

urso Mar 2, 2016

Collaborator

All scalar values will be interpreted as strings

This is not true anymore. Types are currently preserved.

This comment has been minimized.

Copy link
@andrewkroh

andrewkroh Mar 2, 2016

Author Member

I removed this sentence from the docs. I didn't see your comment prior to asking you the question on Slack.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Mar 2, 2016

@urso About tags vs fields: I was thinking the same first but I get the feeling people use them differently. I would keep both if it doesn't add much complexity.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Mar 2, 2016

@andrewkroh

@andrewkroh andrewkroh force-pushed the andrewkroh:feature/fields-and-tags branch from 87e1371 to 4b9e118 Mar 2, 2016

@andrewkroh

This comment has been minimized.

Copy link
Member Author

commented Mar 2, 2016

We could do away with tags since you could implement them with fields. But you couldn't merge the "global" and "local" tags if you used fields to implement tags yourself.

@andrewkroh

This comment has been minimized.

Copy link
Member Author

commented Mar 2, 2016

What happens if inside a prospector fields_under_root:true is set and in the shipper fields_under_root:false?

The prospector fields will be added to the root of the document and the shipper fields will be under the fields key.

There seems to be an issue on windows: https://ci.appveyor.com/project/elastic-beats/beats/build/1568/job/ls2h81bajqmoaghk

This should be fixed now. It was due to scalars not being converted to strings under ucfg. The docs have been updated to reflect this change in behavior and the test fixed.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Mar 3, 2016

@andrewkroh Interesting about the fields_under_root. I like it.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Mar 3, 2016

LGTM. @andrewkroh Can you rebase?

}
*ms = cleanUpInterfaceMap(result)

// Add fields and override.

This comment has been minimized.

Copy link
@ruflin

ruflin Mar 3, 2016

Collaborator

@andrewkroh Seems like the old comment disappeared. The main reason I was thinking about logging is for debugging reason in case we get the info that some fields disappear. Lets leave it out for now and discuss it in case we hit such a case.

Add the ability to set custom fields in shipper config
Add the ability to set tags in Filebeat prospector config
Add the ability to set tags and fields in Winlogbeat event log config
Update docs to remove wording about all scalars being changed to strings.

Closes #726

@andrewkroh andrewkroh force-pushed the andrewkroh:feature/fields-and-tags branch from 4b9e118 to 1043c19 Mar 3, 2016

@andrewkroh

This comment has been minimized.

Copy link
Member Author

commented Mar 3, 2016

It's rebased. Jenkins is failing because of a docker problem.

ruflin added a commit that referenced this pull request Mar 3, 2016

Merge pull request #1092 from andrewkroh/feature/fields-and-tags
Add the ability to set custom fields

@ruflin ruflin merged commit 974c514 into elastic:master Mar 3, 2016

3 of 4 checks passed

default Merged build finished.
Details
CLA Commit author has signed the CLA
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@andrewkroh andrewkroh deleted the andrewkroh:feature/fields-and-tags branch Mar 4, 2016

@danielmotaleite

This comment has been minimized.

Copy link

commented Mar 28, 2016

Any change of back-porting this to the current topbeat and packetbeat releases? i don't want to upgrade live servers with nightly builds, but i need this custom fields to be able to proper filter things on kibana.

@andrewkroh

This comment has been minimized.

Copy link
Member Author

commented Mar 28, 2016

This feature will be included in the upcoming 1.2.0 5.0 release.

@andrewkroh

This comment has been minimized.

Copy link
Member Author

commented Mar 28, 2016

I amended my previous comment. I was thinking of environment variable replacement (which is in 1.2). The custom fields feature is in 5.0.

@PauliniMrtns

This comment has been minimized.

Copy link

commented Mar 21, 2018

Great work,
It would be great if it could be expanded to other beats as well. I opened an issue regarding Metricbeat for instance, where I need to spin-off some additional fields based on existing fields that already exist using metricbeat.

Thanks!

@andrewkroh

This comment has been minimized.

Copy link
Member Author

commented Mar 21, 2018

Every Beat has support for fields, fields_under_root, and tags. And most support setting them at the module level too. Checkout the documentation for Metricbeat:

@PauliniMrtns

This comment has been minimized.

Copy link

commented Mar 21, 2018

Yes but this fields are static right? You define them at the metricbeat.yml if I'm not mistaken. Is there a way to make them variable according to another field pulled from a module? For instance vsphere.virtualmachine.name ?

Thank you!

@andrewkroh

This comment has been minimized.

Copy link
Member Author

commented Mar 21, 2018

These fields are static. You could mutate the event in Logstash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.