Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filebeat field "count" is created directly under the root of an event #778

Closed
kurromat opened this issue Jan 20, 2016 · 9 comments · Fixed by #1166

Comments

@kurromat
Copy link

commented Jan 20, 2016

When forwarding files with filebeat (1.0.1) to logstash, I see a new field called "count" that seems to always contain the value 1.

It would be great if this field could be moved into the beat section of an event, since it right now overlays a field that should be created by logstash.

Remark: I have seen this field on both linux and windows as well as on filebeat and topbeat.

This is what is coming out of logstash:

{
"message" => "2016-01-19 17:00:05,473 INFO loggingtext",
"@Version" => "1",
"@timestamp" => "2016-01-19T16:00:05.473Z",
"beat" => {
"hostname" => "somehostname",
"name" => "somehostname"
},
"count" => 1,
"fields" => {
"type" => "MyType"
},
"input_type" => "log",
"offset" => 28754,
"source" => [
[0] "MyFile"
],
"type" => "log",
"host" => "MyHost",
"MyAttribute" => "Content",
"tags" => [
[0] "MyTag"
]
}

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Mar 15, 2016

@kurromat Sorry for the late reply. I actually think this field is not necessary in filebeat and we can remove it.

@andrewkroh It is set here. Is there a reason we need it? https://github.com/elastic/beats/blob/master/filebeat/input/file.go#L67

@andrewkroh

This comment has been minimized.

Copy link
Member

commented Mar 15, 2016

All Beats have a count field. This is guaranteed by libbeat. My assumption is that this was added for Packetbeat to for the purposes of sampling. It doesn't seem to serve a purpose for most Beats.

If we use count only for Packetbeat I would retest all dashboards to make sure nothing is broken. Using a "sum of count" aggregation is common in the dashboards from what I recall.

@monicasarbu

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2016

Andrew is right. The field count is exported by all the Beats and it is not used by any Beat. If it overlays with another field, I would say to just remove it for now and add it back when we start implementing sampling.
cc-ed @tsg

@monicasarbu

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2016

It's value is always 1.

@tsg

This comment has been minimized.

Copy link
Collaborator

commented Mar 15, 2016

+1 to removing it, but we have to be careful that it might be referenced by the dashboards.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Mar 15, 2016

For the moment I would only remove it for filebeat (which doesn't have dashboards).

ruflin added a commit to ruflin/beats that referenced this issue Mar 16, 2016
Remove count field from filebeat event
This closes elastic#778

Remove count as required field from system tests checks
@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Mar 16, 2016

#1166 removes it for filebeat

@tsg tsg closed this in #1166 Mar 16, 2016

@z0rc

This comment has been minimized.

Copy link

commented Sep 14, 2016

Can we reopen this one? I'm using filebeat 1.3.0 and the count field is here. Same config with filebeat shipping to logstash via beats input. And it's always 1.

@andrewkroh

This comment has been minimized.

Copy link
Member

commented Sep 14, 2016

This was changed only in 5.x because it is non-backwards compatible change to the event schema. So you won't see it in 1.3, but you can try it in the 5.0.0-alpha6 release available on the website.

amomchilov pushed a commit to amomchilov/Filebeat that referenced this issue Apr 19, 2018
Remove count field from filebeat event
This closes elastic/beats#778

Remove count as required field from system tests checks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.