Undefined log.file.* fields breaking tests for filestream inputs

[bhapas]

What does this PR do?

Issue described in #7687

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Relates [8.10] Undefined log.file.* fields breaking tests for filestream inputs #7687

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-09-28T14:52:07.846+0000

• Duration: 26 min 32 sec

Test stats 🧪

Test	Results
Failed	0
Passed	62
Skipped	0
Total	62

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (10/10)	💚
Files	100.0% (11/11)	💚
Classes	100.0% (11/11)	💚
Methods	95.146% (98/103)	❕
Lines	80.69% (1099/1362)	❕
Conditionals	100.0% (0/0)	💚

[andrewkroh]

The more I look at this problem I think we need to go back to the ingest team, and talk about changing the field types to be strings for the reason given in #7716 (comment). Otherwise, this will be an uphill battle to keep a consistent mapping type between integrations. As soon as an integration forgets to map these numeric fields then there will be a conflict with the integrations that do map it as keyword. Today there are 22 separate data streams associated with the filestream input that need mappings for these fields.

Secondly, the fields are uint64¹² in Beats. The default mapping will be long³, and this is not the correct type because a uint64 does not fit into long (i.e. 2^64 > 2^63).

Third, this would be an opportunity to align the definitions between operating systems and propose these to ECS. Like perhaps

	Type	Unix	Windows
log.file.device_id	string	Device ID	NTFS volume
log.file.inode	string	Inode number in decimal	File index number in decimal (high and low order 32-bit values combined to form a uint64)

@rdner @ycombinator Would you be open to some changes to these fields in the input?

Footnotes

1. https://github.com/elastic/beats/blob/8ff6894cab0591cc3a898d7abd6a9103e535efdd/libbeat/common/file/file_windows.go#L32-L34 ↩

2. https://github.com/elastic/beats/blob/f0215ff5cfdbcb0db8ebc1f07af1b4885081389b/libbeat/common/file/file_other.go#L29-L30 ↩

3. https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html ↩

[bhapas]

The more I look at this problem I think we need to go back to the ingest team, and talk about changing the field types to be strings for the reason given in #7716 (comment). Otherwise, this will be an uphill battle to keep a consistent mapping type between integrations. As soon as an integration forgets to map these numeric fields then there will be a conflict with the integrations that do map it as keyword. Today there are 22 separate data streams associated with the filestream input that need mappings for these fields.

Secondly, the fields are uint6412 in Beats. The default mapping will be long3, and this is not the correct type because a uint64 does not fit into long (i.e. 2^64 > 2^63).

Third, this would be an opportunity to align the definitions between operating systems and propose these to ECS. Like perhaps

Type Unix Windows
log.file.device_id string Device ID NTFS volume
log.file.inode string Inode number in decimal File index number in decimal (high and low order 32-bit values combined to form a uint64)
@rdner @ycombinator Would you be open to some changes to these fields in the input?

Footnotes

1. https://github.com/elastic/beats/blob/8ff6894cab0591cc3a898d7abd6a9103e535efdd/libbeat/common/file/file_windows.go#L32-L34 ↩
2. https://github.com/elastic/beats/blob/f0215ff5cfdbcb0db8ebc1f07af1b4885081389b/libbeat/common/file/file_other.go#L29-L30 ↩
3. https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html ↩

Created an issue for this in beats elastic/beats#36695

[rdner]

@andrewkroh I'd like to point out 2 things:

1. initially log.file.device, log.file.inode, log.file.idxhi, log.file.idxlo and log.file.vol were strings and during the review were switched to numbers based on the argument that someone would like to use a range search on these values.

2. Regarding the file.device field in ECS it's not the same as log.file.device_id as I explained during the review here Add file system information to each event beats#36065 (comment)

I'm fine with reverting everything back to strings if @ycombinator is fine with it.

I'm surprised adding these fields caused so many issues, apologies. Next time, I will involve integration developers into a review when we add more fields.

[ycombinator]

Yep, I'm good with using strings / keywords for these fields given how they're intended to be used.

[andrewkroh]

I think we need better technical descriptions of these fields. I have made suggestions. PTAL

@rdner Can you please also review these descriptions to ensure their accuracy? Where are these field descriptions in Filebeat (example)?

[rdner]

@andrewkroh the descriptions you suggested looks perfect to me.

Regarding the example file, should every event field be there? I'm asking because it's missing a few fields, for example, log.file.path is not there either.

What's the purpose of this file exactly?

[andrewkroh]

Regarding the example file, should every event field be there? I'm asking because it's missing a few fields, for example, log.file.path is not there either.

These files drive our documentation and the Elasticsearch index templates. log.file.path is not in that file because it is defined in another file in the repo that provides the ECS fields. On the website you will see log.file.path in the ECS section and log.file.offset in another section because that predates the existence of ECS.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[ishleenk17]

Considering that these fields are currently numeric and would be changed to string as per this comment, is it fine to use keyword here as of now?

[andrewkroh]

Any JSON numeric value can be mapped as keyword in ES without conversion to a JSON string.

[ishleenk17]

Looks good!

[elasticmachine]

Package apache_tomcat - 0.13.0 containing this change is available at https://epr.elastic.co/search?package=apache_tomcat

[elasticmachine]

Package coredns - 0.5.0 containing this change is available at https://epr.elastic.co/search?package=coredns