Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update http and url ECS fields #1813

Merged
merged 1 commit into from Jan 24, 2019
Merged

Update http and url ECS fields #1813

merged 1 commit into from Jan 24, 2019

Conversation

@jalvz
Copy link
Collaborator

jalvz commented Jan 18, 2019

  • loosen intake API for URL port to accept strings and numbers
  • add more tests and make sure to exercise all the fields

This includes all non indexed fields.

Fixes #1752

@jalvz

This comment has been minimized.

Copy link
Collaborator Author

jalvz commented Jan 18, 2019

Missing a few unit tests, putting it up for early review

@jalvz

This comment has been minimized.

Copy link
Collaborator Author

jalvz commented Jan 18, 2019

@elastic/apm-ui you need to be aware for this, non indexed fields root attributes are changing as per ECS:

{event}.context.request.url - > url
{event}.context.request (except url) - > http.request
{event}.context.response - > http.response

@graphaelli

This comment has been minimized.

Copy link
Member

graphaelli commented Jan 18, 2019

what's the plan with url.port ? I was expecting we would switch to a long for that in 7.0.

@jalvz jalvz force-pushed the jalvz:ecs-http branch from 4603bd8 to b461078 Jan 18, 2019
model/context.go Outdated Show resolved Hide resolved
_meta/ecs-migration.yml Show resolved Hide resolved
_meta/ecs-migration.yml Show resolved Hide resolved

- from: context.request.env
to: http.request.env
index: false

This comment has been minimized.

Copy link
@simitt

simitt Jan 21, 2019

Collaborator

not defined in ECS

This comment has been minimized.

Copy link
@simitt

simitt Jan 21, 2019

Collaborator

The standard in ECS is that if the name does not end with text then it is a keyword. This is an object, so not sure how to bring this into ECS convention.

This comment has been minimized.

Copy link
@jalvz

jalvz Jan 22, 2019

Author Collaborator

@graphaelli do you have any opinion on this?

This comment has been minimized.

Copy link
@webmat

webmat Jan 22, 2019

The recommendation is to have all fields as keyword. If you need text datatype, you should have it as a multi-field. This is the reverse of the ES default.

Since this field is not in ECS, you're free to do what you want here.

This comment has been minimized.

Copy link
@webmat

webmat Jan 22, 2019

And about things that aren't in ECS (like http.request.env), they're not automatically a problem. If we don't add it to ECS, there will never be a conflict. And in this case, it's perfectly fine for you to define this field there.

http.request.env is the variables defined in the application's context? If that's the case, I can guarantee it'll never be in ECS (as it has nothing to do with the HTTP protocol), and would not named like this (as we try to use full words as much as possible).

So no conflict here 👍

_meta/ecs-migration.yml Show resolved Hide resolved
index: false

- from: context.request.headers.user-agent
to: http.request.headers.user-agent

This comment has been minimized.

Copy link
@simitt

simitt Jan 21, 2019

Collaborator

ECS defines user_agent. While it is a root field there and not here, for the purpose of aligning I suggest to use http.request.headers.user_agent.original.

model/context.go Outdated Show resolved Hide resolved
delete(e.Context, "request")
delete(e.Context, "response")

utility.Add(fields, "context", e.Context)

This comment has been minimized.

Copy link
@simitt

simitt Jan 21, 2019

Collaborator

If I understand this PR correct, then it aims for getting rid of context. What's left here?

delete(e.Context, "response")

utility.Add(fields, "context", e.Context)

events = append(events, beat.Event{Fields: fields, Timestamp: e.Timestamp})

return events

This comment has been minimized.

Copy link
@simitt

simitt Jan 21, 2019

Collaborator

What happens with the context in span?

This comment has been minimized.

Copy link
@jalvz

jalvz Jan 22, 2019

Author Collaborator

#1804 open to discuss

@watson watson mentioned this pull request Jan 21, 2019
1 of 3 tasks complete
@jalvz jalvz force-pushed the jalvz:ecs-http branch 2 times, most recently from 53cc1fa to b3babfe Jan 22, 2019
@simitt
simitt approved these changes Jan 22, 2019
Copy link
Collaborator

simitt left a comment

I suggest to also wait for @graphaelli 's opinion about those changes, but LGTM.

@simitt

This comment has been minimized.

Copy link
Collaborator

simitt commented Jan 22, 2019

You need to (manually) update the approval files for the system-tests.


# Not in ECS
- from: context.request.headers.user-agent
to: http.request.headers.user-agent.original

This comment has been minimized.

Copy link
@webmat

This comment has been minimized.

Copy link
@jalvz

jalvz Jan 22, 2019

Author Collaborator

what is the reason for that? the actual header name is User-Agent

This comment has been minimized.

Copy link
@graphaelli

graphaelli Jan 22, 2019

Member

I think this should stay as user-agent since it's nested under http.request.headers but I'd also be ok to switch if everyone else prefers that

This comment has been minimized.

Copy link
@webmat

webmat Jan 22, 2019

In ECS it's at the top level, so not really a conflict with how you normalize header names as field names (if you decide to move it at the top level).

If you decide to leave it nested here because you prefer to list it as one of the header entries, then I'd opt to leave as is, to keep the consistent usage of '-' with other headers.

My actual suggestion would be to move this to ECS, however (and I thought I had seen another entry in this file to do so).

Moving this field to the ECS name will ensure the user agents recorded by APM can be found easily by any integration or person that supports the common schema, without having to create an exception for APM. That's the whole point of ECS, after all :-)

This comment has been minimized.

Copy link
@jalvz

jalvz Jan 23, 2019

Author Collaborator

Yes, as you noted there is another entry in this file for that. We have 2 user agents: the one in the request to the apm-server (which is moved to the ECS field at the root) and this one, which is in a request possibly captured by an APM agent.

This comment has been minimized.

Copy link
@webmat

webmat Jan 23, 2019

Are these present in the same event at the same time? If not, then both should break down the user agent in the ECS field, user_agent at the top level.

If they can both be present in the same event, then that's a more complicated question :-)

This comment has been minimized.

Copy link
@jalvz

jalvz Jan 24, 2019

Author Collaborator

Yes, they are present at the same time.

@jalvz jalvz force-pushed the jalvz:ecs-http branch 2 times, most recently from 3ec8074 to 018b0a0 Jan 23, 2019
Also loosen the Intake API so that url.port can be a string or a number.
It will be indexed as number
@jalvz jalvz force-pushed the jalvz:ecs-http branch from 018b0a0 to beec178 Jan 23, 2019
@jalvz

This comment has been minimized.

Copy link
Collaborator Author

jalvz commented Jan 23, 2019

@sqren this updates the elasticsearch/generated files again.

request.method is also lower case now, in case is relevant

Copy link
Member

graphaelli left a comment

minor feedback but lgtm

- name: scheme
type: keyword
description: >
The protocol of the request, e.g. "https:".

This comment has been minimized.

Copy link
@graphaelli

graphaelli Jan 23, 2019

Member

the example should be "https" - ecs is surprisingly specific on this - https://github.com/elastic/ecs#-url-fields.

model/context.go Show resolved Hide resolved
@@ -186,7 +186,7 @@ const ModelSchema = `{
"maxLength": 1024
},
"port": {
"type": ["string", "null"],
"type": ["string", "integer","null"],

This comment has been minimized.

Copy link
@graphaelli

graphaelli Jan 23, 2019

Member

good idea

@jalvz jalvz merged commit af76d15 into elastic:master Jan 24, 2019
1 of 3 checks passed
1 of 3 checks passed
continuous-integration/jenkins/pr-merge This commit cannot be built
Details
apm-ci Build finished.
Details
CLA Commit author is a member of Elasticsearch
Details
@zube zube bot added [zube]: Done and removed [zube]: In Review labels Jan 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.