Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ECS schema for user-agent ingest processor (#37727) #37984

Merged
merged 4 commits into from Jan 30, 2019

Conversation

Projects
None yet
7 participants
@dakrone
Copy link
Member

commented Jan 29, 2019

(This is a forward-port with changes of #37727)

This switches the format of the user agent processor to use the schema from ECS.
So rather than something like this:

{
  "patch" : "3538",
  "major" : "70",
  "minor" : "0",
  "os" : "Mac OS X 10.14.1",
  "os_minor" : "14",
  "os_major" : "10",
  "name" : "Chrome",
  "os_name" : "Mac OS X",
  "device" : "Other"
}

The structure is now like this:

{
  "name" : "Chrome",
  "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36",
  "os" : {
    "name" : "Mac OS X",
    "version" : "10.14.1",
    "full" : "Mac OS X 10.14.1"
  },
  "device" : "Other",
  "version" : "70.0.3538.102"
}

This is now the default for 7.0. The deprecated ecs setting in 6.x is not
supported.

Resolves #37329

Add ECS schema for user-agent ingest processor (#37727)
This switches the format of the user agent processor to use the schema from [ECS](https://github.com/elastic/ecs).
So rather than something like this:

```
{
  "patch" : "3538",
  "major" : "70",
  "minor" : "0",
  "os" : "Mac OS X 10.14.1",
  "os_minor" : "14",
  "os_major" : "10",
  "name" : "Chrome",
  "os_name" : "Mac OS X",
  "device" : "Other"
}
```

The structure is now like this:

```
{
  "name" : "Chrome",
  "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36",
  "os" : {
    "name" : "Mac OS X",
    "version" : "10.14.1",
    "full" : "Mac OS X 10.14.1"
  },
  "device" : "Other",
  "version" : "70.0.3538.102"
}
```

This is now the default for 7.0. The deprecated `ecs` setting in 6.x is not
supported.

Resolves #37329
@elasticmachine

This comment has been minimized.

Copy link
Collaborator

commented Jan 29, 2019

@JoshMock

This comment has been minimized.

Copy link
Member

commented Jan 29, 2019

jenkins retest this please

@dakrone dakrone added the blocker label Jan 30, 2019

@jakelandis jakelandis self-requested a review Jan 30, 2019

@jakelandis
Copy link
Contributor

left a comment

LGTM

@dakrone dakrone merged commit cac6b8e into elastic:master Jan 30, 2019

8 checks passed

CLA Commit author has signed the CLA
Details
elasticsearch-ci/1 Build finished.
Details
elasticsearch-ci/2 Build finished.
Details
elasticsearch-ci/default-distro Build finished.
Details
elasticsearch-ci/docbldesx Build finished.
Details
elasticsearch-ci/docs-check Build finished.
Details
elasticsearch-ci/oss-distro-docs Build finished.
Details
elasticsearch-ci/packaging-sample Build finished.
Details

jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Jan 30, 2019

Merge branch 'master' into replication-tracker-primary-term
* master:
  Remove types from watcher docs (elastic#38002)
  Add test coverage for Painless general casting of boolean and Boolean (elastic#37780)
  Fixed test bug, lastFollowTime is null if there are no follower indices.
  Add ECS schema for user-agent ingest processor (elastic#37727) (elastic#37984)
  Extract TransportRequestDeduplication from ShardStateAction (elastic#37870)
  Expose retention leases in shard stats (elastic#37991)

@webmat webmat referenced this pull request Jan 30, 2019

Merged

Finalize user_agent migration to ECS #10441

5 of 5 tasks complete

jasontedor added a commit to dnhatn/elasticsearch that referenced this pull request Jan 31, 2019

Merge remote-tracking branch 'elastic/master' into pr/37940
* elastic/master:
  ILM setPriority corrections for a 0 value (elastic#38001)
  Temporarily disable BWC for retention lease stats (elastic#38049)
  Skip Shrink when numberOfShards not changed (elastic#37953)
  Add dispatching to `HandledTransportAction` (elastic#38050)
  Update httpclient for JDK 11 TLS engine (elastic#37994)
  Reduce flaxiness of ccr recovery timeouts test (elastic#38035)
  Fix ILM status to allow unknown fields (elastic#38043)
  Fix ILM Lifecycle Policy to allow unknown fields (elastic#38041)
  Update verify repository to allow unknown fields (elastic#37619)
  [ML] Datafeed deprecation checks (elastic#38026)
  Deprecate minimum_master_nodes (elastic#37868)
  Remove types from watcher docs (elastic#38002)
  Add test coverage for Painless general casting of boolean and Boolean (elastic#37780)
  Fixed test bug, lastFollowTime is null if there are no follower indices.
  Add ECS schema for user-agent ingest processor (elastic#37727) (elastic#37984)
  Extract TransportRequestDeduplication from ShardStateAction (elastic#37870)
@simitt

This comment has been minimized.

Copy link

commented Jan 31, 2019

I just tried to use the changed user-agent ingest processor and realized that the device implementation differs from the ECS definition:
device is implemented as keyword but in ECS defined as object with a field name that is a keyword, see https://github.com/elastic/ecs#-user-agent-fields.

@webmat

This comment has been minimized.

Copy link

commented Jan 31, 2019

Oh jeez, I made the mistake in field naming, in my initial issue 🤦‍♂️Sorry about that!

The nesting of device.name is there to allow for other fields about the device eventually. Like device.category (tablet, mobile, etc).

So the nesting is kind of important, even if there's nothing else under device. yet.

@webmat

This comment has been minimized.

Copy link

commented Jan 31, 2019

I created issue #38094 for this

@dakrone dakrone deleted the dakrone:ingest-user-agent-ecs branch Feb 4, 2019

@colings86 colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

jakelandis added a commit to jakelandis/elasticsearch that referenced this pull request Feb 12, 2019

Revert "Add ECS schema for user-agent ingest processor (elastic#37727) (
elastic#37984)"

This reverts commit cac6b8e.

Related: elastic/beats#10650

Will replace this commit with the 6.7 version

jakelandis added a commit that referenced this pull request Feb 13, 2019

Make 7.0 like 6.7 user agent ecs, but default to true (#38757)
This change reverts the initial 7.0 commits and replaces them
with the 6.7 variant that still allows for the ecs flag. 
This commit differs from the 6.7 variants in that ecs flag will 
now default to true. 

6.7: `ecs` : default `false`
7.x: `ecs` : default `true`
8.0: no option, but behaves as `true`

* Revert "Ingest node - user agent, move device to an object (#38115)"
This reverts commit 5b008a3.

* Revert "Add ECS schema for user-agent ingest processor (#37727) (#37984)"
This reverts commit cac6b8e.

* cherry-pick 5dfe193 
Add ECS schema for user-agent ingest processor (#37727)

* cherry-pick ec8ddc8 
Ingest node - user agent, move device to an object (#38115) (#38121)
  
* cherry-pick f63cbdb (with manual merge fixes)
Dep. check for ECS changes to User Agent processor (#38362)

* make true the default for the ecs option, and update 7.0 references and tests

jakelandis added a commit to jakelandis/elasticsearch that referenced this pull request Feb 13, 2019

Make 7.0 like 6.7 user agent ecs, but default to true (elastic#38757)
This change reverts the initial 7.0 commits and replaces them
with the 6.7 variant that still allows for the ecs flag. 
This commit differs from the 6.7 variants in that ecs flag will 
now default to true. 

6.7: `ecs` : default `false`
7.x: `ecs` : default `true`
8.0: no option, but behaves as `true`

* Revert "Ingest node - user agent, move device to an object (elastic#38115)"
This reverts commit 5b008a3.

* Revert "Add ECS schema for user-agent ingest processor (elastic#37727) (elastic#37984)"
This reverts commit cac6b8e.

* cherry-pick 5dfe193 
Add ECS schema for user-agent ingest processor (elastic#37727)

* cherry-pick ec8ddc8 
Ingest node - user agent, move device to an object (elastic#38115) (elastic#38121)
  
* cherry-pick f63cbdb (with manual merge fixes)
Dep. check for ECS changes to User Agent processor (elastic#38362)

* make true the default for the ecs option, and update 7.0 references and tests

jakelandis added a commit that referenced this pull request Feb 13, 2019

Make 7.x like 6.7 user agent ecs, but default to true (#38828)
Forward port of #38757

This change reverts the initial 7.0 commits and replaces them
with the 6.7 variant that still allows for the ecs flag. 
This commit differs from the 6.7 variants in that ecs flag will 
now default to true. 

6.7: `ecs` : default `false`
7.x: `ecs` : default `true`
8.0: no option, but behaves as `true`

* Revert "Ingest node - user agent, move device to an object (#38115)"
This reverts commit 5b008a3.

* Revert "Add ECS schema for user-agent ingest processor (#37727) (#37984)"
This reverts commit cac6b8e.

* cherry-pick 5dfe193 
Add ECS schema for user-agent ingest processor (#37727)

* cherry-pick ec8ddc8 
Ingest node - user agent, move device to an object (#38115) (#38121)
  
* cherry-pick f63cbdb (with manual merge fixes)
Dep. check for ECS changes to User Agent processor (#38362)

* make true the default for the ecs option, and update 7.0 references and tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.