Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metricbeat: Add mapping for docker metrics per cpu #6843

Merged
merged 2 commits into from Apr 19, 2018

Conversation

@jsoriano
Copy link
Member

commented Apr 12, 2018

Fixes #6785

@jsoriano jsoriano requested a review from ruflin Apr 12, 2018
@jsoriano jsoriano force-pushed the jsoriano:docker-per-cpu-fields-mapping branch from 2e3ea14 to 2025f9f Apr 12, 2018
@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Apr 13, 2018

This will probably not work. You have to use dynamic templates for this: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/dynamic-templates.html

So far the way we use dynamic templates is that we specify it as following:

    - name: labels
      type: object
      object_type: keyword
      description: >
        Image labels.

In the template this outputs:

        {
          "docker.image.labels": {
            "mapping": {
              "type": "keyword"
            },
            "match_mapping_type": "string",
            "path_match": "docker.image.labels.*"
          }
        },

I could be that for this case an extension to our template generation is needed. Best have a look at libbeat/template/processor.go and create a test in processor_test.go to see if you get the expected output.

@jsoriano jsoriano force-pushed the jsoriano:docker-per-cpu-fields-mapping branch from 2025f9f to b0fcb07 Apr 13, 2018
@@ -242,12 +258,16 @@ func addDynamicTemplate(f *common.Field, properties common.MapStr, matchType str
if len(f.Path) > 0 {
path = f.Path + "."
}
path_match := path + f.Name
if !strings.ContainsRune(path_match, '*') {
path_match += ".*"

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Apr 13, 2018

don't use underscores in Go names; var path_match should be pathMatch

@@ -242,12 +258,16 @@ func addDynamicTemplate(f *common.Field, properties common.MapStr, matchType str
if len(f.Path) > 0 {
path = f.Path + "."
}
path_match := path + f.Name

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Apr 13, 2018

don't use underscores in Go names; var path_match should be pathMatch

@jsoriano jsoriano force-pushed the jsoriano:docker-per-cpu-fields-mapping branch 3 times, most recently from 11b7a99 to e144525 Apr 13, 2018
@jsoriano

This comment has been minimized.

Copy link
Member Author

commented Apr 13, 2018

Trying to generate the dynamic mapping, still have to test it for real.

@jsoriano jsoriano force-pushed the jsoriano:docker-per-cpu-fields-mapping branch from e144525 to 80a4e3c Apr 13, 2018
@@ -208,6 +209,21 @@ func (p *Processor) object(f *common.Field) common.MapStr {
}

switch f.ObjectType {
case "scaled_float":

This comment has been minimized.

Copy link
@ruflin

ruflin Apr 16, 2018

Collaborator

Probably we will have a request for float and integer here pretty soon. As soon this pops up we should refactor the code to remove the duplicated parts, if possible

This comment has been minimized.

Copy link
@ruflin

ruflin Apr 16, 2018

Collaborator

Just saw further below that long we already have. float should be rather simple to add in the same manner.

This comment has been minimized.

Copy link
@jsoriano

jsoriano Apr 16, 2018

Author Member

For scaled float I could reuse the existing scaledFloat method.

This comment has been minimized.

Copy link
@ruflin

ruflin Apr 17, 2018

Collaborator

We can still clean up the code in a follow up PR.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Apr 16, 2018

Change looks good to, looking forward to your manual testing. The template can be exported with metricbeat export template.

@jsoriano jsoriano force-pushed the jsoriano:docker-per-cpu-fields-mapping branch 3 times, most recently from 74fe717 to 333aa1c Apr 16, 2018
@@ -208,6 +209,9 @@ func (p *Processor) object(f *common.Field) common.MapStr {
}

switch f.ObjectType {
case "scaled_float":
dynProperties = p.scaledFloat(f)
addDynamicTemplate(f, dynProperties, matchType("*"))

This comment has been minimized.

Copy link
@jsoriano

jsoriano Apr 16, 2018

Author Member

I use * by default because we want it to match at least all numeric types.

@jsoriano

This comment has been minimized.

Copy link
Member Author

commented Apr 16, 2018

jenkins, test it again please, it seems that there has been an issue with pypi

@jsoriano

This comment has been minimized.

Copy link
Member Author

commented Apr 16, 2018

jenkins, test it

@jsoriano

This comment has been minimized.

Copy link
Member Author

commented Apr 16, 2018

jenkins, test this again

Copy link
Collaborator

left a comment

Did you manage to test this locally? Code LGTM. PR will need need a rebase because of the CHANGELOG.

@@ -208,6 +209,21 @@ func (p *Processor) object(f *common.Field) common.MapStr {
}

switch f.ObjectType {
case "scaled_float":

This comment has been minimized.

Copy link
@ruflin

ruflin Apr 17, 2018

Collaborator

We can still clean up the code in a follow up PR.

@jsoriano jsoriano force-pushed the jsoriano:docker-per-cpu-fields-mapping branch from 333aa1c to 9553a5e Apr 18, 2018
@jsoriano

This comment has been minimized.

Copy link
Member Author

commented Apr 18, 2018

Yes, I could test it locally, it works fine :)

Branch rebased.

@jsoriano jsoriano removed the in progress label Apr 18, 2018
@@ -79,6 +79,8 @@ https://github.com/elastic/beats/compare/v6.0.0-beta2...master[Check the HEAD di
- Fix Kubernetes calculated fields store. {pull}6564{6564}
- Exclude bind mounts in fsstat and filesystem metricsets. {pull}6819[6819]
- Don't stop Metricbeat if aerospike server is down. {pull}6874[6874]
- Fix Kubernetes calculated fields store. {pull}6564[6564]

This comment has been minimized.

Copy link
@ruflin

ruflin Apr 18, 2018

Collaborator

Seems like that line of changelog sneaked in?

This comment has been minimized.

Copy link
@jsoriano

jsoriano Apr 18, 2018

Author Member

Umm, I don't know why this happens :? removing it

This comment has been minimized.

Copy link
@ruflin

ruflin Apr 18, 2018

Collaborator

rebase with 3 way merges. Unfortunately a common issue.

@jsoriano jsoriano force-pushed the jsoriano:docker-per-cpu-fields-mapping branch from 9553a5e to b4ee1da Apr 18, 2018
@ruflin ruflin merged commit 8e29a8b into elastic:master Apr 19, 2018
6 checks passed
6 checks passed
CLA Commit author has signed the CLA
Details
Hound No violations found. Woof!
beats-ci Build finished.
Details
codecov/patch 100% of diff hit (target 62.97%)
Details
codecov/project 62.99% (+0.01%) compared to 55fe475
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@ctindel

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2018

@ruflin Will this go into any of the 6.x releases as well?

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Apr 19, 2018

@ctindel yes, 6.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.