[Golang] Add Integration Package with expvar Data Stream #4933

niraj-crest · 2023-01-04T18:24:13Z

Enhancement

What does this PR do?

Added 1 data stream (expvar metrics).
Added data collection logic for the data streams.
Added the ingest pipeline for the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files.
Added dashboards and visualizations.
Added system test cases for the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/golang) directory.
Run the following command to run tests. elastic-package test

Related issues

Relates Migration of Golang Integration #4864
Relates [Golang] Add Integration Package with heap Data Stream #4908

Screenshots

elasticmachine · 2023-01-04T18:27:23Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-05T05:12:45.863+0000
Duration: 14 min 37 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6
Skipped	0
Total	6

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

kush-elastic · 2023-01-05T05:12:33Z

/test

kush-elastic

LGTM!

elasticmachine · 2023-01-05T05:26:34Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚
Classes	100.0% (`1/1`)	💚
Methods	100.0% (`8/8`)	💚
Lines	95.745% (`90/94`)	👍 0.181
Conditionals	100.0% (`0/0`)	💚

packages/golang/data_stream/expvar/manifest.yml

ishleenk17 · 2023-01-11T06:28:51Z

packages/golang/data_stream/expvar/fields/fields.yml

+- name: golang
+  type: group
+  fields:
+    - name: expvar


There are many more fields in the beats sample module https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-golang-expvar.html

Are we dropping out some fields for expvar ?

Not actually, We looked into those additional fields but those fields aren’t default golang metric fields, for example, beat.cpu.system.ticks is a field of system system.cpu.system.ticks, we ran go module and its not collecting these fields, go has the functionality to expose custom metrics and these could be custom metrics for which we are providing support. If the user has these or any other custom fields exposed in their application, these will be automatically inserted in our document.

lalit-satapathy · 2023-01-18T12:41:53Z

packages/golang/data_stream/expvar/manifest.yml

@@ -0,0 +1,40 @@
+title: Golang expvar metrics
+type: logs


Wondering if there a simpler way to re-direct these metrics from filebeat httpjson to metrics-* data-stream or metrics-* data view. Explore, if it is feasible.

We tried changing it to “metrics” but we receive the following error. Apparently, it doesn’t work if there is a mismatch between input type and index type, please let us know if there is any other way that we can try.

elastic/elastic-agent#2208: This is the agent issue opened to track the same

packages/golang/data_stream/expvar/agent/stream/httpjson.yml.hbs

ishleenk17 · 2023-01-18T17:26:53Z

packages/golang/data_stream/expvar/agent/stream/httpjson.yml.hbs

+{{/if}}
+{{/if}}
+processors:
+- add_fields:


Why is this processor added here and not as part of ingest pipeline ?

We need service.address to differentiate data among different hosts if the user has added multiple integrations. We can not add this processor in pipeline since we require accessing the input parameter(hostname) here which can not be done from pipeline.

ishleenk17 · 2023-01-31T07:11:27Z

As we have planned to skip some of the gc_pause fields for now.
Please open an issue for the same. So that we have it tracked somewhere, which fields we are skipping and for what reason. @niraj-crest

ishleenk17

Looks good!
Please take care of opening an issue as shared in this comment.

niraj-crest · 2023-01-31T12:11:56Z

As we have planned to skip some of the gc_pause fields for now. Please open an issue for the same. So that we have it tracked somewhere, which fields we are skipping and for what reason. @niraj-crest

As part of this datastream, We don't need to drop any fields. There are some fields in Heap datastream that will be dropped. We'll address them as part of heap PR.

elasticmachine · 2023-02-01T05:41:42Z

Package golang - 0.2.0 containing this change is available at https://epr.elastic.co/search?package=golang

niraj-crest added 3 commits January 4, 2023 23:49

Add Golang package with expvar data stream

e255693

Update changelog entry in changelog.yml

ea01a9a

Update changelog entry in changelog.yml

4989386

kush-elastic assigned niraj-crest Jan 5, 2023

kush-elastic added enhancement New feature or request Integration:golang Golang Team:Service-Integrations Label for the Service Integrations team labels Jan 5, 2023

kush-elastic requested review from lalit-satapathy, SubhrataK and kush-elastic January 5, 2023 05:22

kush-elastic approved these changes Jan 5, 2023

View reviewed changes

ishleenk17 reviewed Jan 11, 2023

View reviewed changes

lalit-satapathy reviewed Jan 18, 2023

View reviewed changes

ishleenk17 reviewed Jan 18, 2023

View reviewed changes

ishleenk17 requested a review from a team January 19, 2023 06:42

SubhrataK approved these changes Jan 31, 2023

View reviewed changes

ishleenk17 approved these changes Jan 31, 2023

View reviewed changes

kush-elastic merged commit ad13051 into elastic:main Feb 1, 2023

andrewkroh added the New Integration label Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Golang] Add Integration Package with expvar Data Stream #4933

[Golang] Add Integration Package with expvar Data Stream #4933

niraj-crest commented Jan 4, 2023 •

edited

Loading

elasticmachine commented Jan 4, 2023 •

edited

Loading

Build stats

Test stats 🧪

kush-elastic commented Jan 5, 2023

kush-elastic left a comment

elasticmachine commented Jan 5, 2023

ishleenk17 Jan 11, 2023

harnish-elastic Jan 17, 2023

lalit-satapathy Jan 18, 2023 •

edited

Loading

niraj-crest Jan 20, 2023

ishleenk17 Jan 31, 2023

ishleenk17 Jan 18, 2023

niraj-crest Jan 20, 2023

ishleenk17 commented Jan 31, 2023 •

edited

Loading

ishleenk17 left a comment

niraj-crest commented Jan 31, 2023 •

edited

Loading

elasticmachine commented Feb 1, 2023

[Golang] Add Integration Package with expvar Data Stream #4933

[Golang] Add Integration Package with expvar Data Stream #4933

Conversation

niraj-crest commented Jan 4, 2023 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jan 4, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

kush-elastic commented Jan 5, 2023

kush-elastic left a comment

Choose a reason for hiding this comment

elasticmachine commented Jan 5, 2023

🌐 Coverage report

ishleenk17 Jan 11, 2023

Choose a reason for hiding this comment

harnish-elastic Jan 17, 2023

Choose a reason for hiding this comment

lalit-satapathy Jan 18, 2023 • edited Loading

Choose a reason for hiding this comment

niraj-crest Jan 20, 2023

Choose a reason for hiding this comment

ishleenk17 Jan 31, 2023

Choose a reason for hiding this comment

ishleenk17 Jan 18, 2023

Choose a reason for hiding this comment

niraj-crest Jan 20, 2023

Choose a reason for hiding this comment

ishleenk17 commented Jan 31, 2023 • edited Loading

ishleenk17 left a comment

Choose a reason for hiding this comment

niraj-crest commented Jan 31, 2023 • edited Loading

elasticmachine commented Feb 1, 2023

niraj-crest commented Jan 4, 2023 •

edited

Loading

elasticmachine commented Jan 4, 2023 •

edited

Loading

lalit-satapathy Jan 18, 2023 •

edited

Loading

ishleenk17 commented Jan 31, 2023 •

edited

Loading

niraj-crest commented Jan 31, 2023 •

edited

Loading