Add metric_type mapping to stat datastream of HA Proxy #7183

agithomas · 2023-07-28T16:59:36Z

Type of Change

Enhancement

What does this PR do?

Add metric_type mapping to stat datastream of HA Proxy

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

elastic-package build
elastic-package stack up -v -d --services package-registry

Related issues

relates [HA Proxy] TSDB Enablement #7138

elasticmachine · 2023-07-28T17:17:32Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-08-01T09:21:21.455+0000
Duration: 15 min 1 sec

Test stats 🧪

Test	Results
Failed	0
Passed	40
Skipped	0
Total	40

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-07-28T17:17:42Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚
Classes	100.0% (`1/1`)	💚
Methods	90.476% (`19/21`)	👎 -6.494
Lines	97.297% (`180/185`)	👍 2.979
Conditionals	100.0% (`0/0`)	💚

lalit-satapathy · 2023-07-31T10:04:43Z

packages/haproxy/data_stream/stat/fields/fields.yml

          description: |
            Number of connection reuses.
        - name: idle
          type: group
          fields:
            - name: total
              type: long
+              metric_type: gauge


idle.total is it a counter instead?

The schema definition is srv_icur.1:MGP. So, it is a gauge type.

should it be named like idle.current instead?

Yes, it would have been optimal. The description goes as Current number of idle connections available for reuse on this server.

If it is to be changed, I would prefer that taken up as a separate enhancement with an assessment of impact (say dashboard changes). Agree?

Yes I agree, any un-related changes can be filed as a separate backlogs. In this doc also: idle section coming empty.

"idle": {},

I generated some traffic to populate the value for idle. Please find the extract from the document below

"connection": { "reuse.total": 7, "retried": 0, "cache": {}, "idle": { "total": 1 }, "attempt.total": 290, "time.avg": 1 }, "queue": { "time.avg": 0 }, "status": "UP", "client.aborted": 1, "compressor": {} }

lalit-satapathy · 2023-07-31T10:08:49Z

@agithomas, Can you share a sample "stat" document to cross-check the metric types once?

agithomas · 2023-07-31T10:17:19Z

@agithomas, Can you share a sample "stat" document to cross-check the metric types once?

{
    "haproxy": {
      "stat": {
        "server": {
          "id": 0
        },
        "request": {
          "total": 0,
          "rate": {
            "max": 0,
            "value": 0
          },
          "denied_by_connection_rules": 0,
          "denied_by_session_rules": 0,
          "denied": 0,
          "intercepted": 0,
          "errors": 0
        },
        "agent": {
          "fall": "",
          "health": "",
          "description": "",
          "check": {
            "health": "",
            "description": "",
            "rise": ""
          },
          "rise": "",
          "status": ""
        },
        "cookie": "",
        "session": {
          "current": 0,
          "total": 0,
          "max": 0,
          "rate": {
            "max": 0,
            "limit": 0,
            "value": 0
          },
          "limit": 524262
        },
        "service_name": "FRONTEND",
        "in.bytes": 0,
        "source": {
          "address": ""
        },
        "check": {
          "agent.last": "",
          "health.last": "",
          "status": ""
        },
        "out.bytes": 0,
        "proxy": {
          "mode": "http",
          "name": "stats",
          "id": 2
        },
        "component_type": 0,
        "response": {
          "http": {
            "1xx": 0,
            "2xx": 0,
            "other": 0,
            "3xx": 0,
            "4xx": 0,
            "5xx": 0
          },
          "denied": 0
        },
        "load_balancing_algorithm": "",
        "header": {
          "rewrite": {
            "failed": {
              "total": 0
            }
          }
        },
        "connection": {
          "total": 0,
          "cache": {
            "hits": 0,
            "lookup.total": 0
          },
          "rate": 0,
          "idle": {},
          "rate_max": 0
        },
        "queue": {},
        "status": "OPEN",
        "compressor": {
          "response.bytes": 0,
          "in.bytes": 0,
          "bypassed.bytes": 0,
          "out.bytes": 0
        }
      }
    }
}

agithomas · 2023-07-31T11:11:07Z

Document extract after traffic inflow

"haproxy": {
      "stat": {
        "server": {
          "backup": 0,
          "aborted": 0,
          "active": 1,
          "id": 1
        },
        "request": {
          "connection.errors": 0,
          "total": 297,
          "queued.current": 0,
          "rate": {},
          "queued.max": 0,
          "redispatched": 0
        },
        "agent": {
          "fall": "",
          "description": "",
          "health": "",
          "check": {
            "health": "4",
            "description": "Layer4 check passed",
            "rise": "2"
          },
          "rise": "",
          "status": ""
        },
        "cookie": "",
        "last_change": 349012,
        "session": {
          "total": 297,
          "current": 0,
          "max": 1,
          "rate": {
            "max": 3,
            "value": 0
          }
        },
        "service_name": "s1",
        "in.bytes": 64596,
        "weight": 1,
        "check": {
          "duration": 0,
          "agent.last": "",
          "failed": 0,
          "health.last": "",
          "down": 0,
          "status": "L4OK"
        },
        "source": {
          "address": "192.168.224.2:8080"
        },
        "out.bytes": 104840,
        "proxy": {
          "mode": "http",
          "name": "webservers",
          "id": 4
        },
        "component_type": 2,
        "downtime": 0,
        "selected.total": 297,
        "response": {
          "http": {
            "1xx": 0,
            "other": 0,
            "2xx": 297,
            "3xx": 0,
            "4xx": 0,
            "5xx": 0
          },
          "denied": 0,
          "time.avg": 1,
          "errors": 0
        },
        "load_balancing_algorithm": "",
        "header": {
          "rewrite": {
            "failed": {
              "total": 0
            }
          }
        },
        "connection": {
          "reuse.total": 7,
          "retried": 0,
          "cache": {},
          "idle": {
            "total": 1
          },
          "attempt.total": 290,
          "time.avg": 1
        },
        "queue": {
          "time.avg": 0
        },
        "status": "UP",
        "client.aborted": 1,
        "compressor": {}
      }
    },

lalit-satapathy · 2023-08-01T05:06:14Z

packages/haproxy/data_stream/stat/fields/fields.yml

@@ -7,10 +7,12 @@
        Status (UP, DOWN, NOLB, MAINT, or MAINT(via)...).
    - name: weight
      type: long
+      metric_type: gauge


@agithomas, Please cross check once if it is counter/gauge.

haproxy doc:

weight [..BS]: total effective weight (backend), effective weight (server)

The mapping is MaP, with a indicating an average value. So, it is a gauge type.

lalit-satapathy · 2023-08-01T05:06:29Z

packages/haproxy/data_stream/stat/fields/fields.yml

@@ -306,15 +354,18 @@
        - name: agent.last
          type: integer
        - name: failed
+          metric_type: counter


This looks like a wrong type. Can we cross check?

"check": { "agent.last": "", }

I know that it is a bug and hence i didn't assign any metric_type mapping to check.agent.last field.

Bug reported here : #7202

lalit-satapathy · 2023-08-01T05:07:23Z

packages/haproxy/data_stream/stat/fields/fields.yml

          description: |
            Number of data transfers aborted by the server. This value is included in haproxy.stat.response.errors.
        - name: active
          type: integer
+          metric_type: gauge


Need to cross check this is counter or gauge, since aborted and backup are counter for server group.

The mapping is SGP. So, it is a gauge type

…githomas/integrations into issue-7138-haproxy-metric-stat

agithomas · 2023-08-09T04:38:12Z

@lalit-satapathy , can you please approve if all looks good?

elasticmachine · 2023-08-09T05:51:45Z

Package haproxy - 1.8.1 containing this change is available at https://epr.elastic.co/search?package=haproxy

Added changelog and manifest.yml

8e6827b

agithomas mentioned this pull request Jul 28, 2023

[HA Proxy] TSDB Enablement #7138

Closed

20 tasks

agithomas and others added 2 commits July 28, 2023 17:01

Updated PR link

686d227

Merge branch 'main' into issue-7138-haproxy-metric-stat

a229201

agithomas marked this pull request as ready for review July 29, 2023 05:48

agithomas requested a review from a team as a code owner July 29, 2023 05:48

lalit-satapathy reviewed Jul 31, 2023

View reviewed changes

agithomas requested review from ritalwar and a team August 1, 2023 04:44

lalit-satapathy reviewed Aug 1, 2023

View reviewed changes

agithomas added 2 commits August 1, 2023 06:02

Change the metric type mapping for server.backup

903bb5c

Merge branch 'issue-7138-haproxy-metric-stat' of https://github.com/a…

c0779f9

…githomas/integrations into issue-7138-haproxy-metric-stat

agithomas marked this pull request as draft August 1, 2023 06:04

agithomas marked this pull request as ready for review August 1, 2023 06:05

Updated Readme file

1ed3611

andrewkroh added the Integration:haproxy HAProxy label Aug 1, 2023

lalit-satapathy approved these changes Aug 9, 2023

View reviewed changes

agithomas merged commit c8bb1a9 into elastic:main Aug 9, 2023
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add metric_type mapping to stat datastream of HA Proxy #7183

Add metric_type mapping to stat datastream of HA Proxy #7183

agithomas commented Jul 28, 2023

elasticmachine commented Jul 28, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jul 28, 2023 •

edited

Loading

lalit-satapathy Jul 31, 2023

agithomas Jul 31, 2023

lalit-satapathy Jul 31, 2023

agithomas Jul 31, 2023

agithomas Jul 31, 2023

lalit-satapathy Jul 31, 2023

agithomas Jul 31, 2023

lalit-satapathy commented Jul 31, 2023

agithomas commented Jul 31, 2023 •

edited

Loading

agithomas commented Jul 31, 2023

lalit-satapathy Aug 1, 2023

agithomas Aug 1, 2023

lalit-satapathy Aug 1, 2023

agithomas Aug 1, 2023

lalit-satapathy Aug 1, 2023

agithomas Aug 1, 2023

agithomas commented Aug 9, 2023

elasticmachine commented Aug 9, 2023

Add metric_type mapping to stat datastream of HA Proxy #7183

Add metric_type mapping to stat datastream of HA Proxy #7183

Conversation

agithomas commented Jul 28, 2023

What does this PR do?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Jul 28, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jul 28, 2023 • edited Loading

🌐 Coverage report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lalit-satapathy commented Jul 31, 2023

agithomas commented Jul 31, 2023 • edited Loading

agithomas commented Jul 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agithomas commented Aug 9, 2023

elasticmachine commented Aug 9, 2023

elasticmachine commented Jul 28, 2023 •

edited

Loading

elasticmachine commented Jul 28, 2023 •

edited

Loading

agithomas commented Jul 31, 2023 •

edited

Loading