[GCP] Add dimensions for metrics data streams #8314
Conversation
gpop63
force-pushed
the
add_gcp_dimensions
branch
from
49c48f8
to
d75e2e3
Compare
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
🤖 GitHub commentsExpand to view the GitHub comments
To re-run your PR in the CI, just comment with:
|
changes gcp field type to group add labels ffingerprint in ingest pipeline
gpop63
force-pushed
the
add_gcp_dimensions
branch
from
d75e2e3
to
4dcf819
Compare
🌐 Coverage report
|
|
Hey @gpop63 , thanks for opening this PR! This is the related issue for GCP: #7555. Do you think we need all those ECS fields in every data stream? I checked (not for all data streams), and I found this:
And then we had some other data streams with values for all those, like Compute data stream. |
|
Does the |
It should be the account ID for the metric, basically the GCP project id. I actually should have added "cloud": {
"provider": "gcp",
"account": {
"name": "elastic-obs-integrations-dev",
"id": "elastic-obs-integrations-dev"
}
}, |
|
Good to know. I noticed some inputs/metricsets set the |
Are we adding the |
|
@constanca-m you're right. Could be due to having additional metadata logic in beats for Should we maybe keep all of them as dimensions only for |
I think it would be best. I remember that some data streams in other packages already followed this logic. |
|
@lalit-satapathy can you please help getting this one reviewed to unblock GCP packages |
|
|
@gpop63 , kindly refer to the comment here to find the reasoning behind selecting some of the dimension PRs. Also, you can find that , GCP resource identifier such as |
I guess you are referring to the |
| @@ -28,7 +28,7 @@ streams: | |||
| - name: period | |||
| type: text | |||
| title: Period | |||
| default: 60s | |||
| default: 5m | |||
There was a problem hiding this comment.
Is the default perios of collection 5 minutes and not 1 minute unlike other GCP datastreams ?
There was a problem hiding this comment.
The data stream period for storage had to be adjusted from 60s to 5m as some metrics have a sampling period of 5m. A shorter period would result in duplicated documents (a metric is generated every 5 minutes on GCP side but we fetch it every minute), which would get dropped as the only difference between them would be event.ingested.
|
@gpop63 , We discussed that not all of the common dimension fields are not included as dimensions. I learnt from you that, GCP does not have values populated for these common dimension fields even when there exist entry in As discussed, kindly share the list of fields that have values and those which doesn't have value for the fields in the common dimension fields. You may choose a table format to share this information. |
I will add actual documents for each data stream so we can check them at a later time if needed. compute
{
"cloud": {
"availability_zone": "us-central1-c",
"instance": {
"name": "gke-miguel-kubecon-default-pool-8d586473-uj1d",
"id": "7442613444431818544"
},
"provider": "gcp",
"machine": {
"type": "e2-medium"
},
"region": "us-central1",
"account": {
"name": "elastic-obs-integrations-dev",
"id": "elastic-obs-integrations-dev"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"ephemeral_id": "1de59397-2d84-4e9e-8ec2-8becfa2dccd3",
"type": "metricbeat",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:13:00.000Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"compute": {
"instance": {
"network": {
"ingress": {
"bytes": 9414,
"packets": {
"count": 134
}
},
"egress": {
"bytes": 11485,
"packets": {
"count": 89
}
}
}
}
},
"labels": {
"metrics": {
"loadbalanced": "true"
},
"user": {
"division": "engineering",
"goog-k8s-cluster-name": "miguel-kubecon",
"org": "obs",
"goog-k8s-node-pool-name": "default-pool",
"goog-k8s-cluster-location": "us-central1-c",
"project": "miguelluna",
"team": "cloud-native-monitoring",
"goog-gke-node": ""
}
}
},
"service": {
"type": "gcp"
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.compute"
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"type": "linux",
"family": "debian",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"containerized": true,
"ip": [
"172.22.0.7"
],
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 3126038318,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:17:09Z",
"module": "gcp",
"dataset": "gcp.compute"
}
}gke
{
"cloud": {
"provider": "gcp",
"account": {
"name": "elastic-obs-integrations-dev",
"id": "elastic-obs-integrations-dev"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"ephemeral_id": "1de59397-2d84-4e9e-8ec2-8becfa2dccd3",
"type": "metricbeat",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:14:05.736Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"gke": {
"container": {
"memory": {
"request": {
"bytes": 10485760
},
"limit": {
"bytes": 52428800
}
},
"restart": {
"count": 0
},
"cpu": {
"request_cores": {
"value": 0.005
},
"limit_cores": {
"value": 0
}
},
"ephemeral_storage": {
"request": {
"bytes": 0
},
"limit": {
"bytes": 0
}
}
}
},
"labels": {
"resource": {
"cluster_name": "tetiana-prometheus",
"container_name": "csi-driver-registrar",
"location": "europe-west1",
"pod_name": "pdcsi-node-7b6hq",
"namespace_name": "kube-system"
}
}
},
"service": {
"type": "gcp"
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.gke"
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"type": "linux",
"family": "debian",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"containerized": true,
"ip": [
"172.22.0.7"
],
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 1244946929,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:18:19Z",
"module": "gcp",
"dataset": "gcp.gke"
}
}loadbalancing_metrics
{
"cloud": {
"provider": "gcp",
"account": {
"name": "elastic-obs-integrations-dev",
"id": "elastic-obs-integrations-dev"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"ephemeral_id": "1de59397-2d84-4e9e-8ec2-8becfa2dccd3",
"type": "metricbeat",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:11:00.000Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"loadbalancing_metrics": {
"https": {
"backend_request": {
"bytes": 417,
"count": 1
},
"backend_response": {
"bytes": 488
}
}
},
"labels": {
"resource": {
"backend_scope_type": "INVALID_BACKEND",
"matched_url_path_rule": "UNMATCHED",
"backend_target_name": "tas-demo-elastic-http-lb",
"backend_type": "INVALID_BACKEND",
"backend_scope": "INVALID_BACKEND",
"target_proxy_name": "tas-demo-elastic-http-lb",
"forwarding_rule_name": "tas-demo-elastic-http-lb",
"backend_name": "INVALID_BACKEND",
"url_map_name": "tas-demo-elastic-https-lb",
"backend_target_type": "BACKEND_SERVICE",
"region": "global"
},
"metrics": {
"response_code": "502",
"proxy_continent": "America",
"cache_result": "DISABLED",
"response_code_class": "500"
}
}
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.loadbalancing_metrics"
},
"service": {
"type": "gcp"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"family": "debian",
"type": "linux",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"containerized": true,
"ip": [
"172.22.0.7"
],
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 476084876,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:15:18Z",
"module": "gcp",
"dataset": "gcp.loadbalancing_metrics"
}
}pubsub
{
"cloud": {
"provider": "gcp",
"account": {
"name": "elastic-obs-integrations-dev",
"id": "elastic-obs-integrations-dev"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"ephemeral_id": "1de59397-2d84-4e9e-8ec2-8becfa2dccd3",
"type": "metricbeat",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:15:00.000Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"pubsub": {
"subscription": {
"retained_acked_bytes_by_region": {
"bytes": 0
},
"unacked_bytes_by_region": {
"bytes": 0
},
"oldest_unacked_message_age_by_region": {
"value": 0
},
"oldest_retained_acked_message_age_by_region": {
"value": 0
}
}
},
"labels": {
"resource": {
"subscription_id": "filebeat-gcp-audit"
},
"metrics": {
"region": "northamerica-northeast2"
}
}
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.pubsub"
},
"service": {
"type": "gcp"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"family": "debian",
"type": "linux",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"containerized": true,
"ip": [
"172.22.0.7"
],
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 434512489,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:19:18Z",
"module": "gcp",
"dataset": "gcp.pubsub"
}
}redis
{
"cloud": {
"instance": {
"name": "redis1",
"id": "projects/elastic-obs-integrations-dev/locations/us-central1/instances/redis1"
},
"provider": "gcp",
"machine": {
"type": "BASIC"
},
"account": {
"name": "elastic-obs-integrations-dev",
"id": "elastic-obs-integrations-dev"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"type": "metricbeat",
"ephemeral_id": "1de59397-2d84-4e9e-8ec2-8becfa2dccd3",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:15:26.810Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"metrics": {
"stats": {
"reject_connections_count": {}
}
},
"redis": {
"stats": {
"reject_connections": {
"count": 0
}
}
},
"labels": {
"resource": {
"region": "us-central1",
"node_id": "node-0"
},
"metrics": {
"role": "primary"
}
}
},
"service": {
"type": "gcp"
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.redis"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"family": "debian",
"type": "linux",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"containerized": true,
"ip": [
"172.22.0.7"
],
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 1381430449,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:19:39Z",
"module": "gcp",
"dataset": "gcp.redis"
}
}storage
{
"cloud": {
"provider": "gcp",
"account": {
"name": "elastic-obs-integrations-dev",
"id": "elastic-obs-integrations-dev"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"type": "metricbeat",
"ephemeral_id": "1de59397-2d84-4e9e-8ec2-8becfa2dccd3",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:10:00.000Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"storage": {
"storage": {
"object": {
"count": 6
}
}
},
"labels": {
"resource": {
"bucket_name": "dataproc-temp-us-central1-774712120909-e8nuxv73",
"location": "us-central1"
},
"metrics": {
"storage_class": "REGIONAL"
}
}
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.storage"
},
"service": {
"type": "gcp"
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"type": "linux",
"family": "debian",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"containerized": true,
"ip": [
"172.22.0.7"
],
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 815694657,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:20:18Z",
"module": "gcp",
"dataset": "gcp.storage"
}
}cloudrun_metrics
{
"cloud": {
"provider": "gcp",
"account": {
"name": "elastic-obs-integrations-dev",
"id": "elastic-obs-integrations-dev"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"type": "metricbeat",
"ephemeral_id": "1de59397-2d84-4e9e-8ec2-8becfa2dccd3",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:18:21.590Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"cloudrun_metrics": {
"container": {
"billable_instance_time": 2.7
}
},
"labels": {
"resource": {
"revision_name": "damien-test-hello-00001-ped",
"service_name": "damien-test-hello",
"location": "us-central1",
"configuration_name": "damien-test-hello"
}
}
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.cloudrun_metrics"
},
"service": {
"type": "gcp"
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"type": "linux",
"family": "debian",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"ip": [
"172.22.0.7"
],
"containerized": true,
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 503983018,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:21:34Z",
"module": "gcp",
"dataset": "gcp.cloudrun_metrics"
}
}cloudsql_mysql
{
"cloud": {
"provider": "gcp",
"account": {
"name": "elastic-observability",
"id": "elastic-observability"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"ephemeral_id": "1de59397-2d84-4e9e-8ec2-8becfa2dccd3",
"type": "metricbeat",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:20:00.000Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"cloudsql_mysql": {
"database": {
"instance_state": false
}
},
"labels": {
"resource": {
"database_id": "elastic-observability:mysql",
"region": "us-central"
},
"cloudsql": {
"name": "mysql",
"version": "8.0.31"
},
"metrics": {
"state": "RUNNABLE"
}
}
},
"service": {
"type": "gcp"
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.cloudsql_mysql"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"family": "debian",
"type": "linux",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"containerized": true,
"ip": [
"172.22.0.7"
],
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 1074356670,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:23:29Z",
"module": "gcp",
"dataset": "gcp.cloudsql_mysql"
}
}cloudsql_postgresql
{
"cloud": {
"provider": "gcp",
"account": {
"name": "elastic-observability",
"id": "elastic-observability"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"ephemeral_id": "1de59397-2d84-4e9e-8ec2-8becfa2dccd3",
"type": "metricbeat",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:20:00.000Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"cloudsql_postgresql": {
"database": {
"num_backends": {
"count": 2
}
}
},
"labels": {
"resource": {
"database_id": "elastic-observability:postgres",
"region": "us-central"
},
"cloudsql": {
"name": "postgres",
"version": "14"
},
"metrics": {
"database": "cloudsqladmin"
}
}
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.cloudsql_postgresql"
},
"service": {
"type": "gcp"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"type": "linux",
"family": "debian",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"containerized": true,
"ip": [
"172.22.0.7"
],
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 1140922692,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:23:18Z",
"module": "gcp",
"dataset": "gcp.cloudsql_postgresql"
}
}cloudsql_sqlserver
{
"cloud": {
"provider": "gcp",
"account": {
"name": "elastic-observability",
"id": "elastic-observability"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"type": "metricbeat",
"ephemeral_id": "1de59397-2d84-4e9e-8ec2-8becfa2dccd3",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:20:00.000Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"cloudsql_sqlserver": {
"database": {
"disk": {
"write_ops": {
"count": 110
},
"bytes_used": {
"bytes": 651927552
},
"read_ops": {
"count": 0
},
"quota": {
"bytes": 105089261568
},
"utilization": {
"pct": 0.006203560119015185
}
},
"memory": {
"quota": {
"bytes": 27331235840
},
"total_usage": {
"bytes": 3081568256
},
"usage": {
"bytes": 2311217152
},
"utilization": {
"pct": 0.08456321424798038
}
},
"cpu": {
"usage_time": {
"sec": 11.019461885036435
},
"utilization": {
"pct": 0.045281727500696436
},
"reserved_cores": {
"count": 4
}
},
"up": 1,
"network": {
"received_bytes": {
"count": 315087
},
"connections": {
"count": 9
},
"sent_bytes": {
"count": 3681782
}
},
"uptime": {
"sec": 60
}
}
},
"labels": {
"resource": {
"database_id": "elastic-observability:ms-sql",
"region": "us-west1"
},
"cloudsql": {
"name": "sqlserver",
"version": "2019_standard"
}
}
},
"service": {
"type": "gcp"
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.cloudsql_sqlserver"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"type": "linux",
"family": "debian",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"ip": [
"172.22.0.7"
],
"containerized": true,
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 1086857363,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:23:29Z",
"module": "gcp",
"dataset": "gcp.cloudsql_sqlserver"
}
}firestore
{
"cloud": {
"provider": "gcp",
"account": {
"name": "robust-catalyst-399814",
"id": "robust-catalyst-399814"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"type": "metricbeat",
"ephemeral_id": "1de59397-2d84-4e9e-8ec2-8becfa2dccd3",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:28:00.000Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"firestore": {
"document": {
"delete": {
"count": 18
}
}
},
"labels": {
"metrics": {
"module": "__unknown__",
"version": "__unknown__"
}
}
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.firestore"
},
"service": {
"type": "gcp"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"type": "linux",
"family": "debian",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"containerized": true,
"ip": [
"172.22.0.7"
],
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 397651706,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:32:39Z",
"module": "gcp",
"dataset": "gcp.firestore"
}
}dataproc
{
"cloud": {
"provider": "gcp",
"account": {
"name": "elastic-obs-integrations-dev",
"id": "elastic-obs-integrations-dev"
}
},
"agent": {
"name": "docker-fleet-agent",
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"ephemeral_id": "1f9a913c-dd2f-4347-a5fc-5746e4b9f819",
"type": "metricbeat",
"version": "8.12.0"
},
"@timestamp": "2023-11-28T11:47:00.000Z",
"ecs": {
"version": "8.0.0"
},
"gcp": {
"labels": {
"resource": {
"cluster_name": "cluster-aa34",
"cluster_uuid": "89f8d591-cb18-4778-9c14-8be71fb578f1",
"region": "us-central1"
},
"metrics": {
"operation_type": "START_CLUSTER",
"error_type": "UNKNOWN_ERROR"
}
},
"dataproc": {
"cluster": {
"operation": {
"failed": {
"count": 0
}
}
}
}
},
"service": {
"type": "gcp"
},
"data_stream": {
"namespace": "default",
"type": "metrics",
"dataset": "gcp.dataproc"
},
"elastic_agent": {
"id": "8442dd66-0f0b-4b54-93f6-6dcf25f45296",
"version": "8.12.0",
"snapshot": true
},
"host": {
"hostname": "docker-fleet-agent",
"os": {
"kernel": "5.10.102.1-microsoft-standard-WSL2",
"codename": "focal",
"name": "Ubuntu",
"type": "linux",
"family": "debian",
"version": "20.04.6 LTS (Focal Fossa)",
"platform": "ubuntu"
},
"containerized": true,
"ip": [
"172.22.0.7"
],
"name": "docker-fleet-agent",
"id": "d03b51e638e64b05b5cf16c41d2058c0",
"mac": [
"02-42-AC-16-00-07"
],
"architecture": "x86_64"
},
"metricset": {
"period": 60000,
"name": "metrics"
},
"event": {
"duration": 539443435,
"agent_id_status": "verified",
"ingested": "2023-11-28T11:50:03Z",
"module": "gcp",
"dataset": "gcp.dataproc"
}
}Some data streams have the region/AZ under |
|
Hi @gpop63 , I have created two resources as below 
Two services having the same name |
I see that the labels are having the location information as below. As labels.* are part of dimensions, i believe, this case is covered. |
|
@gpop63 , Can you check firestore? I see no labels here. |
|
Testing Firestore: Our internal GCP projects use Firestore in Datastore mode so I had to use my personal GCP account. Within a GCP project, you can create multiple Firestore databases (we are using native mode). Each project comes with a default Firestore database named I added two additional databases, so my current databases are:
Command to create a Firestore database: Generating Metrics with Go:
Firestore Library Code
package main
import (
"context"
"fmt"
"log"
"time"
"cloud.google.com/go/firestore"
"google.golang.org/api/option"
)
func main() {
ctx := context.Background()
// service account key
saKey := "robust-catalyst-399814-e6dbc28a6a02.json"
projectID := "robust-catalyst-399814"
client, err := firestore.NewClient(ctx, projectID, option.WithCredentialsFile(saKey))
if err != nil {
log.Fatalf("Failed to create client: %v", err)
}
defer client.Close()
for {
time.Sleep(3 * time.Second)
docRef, _, err := client.Collection("users").Add(ctx, map[string]interface{}{
"first": "Ada",
"last": "Lovelace",
"born": 1815,
})
if err != nil {
log.Fatalf("Failed adding a new user: %v", err)
}
_, err = docRef.Delete(ctx)
if err != nil {
log.Fatalf("Failed deleting user: %v", err)
}
iter := client.Collection("users").Documents(ctx)
for {
doc, err := iter.Next()
if err != nil {
break
}
fmt.Println(doc.Data())
}
}
}HTTP Requests Code
package main
import (
"bytes"
"context"
"fmt"
"io"
"log"
"net/http"
"os"
"time"
"golang.org/x/oauth2/google"
)
var db = "mydb2"
func createJWT(serviceAccountPath string) (string, error) {
data, err := os.ReadFile(serviceAccountPath)
if err != nil {
return "", err
}
config, err := google.JWTConfigFromJSON(data, "https://www.googleapis.com/auth/datastore")
if err != nil {
return "", err
}
token, err := config.TokenSource(context.Background()).Token()
if err != nil {
return "", err
}
return token.AccessToken, nil
}
func getDocuments(url string, data []byte, jwt string) error {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+jwt)
req.Header.Set("x-goog-request-params", "project_id=robust-catalyst-399814&database_id="+db)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return err
}
fmt.Println(len(body))
return nil
}
func createDocument(url string, data []byte, jwt string) error {
req, err := http.NewRequest("POST", url, bytes.NewBuffer(data))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+jwt)
req.Header.Set("x-goog-request-params", "project_id=robust-catalyst-399814&database_id="+db)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return err
}
fmt.Println(string(body))
return nil
}
func main() {
jwt, err := createJWT("robust-catalyst-399814-e6dbc28a6a02.json")
if err != nil {
log.Fatal(err)
}
for {
time.Sleep(3 * time.Second)
err := createDocument(fmt.Sprintf(`https://firestore.googleapis.com/v1/projects/robust-catalyst-399814/databases/%s/documents/artists`, db), []byte(`{"fields": {"first": {"stringValue": "Ada"}, "last": {"stringValue": "Lovelace"}}}`), jwt)
if err != nil {
log.Fatal(err)
}
err = getDocuments(fmt.Sprintf(`https://firestore.googleapis.com/v1/projects/robust-catalyst-399814/databases/%s/documents/artists`, db), nil, jwt)
if err != nil {
log.Fatal(err)
}
}
}I executed two instances of direct HTTP requests for I also noticed a variation of "labels": {
"metrics": {
"module": "__unknown__",
"type": "QUERY",
"version": "__unknown__"
}
}"labels": {
"metrics": {
"op": "CREATE",
"module": "__unknown__",
"version": "__unknown__"
}
}TSDB Test
|
|
Regarding the Firestore data stream, @agithomas identified a significant issue: we lack a database or location identifier in either If a user has multiple Firestore databases and performs the same action (QUERY/LOOKUP/CREATE) on two or more databases at any given time, it could lead to dropped documents. "labels": {
"metrics": {
"module": "__unknown__",
"type": "QUERY",
"version": "__unknown__"
}
}Having several Firestore databases under one account is a new feature currently in preview mode. However, it's safer to not enable TSDB for this data stream until we can add database id and location identifiers. |
gpop63
force-pushed
the
add_gcp_dimensions
branch
from
da1f239
to
b37f946
Compare
@gpop63 , Can you create a backlog issue for Firestore and include it under the meta under the section Blocked packages? Please consider including the reason for the blocked status, the same as you mentioned in the previous comment. This will help the team be aware of the pending TSDB packages/dataset. |
|
@ishleenk17 , can you please do a final review of the PR? |
Agree. @gpop63, I am fine to keep "firestore" as a backlog. Let's also update the top-level gcp issue, with the details, so that looking at it, we get a clear picture. |
Are these new fields to be added? We can track in the backlog issue, what needs to be done to have TSDB working on this data stream. |
Yes, we would have to add new fields in beats specifically for the |
|
@elastic/obs-cloud-monitoring could someone please take a look? TSDB review was done by @agithomas |
|
Package gcp - 2.32.1 containing this change is available at https://epr.elastic.co/search?package=gcp |
Overview
This change introduces dimension mappings, new fields and fixes, while also changing the
gcpfield type togroupacross all metrics data streams.Without specifying
groupas the type, Elasticsearch was unable to recognizegcpas an object containing these subfields, resulting in these fields not being included in the mapping.The data stream period for
storagehad to be adjusted from60sto5mas some metrics have a sampling period of5m. A shorter period would result in duplicated documents (a metric is generated every 5 minutes on GCP side but we fetch it every minute), which would get dropped as the only difference between them would beevent.ingested.Duplicated
storagedocuments example:Document 1
{ "_index": ".ds-metrics-gcp.storage-default-2023.10.17-000001", "_id": "Cou9PosBY5p1KHvoLiu7", "_score": null, "_source": { "cloud": { "provider": "gcp", "account": { "name": "elastic-obs-integrations-dev", "id": "elastic-obs-integrations-dev" } }, "agent": { "name": "5506d89e26e7", "id": "64d8d07b-1695-4b23-ac74-b10969e07a9c", "type": "metricbeat", "ephemeral_id": "75afe95d-f1bb-4154-8bfc-50ca3f0b0da0", "version": "8.10.4" }, "@timestamp": "2023-10-17T17:30:00.000Z", "ecs": { "version": "8.0.0" }, "gcp": { "storage": { "storage": { "total": { "bytes": 2584690366.0 }, "total_byte_seconds": { "bytes": 223317247622144.0 }, "object": { "count": 84 } } }, "labels_hash": "3aOy2p1nLq3mU9OOBkDbjTJYc6u02W8+sWGHVMSTODE=", "labels": { "resource": { "bucket_name": "elastic-obs-integrations-dev-giz-2-packages-99412", "location": "us" }, "metrics": { "storage_class": "MULTI_REGIONAL" } } }, "data_stream": { "namespace": "default", "type": "metrics", "dataset": "gcp.storage" }, "service": { "type": "gcp" }, "elastic_agent": { "id": "64d8d07b-1695-4b23-ac74-b10969e07a9c", "version": "8.10.4", "snapshot": true }, "host": { "hostname": "5506d89e26e7", "os": { "kernel": "5.10.102.1-microsoft-standard-WSL2", "codename": "focal", "name": "Ubuntu", "type": "linux", "family": "debian", "version": "20.04.6 LTS (Focal Fossa)", "platform": "ubuntu" }, "containerized": true, "ip": [ "172.21.0.4" ], "name": "5506d89e26e7", "mac": [ "02-42-AC-15-00-04" ], "architecture": "aarch64" }, "metricset": { "period": 60000, "name": "metrics" }, "event": { "duration": 873564254, "agent_id_status": "verified", "ingested": "2023-10-17T17:44:27Z", "module": "gcp", "dataset": "gcp.storage", "metric_names_hash": "9eE4YKXcP7hKZqYPVrTCDmySLL4jTymXE0bPcxnLzrs=" } }, "sort": [ 1697563800000 ] }Document 2
{ "_index": ".ds-metrics-gcp.storage-default-2023.10.17-000001", "_id": "iou8PosBY5p1KHvoQiH8", "_score": null, "_source": { "cloud": { "provider": "gcp", "account": { "name": "elastic-obs-integrations-dev", "id": "elastic-obs-integrations-dev" } }, "agent": { "name": "5506d89e26e7", "id": "64d8d07b-1695-4b23-ac74-b10969e07a9c", "type": "metricbeat", "ephemeral_id": "75afe95d-f1bb-4154-8bfc-50ca3f0b0da0", "version": "8.10.4" }, "@timestamp": "2023-10-17T17:30:00.000Z", "ecs": { "version": "8.0.0" }, "gcp": { "storage": { "storage": { "total": { "bytes": 2584690366.0 }, "total_byte_seconds": { "bytes": 223317247622144.0 }, "object": { "count": 84 } } }, "labels_hash": "3aOy2p1nLq3mU9OOBkDbjTJYc6u02W8+sWGHVMSTODE=", "labels": { "resource": { "bucket_name": "elastic-obs-integrations-dev-giz-2-packages-99412", "location": "us" }, "metrics": { "storage_class": "MULTI_REGIONAL" } } }, "data_stream": { "namespace": "default", "type": "metrics", "dataset": "gcp.storage" }, "service": { "type": "gcp" }, "elastic_agent": { "id": "64d8d07b-1695-4b23-ac74-b10969e07a9c", "version": "8.10.4", "snapshot": true }, "host": { "hostname": "5506d89e26e7", "os": { "kernel": "5.10.102.1-microsoft-standard-WSL2", "codename": "focal", "name": "Ubuntu", "type": "linux", "family": "debian", "version": "20.04.6 LTS (Focal Fossa)", "platform": "ubuntu" }, "containerized": true, "ip": [ "172.21.0.4" ], "name": "5506d89e26e7", "mac": [ "02-42-AC-15-00-04" ], "architecture": "aarch64" }, "metricset": { "period": 60000, "name": "metrics" }, "event": { "duration": 547719500, "agent_id_status": "verified", "ingested": "2023-10-17T17:43:27Z", "module": "gcp", "dataset": "gcp.storage", "metric_names_hash": "9eE4YKXcP7hKZqYPVrTCDmySLL4jTymXE0bPcxnLzrs=" } }, "sort": [ 1697563800000 ] }Dimensions and fields added:
Data streams updated
cloudrun_metricscloudsql_mysqlcloudsql_postgresqlcloudsql_sqlservercomputedataprocfirestoregkeloadbalancing_metricspubsubredisstorageECS fields:
cloud.account.idcloud.account.namecloud.availability_zonecloud.instance.idcloud.machine.typecloud.regionPackage fields:
gcp.metric_names_fingerprintThis field was added as a dimension to ensure that each document, even those with the same timestamp from different batches, can be uniquely identified. The hashing is done on the beats side.gcp.labels_fingerprintgcp.labelsfield using thefingerprintprocessor.TSDB toolkit tests (outdated)
pubsub
gke
compute
redis
cloudrun_metrics
loadbalancing_metrics
cloudsql_postgresql
cloudsql_mysql
cloudsql_sqlserver
dataproc
firestore
storage
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots