You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While working on enabling TSDB for GCP metrics data streams, @agithomas found that it's impossible to uniquely identify the Firestore database that produced metrics.
Each GCP Firestore document has labels, but they're generic and don't specify a particular Firestore database.
If a user has several Firestore databases and performs the same action (QUERY/LOOKUP/CREATE) on two or more databases at the same time, it can result in lost documents.
To solve this, we need to add more metadata about the Firestore databases in the documents, like database id and location.
I see two potential solutions:
Modify the agent configuration for Firestore to gather different metrics that contain these labels. This can be done by:
Replacing document/delete_count with document/delete_ops_count
Replacing document/read_count with document/read_ops_count
Replacing document/write_count with document/write_ops_count
This approach would resolve the issue as these new metrics include database metadata. The downside is that these new metrics are currently in BETA, since having several Firestore databases under one account seems like a new feature.
Implement additional code in beats to retrieve metadata about Firestore databases (might not be possible).
Test with first approach
Changes I made to test this:
diff --git a/packages/gcp/data_stream/firestore/agent/stream/stream.yml.hbs b/packages/gcp/data_stream/firestore/agent/stream/stream.yml.hbs
index eaf3f9ab68..644dd1889e 100644
--- a/packages/gcp/data_stream/firestore/agent/stream/stream.yml.hbs+++ b/packages/gcp/data_stream/firestore/agent/stream/stream.yml.hbs@@ -23,6 +23,6 @@ exclude_labels: {{exclude_labels}}
metrics:
- service: firestore
metric_types:
- - "document/delete_count"- - "document/read_count"- - "document/write_count"
\ No newline at end of file
+ - "document/delete_ops_count"+ - "document/read_ops_count"+ - "document/write_ops_count"
\ No newline at end of file
diff --git a/packages/gcp/data_stream/firestore/elasticsearch/ingest_pipeline/default.yml b/packages/gcp/data_stream/firestore/elasticsearch/ingest_pipeline/default.yml
index c79d82a622..fa779926c0 100644
--- a/packages/gcp/data_stream/firestore/elasticsearch/ingest_pipeline/default.yml+++ b/packages/gcp/data_stream/firestore/elasticsearch/ingest_pipeline/default.yml@@ -2,15 +2,15 @@
description: Pipeline for parsing GCP Firestore metrics.
processors:
- rename:
- field: gcp.metrics.document.delete.count+ field: gcp.metrics.document.delete_ops_count.value
target_field: gcp.firestore.document.delete.count
ignore_missing: true
- rename:
- field: gcp.metrics.document.read.count+ field: gcp.metrics.document.read_ops_count.value
target_field: gcp.firestore.document.read.count
ignore_missing: true
- rename:
- field: gcp.metrics.document.write.count+ field: gcp.metrics.document.write_ops_count.value
target_field: gcp.firestore.document.write.count
ignore_missing: true
- remove:
@agithomas Currently, the new metrics are in BETA. Typically, we don't include BETA metrics in GA data streams. Not sure when exceptions are acceptable. We might have to wait for them to become GA. However, since having multiple Firebase databases under a GCP project is a new feature, it might take some time.
While working on enabling TSDB for GCP metrics data streams, @agithomas found that it's impossible to uniquely identify the Firestore database that produced metrics.
Each GCP Firestore document has labels, but they're generic and don't specify a particular Firestore database.
If a user has several Firestore databases and performs the same action (QUERY/LOOKUP/CREATE) on two or more databases at the same time, it can result in lost documents.
The metrics we collect in the current configuration don't include database metadata.
To solve this, we need to add more metadata about the Firestore databases in the documents, like database id and location.
I see two potential solutions:
Modify the agent configuration for Firestore to gather different metrics that contain these labels. This can be done by:
document/delete_count
withdocument/delete_ops_count
document/read_count
withdocument/read_ops_count
document/write_count
withdocument/write_ops_count
This approach would resolve the issue as these new metrics include database metadata. The downside is that these new metrics are currently in BETA, since having several Firestore databases under one account seems like a new feature.
Implement additional code in beats to retrieve metadata about Firestore databases (might not be possible).
Test with first approach
Changes I made to test this:
Document sample:
@agithomas @lalit-satapathy
The text was updated successfully, but these errors were encountered: