exporter/stackdriver: creation fails with AlreadyExists for new metrics #147
Comments
|
This is coming from CreateMetricDescriptor if the metric descriptor is just newly created. Needs more investigation... |
|
I suppose #162 should fix this? For Stackdriver, it's ok to create the exact same metric descriptor multiple times, however we'll get an AlreadyExists error if we try to create a metric descriptor that has the same name but with different attributes (e.g aggregation or window). |
|
I cannot reproduce it anymore hence closing, I will reopen it if I see it again. |
|
We get the following errors: 2018/04/03 07:02:37 Failed to export to Stackdriver Monitoring: rpc error: code = AlreadyExists desc = Errors during metric descriptor creation: {(metric: custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count, error: The metric already exists.)}. When using code like this with go.opencensus.io master and cloud.google.com/go master deps: stackdriverExporter, err := stackdriver.NewExporter(stackdriver.Options{ProjectID: os.Getenv("GCP_PROJECT_ID")})
if err != nil {
share_log.Logger.Fatal(ctx, "Failed initializing statistics - opencensus", share_log.Logger.FmtError(err))
}
view.RegisterExporter(stackdriverExporter)
trace.RegisterExporter(stackdriverExporter)
for _, v := range []*view.View{
pubsub.PullCountView,
pubsub.AckCountView,
pubsub.NackCountView,
pubsub.ModAckCountView,
pubsub.StreamOpenCountView,
pubsub.StreamRetryCountView,
pubsub.StreamRequestCountView,
pubsub.StreamResponseCountView,
} {
if err := v.Subscribe(); err != nil {
share_log.Logger.Fatal(ctx, "Failed subscribing to statistics view - opencensus", share_log.Logger.FmtError(err))
}
share_log.Logger.Info(ctx, "Subscribed to statistics view - opencensus", share_log.Logger.FmtString("View", v.Name))
}
share_log.Logger.Info(ctx, "Initialized statistics - opencensus") |
|
Hi @ChrisBartonLC , this error is probably because you updated your view definition after exporting it to Stackdriver Monitoring. Currently our exporter doesn't support updating a view (or creating a new view with existing names). For now please try deleting the conflicted metric descriptors from Stackdriver before exporting. |
|
Ok deleted them after modifying the demo code from "google.golang.org/api/monitoring/v3" to "monitoring "google.golang.org/api/monitoring/v3" and now the error is: Failed to export to Stackdriver Monitoring: stackdriver metric descriptor was not created with aggregation type view.AggType |
|
@ChrisBartonLC Thanks for reporting, this is related to a recent change to Aggregation. #664 should fix it. |
|
Thanks! Works now after using your latest code on master :) |
|
Hi I updated this morning and swapped to to using the new 'register' api for setting up the view. Unfortunately I can no longer delete metrics even though the delete does not return an error: INFO 2018/04/05 09:01:31.888517 BACKGROUND Deleting metric { "metric": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count" } main.main line 64 /Users/tanh/projects/localcover/dev/code_lc/lc/lc-api/lc-api-customer/src/lc-api-customer/tools/stackdriver-metric-delete/main.go
INFO 2018/04/05 09:01:34.330270 BACKGROUND Deleted metric { "metric": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count", "response": {} } main.main line 71 /Users/tanh/projects/localcover/dev/code_lc/lc/lc-api/lc-api-customer/src/lc-api-customer/tools/stackdriver-metric-delete/main.goMetric still exists: 2018/04/05 09:00:45 Failed to export to Stackdriver Monitoring: rpc error: code = AlreadyExists desc = Errors during metric descriptor creation: {(metric: custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count, error: The metric already exists.)}. |
|
Hi @ChrisBartonLC , can you show us how you defined |
stackdriverExporter, err := stackdriver.NewExporter(stackdriver.Options{ProjectID: os.Getenv("GCP_PROJECT_ID")})
if err != nil {
share_log.Logger.Fatal(ctx, "Failed initializing stackdriver", share_log.Logger.FmtError(err))
}
trace.RegisterExporter(stackdriverExporter)
share_log.Logger.Info(ctx, "Registered stackdriver tracing")
view.RegisterExporter(stackdriverExporter)
if err := view.Register(
pubsub.PullCountView,
pubsub.AckCountView,
pubsub.NackCountView,
pubsub.ModAckCountView,
pubsub.StreamOpenCountView,
pubsub.StreamRetryCountView,
pubsub.StreamRequestCountView,
pubsub.StreamResponseCountView,
); err != nil {
share_log.Logger.Fatal(ctx, "Failed registering stackdriver statistics", share_log.Logger.FmtError(err))
}
share_log.Logger.Info(ctx, "Registered stackdriver statistics") |
|
Is this issue specific to I suspect this is because /cc @rakyll |
|
Yep we are only getting errors related to StreamRequestCountView. Here are all the metrics from "https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.metricDescriptors/list" using the filter "metric.type = starts_with("custom.googleapis.com/")" {
"metricDescriptors": [
{
"name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/ack_count",
"labels": [
{
"key": "subscription"
},
{
"key": "opencensus_task",
"description": "Opencensus task identifier"
}
],
"metricKind": "CUMULATIVE",
"valueType": "INT64",
"unit": "1",
"description": "Number of PubSub messages acked",
"displayName": "OpenCensus/cloud.google.com/go/pubsub/ack_count",
"type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/ack_count"
},
{
"name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/mod_ack_count",
"labels": [
{
"key": "subscription"
},
{
"key": "opencensus_task",
"description": "Opencensus task identifier"
}
],
"metricKind": "CUMULATIVE",
"valueType": "INT64",
"unit": "1",
"description": "Number of ack-deadlines modified",
"displayName": "OpenCensus/cloud.google.com/go/pubsub/mod_ack_count",
"type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/mod_ack_count"
},
{
"name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/nack_count",
"labels": [
{
"key": "subscription"
},
{
"key": "opencensus_task",
"description": "Opencensus task identifier"
}
],
"metricKind": "CUMULATIVE",
"valueType": "INT64",
"unit": "1",
"description": "Number of PubSub messages nacked",
"displayName": "OpenCensus/cloud.google.com/go/pubsub/nack_count",
"type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/nack_count"
},
{
"name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/pull_count",
"labels": [
{
"key": "subscription"
},
{
"key": "opencensus_task",
"description": "Opencensus task identifier"
}
],
"metricKind": "CUMULATIVE",
"valueType": "INT64",
"unit": "1",
"description": "Number of PubSub messages pulled",
"displayName": "OpenCensus/cloud.google.com/go/pubsub/pull_count",
"type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/pull_count"
},
{
"name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_open_count",
"labels": [
{
"key": "subscription"
},
{
"key": "opencensus_task",
"description": "Opencensus task identifier"
}
],
"metricKind": "CUMULATIVE",
"valueType": "INT64",
"unit": "1",
"description": "Number of calls opening a new streaming pull",
"displayName": "OpenCensus/cloud.google.com/go/pubsub/stream_open_count",
"type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_open_count"
},
{
"name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count",
"labels": [
{
"key": "subscription"
},
{
"key": "opencensus_task",
"description": "Opencensus task identifier"
}
],
"metricKind": "CUMULATIVE",
"valueType": "INT64",
"unit": "1",
"description": "Number gRPC StreamingPull request messages sent",
"displayName": "OpenCensus/cloud.google.com/go/pubsub/stream_request_count",
"type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count"
},
{
"name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_response_count",
"labels": [
{
"key": "subscription"
},
{
"key": "opencensus_task",
"description": "Opencensus task identifier"
}
],
"metricKind": "CUMULATIVE",
"valueType": "INT64",
"unit": "1",
"description": "Number of gRPC StreamingPull response messages received",
"displayName": "OpenCensus/cloud.google.com/go/pubsub/stream_response_count",
"type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_response_count"
},
{
"name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_retry_count",
"labels": [
{
"key": "subscription"
},
{
"key": "opencensus_task",
"description": "Opencensus task identifier"
}
],
"metricKind": "CUMULATIVE",
"valueType": "INT64",
"unit": "1",
"description": "Number of retries of a stream send or receive",
"displayName": "OpenCensus/cloud.google.com/go/pubsub/stream_retry_count",
"type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_retry_count"
},
{
"name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/grpc.io/client/error_count/cumulative",
"labels": [
{
"key": "canonical_status"
},
{
"key": "method"
},
{
"key": "opencensus_task",
"description": "Opencensus task identifier"
}
],
"metricKind": "CUMULATIVE",
"valueType": "DISTRIBUTION",
"unit": "1",
"description": "RPC Errors",
"displayName": "OpenCensus/grpc.io/client/error_count/cumulative",
"type": "custom.googleapis.com/opencensus/grpc.io/client/error_count/cumulative"
},
{
"name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/grpc.io/client/request_count/cumulative",
"labels": [
{
"key": "method"
},
{
"key": "opencensus_task",
"description": "Opencensus task identifier"
}
],
"metricKind": "CUMULATIVE",
"valueType": "DISTRIBUTION",
"unit": "1",
"description": "Number of client RPC request messages",
"displayName": "OpenCensus/grpc.io/client/request_count/cumulative",
"type": "custom.googleapis.com/opencensus/grpc.io/client/request_count/cumulative"
}
]
} |
|
Actually it is ok now, sorry :( seems another developer must have been using the old code so we kept clashing |
|
Cool :) |
Add a Skeleton for Collector
Metrics description creation is failing with "AlreadyExists" if the metric descriptor is being created for the first time:
Wait longer than the reporting duration...
2017/11/21 15:03:27 Failed to export to Stackdriver Monitoring: rpc error: code = AlreadyExists desc = Errors during metric descriptor creation: {(metric: custom.googleapis.com/opencensus%2Fmy.org%2Fviews%2Fvideoooosizeovertime, error: The metric already exists.)}.
The text was updated successfully, but these errors were encountered: