[QUESTION] Getting error when using customNamespace

[bc-ibiks]

I am getting the following error when attempting to export metric from a customnamespace

"custom_metric_namespace":"ANI","error":"ValidationError: The parameter StartTime must not equal parameter EndTime.\n\tstatus code: 400

Config
customNamespace:

• name: ani
  namespace: ANI
  regions:
  • us-west-2
    roles:
  • roleArn
    enableMetricData: true
    metrics:
  • name: PROCESSED_BATCHES
    statistics:
    • Sum
      period: 60
      length: 60
      nilToZero: true
  • name: PROCESSED_BATCHES_ITEMS
    statistics:
    • Sum
      period: 60
      length: 60
      nilToZero: true

[cristiangreco]

@bc-ibiks what version of YACE are you running? Usage of custom namespaces is deprecated and might be removed or changed in future versions.

I think the error might be due to the period and length you're using, which are shorter than the scrape interval. If you definitely need to keep the value of 60, I'd suggest you run yace with -scraping-interval=60.

Please replace your custom namespace job with a static job as well.

[bc-ibiks]

@cristiangreco thanks for the update
v0.43.0-alpha
I will make changes as you suggested. will let you know how this goes
cheers

[cristiangreco]

@bc-ibiks great, please let me know 👍

[bc-ibiks]

@cristiangreco I have move my config to static job
The errors have gone but no metrics captured. I can see these data in cloudwatch

• name: ani
  namespace: ANI
  regions:
  • us-west-2
  • us-east-2
    roles:
  • roleArn: arn:aws:iam::<>:role/prometheus
    enableMetricData: true
    metrics:
  • name: PROCESSED_BATCHES
    statistics:
    • Sum
    • Average
      period: 300
      length: 300
      nilToZero: true
  • name: PROCESSED_BATCHES_ITEMS
    statistics:
    • Sum
    • Average
      period: 300
      length: 300
      nilToZero: true

[cristiangreco]

@bc-ibiks what version of YACE are you using? Can you make sure it's the latest please?

Also, can you run with -debug and paste the relevant output here? (just make sure it contains no auth details)

[bc-ibiks]

I have updated the latest version of YACE - v0.51.0

I am seeing the error below: is enableMetricData deprecated?

{"level":"info","msg":"Parsing config","time":"2023-05-08T13:59:27Z"}
{"err":"line 32: field enableMetricData not found in type config.Job","level":"warning","msg":"config file syntax error","time":"2023-05-08T13:59:27Z"}
{"err":"line 41: field enableMetricData not found in type config.Job","level":"warning","msg":"config file syntax error","time":"2023-05-08T13:59:27Z"}
{"err":"line 56: field enableMetricData not found in type config.Job","level":"warning","msg":"config file syntax error","time":"2023-05-08T13:59:27Z"}
{"err":"line 118: field enableMetricData not found in type config.Job","level":"warning","msg":"config file syntax error","time":"2023-05-08T13:59:27Z"}
{"err":"line 214: field enableMetricData not found in type config.Static","level":"warning","msg":"config file syntax error","time":"2023-05-08T13:59:27Z"}
{"err":"line 320: field enableMetricData not found in type config.Static","level":"warning","msg":"config file syntax error","time":"2023-05-08T13:59:27Z"}
{"err":"line 418: field enableMetricData not found in type config.Static","level":"warning","msg":"config file syntax error","time":"2023-05-08T13:59:27Z"}
{"err":"line 452: field enableMetricData not found in type config.Static","level":"warning","msg":"config file syntax error","time":"2023-05-08T13:59:27Z"}
{"err":"line 476: field enableMetricData not found in type config.Static","level":"warning","msg":"config file syntax error","time":"2023-05-08T13:59:27Z"}

I removed enableMetricData = true from my config

I am looking through the logs with -debug switched on

saw the below for the metric in the custom namespace

-----------------------------------------------------
2023/05/08 13:41:09 <GetMetricStatisticsResponse xmlns="http://monitoring.amazonaws.com/doc/2010-08-01/">
  <GetMetricStatisticsResult>
    <Datapoints/>
    <Label>PROCESSED_BATCHES</Label>
  </GetMetricStatisticsResult>
  <ResponseMetadata>
    <RequestId>2761db6e-474c-46cc-b5d7-55382768590d</RequestId>
  </ResponseMetadata>
</GetMetricStatisticsResponse>

{"account":"244731367290","arn":"arn:aws:iam::1234567890:role/prometheus","level":"debug","msg":"GetMetricStatistics","output":"{\n  Label: \"PROCESSED_BATCHES\"\n}","region":"us-east-2","static_job_name":"ani","time":"2023-05-08T13:41:09Z"}
2023/05/08 13:41:09 DEBUG: Request monitoring/GetMetricStatistics Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: monitoring.us-east-2.amazonaws.com
User-Agent: aws-sdk-go/1.44.249 (go1.20.3; linux; amd64)
Content-Length: 268
Authorization: [redacted]
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20230508T134109Z
X-Amz-Security-Token: [redacted]
Accept-Encoding: gzip

Action=GetMetricStatistics&Dimensions.member.1.Name=Protocol&Dimensions.member.1.Value=MQTT&EndTime=2023-05-08T13%3A41%3A09.004Z&MetricName=Ping.Success&Namespace=AWS%2FIoT&Period=60&StartTime=2023-05-08T13%3A31%3A09.004Z&Statistics.member.1=Average&Version=2010-08-01
-----------------------------------------------------
2023/05/08 13:41:09 DEBUG: Response monitoring/GetMetricStatistics Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 338
Content-Type: text/xml
Date: Mon, 08 May 2023 13:41:09 GMT
X-Amzn-Requestid: eb33b780-2d38-4c53-b5ca-48ef25958258


-----------------------------------------------------
2023/05/08 13:41:09 <GetMetricStatisticsResponse xmlns="http://monitoring.amazonaws.com/doc/2010-08-01/">
  <GetMetricStatisticsResult>
    <Datapoints/>
    <Label>TimeToFirstByte</Label>
  </GetMetricStatisticsResult>
  <ResponseMetadata>
    <RequestId>eb33b780-2d38-4c53-b5ca-48ef25958258</RequestId>
  </ResponseMetadata>
</GetMetricStatisticsResponse>

[cristiangreco]

I am seeing the error below: is enableMetricData deprecated?

Yes that field has been removed long time ago. CustomNamespace jobs are also deprecated, that's why I was suggesting to swithc to a Static job.

[cristiangreco]

@bc-ibiks it seems the response from AWS GetMetricStatistics does not contain any datapoint.

You mentioned that you can see metrics in CloudWatch. What happens if you run the AWS cli aws cloudwatch get-metric-statistics? (you can find an example in YACE debug logs: YACE emits a log line like CLI helper - ... which tells you how to reproduce with aws cli tool what YACE is requesting).

[bc-ibiks]

I will let you know what I find out - thanks

[mhester-nutrien]

I have YACE running in >60 accounts and ran into this exact same issue. What solved it for me was adding the delay attribute.

        period: 60
        length: 600
        delay: 60

I also found that the addCloudwatchTimestamp attribute helped with this (but did not completely fix), but it did also align the metrics with native prometheus metrics on the Grafana dashboard. Without this attribute the metrics coming from CW/YACE were misaligned.

Also, I ran into another issue with "duplicate" metrics when using nilToZero, and I didn't find any real benefit to using that attribute. I would suggest playing with removing that attribute and seeing if you still get what you expect/want.

[bc-ibiks]

thanks! @mhester-nutrien. I will give this a test and let you know

[bc-ibiks]

@cristiangreco and @mhester-nutrien I was able to retrieve the custom metric using prometheus cloudwatch exporter. (which is a tad annoying to use)
I still wondering why I can pull this with YACE. I am able to pull the metrics using aws cli

Seems I am missing some settings in my config