CloudWatch Joda Decode Issue

[rdifalco]

Just started having a ton of these, anyone seen them yet? Halp!

        <aws-java-sdk.version>1.11.282</aws-java-sdk.version>

Unable to parse date '2018-03-22T08:00:00+00:00':  Invalid format: "2018-03-22T08:00:00+00:00" is malformed at "+00:00"
java.lang.IllegalArgumentException: Invalid format: "2018-03-22T08:00:00+00:00" is malformed at "+00:00"
    at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.util.DateUtils.doParseISO8601Date(DateUtils.java:98) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.util.DateUtils.parseISO8601Date(DateUtils.java:77) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.transform.SimpleTypeStaxUnmarshallers$DateStaxUnmarshaller.unmarshall(SimpleTypeStaxUnmarshallers.java:184) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.services.cloudwatch.model.transform.DatapointStaxUnmarshaller.unmarshall(DatapointStaxUnmarshaller.java:91) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.services.cloudwatch.model.transform.GetMetricStatisticsResultStaxUnmarshaller.unmarshall(GetMetricStatisticsResultStaxUnmarshaller.java:59) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.services.cloudwatch.model.transform.GetMetricStatisticsResultStaxUnmarshaller.unmarshall(GetMetricStatisticsResultStaxUnmarshaller.java:30) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:100) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:40) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1545) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1270) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:1340) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:1316) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executeGetMetricStatistics(AmazonCloudWatchClient.java:830) ~[aws-stats-worker-2.35.319.jar:2.35.319]
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.getMetricStatistics(AmazonCloudWatchClient.java:805) ~[aws-stats-worker-2.35.319.jar:2.35.319]

[mheffner]

We are also seeing this and it is leading to outages.

[varunnvs92]

Are you seeing this issue after upgrading the SDK? If so, what version of the SDK were you using before?
What version of JDK are you on?

[varunnvs92]

we had joda time issue before with certain version of JDK and JodaTime. Please look into the follow issues and see if you are using these affected versions.
#444
#484

[rdifalco]

@varunnvs92, the JDK version is not the issue.

[lucky]

It appears to be isolated to us-west-1

[lucky]

Also, we've confirmed that we're on the latest version of joda (2.9.9)

[varunnvs92]

@rdifalco @mheffner
Can you confirm if you are seeing this issue in only us-west-1 region? We are trying to reproduce the issue and want to isolate if this is a SDK issue or service issue.

[lucky]

@varunnvs92 To clarify I work with @mheffner and I just confirmed with our team it's definitely only us-west-1.

[rdifalco]

Unfortunately we do a bunch of regions at once and I don't have the region in the logs. I can always mock up a test. But it will have to be in an hour or so (meetings).

[varunnvs92]

The issue is related to a deployment on service end in us-west-1. They are rolling back the change. We will update you once the rollback is complete

[lucky]

We can confirm a reduction of errors over time (3hr window)

[lucky]

It appears the errors have completely abated within the last minute or so.

[zoewangg]

Glad to hear that! Thanks for confirming. The rollback is still in progress but should finish soon.

[zoewangg]

The rollback was complete. Please let us know if you see any errors.

[lucky]

I have set up a new (non-Cloudwatch) metric that tracks these failures per region and AWS service, so if we see this again we'll know pretty quickly.

[zoewangg]

Closing the issue, feel free to reopen it if anything comes up.