Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AmazonCloudWatchLogsClient.ListAnomaliesAsync throws Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart #3205

Closed
danispir opened this issue Feb 29, 2024 · 14 comments
Labels
bug This issue is a bug. module/sdk-custom p1 This is a high priority issue service-api This issue is due to a problem in a service API, not the SDK implementation.

Comments

@danispir
Copy link

Describe the bug

Hello,
On our AWS business account we have Log Groups created to group various logs coming from our services. One of the log group has Anomaly Detector set to ON.
In our code repository, we want to implement a .NET 8 Micro-Service that pulls Log Anomalies from a specified Anomaly Detector ARN.

We have the following setup:
TargetFramework: net8.0

We are using latest nuget packages:
AWSSDK.CloudWatch Version="3.7.302.5"
AWSSDK.CloudWatchLogs Version="3.7.304.7"

We have created properly a IAM user with correct rights and we are executing the following code:

var cloudWatchClient = new AmazonCloudWatchClient(credentials, Amazon.RegionEndpoint.MY_REGION_HERE);
var result = await client.ListAnomaliesAsync(new ListAnomaliesRequest { AnomalyDetectorArn = serverArn, Limit = 10, });

serverArn is valid ARN from our subscription.

The above code throws the following exception: Amazon.Runtime.AmazonUnmarshallingException

With the following message:
Error unmarshalling response back from AWS. Request ID: 9*****-ffd8ed4093b7, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK**

And this stacktrace
at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
at Amazon.Runtime.Internal.Unmarshaller.d__5.MoveNext()
at Amazon.Runtime.Internal.Unmarshaller.d__31.MoveNext() at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__51.MoveNext()
at Amazon.Runtime.Internal.ErrorHandler.d__51.MoveNext() at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__91.MoveNext()
at Amazon.Runtime.Internal.Signer.d__11.MoveNext() at Amazon.Runtime.Internal.EndpointDiscoveryHandler.<InvokeAsync>d__21.MoveNext()
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.d__21.MoveNext() at Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__71.MoveNext()
at Amazon.Runtime.Internal.RetryHandler.d__101.MoveNext() at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__101.MoveNext()
at Amazon.Runtime.Internal.CallbackHandler.d__91.MoveNext() at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__91.MoveNext()
at Amazon.Runtime.Internal.ErrorCallbackHandler.d__51.MoveNext() at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__11.MoveNext()
at Program.<

$>d__0.MoveNext() in D:\ZCode\AwsAnomaliesDetector\Program.cs:line 71

And this inner exception:
We expected a VALUE token but got: ObjectStart

If a call method ListAnomaliesAsync without any ARN specified, I get the same exception and stacktrace as above.

If I call other methods such as ListLogAnomalyDetectorsAsync or GetLogAnomalyDetectorAsync I do get successful results.

I tried also AWS Support Central, however they don't provide code assistance.

Any help will be much appreciated.

Expected Behavior

I should receive a successful response as described in this document

https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_ListAnomalies.html

Current Behavior

AmazonCloudWatchLogsClient.ListAnomaliesAsync throws Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart

Reproduction Steps

Specified in bug description.

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.CloudWatchLogs 3.7.304.7

Targeted .NET Platform

.NET 8

Operating System and version

Windows 10/11, Linux

@danispir danispir added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 29, 2024
@dscpinheiro
Copy link
Contributor

Do you see the same error when using the AWS CLI? For example: aws logs list-anomalies --anomaly-detector-arn "serverArn". Our CloudWatch Logs client is generated, we don't modify the response that's coming from the service.

@dscpinheiro dscpinheiro added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Feb 29, 2024
@danispir
Copy link
Author

Hi @dscpinheiro. Thanks for your reply.
I tried, but the method list-anomalies is not available at all. Error aws: error: argument operation: Invalid choice
I also tried it with Python Boto3 and I'm getting the message list_anomalies attribute does not exists. Although it is available in their documentation.

My assumption is that this method has been hidden entirely?

@dscpinheiro
Copy link
Contributor

No, the API is public but it was released recently, during last year's re:Invent: https://github.com/aws/aws-sdk-net/blob/main/changelogs/SDK.CHANGELOG.2023.md#376930-2023-11-27-0424-utc.

It's available in the last version of the CLI, so you may have to update your version:

PS> aws --version
aws-cli/2.15.24 Python/3.11.6 Windows/10 exe/AMD64 prompt/off

PS> aws logs list-anomalies
{
    "anomalies": []
}

@danispir
Copy link
Author

You were right. Now I can execute correctly aws logs list-anomalies command.

But from AWSSDK.NET in my .net 8 console app, I'm still getting same error. Any idea?
image

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Mar 1, 2024
@bhoradc bhoradc added module/sdk-custom p3 This is a minor priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Mar 1, 2024
@normj
Copy link
Member

normj commented Mar 1, 2024

@danispir Can you enable response logging by adding the following code at the startup of your test application and share the response JSON from the call. Be sure to redact anything sensitive in the response.

AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;

@danispir
Copy link
Author

danispir commented Mar 1, 2024

Hello @normj. Below is the output I get. Since it seems that I do get the response, could be a deserialization issue when trying to deserialize LogSamples array of objects?

EnvironmentVariableInternalConfiguration 1|2024-03-01T10:49:25.354Z|INFO|The environment variable AWS_ENABLE_ENDPOINT_DISCOVERY was not set with a value.
EnvironmentVariableInternalConfiguration 2|2024-03-01T10:49:25.392Z|INFO|The environment variable AWS_MAX_ATTEMPTS was not set with a value.
EnvironmentVariableInternalConfiguration 3|2024-03-01T10:49:25.393Z|INFO|The environment variable AWS_RETRY_MODE was not set with a value.
EnvironmentVariableInternalConfiguration 4|2024-03-01T10:49:25.398Z|INFO|The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT was not set with a value.
EnvironmentVariableInternalConfiguration 5|2024-03-01T10:49:25.398Z|INFO|The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE was not set with a value.
EnvironmentVariableInternalConfiguration 6|2024-03-01T10:49:25.398Z|INFO|The environment variable AWS_EC2_METADATA_V1_DISABLED was not set with a value.
EnvironmentVariableInternalConfiguration 7|2024-03-01T10:49:25.398Z|INFO|The environment variable AWS_USE_DUALSTACK_ENDPOINT was not set with a value.
EnvironmentVariableInternalConfiguration 8|2024-03-01T10:49:25.398Z|INFO|The environment variable AWS_USE_FIPS_ENDPOINT was not set with a value.
EnvironmentVariableInternalConfiguration 9|2024-03-01T10:49:25.398Z|INFO|The environment variable AWS_IGNORE_CONFIGURED_ENDPOINT_URLS was not set with a value.
EnvironmentVariableInternalConfiguration 10|2024-03-01T10:49:25.399Z|INFO|The environment variable AWS_DISABLE_REQUEST_COMPRESSION was not set with a value.
EnvironmentVariableInternalConfiguration 11|2024-03-01T10:49:25.399Z|INFO|The environment variable AWS_REQUEST_MIN_COMPRESSION_SIZE_BYTES was not set with a value.
EnvironmentVariableInternalConfiguration 12|2024-03-01T10:49:25.399Z|INFO|The environment variable AWS_SDK_UA_APP_ID was not set with a value.
ProfileInternalConfiguration 13|2024-03-01T10:49:25.442Z|INFO|There is no defaults_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 14|2024-03-01T10:49:25.443Z|INFO|There is no endpoint_discovery_enabled set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 15|2024-03-01T10:49:25.443Z|INFO|There is no retry_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 16|2024-03-01T10:49:25.443Z|INFO|There is no max_attempts set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 17|2024-03-01T10:49:25.443Z|INFO|There is no ec2_metadata_service_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 18|2024-03-01T10:49:25.444Z|INFO|There is no ec2_metadata_service_endpoint_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 19|2024-03-01T10:49:25.444Z|INFO|There is no ec2_metadata_v1_disabled set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 20|2024-03-01T10:49:25.445Z|INFO|There is no use_dualstack_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 21|2024-03-01T10:49:25.445Z|INFO|There is no use_fips_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 22|2024-03-01T10:49:25.445Z|INFO|ignore_configured_endpoint_urls found in profile 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 23|2024-03-01T10:49:25.446Z|INFO|There is no endpoint_url set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 24|2024-03-01T10:49:25.446Z|INFO|There is no disable_request_compression set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 25|2024-03-01T10:49:25.446Z|INFO|There is no request_min_compression_size_bytes set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 26|2024-03-01T10:49:25.446Z|INFO|There is no sdk_ua_app_id set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
DefaultConfigurationProvider 27|2024-03-01T10:49:25.449Z|INFO|Resolved DefaultConfigurationMode for RegionEndpoint [*****] to [Legacy].
AmazonCloudWatchLogsClient 28|2024-03-01T10:49:27.074Z|DEBUG|Received response (truncated to 1024 bytes): [{"anomalies":[{"active":false,"anomalyDetectorArn":"arn:aws:logs:*********************","anomalyId":"a*******************************3c","description":"Unexpected pattern detected with severity ERROR","firstSeen":1708541760000,"histogram":{"1708524000000":3,"1708542000000":230,"1708560000000":269,"1708578000000":381,"1708596000000":381,"1708614000000":715,"1708632000000":725,"1708650000000":640,"1708668000000":718,"1708686000000":646,"1708704000000":716,"1708722000000":719,"1708740000000":712,"1708758000000":712,"1708776000000":719,"1708794000000":725,"1708812000000":710,"1708830000000":711,"1708848000000":716,"1708866000000":709,"1708884000000":717,"1708902000000":563,"1708920000000":235,"1708938000000":245,"1708956000000":240,"1708974000000":240,"1708992000000":241,"1709010000000":240,"1709028000000":240,"1709046000000":239,"1709064000000":233,"1709082000000":238,"1709100000000":239,"1709118000000":269,"1709136000000":479,"1709154000000":478,"17091720]
AmazonCloudWatchLogsClient 29|2024-03-01T10:49:27.128Z|ERROR|An exception of type AmazonUnmarshallingException was handled in ErrorHandler. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: ********-****-4b72-81db-b68**465d2**, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.StringUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonCloudWatchLogsClient 30|2024-03-01T10:49:27.365Z|INFO|AmazonUnmarshallingException making request ListAnomaliesRequest to https://logs.*****.amazonaws.com/. Attempting retry 1 of 4.
AmazonCloudWatchLogsClient 31|2024-03-01T10:49:28.190Z|DEBUG|Received response (truncated to 1024 bytes): [{"anomalies":[{"active":false,"anomalyDetectorArn":"arn:aws:logs:*********************","anomalyId":"a*******************************3c","description":"Unexpected pattern detected with severity ERROR","firstSeen":1708541760000,"histogram":{"1708524000000":3,"1708542000000":230,"1708560000000":269,"1708578000000":381,"1708596000000":381,"1708614000000":715,"1708632000000":725,"1708650000000":640,"1708668000000":718,"1708686000000":646,"1708704000000":716,"1708722000000":719,"1708740000000":712,"1708758000000":712,"1708776000000":719,"1708794000000":725,"1708812000000":710,"1708830000000":711,"1708848000000":716,"1708866000000":709,"1708884000000":717,"1708902000000":563,"1708920000000":235,"1708938000000":245,"1708956000000":240,"1708974000000":240,"1708992000000":241,"1709010000000":240,"1709028000000":240,"1709046000000":239,"1709064000000":233,"1709082000000":238,"1709100000000":239,"1709118000000":269,"1709136000000":479,"1709154000000":478,"17091720]
AmazonCloudWatchLogsClient 32|2024-03-01T10:49:28.236Z|ERROR|An exception of type AmazonUnmarshallingException was handled in ErrorHandler. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: *****-a663-46fc-aa5c-*********, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.StringUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonCloudWatchLogsClient 33|2024-03-01T10:49:28.421Z|INFO|AmazonUnmarshallingException making request ListAnomaliesRequest to https://logs.*****.amazonaws.com/. Attempting retry 2 of 4.
AmazonCloudWatchLogsClient 34|2024-03-01T10:49:30.621Z|DEBUG|Received response (truncated to 1024 bytes): [{"anomalies":[{"active":false,"anomalyDetectorArn":"arn:aws:logs:*********************","anomalyId":"a*******************************3c","description":"Unexpected pattern detected with severity ERROR","firstSeen":1708541760000,"histogram":{"1708524000000":3,"1708542000000":230,"1708560000000":269,"1708578000000":381,"1708596000000":381,"1708614000000":715,"1708632000000":725,"1708650000000":640,"1708668000000":718,"1708686000000":646,"1708704000000":716,"1708722000000":719,"1708740000000":712,"1708758000000":712,"1708776000000":719,"1708794000000":725,"1708812000000":710,"1708830000000":711,"1708848000000":716,"1708866000000":709,"1708884000000":717,"1708902000000":563,"1708920000000":235,"1708938000000":245,"1708956000000":240,"1708974000000":240,"1708992000000":241,"1709010000000":240,"1709028000000":240,"1709046000000":239,"1709064000000":233,"1709082000000":238,"1709100000000":239,"1709118000000":269,"1709136000000":479,"1709154000000":478,"17091720]
AmazonCloudWatchLogsClient 35|2024-03-01T10:49:30.669Z|ERROR|An exception of type AmazonUnmarshallingException was handled in ErrorHandler. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: *******-****-43f5-*****-*****, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.StringUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonCloudWatchLogsClient 36|2024-03-01T10:49:30.850Z|INFO|AmazonUnmarshallingException making request ListAnomaliesRequest to https://logs.*****.amazonaws.com/. Attempting retry 3 of 4.
AmazonCloudWatchLogsClient 37|2024-03-01T10:49:38.460Z|DEBUG|Received response (truncated to 1024 bytes): [{"anomalies":[{"active":false,"anomalyDetectorArn":"arn:aws:logs:*********************","anomalyId":"a*******************************3c","description":"Unexpected pattern detected with severity ERROR","firstSeen":1708541760000,"histogram":{"1708524000000":3,"1708542000000":230,"1708560000000":269,"1708578000000":381,"1708596000000":381,"1708614000000":715,"1708632000000":725,"1708650000000":640,"1708668000000":718,"1708686000000":646,"1708704000000":716,"1708722000000":719,"1708740000000":712,"1708758000000":712,"1708776000000":719,"1708794000000":725,"1708812000000":710,"1708830000000":711,"1708848000000":716,"1708866000000":709,"1708884000000":717,"1708902000000":563,"1708920000000":235,"1708938000000":245,"1708956000000":240,"1708974000000":240,"1708992000000":241,"1709010000000":240,"1709028000000":240,"1709046000000":239,"1709064000000":233,"1709082000000":238,"1709100000000":239,"1709118000000":269,"1709136000000":479,"1709154000000":478,"17091720]
AmazonCloudWatchLogsClient 38|2024-03-01T10:49:38.581Z|ERROR|An exception of type AmazonUnmarshallingException was handled in ErrorHandler. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: *****-95f1-43ab-*****-********, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonCloudWatchLogsClient 39|2024-03-01T10:49:39.041Z|INFO|AmazonUnmarshallingException making request ListAnomaliesRequest to https://logs.*****.amazonaws.com/. Attempting retry 4 of 4.
AmazonCloudWatchLogsClient 40|2024-03-01T10:50:05.973Z|DEBUG|Received response (truncated to 1024 bytes): [{"anomalies":[{"active":false,"anomalyDetectorArn":"arn:aws:logs:*********************","anomalyId":"a*******************************3c","description":"Unexpected pattern detected with severity ERROR","firstSeen":1708541760000,"histogram":{"1708524000000":3,"1708542000000":230,"1708560000000":269,"1708578000000":381,"1708596000000":381,"1708614000000":715,"1708632000000":725,"1708650000000":640,"1708668000000":718,"1708686000000":646,"1708704000000":716,"1708722000000":719,"1708740000000":712,"1708758000000":712,"1708776000000":719,"1708794000000":725,"1708812000000":710,"1708830000000":711,"1708848000000":716,"1708866000000":709,"1708884000000":717,"1708902000000":563,"1708920000000":235,"1708938000000":245,"1708956000000":240,"1708974000000":240,"1708992000000":241,"1709010000000":240,"1709028000000":240,"1709046000000":239,"1709064000000":233,"1709082000000":238,"1709100000000":239,"1709118000000":269,"1709136000000":479,"1709154000000":478,"17091720]
AmazonCloudWatchLogsClient 41|2024-03-01T10:50:06.067Z|ERROR|An exception of type AmazonUnmarshallingException was handled in ErrorHandler. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: 5*****-*****-*****-****-********, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonCloudWatchLogsClient 42|2024-03-01T10:50:06.520Z|ERROR|AmazonUnmarshallingException making request ListAnomaliesRequest to https://logs.*****.amazonaws.com/. Attempt 5. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: *****-1758-47f4-8bfd-**************, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)

@bhoradc bhoradc added p1 This is a high priority issue and removed p3 This is a minor priority issue needs-review labels Mar 1, 2024
@normj
Copy link
Member

normj commented Mar 1, 2024

@danispir Thanks for the log. Unfortunately it cut off the response payload at 1K and I think we are missing the part that the unmarshaller is having a problem. Can you add the following line increase the size of the response payload logging?

AWSConfigs.LoggingConfig.LogResponsesSizeLimit = 10485760;

@danispir
Copy link
Author

danispir commented Mar 2, 2024

I increased it to 20_000_000.

EnvironmentVariableInternalConfiguration 1|2024-03-02T06:29:12.601Z|INFO|The environment variable AWS_ENABLE_ENDPOINT_DISCOVERY was not set with a value.
EnvironmentVariableInternalConfiguration 2|2024-03-02T06:29:12.701Z|INFO|The environment variable AWS_MAX_ATTEMPTS was not set with a value.
EnvironmentVariableInternalConfiguration 3|2024-03-02T06:29:12.702Z|INFO|The environment variable AWS_RETRY_MODE was not set with a value.
EnvironmentVariableInternalConfiguration 4|2024-03-02T06:29:12.716Z|INFO|The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT was not set with a value.
EnvironmentVariableInternalConfiguration 5|2024-03-02T06:29:12.717Z|INFO|The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE was not set with a value.
EnvironmentVariableInternalConfiguration 6|2024-03-02T06:29:12.717Z|INFO|The environment variable AWS_EC2_METADATA_V1_DISABLED was not set with a value.
EnvironmentVariableInternalConfiguration 7|2024-03-02T06:29:12.717Z|INFO|The environment variable AWS_USE_DUALSTACK_ENDPOINT was not set with a value.
EnvironmentVariableInternalConfiguration 8|2024-03-02T06:29:12.718Z|INFO|The environment variable AWS_USE_FIPS_ENDPOINT was not set with a value.
EnvironmentVariableInternalConfiguration 9|2024-03-02T06:29:12.718Z|INFO|The environment variable AWS_IGNORE_CONFIGURED_ENDPOINT_URLS was not set with a value.
EnvironmentVariableInternalConfiguration 10|2024-03-02T06:29:12.719Z|INFO|The environment variable AWS_DISABLE_REQUEST_COMPRESSION was not set with a value.
EnvironmentVariableInternalConfiguration 11|2024-03-02T06:29:12.719Z|INFO|The environment variable AWS_REQUEST_MIN_COMPRESSION_SIZE_BYTES was not set with a value.
EnvironmentVariableInternalConfiguration 12|2024-03-02T06:29:12.720Z|INFO|The environment variable AWS_SDK_UA_APP_ID was not set with a value.
ProfileInternalConfiguration 13|2024-03-02T06:29:12.835Z|INFO|There is no defaults_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 14|2024-03-02T06:29:12.836Z|INFO|There is no endpoint_discovery_enabled set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 15|2024-03-02T06:29:12.836Z|INFO|There is no retry_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 16|2024-03-02T06:29:12.836Z|INFO|There is no max_attempts set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 17|2024-03-02T06:29:12.839Z|INFO|There is no ec2_metadata_service_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 18|2024-03-02T06:29:12.840Z|INFO|There is no ec2_metadata_service_endpoint_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 19|2024-03-02T06:29:12.843Z|INFO|There is no ec2_metadata_v1_disabled set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 20|2024-03-02T06:29:12.844Z|INFO|There is no use_dualstack_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 21|2024-03-02T06:29:12.845Z|INFO|There is no use_fips_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 22|2024-03-02T06:29:12.847Z|INFO|ignore_configured_endpoint_urls found in profile 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 23|2024-03-02T06:29:12.847Z|INFO|There is no endpoint_url set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 24|2024-03-02T06:29:12.849Z|INFO|There is no disable_request_compression set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 25|2024-03-02T06:29:12.852Z|INFO|There is no request_min_compression_size_bytes set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
ProfileInternalConfiguration 26|2024-03-02T06:29:12.853Z|INFO|There is no sdk_ua_app_id set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
DefaultConfigurationProvider 27|2024-03-02T06:29:12.861Z|INFO|Resolved DefaultConfigurationMode for RegionEndpoint [**-****-*] to [Legacy].
AmazonCloudWatchLogsClient 28|2024-03-02T06:29:14.928Z|DEBUG|Received response (truncated to 20000000 bytes): [{"anomalies":[{"active":false,"anomalyDetectorArn":"arn:aws:logs:*********************","anomalyId":"a55219e6-92f0-4e21-b94c-a4267311613c","description":"Unexpected pattern detected with severity ERROR","firstSeen":1708541760000,"histogram":{"1708524000000":3,"1708542000000":230,"1708560000000":269,"1708578000000":381,"1708596000000":381,"1708614000000":715,"1708632000000":725,"1708650000000":640,"1708668000000":718,"1708686000000":646,"1708704000000":716,"1708722000000":719,"1708740000000":712,"1708758000000":712,"1708776000000":719,"1708794000000":725,"1708812000000":710,"1708830000000":711,"1708848000000":716,"1708866000000":709,"1708884000000":717,"1708902000000":563,"1708920000000":235,"1708938000000":245,"1708956000000":240,"1708974000000":240,"1708992000000":241,"1709010000000":240,"1709028000000":240,"1709046000000":239,"1709064000000":233,"1709082000000":238,"1709100000000":239,"1709118000000":269,"1709136000000":479,"1709154000000":478,"1709172000000":475,"1709190000000":472,"1709208000000":475,"1709226000000":477,"1709244000000":478,"1709262000000":475,"1709280000000":483,"1709298000000":707,"1709316000000":716,"1709334000000":713,"1709352000000":61},"isPatternLevelSuppression":false,"lastSeen":1709353500000,"logGroupArnList":["arn:aws:logs:**-****-*:818373788390:log-group:Prod-Alert-******"],"logSamples":[{"message":"{\"Timestamp\":\"2024-02-21T19:04:47.4963866+00:00\",\"Level\":\"Error\",\"MessageTemplate\":\"System.Net.Http.HttpRequestException: An error occurred while sending the request.\\n ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.\\n ---> System.Net.Sockets.SocketException (104): Connection reset by peer\\n   --- End of inner exception stack trace ---\\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)\\n   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)\\n   at S]
AmazonCloudWatchLogsClient 29|2024-03-02T06:29:15.071Z|ERROR|An exception of type AmazonUnmarshallingException was handled in ErrorHandler. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: d8aaa943-774e-4d9b-b1f6-7b0f3a3a8c15, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.StringUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonCloudWatchLogsClient 30|2024-03-02T06:29:15.641Z|INFO|AmazonUnmarshallingException making request ListAnomaliesRequest to https://logs.*****.amazonaws.com/. Attempting retry 1 of 4.
AmazonCloudWatchLogsClient 31|2024-03-02T06:29:16.490Z|DEBUG|Received response (truncated to 20000000 bytes): [{"anomalies":[{"active":false,"anomalyDetectorArn":"arn:aws:logs:*********************","anomalyId":"a55219e6-92f0-4e21-b94c-a4267311613c","description":"Unexpected pattern detected with severity ERROR","firstSeen":1708541760000,"histogram":{"1708524000000":3,"1708542000000":230,"1708560000000":269,"1708578000000":381,"1708596000000":381,"1708614000000":715,"1708632000000":725,"1708650000000":640,"1708668000000":718,"1708686000000":646,"1708704000000":716,"1708722000000":719,"1708740000000":712,"1708758000000":712,"1708776000000":719,"1708794000000":725,"1708812000000":710,"1708830000000":711,"1708848000000":716,"1708866000000":709,"1708884000000":717,"1708902000000":563,"1708920000000":235,"1708938000000":245,"1708956000000":240,"1708974000000":240,"1708992000000":241,"1709010000000":240,"1709028000000":240,"1709046000000":239,"1709064000000":233,"1709082000000":238,"1709100000000":239,"1709118000000":269,"1709136000000":479,"1709154000000":478,"1709172000000":475,"1709190000000":472,"1709208000000":475,"1709226000000":477,"1709244000000":478,"1709262000000":475,"1709280000000":483,"1709298000000":707,"1709316000000":716,"1709334000000":713,"1709352000000":61},"isPatternLevelSuppression":false,"lastSeen":1709353500000,"logGroupArnList":["arn:aws:logs:******-*******-group:**************"],"logSamples":[{"message":"{\"Timestamp\":\"2024-02-21T19:04:47.4963866+00:00\",\"Level\":\"Error\",\"MessageTemplate\":\"System.Net.Http.HttpRequestException: An error occurred while sending the request.\\n ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.\\n ---> System.Net.Sockets.SocketException (104): Connection reset by peer\\n   --- End of inner exception stack trace ---\\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)\\n   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)\\n   at S]
AmazonCloudWatchLogsClient 32|2024-03-02T06:29:16.602Z|ERROR|An exception of type AmazonUnmarshallingException was handled in ErrorHandler. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: *******-b0c3-4e74-b9cd-1c2d936b552a, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.StringUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonCloudWatchLogsClient 33|2024-03-02T06:29:17.085Z|INFO|AmazonUnmarshallingException making request ListAnomaliesRequest to https://logs.*****.amazonaws.com/. Attempting retry 2 of 4.
AmazonCloudWatchLogsClient 34|2024-03-02T06:29:19.346Z|DEBUG|Received response (truncated to 20000000 bytes): [{"anomalies":[{"active":false,"anomalyDetectorArn":"arn:aws:logs:*********************","anomalyId":"a55219e6-92f0-4e21-b94c-a4267311613c","description":"Unexpected pattern detected with severity ERROR","firstSeen":1708541760000,"histogram":{"1708524000000":3,"1708542000000":230,"1708560000000":269,"1708578000000":381,"1708596000000":381,"1708614000000":715,"1708632000000":725,"1708650000000":640,"1708668000000":718,"1708686000000":646,"1708704000000":716,"1708722000000":719,"1708740000000":712,"1708758000000":712,"1708776000000":719,"1708794000000":725,"1708812000000":710,"1708830000000":711,"1708848000000":716,"1708866000000":709,"1708884000000":717,"1708902000000":563,"1708920000000":235,"1708938000000":245,"1708956000000":240,"1708974000000":240,"1708992000000":241,"1709010000000":240,"1709028000000":240,"1709046000000":239,"1709064000000":233,"1709082000000":238,"1709100000000":239,"1709118000000":269,"1709136000000":479,"1709154000000":478,"1709172000000":475,"1709190000000":472,"1709208000000":475,"1709226000000":477,"1709244000000":478,"1709262000000":475,"1709280000000":483,"1709298000000":707,"1709316000000":716,"1709334000000":713,"1709352000000":61},"isPatternLevelSuppression":false,"lastSeen":1709353500000,"logGroupArnList":["arn:aws:logs:**-****-*:818373788390:log-group:Prod-Alert-******"],"logSamples":[{"message":"{\"Timestamp\":\"2024-02-21T19:04:47.4963866+00:00\",\"Level\":\"Error\",\"MessageTemplate\":\"System.Net.Http.HttpRequestException: An error occurred while sending the request.\\n ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.\\n ---> System.Net.Sockets.SocketException (104): Connection reset by peer\\n   --- End of inner exception stack trace ---\\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)\\n   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)\\n   at S]
AmazonCloudWatchLogsClient 35|2024-03-02T06:29:19.468Z|ERROR|An exception of type AmazonUnmarshallingException was handled in ErrorHandler. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: 13978e97-d8a5-4aaf-8ddd-71995fc87ad4, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.StringUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonCloudWatchLogsClient 36|2024-03-02T06:29:19.945Z|INFO|AmazonUnmarshallingException making request ListAnomaliesRequest to https://logs.*****.amazonaws.com/. Attempting retry 3 of 4.
AmazonCloudWatchLogsClient 37|2024-03-02T06:29:27.559Z|DEBUG|Received response (truncated to 20000000 bytes): [{"anomalies":[{"active":false,"anomalyDetectorArn":"arn:aws:logs:*********************","anomalyId":"a55219e6-92f0-4e21-b94c-a4267311613c","description":"Unexpected pattern detected with severity ERROR","firstSeen":1708541760000,"histogram":{"1708524000000":3,"1708542000000":230,"1708560000000":269,"1708578000000":381,"1708596000000":381,"1708614000000":715,"1708632000000":725,"1708650000000":640,"1708668000000":718,"1708686000000":646,"1708704000000":716,"1708722000000":719,"1708740000000":712,"1708758000000":712,"1708776000000":719,"1708794000000":725,"1708812000000":710,"1708830000000":711,"1708848000000":716,"1708866000000":709,"1708884000000":717,"1708902000000":563,"1708920000000":235,"1708938000000":245,"1708956000000":240,"1708974000000":240,"1708992000000":241,"1709010000000":240,"1709028000000":240,"1709046000000":239,"1709064000000":233,"1709082000000":238,"1709100000000":239,"1709118000000":269,"1709136000000":479,"1709154000000":478,"1709172000000":475,"1709190000000":472,"1709208000000":475,"1709226000000":477,"1709244000000":478,"1709262000000":475,"1709280000000":483,"1709298000000":707,"1709316000000":716,"1709334000000":713,"1709352000000":61},"isPatternLevelSuppression":false,"lastSeen":1709353500000,"logGroupArnList":["arn:aws:logs:**-****-*:818373788390:log-group:Prod-Alert-******"],"logSamples":[{"message":"{\"Timestamp\":\"2024-02-21T19:04:47.4963866+00:00\",\"Level\":\"Error\",\"MessageTemplate\":\"System.Net.Http.HttpRequestException: An error occurred while sending the request.\\n ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.\\n ---> System.Net.Sockets.SocketException (104): Connection reset by peer\\n   --- End of inner exception stack trace ---\\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)\\n   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)\\n   at S]
AmazonCloudWatchLogsClient 38|2024-03-02T06:29:27.683Z|ERROR|An exception of type AmazonUnmarshallingException was handled in ErrorHandler. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: b0df53fa-6696-4429-a24c-1d241ab1e420, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonCloudWatchLogsClient 39|2024-03-02T06:29:28.178Z|INFO|AmazonUnmarshallingException making request ListAnomaliesRequest to https://logs.*****.amazonaws.com/. Attempting retry 4 of 4.
AmazonCloudWatchLogsClient 40|2024-03-02T06:29:54.987Z|DEBUG|Received response (truncated to 20000000 bytes): [{"anomalies":[{"active":false,"anomalyDetectorArn":"arn:aws:logs:*********************","anomalyId":"a55219e6-92f0-4e21-b94c-a4267311613c","description":"Unexpected pattern detected with severity ERROR","firstSeen":1708541760000,"histogram":{"1708524000000":3,"1708542000000":230,"1708560000000":269,"1708578000000":381,"1708596000000":381,"1708614000000":715,"1708632000000":725,"1708650000000":640,"1708668000000":718,"1708686000000":646,"1708704000000":716,"1708722000000":719,"1708740000000":712,"1708758000000":712,"1708776000000":719,"1708794000000":725,"1708812000000":710,"1708830000000":711,"1708848000000":716,"1708866000000":709,"1708884000000":717,"1708902000000":563,"1708920000000":235,"1708938000000":245,"1708956000000":240,"1708974000000":240,"1708992000000":241,"1709010000000":240,"1709028000000":240,"1709046000000":239,"1709064000000":233,"1709082000000":238,"1709100000000":239,"1709118000000":269,"1709136000000":479,"1709154000000":478,"1709172000000":475,"1709190000000":472,"1709208000000":475,"1709226000000":477,"1709244000000":478,"1709262000000":475,"1709280000000":483,"1709298000000":707,"1709316000000":716,"1709334000000":713,"1709352000000":61},"isPatternLevelSuppression":false,"lastSeen":1709353500000,"logGroupArnList":["arn:aws:logs:**-****-*:818373788390:log-group:Prod-Alert-******"],"logSamples":[{"message":"{\"Timestamp\":\"2024-02-21T19:04:47.4963866+00:00\",\"Level\":\"Error\",\"MessageTemplate\":\"System.Net.Http.HttpRequestException: An error occurred while sending the request.\\n ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.\\n ---> System.Net.Sockets.SocketException (104): Connection reset by peer\\n   --- End of inner exception stack trace ---\\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)\\n   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)\\n   at S]
AmazonCloudWatchLogsClient 41|2024-03-02T06:29:55.117Z|ERROR|An exception of type AmazonUnmarshallingException was handled in ErrorHandler. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: b194a96a-3446-47b1-84c4-8760f0070b19, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
AmazonCloudWatchLogsClient 42|2024-03-02T06:29:55.594Z|ERROR|AmazonUnmarshallingException making request ListAnomaliesRequest to https://logs.*****.amazonaws.com/. Attempt 5. --> Amazon.Runtime.AmazonUnmarshallingException: Error unmarshalling response back from AWS. Request ID: b194a96a-3446-47b1-84c4-8760f0070b19, Last Parsed Path: /anomalies//logSamples//, HTTP Status Code: 200 OK
 ---> Amazon.Runtime.AmazonClientException: We expected a VALUE token but got: ObjectStart
   at Amazon.Runtime.Internal.Transform.JsonUnmarshallerContext.ReadText()
   at Amazon.Runtime.Internal.Transform.SimpleTypeUnmarshaller`1.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.AnomalyUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.ListUnmarshaller`2.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.CloudWatchLogs.Model.Internal.MarshallTransformations.ListAnomaliesResponseUnmarshaller.Unmarshall(JsonUnmarshallerContext context)
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.Transform.JsonResponseUnmarshaller.Unmarshall(UnmarshallerContext input)
   at Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext)
   at Amazon.Runtime.Internal.Unmarshaller.UnmarshallAsync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)

@normj
Copy link
Member

normj commented Mar 4, 2024

@danispir Thanks for the bigger snippet. The logSamples is coming back different then the .NET SDK is expecting. I have a query to the CloudWatch Logs team going on to understand why that is.

@ashishdhingra ashishdhingra added service-api This issue is due to a problem in a service API, not the SDK implementation. needs-review labels Mar 7, 2024
@ashishdhingra
Copy link
Contributor

P120800175

@dscpinheiro
Copy link
Contributor

@danispir Could you test with the new version of the CloudWatch package that was just released? https://www.nuget.org/packages/AWSSDK.CloudWatchLogs/3.7.305

It includes an update to the ListAnomalies API that hopefully fixes your issue.

@danispir
Copy link
Author

Hi @dscpinheiro. I can confirm that with the latest version it is working. Thanks everyone for your support!

I would like to point another thing, but is not related in this case to AWS SDK, because it's behaving the same when running through aws cli command line. Maybe somebody can pass this information to CloudWatch logs team.

When executing list anomalies, there is LogSamples array which contains messages and templates. If the message is above a specific length, it is truncated. This is a bit frustrating for us developers. However, in the browser on AWS Console, when navigating the anomalies page in CloudWatch, log samples tab displays entire message. One work around is to take the timestamp and execute some requests over CloudWatch logs in order to fetch entire message. But of course, this implies other API calls on our side, which increases execution time when we have large amount of log data.

@dscpinheiro
Copy link
Contributor

Glad that resolved your issue, I'll pass the feedback to the CloudWatch team.

Copy link

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. module/sdk-custom p1 This is a high priority issue service-api This issue is due to a problem in a service API, not the SDK implementation.
Projects
None yet
Development

No branches or pull requests

5 participants