Getting “The security token included in the request is invalid” while deploying in EKS in GovCloud

[prabinsh]

Description

I am getting the following error security token included in the request is invalid when attempting to discover AWS RDS using a pod that has administrator rights assigned to its role.
The error message I am receiving is
pmm-managed.log:WARN[2023-03-02T18:12:00.044+00:00] RPC /management.RDS/DiscoverRDS done in 1.244182818s with gRPC error: rpc error: code = InvalidArgument desc = The security token included in the request is invalid request=b90e3874-b925-11ed-889c-626f4a78ebbc ,

and I am unsure where to look to find more information about the issue. Can anyone provide guidance on how to troubleshoot this error?

Expected Results

It should discover the databases

Actual Results

Ran into an error

Version

PMM Server 2.35.0

Steps to reproduce

No response

Relevant logs

No response

Code of Conduct

• I agree to follow Percona Community Code of Conduct

[ritbl]

Hey @prabinsh, thank you for contacting. I need more details on your setup so that I can assist you. How do you run PMM and you setup monitoring?

[prabinsh]

Hello @ritbl ,
I am currently running PMM in AWS EKS and have successfully attached the required role to the EKS node. While adding the CloudWatch datasource, I did not encounter any problems. However, I am experiencing some difficulties with the discovery process as it seems to be encountering errors. I would appreciate any advice or recommendations you may have in resolving
this issue.

I also tested by running the docker in my local and trying to discover the AWS instances and ran into the same issue.

I believe you can replicate this easily on your local machine as well.

[prabinsh]

I was running this in GovCloud. I came across this link and was able to configure this https://poc2.zoomgovdev.com/graph/add-instance https://docs.percona.com/percona-monitoring-and-management/faq.html#can-i-add-an-aws-rds-mysql-or-aurora-mysql-instance-from-a-non-default-aws-partition