Not able to create PV for cost analyzer and prometheus pod

[kishore0709]

Hi Team,

My kubecost deployment is not able to create PV & PVC on my EKS cluster. It shows me unauthorized error.

Could you please help suggest what could be the reason.

PV output:
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
default efs-pvc-test Bound efs-pv-test 1Gi RWO aws-efs 12d
kubecost kubecost-cost-analyzer Pending gp2 179m
kubecost kubecost-prometheus-server Pending gp2 179m

Error:

kubecost 54s Warning ProvisioningFailed persistentvolumeclaim/kubecost-cost-analyzer (combined from similar events): Failed to provision volume with StorageClass "gp2": UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: BNt6NUyO1G8R-Q-zsn9qc8g40HPz3mr4tmz9DmxmWtfPY-B4zL14dfvVo15CJu5UUlupp2feA5-QYOR5blJI4dAFgX7V3SlWrB2f49j6GsSLrBRF0o8iiECwuEPxZ_tEaBl8ngR7CS9k1zQZOQVrkN5LGgp0GMO4jrH7Ba4Bs_1DqqyWfHKxBnAPIqcYcpz-QZbGWzvvvU2v4Ty3zLEqJXSMOPrRLyiPbYbBTKLcrIn7hLFg5up91xkB-io1YlAAjAQvNDRVNIBLzr4phR0NwySONy97wA1p6H2kCkvo8R8Um3FgTrMFpzTneWrdZ7GcO6jPK-vC3e0caCwlg-KSYivUjcYjwmOWKbAqEL0IWUxEEb2Xe6RIhW1osKNysoy7nK_41omvMtmUcispcXvTIfUDw-IYuuh2vyozB3gUDnHfuXnMEO9C0i7WChjhd3b-bdmi_wYdyO6S1kyupI_P5S-NTtpOt4c2IeqKVK7iOOYxBmFHhdKDiFIbEIi8T2FAY2nuFc7THnjXCnWSPIECSxku2rmfN7P8YxgROB5LQKZh_OZttbAq2OF71b53e53TAUsUd7jryMFwWTuUexdP-mK5Pr5UR-FvYM7Y0YIAOILZB8RugX2Mrl_A_n6WOS8BIvFYK-ub-XURobBXiNzoXv0Qu4BMislsXO2qz8UjHgs9tJ5g8nJHSOgjZlJ7lcTH8D7EasND5OpxxcUw-tlir3fBw5CqVxRTPJK46fBzH1YjfsOkgk6efJSw8g5arLWrH0Jf523VbkrLE806s0OaWIMn_D1oAIcNLAb_V_z8KejhRFafwC6lkV8SRaFZBzDDLwYsTLRTrCcRcEMfX7YYeLmkSn-c5uCwqbp572uVED_p6ao9wLtwm74JkOIiVuR8RoGNDyQriGQ9r8nHVOQfHu9hYz6TcEbhyRN-mu-TQ3keWi4AFmkdYSUjMwP9gGFAtvpaPplrbYeAsdu--TFjvxsTkq0NGlxOqlLa0bc8WG49k0n4jB1KyRF07mz5tidm2cIOoCujK3MSpfaJhoc8ZiRs57PAWqbWbXDNELv7XEyT5OjNqsal0sR_S5ESF-Q55KN5fI9mJUO0Hx0KycYqflkkXCKYLbx51PKHU4BEXfCjBaPQ3_7N3soHWCjcI-gFBx30-plHKZ-9yll6Jpqq8hpuWX7O2g
status code: 403, request id: f3b35409-62bc-44f0-a4ca-9f0a76fa1032

Thanks!
Kishor.

[kishore0709]

Also, can we use EFS storage class for kubecost-cost-analyzer-fxxxx and kubecost-prometheus-server-xxx pods

[michaelmdresser]

Hi @kishore0709,

The UnauthorizedOperation error looks like its related to AWS IAM permissions. See https://aws.amazon.com/premiumsupport/knowledge-center/ec2-not-auth-launch/ for AWS documentation related to the error, which includes instructions for decoding that encoded failure message which should help you debug.

We'll get back to you about your EFS storage class question soon.

[dwbrown2]

@kishore0709 were you able to solve your permissions issue? If not you can run Kubecost without PVs if you are just trialing.

You can see storageClass for kubecost PV here: https://github.com/kubecost/cost-analyzer-helm-chart/blob/master/cost-analyzer/values.yaml#L264

[kishore0709]

Hi @michaelmdresser

Thanks for your quick response here.

[kishore0709]

Hi @dwbrown2

Firstly thanks for the response.
Actually we are testing kubecost for POC and if everything goes fine will use(kubecost enterprise one) for production workloads.

Can we use EFS storageclass in kubecost PV section:

here: https://github.com/kubecost/cost-analyzer-helm-chart/blob/master/cost-analyzer/values.yaml#L264

[kishore0709]

@dwbrown2

Also, if I make changes in kubecost PV section should we need to explicitly execute that values.yaml file or that helm installation will invoke this storageclass changes ?

Please confirm this.

Thanks!
Kishor

[kishore0709]

@dwbrown2
I tried to use following command to invoke kubecost PV values.yaml but no luck that is still taking the gp2 volumes.

helm upgrade kubecost kubecost/cost-analyzer --cleanup-on-fail --install --namespace kubecost --create-namespace --set kubecostToken="xxxxxxxxxxxxxxxxxxxxx=xm343yadf72" --values cost-analyzer/values.yaml --values kubecost_values.yaml

Can you please help suggest.

Thanks!
Kishor.

[dwbrown2]

Followed up in kubecost/docs#77

[mattray]

This issue is being closed because it may not be relevant to the OpenCost project and appears stale. If you feel this was closed in error, please open a new OpenCost issue with updated details or if it is still relevant for Kubecost please open an issue with Kubecost Support.