Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Cloudwatch datasource with "Assume Role ARN" doesn't work in Openshift #374

Closed
alrf opened this issue Mar 22, 2021 · 2 comments
Closed
Labels
bug Something isn't working needs triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@alrf
Copy link

alrf commented Mar 22, 2021

Describe the bug
Getting these logs during Cloudwatch datasource with "Assume Role ARN" configuration in Openshift:

failed to call cloudwatch:ListMetrics: NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: 
failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, .\nEC2RoleRequestError: 
no EC2 instance role found\ncaused by: RequestError: 
send request failed\ncaused by: Get \"http://169.254.169.254/latest/meta-data/iam/security-credentials/\": 
dial tcp 169.254.169.254:80: connect: connection refused"

169.254.169.254:80 is available from Openshift node, but not from Grafana pod/container.

The problem could probably have been solved using the hostNetwork: true:
https://bugzilla.redhat.com/show_bug.cgi?id=1745720
but there is no option to configure it (or configuration is ignored ?).
e.g. I tried to define it as:

spec:
  deployment:
    securityContext:
      hostNetwork: true

and

spec:
  securityContext:
    hostNetwork: true

The deployment, replicaset, pod were re-deployed, but there was no hostNetwork option.
It looks like the securityContext option is ignored.

Expected behavior
Cloudwatch datasource with "Assume Role ARN" works in Openshift.

Runtime (please complete the following information):

  • OS: Linux
  • Grafana Operator Version: 3.9.0
  • Environment: Openshift in AWS

Additional context
The same configuration (IAM role, grafana-operator, without hostNetwork: true) perfectly works in AWS EKS cluster.

@alrf alrf added bug Something isn't working needs triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 22, 2021
@alrf
Copy link
Author

alrf commented Mar 29, 2021

I was able to rebuild operator and fix it.
Added hostNetwork option to deployment configuration.

hostNetwork: true should be used with DNSPolicy: ClusterFirstWithHostNet (pod default DNSPolicy is ClusterFirst)

@alrf alrf mentioned this issue Mar 29, 2021
6 tasks
@HVBE
Copy link
Collaborator

HVBE commented Mar 30, 2021

Closing, fixed in #380, Thanks for contributing @alrf ! 👍

@HVBE HVBE closed this as completed Mar 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
None yet
Development

No branches or pull requests

2 participants