Can't create a Dashboard with Grafana Cloud

[luebken]

Hey folks

I'm getting started with the Grafana Crossplane provider and I am testing it against Grafana Cloud. I was able to create a dashboard with the HTTP API. But when I try to replicate it with Crossplane I get the following error apply failed: the Grafana client is required for grafana_dashboard. Set the auth and url provider attributes:

Most likely, I've configured something wrong. Any pointers would be appreciated.

Working example with curl

Created API key at https://luebken.grafana.net/org/apikeys

curl -X POST --insecure -H "Authorization: Bearer <MY_API_KEY>" -H "Content-Type: application/json" -d '{
  "dashboard": {
    "title": "Test Dashboard"
  }
}' https://luebken.grafana.net/api/dashboards/db
{"id":15,"slug":"test-dashboard","status":"success","uid":"9OEyRFaVk","url":"/d/9OEyRFaVk/test-dashboard","version":1}

Broken example with Crossplane

kind create cluster
helm install crossplane --namespace crossplane-system --create-namespace crossplane-stable/crossplane
kubectl crossplane install provider xpkg.upbound.io/grafana/provider-grafana:v0.3.0

cat grafana/grafana-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: grafana-cloud-creds
  namespace: crossplane-system
type: Opaque
stringData:
  credentials: |
    {
      "cloud_api_key": <MY_API_KEY>
    }
    
kubectl apply -f grafana/grafana-secret.yaml

cat grafana/provider-config.yaml
apiVersion: grafana.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: grafana-cloud-provider
spec:
  credentials:
    source: Secret
    secretRef:
      name: grafana-cloud-creds
      namespace: crossplane-system
      key: credentials
      
kubectl apply -f grafana/provider-config.yaml

kubectl get providers
NAME                       INSTALLED   HEALTHY   PACKAGE                                           AGE
grafana-provider-grafana   True        True      xpkg.upbound.io/grafana/provider-grafana:v0.3.0   13m

cat grafana/dashboard.yaml
apiVersion: oss.grafana.crossplane.io/v1alpha1
kind: Dashboard
metadata:
  name: cloud-stack-dashboard
spec:
  forProvider:
    configJson: |
      {
        "title": "Crossplane Test!"
      }
  providerConfigRef:
    name: grafana-cloud-provider
    
kubectl create -f grafana/dashboard.yaml
dashboard.oss.grafana.crossplane.io/cloud-stack-dashboard created

kubectl get dashboards
NAME                    READY   SYNCED   EXTERNAL-NAME   AGE
cloud-stack-dashboard   False   True                     38s

kubectl describe dashboards cloud-stack-dashboard
Name:         cloud-stack-dashboard
Namespace:    
Labels:       <none>
Annotations:  crossplane.io/external-create-pending: 2023-03-11T12:04:53Z
              crossplane.io/external-create-succeeded: 2023-03-11T12:04:53Z
API Version:  oss.grafana.crossplane.io/v1alpha1
Kind:         Dashboard
Metadata:
  Creation Timestamp:  2023-03-11T12:03:17Z
  Finalizers:
    finalizer.managedresource.crossplane.io
  Generation:  1
  Managed Fields:
    API Version:  oss.grafana.crossplane.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:deletionPolicy:
        f:forProvider:
          .:
          f:configJson:
        f:providerConfigRef:
          .:
          f:name:
    Manager:      kubectl-create
    Operation:    Update
    Time:         2023-03-11T12:03:17Z
    API Version:  oss.grafana.crossplane.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:crossplane.io/external-create-pending:
          f:crossplane.io/external-create-succeeded:
        f:finalizers:
          .:
          v:"finalizer.managedresource.crossplane.io":
    Manager:      provider
    Operation:    Update
    Time:         2023-03-11T12:04:53Z
    API Version:  oss.grafana.crossplane.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:atProvider:
        f:conditions:
    Manager:         provider
    Operation:       Update
    Subresource:     status
    Time:            2023-03-11T12:04:53Z
  Resource Version:  3411
  UID:               53b11290-2c66-4578-b935-42d0e14ebb21
Spec:
  Deletion Policy:  Delete
  For Provider:
    Config Json:  {
  "title": "Crossplane Test!"
}

  Provider Config Ref:
    Name:  grafana-cloud-provider
Status:
  At Provider:
  Conditions:
    Last Transition Time:  2023-03-11T12:03:17Z
    Reason:                Creating
    Status:                False
    Type:                  Ready
    Last Transition Time:  2023-03-11T12:03:17Z
    Reason:                ReconcileSuccess
    Status:                True
    Type:                  Synced
    Last Transition Time:  2023-03-11T12:04:53Z
    Reason:                Finished
    Status:                True
    Type:                  AsyncOperation
    Last Transition Time:  2023-03-11T12:03:17Z
    Message:               apply failed: the Grafana client is required for `grafana_dashboard`. Set the auth and url provider attributes: 
    Reason:                ApplyFailure
    Status:                False
    Type:                  LastAsyncOperation
Events:
  Type    Reason                   Age                 From                                                        Message
  ----    ------                   ----                ----                                                        -------
  Normal  CreatedExternalResource  4s (x4 over 100s)   managed/oss.grafana.crossplane.io/v1alpha1, kind=dashboard  Successfully requested creation of external resource
  Normal  PendingExternalResource  3s (x17 over 100s)  managed/oss.grafana.crossplane.io/v1alpha1, kind=dashboard  Waiting for external resource existence to be confirmed

kubectl logs -n crossplane-system grafana-provider-grafana-2e51ed51dc69-56c8bcb46-6bn2w
(empty)

kubectl get events
LAST SEEN   TYPE      REASON                    OBJECT                                                   MESSAGE
11m         Normal    CreatedExternalResource   dashboard/cloud-stack-dashboard                          Successfully requested creation of external resource
16m         Normal    PendingExternalResource   dashboard/cloud-stack-dashboard                          Waiting for external resource existence to be confirmed

[julienduchesne]

Check out the examples folder. There’s two providers required to do this because it works with two different APIs (cloud API + the instance’s API)

[luebken]

Thanks. I've missed 4.grafana-provider.yaml.

So it's secret seems to be created through 3.api-key-from-stack.yaml which in return needs 2.stack.yaml.

Unfortunatly, I can't create this stack:

apiVersion: cloud.grafana.crossplane.io/v1alpha1
kind: Stack
metadata:
  name: my-stack
spec:
  forProvider:
    slug: 'luebken'
    name: 'luebken'
    regionSlug: "us"
  providerConfigRef:
    name: grafana-cloud-provider

kubectl get stacks
NAME       READY   SYNCED   EXTERNAL-NAME   AGE
my-stack   False   True                     3m58s

kubectl describe stacks my-stack
...
Last Transition Time:  2023-03-11T14:48:05Z
    Message:               apply failed: status: 401, body: {
  "code": "InvalidCredentials",
  "message": "Token invalid",
  "requestId": "70a09e9c-012e-4e4a-b7b0-2f98a9042612"
}:
    Reason:  ApplyFailure
    Status:  False
    Type:    LastAsyncOperation

Note that I'm using the same token as with the curl command mentioned above.

[luebken]

Still working on this. I've now created a key at https://grafana.com/orgs/luebken/api-keys instead of https://luebken.grafana.net/org/apikeys. Now I'm running into "Hosted instance limit reached":

kubectl describe stacks.cloud.grafana.crossplane.io
...
Status:
  At Provider:
  Conditions:
...
    Last Transition Time:  2023-03-18T08:03:52Z
    Message:               apply failed: status: 403, body: {
  "code": "Forbidden",
  "message": "Hosted instance limit reached",
  "requestId": "1353fb29-c58e-4446-8630-001bcf93af88"
}:

[luebken]

Next step. I've created a Grafana Pro account now I get an That url is not available error:

kubectl describe stacks.cloud.grafana.crossplane.io
...
Status:
  At Provider:
  Conditions:
...
    Last Transition Time:  2023-03-18T08:20:05Z
    Message:               apply failed: status: 409, body: {
  "code": "Conflict",
  "message": "That url is not available",
  "requestId": "45bcefc6-f2d8-4ab5-a536-ef61eb6fd675"
}:
}:

[luebken]

I got it running. The last error is because I've tried to create a stack with the same name.

[julienduchesne]

Awesome, sorry for the lack of help here 😬 . I had to focus on something else last week unfortunately