kube-prometheus: need to add Toleration to Prometheus

[jamroks]

Hi,

I'am trying to add a toleration on prometheus, to make sure it's scheduled on specific nodes on the Kubernetes cluster. Those nodes are specifically provision For Prometheus hence the need for "Toleration" and "NodeAffinity" because those nodes have been tainted.

local k  = import 'ksonnet/ksonnet.beta.3/k.libsonnet';

local pvc = k.core.v1.persistentVolumeClaim;
local toleration = k.core.v1.tolerationsType;

local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') + 
           (import 'kube-prometheus/kube-prometheus-kops.libsonnet') + {
  _config+:: {
    namespace: 'monitoring',
    prometheus+:: {
      namespaces+: ['dev', 'int', 'uat'],
      replicas+: 1,
      prometheus+: {
         retention: "30d",
         storage: {
           volumeClaimTemplate:
              pvc.new() +
              pvc.mixin.spec.withAccessModes('ReadWriteOnce') +
              pvc.mixin.spec.resources.withRequests({ storage: '50Gi' }) +
              pvc.mixin.spec.withStorageClassName('gp2'),
         },
         tolerations:
              toleration.new() +
              toleration.withKey('') +
              toleration.withValue('') +
              toleration.withEffect('')

      }
    },
  },
};

I'm not really sure I'm doing the right thing here as I'm new to Jsonnet and all that Jazz . but I'm looking forward to learn Jsonnet syntaxe and concept & use it more and more in my project. How would you go about defining Toleration and Affinity in this contexte ?

cheers

[metalmatze]

The antiaffinity should be a good example on how to add these tolerations:
https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/jsonnet/kube-prometheus/kube-prometheus-anti-affinity.libsonnet

The PrometheusSpec allows to set the tolerations directly (same for AlertmanagerSpec):
https://github.com/coreos/prometheus-operator/blob/a7eb782ea174af819bd82c72be54946ed83bd7b3/pkg/client/monitoring/v1/types.go#L138

Instead of adding the tolerations to the config you have to merge them with the real Prometheus and Alertmanager objects.

Maybe we could also make this an example like antiAffinity, @brancz?!

[brancz]

@metalmatze I think that's a good idea. Either of you want to create a PR and add a mixin?

[jamroks]

Thanks a lot @metalmatze ...that most definitely gave me a direction on how to proceed, even if I don't know golang, reading the code helped !

@brancz thanks i will try to put something together and submit a PR for it ...but please bare with me here it's gonna be my first one ever, I let you know as soon as I have something

[metalmatze]

Very cool. 👍
Don't hesitate to ask in the Kubernetes Slack in #monitoring-mixins, if you have questions about writing the example.

[jamroks]

wow ...there's a channel for that !? that's dope !! will do.
by the way i have something working as i wanted..now i'm just gonna craft that mixin thing and a followed up PR

Thanks guys

Cheers

[brancz]

We're always very happy to assist new-comers, please feel free to ask any question 🙂 .

[hamid2013]

Hello Guys,

I am trying to increase the retention and use the pvc (persistent volume) for prometheus. And we do have example for it. When i copy and pasted that on the jsonnet/kube-prometheus folder and running the build.sh its giving the error:

+ rm -rf manifests
+ mkdir manifests
+ jsonnet -J vendor -m manifests example.jsonnet
+ xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml; rm -f {}' -- '{}'
RUNTIME ERROR: couldn't open import "kube-prometheus/kube-prometheus.libsonnet": no match locally or in the Jsonnet library paths
        example.jsonnet:2:4-54  thunk <kp> from <$>
        example.jsonnet:8:81-83 thunk from <$>
        <std>:1176:24-25        thunk from <function <anonymous>>
        <std>:1176:5-33 function <anonymous>
        example.jsonnet:8:64-99 $


        During evaluation```

Any idea how to resolve that. 

[metalmatze]

You need to have the jsonnetfile.json in your same directory and then run jb install (jb being the jsonnet-bundler) to fetch all necessary dependencies to build kube-prometheus.

[hamid2013]

@metalmatze : Thanks for the prompt reply.
To achieve that, i have copied the examples/prometheus-pvc.jsonnet to jsonnet/kube-prometheus and run the below commands:

1. jb install
2. ./build.sh

Now from where i can confirm that the retention and pvc has been added in the manifests so that i can apply on the cluster.

Apology, but new to jsonnet and kube-prometheus both.

[metalmatze]

No worries. Glad it works now!

[hamid2013]

@metalmatze: I have not applied in the cluster, just wanted to know where can i confirm the retention period and pvc configuration has been modified before i apply.

Also if we have any slack channel for this kind of issue, can you please provide the URL to join that.

[metalmatze]

Yes. We have. There's #prometheus-operator in the Kubernetes Slack. See you there! :)

[hamid2013]

Can you please provide the link, while searching the workspace kubernetes.slack.com, its asked to login and i am using my github registered email to do that or any other way to login to the kubernetes slack

[hamid2013]

Please ignore i was able to join the slack now

[stale]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.