Bump kube-prometheus-stack for multi-cluster support #162
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,130 @@ | ||
| # Tobs helm values config | ||
|
|
||
| ## Prometheus High-Availability | ||
|
|
||
| **Note**: This is unnecessary if using the tobs CLI, To enable Prometheus high-availability with tobs CLI use `--enable-prometheus-ha`. | ||
|
|
||
| The following steps will explain how to enable Prometheus high-availability with Promscale when using tobs helm chart (without tobs CLI). | ||
|
|
||
| Update the tobs `values.yaml` with below HA configuration. | ||
|
|
||
| Increase the TimescaleDB connection pool i.e. | ||
|
|
||
| ``` | ||
| timescaledb-single: | ||
| patroni: | ||
| bootstrap: | ||
| dcs: | ||
| postgresql: | ||
| parameters: | ||
| max_connections: 400 | ||
| ``` | ||
|
|
||
| Update the Promscale configuration to enable HA mode and increase the replicas to 3: | ||
|
|
||
| ``` | ||
| promscale: | ||
| replicaCount: 3 | ||
| args: | ||
| - --high-availability | ||
| ``` | ||
|
|
||
| Update Prometheus configuration to send prometheus pod name with `__replica__` and prometheus cluster name as `cluster` labelSets in the form of external labels and run Prometheus as 3 replicas for HA. | ||
|
|
||
| ``` | ||
| kube-prometheus-stack: | ||
| prometheus: | ||
| prometheusSpec: | ||
| replicaExternalLabelName: "__replica__" | ||
| prometheusExternalLabelName: "cluster" | ||
| replicas: 3 | ||
| ``` | ||
|
|
||
| ## Multi-cluster support | ||
|
There was a problem hiding this comment. can we have a diagram for this section showing the various clusters etc. |
||
|
|
||
| <img src="./../../docs/assets/multi-cluster.png" alt="multi-cluster setup diagram" width="800"/> | ||
|
There was a problem hiding this comment. I believe you forgot to add this image in the PR? There was a problem hiding this comment. Added it to the PR here is the file: https://github.com/timescale/tobs/pull/162/files#diff-1a7cd77999e454c5a336fcb163eb5108d70057981aa40d648c619671eba20b3e |
||
|
|
||
| In tobs you can enable multi-cluster support to install a data aggregation cluster to collect observability data coming from different observer clusters. | ||
|
|
||
| With tobs you can deploy both observer clusters and data aggregation cluster. | ||
|
|
||
| ### Deploying the data aggregation cluster | ||
|
|
||
| The data aggregation cluster acts as central observability cluster which stores, visualizes data flowing from observer clusters. | ||
|
|
||
| Steps to deploy data aggregation cluster | ||
|
|
||
| * Enable LoadBalancer service to Promscale to remote-wrtie, remote-read data from observer clusters. | ||
|
|
||
| ``` | ||
| promscale: | ||
| service: | ||
| loadBalancer: | ||
| enabled: true | ||
| ``` | ||
|
|
||
| * Add external cluster label to differentiate the visualisation in Grafana dashboards per cluster level. | ||
|
|
||
| ``` | ||
| kube-prometheus-stack: | ||
| prometheus: | ||
| prometheusSpec: | ||
| externalLabels: | ||
| cluster: <clusterName> | ||
| ``` | ||
|
|
||
| * Enable multi-cluster support in Grafana dashboards | ||
|
|
||
| ``` | ||
| kube-prometheus-stack: | ||
| grafana: | ||
| sidecar: | ||
| dashboards: | ||
| multicluster: | ||
| global: | ||
| enabled: true | ||
| ``` | ||
|
|
||
| ### Deploying the observer cluster | ||
|
|
||
| The observer cluster forwards the metrics to data-aggregation/centralised monitoring cluster which supports ingesting, visualising of metrics flowing from different observer clusters. | ||
|
|
||
| Steps to install an observer cluster | ||
|
|
||
| * Disable TimescaleDB | ||
|
|
||
| ``` | ||
| timescaledb-single: | ||
| enabled: false | ||
| ``` | ||
|
|
||
| * Disable Promscale | ||
|
|
||
| ``` | ||
| promscale: | ||
| enabled: false | ||
| ``` | ||
|
|
||
|
|
||
| * Disable Grafana | ||
|
|
||
| ``` | ||
| kube-prometheus-stack: | ||
| grafana: | ||
| enabled: false | ||
| ``` | ||
|
|
||
| * Configure Prometheus remote-write to Promscale loadbalancer service in data aggregation cluster and add the external label representing the current cluster name to differentiate the visualisation on per cluster basis in Grafana. | ||
|
|
||
| ``` | ||
| kube-prometheus-stack: | ||
| prometheus: | ||
| prometheusSpec: | ||
| externalLabels: | ||
| cluster: <clusterName> | ||
| remoteRead: | ||
| - url: "<PROMSCALE_SERVICE_ENDPOINT_OF_DATA_AGGREGATION_CLUSTER>/read" | ||
| readRecent: true | ||
| remoteWrite: | ||
| - url: "<PROMSCALE_ENDPOINT>/write" | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| package dependency_tests | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "io/ioutil" | ||
| "log" | ||
| "net/http" | ||
| "testing" | ||
|
|
||
| "github.com/timescale/tobs/cli/cmd/upgrade" | ||
| "github.com/timescale/tobs/cli/pkg/helm" | ||
| "sigs.k8s.io/yaml" | ||
| ) | ||
|
|
||
| func TestMain(m *testing.M) { | ||
| validateKubePrometheusVersions() | ||
| } | ||
|
|
||
| func validateKubePrometheusVersions() { | ||
| // Get existing tobs helm chart metadata | ||
| b, err := ioutil.ReadFile("./../../../chart/Chart.yaml") | ||
| if err != nil { | ||
| log.Fatal(err) | ||
| } | ||
| existingTobsChart := &helm.ChartMetadata{} | ||
| err = yaml.Unmarshal(b, existingTobsChart) | ||
| if err != nil { | ||
| log.Fatal(err) | ||
| } | ||
| var kubePrometheusVersion string | ||
| for _, i := range existingTobsChart.Dependencies { | ||
| if i.Name == "kube-prometheus-stack" { | ||
| kubePrometheusVersion = i.Version | ||
| break | ||
| } | ||
| } | ||
|
|
||
| // Get upstream kube-prometheus chart metadata using kube-prometheus version used in tobs local Chart.yaml | ||
| resp, err := http.Get("https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-" + kubePrometheusVersion + "/charts/kube-prometheus-stack/Chart.yaml") | ||
| if err != nil { | ||
| log.Fatalf("failed to get the kube-prometheus CHart.yaml info %v", err) | ||
| } | ||
| defer resp.Body.Close() | ||
|
|
||
| bodyBytes, err := ioutil.ReadAll(resp.Body) | ||
| if err != nil { | ||
| log.Fatal(err) | ||
| } | ||
|
|
||
| upstreamKPChart := &helm.ChartMetadata{} | ||
| err = yaml.Unmarshal(bodyBytes, upstreamKPChart) | ||
| if err != nil { | ||
| log.Fatal(err) | ||
| } | ||
|
|
||
| upstreamKPChart.AppVersion = "v" + upstreamKPChart.AppVersion | ||
| // validate existing tobs kube-prometheus helm chart version & CRDs version with upstream version and CRDs that are being used | ||
| if upstreamKPChart.Version != kubePrometheusVersion || upstreamKPChart.AppVersion != upgrade.KubePrometheusCRDVersion { | ||
| log.Fatalf("failed to validate tobs kube-prometheus helm chart version and CRDs version with upstream versions."+ | ||
| "Mismatch in validation, tobs Kube-Prometheus version: %s, tobs kube-prometheus CRD version: %s, "+ | ||
| "upstream kube-prometheus CRD version: %s", kubePrometheusVersion, upgrade.KubePrometheusCRDVersion, upstreamKPChart.AppVersion) | ||
| } | ||
| fmt.Println("successfully validated kube-prometheus CRD versions with upstream versions.") | ||
| } |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm. I think there are two different types of config here.
Type 1 is ok here. I'd move type 2 stuff into a new top-level /docs directory. It does not belong under /chart and I'd add new links from the front-page readme here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So currently using CLI we offer only abstract features with flags like
HA,integration with forge, etc... all these features can also be configured withinvalues.yaml. All the flexibility lies in thevalues.yamlso the intention behind creating this file is to document all possible configurations across different components in tobs.Yes, I agree with you only config related to
values.yamlwill reside in this file. Anything about CLI will be in a new file in the high level root directory of this repo.