-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Prepare next release * Docs: Monitoring * K8s: v1apps.list_namespaced_deployment (new K8s version in cluster) * K8s: Resultfolder as attribute of cluster object * Masterscript: K8s store number of requested gpus * Masterscript: K8s run shell scripts for loading data * Masterscript: Allow list of jars per dbms * Requirements: Allow current versions * TPC-H: New specifics * Masterscript: K8s monitoring as option * Masterscript: K8s optional UPPER parameter of db and schema * Masterscript: K8s DEPRECATED: we must know all jars upfront * TPC-H: DDL for MariaDB Columnstore * TPC-H: Bigint at MonetDB * TPC-H: SQL Server precision and DB * TPC-H: OmniSci sharding * TPC-H: OmniSci template
- Loading branch information
Showing
14 changed files
with
1,136 additions
and
60 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| # Monitoring | ||
|
|
||
| To include monitoring you will need | ||
| * a Prometheus server scraping a fixed IP / Port | ||
| * a Grafana server collecting metrics from the Prometheus server | ||
| * some [configuration](#configuration) what metrics to collect | ||
|
|
||
| This document contains information about the | ||
| * [Concept](#concept) | ||
| * [Installation](#installation) | ||
| * [Configuration](#configuration) | ||
|
|
||
| ## Concept | ||
|
|
||
| <p align="center"> | ||
| <img src="architecture.png" width="640"> | ||
| </p> | ||
|
|
||
| There is | ||
| * an Experiment Host - this needs Prometheus exporters | ||
| * a Monitor - this needs a Prometheus server and a Grafana server scraping the Experiment Host | ||
| * a Manager - this needs a configuration (which metrics to collect and where from) | ||
|
|
||
| ## Installation | ||
|
|
||
| To be documented | ||
|
|
||
| ### Kubernetes | ||
|
|
||
| * Experiment Host: Exporters are part of the [deployments](Deployments.md) | ||
| * Monitor: Servers are deployed using Docker images, fixed on a separate monitoring instance | ||
| * Manager: See [configuration](#configuration) | ||
|
|
||
| ### AWS | ||
|
|
||
| * Experiment Host: Exporters are deployed using Docker images, fixed on the benchmarked instance | ||
| * Monitor: Servers are deployed using Docker images, fixed on a separate monitoring instance | ||
| * Manager: See [configuration](#configuration) | ||
|
|
||
| ## Configuration | ||
|
|
||
| We insert information about | ||
| * the Grafana server | ||
| * access token | ||
| * URL | ||
| * the collection | ||
| * extension of measure intervals | ||
| * time shift | ||
| * metrics definitions | ||
|
|
||
| into the cluster configuration. | ||
| This is handed over to the [DBMS configuration](https://github.com/Beuth-Erdelt/DBMS-Benchmarker/blob/master/docs/Options.md#connection-file) of the [benchmarker](https://github.com/Beuth-Erdelt/DBMS-Benchmarker/blob/master/docs/Concept.md#monitoring-hardware-metrics). | ||
|
|
||
| ### Example | ||
|
|
||
| The details of the metrics correspond to the YAML configuration of the [deployments](Deployments.md): | ||
| * `job="monitor-node"` | ||
| * `container_name="dbms"` | ||
|
|
||
|
|
||
| ``` | ||
| 'monitor': { | ||
| 'grafanatoken': 'Bearer ABCDE==', | ||
| 'grafanaurl': 'http://localhost:3000/api/datasources/proxy/1/api/v1/', | ||
| 'grafanaextend': 20, | ||
| 'grafanashift': 0, | ||
| 'metrics': { | ||
| 'total_cpu_memory': { | ||
| 'query': 'container_memory_working_set_bytes{{job="monitor-node", container_label_io_kubernetes_container_name="dbms"}}', | ||
| 'title': 'CPU Memory [MiB]' | ||
| }, | ||
| 'total_cpu_memory_cached': { | ||
| 'query': 'container_memory_usage_bytes{{job="monitor-node", container_label_io_kubernetes_container_name="dbms"}}', | ||
| 'title': 'CPU Memory Cached [MiB]' | ||
| }, | ||
| 'total_cpu_util': { | ||
| 'query': 'sum(irate(container_cpu_usage_seconds_total{{job="monitor-node", container_label_io_kubernetes_container_name="dbms"}}[1m]))', | ||
| 'title': 'CPU Util [%]' | ||
| }, | ||
| 'total_cpu_throttled': { | ||
| 'query': 'sum(irate(container_cpu_cfs_throttled_seconds_total{{job="monitor-node", container_label_io_kubernetes_container_name="dbms"}}[1m]))', | ||
| 'title': 'CPU Throttle [%]' | ||
| }, | ||
| 'total_cpu_util_others': { | ||
| 'query': 'sum(irate(container_cpu_usage_seconds_total{{job="monitor-node", container_label_io_kubernetes_container_name!="dbms",id!="/"}}[1m]))', | ||
| 'title': 'CPU Util Others [%]' | ||
| }, | ||
| 'total_cpu_util_s': { | ||
| 'query': 'sum(container_cpu_usage_seconds_total{{job="monitor-node", container_label_io_kubernetes_container_name="dbms"}})', | ||
| 'title': 'CPU Util [s]' | ||
| }, | ||
| 'total_cpu_throttled_s': { | ||
| 'query': 'sum(container_cpu_cfs_throttled_seconds_total{{job="monitor-node", container_label_io_kubernetes_container_name="dbms"}})', | ||
| 'title': 'CPU Throttle [s]' | ||
| }, | ||
| 'total_cpu_util_others_s': { | ||
| 'query': 'sum(container_cpu_usage_seconds_total{{job="monitor-node", container_label_io_kubernetes_container_name!="dbms",id!="/"}})', | ||
| 'title': 'CPU Util Others [s]' | ||
| }, | ||
| 'total_network_rx': { | ||
| 'query': 'sum(container_network_receive_bytes_total{{container_label_app="dbmsbenchmarker", job="monitor-node"}})', | ||
| 'title': 'Net Rx [b]' | ||
| }, | ||
| 'total_network_tx': { | ||
| 'query': 'sum(container_network_transmit_bytes_total{{container_label_app="dbmsbenchmarker", job="monitor-node"}})', | ||
| 'title': 'Net Tx [b]' | ||
| }, | ||
| 'total_fs_read': { | ||
| 'query': 'sum(container_fs_reads_bytes_total{{job="monitor-node", container_label_io_kubernetes_container_name="dbms"}})', | ||
| 'title': 'FS Read [b]' | ||
| }, | ||
| 'total_fs_write': { | ||
| 'query': 'sum(container_fs_writes_bytes_total{{job="monitor-node", container_label_io_kubernetes_container_name="dbms"}})', | ||
| 'title': 'FS Write [b]' | ||
| }, | ||
| 'total_gpu_util': { | ||
| 'query': 'sum(DCGM_FI_DEV_GPU_UTIL{{UUID=~"{gpuid}"}})', | ||
| 'title': 'GPU Util [%]' | ||
| }, | ||
| 'total_gpu_power': { | ||
| 'query': 'sum(DCGM_FI_DEV_POWER_USAGE{{UUID=~"{gpuid}"}})', | ||
| 'title': 'GPU Power Usage [W]' | ||
| }, | ||
| 'total_gpu_memory': { | ||
| 'query': 'sum(DCGM_FI_DEV_FB_USED{{UUID=~"{gpuid}"}})', | ||
| 'title': 'GPU Memory [MiB]' | ||
| }, | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| #### Fine Tuning | ||
|
|
||
| If the Grafana server has metrics coming from general Prometheus server, that is it scrapes more exporters than just the bexhoma related, we will need to specify further which metrics we are interested in. | ||
|
|
||
| There is a placeholder `{gpuid}` that is substituted automatically by a list of GPUs present in the pod. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| cpimport tpch nation /data/tpch/SF1/nation.tbl | ||
| cpimport tpch region /data/tpch/SF1/region.tbl | ||
| cpimport tpch part /data/tpch/SF1/part.tbl | ||
| cpimport tpch supplier /data/tpch/SF1/supplier.tbl | ||
| cpimport tpch partsupp /data/tpch/SF1/partsupp.tbl | ||
| cpimport tpch customer /data/tpch/SF1/customer.tbl | ||
| cpimport tpch orders /data/tpch/SF1/orders.tbl | ||
| cpimport tpch lineitem /data/tpch/SF1/lineitem.tbl |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| CREATE DATABASE tpch; | ||
| CREATE USER 'benchmark'@'%'; | ||
| GRANT ALL PRIVILEGES ON *.* TO 'benchmark'@'%' WITH GRANT OPTION; | ||
|
|
||
| -- sccsid: @(#)dss.ddl 2.1.8.1 | ||
| create table tpch.nation ( n_nationkey integer not null, | ||
| n_name char(25) not null, | ||
| n_regionkey integer not null, | ||
| n_comment varchar(152)) engine=columnstore; | ||
|
|
||
| create table tpch.region ( r_regionkey integer not null, | ||
| r_name char(25) not null, | ||
| r_comment varchar(152)) engine=columnstore; | ||
|
|
||
| create table tpch.part ( p_partkey integer not null, | ||
| p_name varchar(55) not null, | ||
| p_mfgr char(25) not null, | ||
| p_brand char(10) not null, | ||
| p_type varchar(25) not null, | ||
| p_size integer not null, | ||
| p_container char(10) not null, | ||
| p_retailprice decimal(15,2) not null, | ||
| p_comment varchar(23) not null ) engine=columnstore; | ||
|
|
||
| create table tpch.supplier ( s_suppkey integer not null, | ||
| s_name char(25) not null, | ||
| s_address varchar(40) not null, | ||
| s_nationkey integer not null, | ||
| s_phone char(15) not null, | ||
| s_acctbal decimal(15,2) not null, | ||
| s_comment varchar(101) not null) engine=columnstore; | ||
|
|
||
| create table tpch.partsupp ( ps_partkey integer not null, | ||
| ps_suppkey integer not null, | ||
| ps_availqty integer not null, | ||
| ps_supplycost decimal(15,2) not null, | ||
| ps_comment varchar(199) not null ) engine=columnstore; | ||
|
|
||
| create table tpch.customer ( c_custkey integer not null, | ||
| c_name varchar(25) not null, | ||
| c_address varchar(40) not null, | ||
| c_nationkey integer not null, | ||
| c_phone char(15) not null, | ||
| c_acctbal decimal(15,2) not null, | ||
| c_mktsegment char(10) not null, | ||
| c_comment varchar(117) not null) engine=columnstore; | ||
|
|
||
| create table tpch.orders ( o_orderkey integer not null, | ||
| o_custkey integer not null, | ||
| o_orderstatus char(1) not null, | ||
| o_totalprice decimal(15,2) not null, | ||
| o_orderdate date not null, | ||
| o_orderpriority char(15) not null, | ||
| o_clerk char(15) not null, | ||
| o_shippriority integer not null, | ||
| o_comment varchar(79) not null) engine=columnstore; | ||
|
|
||
| create table tpch.lineitem ( l_orderkey integer not null, | ||
| l_partkey integer not null, | ||
| l_suppkey integer not null, | ||
| l_linenumber integer not null, | ||
| l_quantity decimal(15,2) not null, | ||
| l_extendedprice decimal(15,2) not null, | ||
| l_discount decimal(15,2) not null, | ||
| l_tax decimal(15,2) not null, | ||
| l_returnflag char(1) not null, | ||
| l_linestatus char(1) not null, | ||
| l_shipdate date not null, | ||
| l_commitdate date not null, | ||
| l_receiptdate date not null, | ||
| l_shipinstruct char(25) not null, | ||
| l_shipmode char(10) not null, | ||
| l_comment varchar(44) not null) engine=columnstore; | ||
|
|
Oops, something went wrong.