Skip to content

Commit

Permalink
first round of content updates
Browse files Browse the repository at this point in the history
  • Loading branch information
evgeni committed Aug 31, 2023
1 parent fad1e88 commit 5532c86
Show file tree
Hide file tree
Showing 14 changed files with 98 additions and 175 deletions.
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ The storage capacity required by PCP data logging is determined by the following

The default logging (sampling) interval is 60 seconds.

The default retention policy is to keep archives for the last 14 days, compressing archives older than one day. PCP archive logs are stored in the `/var/log/pcp/pmlogger/_hostname_` directory.
The default retention policy is to keep archives for the last 14 days, compressing archives older than one day. PCP archive logs are stored in the `/var/log/pcp/pmlogger/_{foreman-example-com}_` directory.
15 changes: 11 additions & 4 deletions guides/doc-Monitoring_Project/topics/con_pcp-metrics.adoc
Original file line number Diff line number Diff line change
@@ -1,11 +1,18 @@
[id='pcp-metrics_{context}']
= PCP Metrics

Metrics are stored in a tree-like structure. For example, all network metrics are stored in a node named `network`. Each metric may be a single value, or a list of values, known as instances. For example, kernel load has three instances, a 1-minute, 5-minute, and 15-minute average.
Metrics are stored in a tree-like structure.
For example, all network metrics are stored in a node named `network`.
Each metric may be a single value, or a list of values, known as instances.
For example, kernel load has three instances, a 1-minute, 5-minute, and 15-minute average.

For every metric entry, PCP stores both its data and metadata. This includes the metrics description, data type, units, and dimensions. For example, the metadata enables PCP to output multiple metrics with different dimensions.
For every metric entry, PCP stores both its data and metadata.
This includes the metrics description, data type, units, and dimensions.
For example, the metadata enables PCP to output multiple metrics with different dimensions.

The value of a counter metric only increases. For example, a count of disk write operations on a specific device only increases. When you query the value of a counter metric, PCP converts this into a rate value by default.
The value of a counter metric only increases.
For example, a count of disk write operations on a specific device only increases.
When you query the value of a counter metric, PCP converts this into a rate value by default.

In addition to system metrics such as CPU, memory, kernel, XFS, disk, and network, the following metrics are configured:

Expand All @@ -23,6 +30,6 @@ In addition to system metrics such as CPU, memory, kernel, XFS, disk, and networ
|postgresql.*
|Basic PostgreSQL statistics

|mmv.fm_rails_*
|openmetrics.foreman.fm_rails_*
|{Project} metrics
|===
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
[id='project-metrics-overview_{context}']
= Overview

Obtaining metrics from {Project} is useful for troubleshooting a current issue, and capacity planning. This guide describes how to collect live metrics and archive them for a fixed period of time. If you need to raise a support case with Red{nbsp}Hat to resolve a performance issue, the archived data provides valuable insight. Note that Red Hat Support can only access the archived data if you upload it to a Support Case.
Obtaining metrics from {Project} is useful for troubleshooting a current issue, and capacity planning.
This guide describes how to collect live metrics and archive them for a fixed period of time.
ifdef::satellite[]
If you need to raise a support case with Red{nbsp}Hat to resolve a performance issue, the archived data provides valuable insight.
Note that Red Hat Support can only access the archived data if you upload it to a Support Case.
endif::[]

You can collect the following metrics from {Project}:

* Basic statistics from Red{nbsp}Hat Enterprise Linux, including system load, memory utilization, and input/output operations;
* Basic statistics from the operating system, including system load, memory utilization, and input/output operations;
* Process statistics, including memory and CPU utilization;
* Apache HTTP Server activity statistics;
* PostgreSQL activity statistics;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
[id='retrieving-metrics_{context}']
= Retrieving Metrics

You can retrieve metrics from PCP using the CLI or the web UI interfaces. A number of CLI tools are provided with PCP, which can either output live data, or data from archived sources. The web UI interfaces are provided by the Grafana and Vector web applications. Vector connects directly to the PCP daemon, and can only display live data. Grafana reads from PCP archive files and can display data to up to 1 year old.
You can retrieve metrics from PCP using the CLI or the web UI interface.
A number of CLI tools are provided with PCP, which can either output live data, or data from archived sources.
ifndef::foreman-deb[]
The web UI interface is provided by the Grafana web application.
endif::[]
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,20 @@ This procedure describes how to change the data retention policy.

.Procedure

. Edit the `/etc/cron.d/pcp-pmlogger` file.
ifndef::foreman-deb[]
. Edit the `/etc/sysconfig/pmlogger_timers` file.
endif::[]
ifdef::foreman-deb[]
. Edit the `/etc/default/pmlogger_timers` file.
endif::[]

. Find the line containing `pmlogger_daily`.
. Find the line containing `PMLOGGER_DAILY_PARAMS`.

. Change the value for parameter `-x` to the desired number of days after which data is archived.
. Uncomment the line, if it is commented out.

. Ensure the default parameter `-E` is present.

. Add parameter `-x`, and add a value for the desired number of days after which data is archived.

. Add parameter `-k`, and add a value for the number of days after which data is deleted.
+
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,35 +11,31 @@ This procedure describes how to configure PCP to collect metrics about processes
#pmdahotproc
Version 1.0
# processes with load 0.9 or 1GB RSS memory
(cpuburn > 0.9 || residentsize > 1000000) &&
(
fname == "java" ||
fname == "qdrouterd" ||
fname == "qpidd" ||
fname == "redis-server" ||
(fname == "postgres" && psargs ~ /-D/) ||
fname == "mongod" ||
fname == "dynflow_executor" ||
fname == "dynflow_executor_monitor" ||
fname ~ /smart_proxy_dynflow_core/ ||
psargs ~ /Passenger RackApp/ ||
psargs ~ /celery beat/ ||
psargs ~ /celery worker/ ||
psargs ~ /pulp/ ||
psargs ~ /smart-proxy/ ||
psargs ~ /squid.conf/
psargs ~ /puma/ ||
ifndef::foreman-el,foreman-deb[]
psargs ~ /pulpcore.app.wsgi:application/ ||
psargs ~ /pulpcore.content:server/ ||
psargs ~ /pulpcore-worker/ ||
endif::[]
psargs ~ /sidekiq/ ||
psargs ~ /smart-proxy/
)
----
+
By default, PCP collects basic system metrics. This step enables detailed metrics about the following {Project} processes:
+
* Java
* PostgreSQL
* MongoDB
* Redis
* Dynflow
* Passenger
* Pulp
* Qpid
* Puma
ifndef::foreman-el,foreman-deb[]
* Pulpcore
endif::[]

. Configure PCP to log the process metrics being collected.
+
Expand Down Expand Up @@ -76,130 +72,43 @@ EOF
+
.. Enable the Apache HTTP Server extended status module.
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
#cat >/etc/httpd/conf.d/01-status.conf <<EOF
ExtendedStatus On
LoadModule status_module modules/mod_status.so
<Location "/server-status">
PassengerEnabled off
SetHandler server-status
Order deny,allow
Deny from all
Allow from localhost
</Location>
EOF
# {foreman-installer} --enable-apache-mod-status
----
.. Enable the Apache HTTP Server PMDA.
+
----
# cd /var/lib/pcp/pmdas/apache
# ./Install
----
.. Prevent the {Project} installer overwriting the extended status module’s configuration file.
+
Add the following line to the `/etc/foreman-installer/custom-hiera.yaml` configuration file.
+
----
apache::purge_configs: false
----

. Configure PCP to collect metrics from PostgreSQL.

.. Change to the `/var/lib/pcp/pmdas/postgresql` directory.
+
-----
# cd /var/lib/pcp/pmdas/postgresql
-----

.. Run the installer.
+
----
# cd /var/lib/pcp/pmdas/postgresql
# ./Install
----

.. Configure the PCP database interface to permit access to the PostgreSQL database.
+
Edit the `/etc/pcpdbi.conf` configuration file, inserting the following lines:
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
$database = "dbi:Pg:dbname=foreman;host=localhost";
$username = "foreman";
$password = "_6qXfN9m5nii5iEcbz8nuiJBNsyjjdRHA_"; <1>
$os_user = "foreman";
----
+
<1> The value for _$password_ is stored in `/etc/foreman/database.yml` configuration file.

.. Change the SELinux `pcp_pmcd_t` domain permission to permit PCP access to the PostgreSQL database.
+
----
# semanage permissive -a pcp_pmcd_t
----

.. Verify the PostgreSQL PMDA is able to connect to PostgreSQL.
+
Examine the `/var/log/pcp/pmcd/postgresql.log` file to confirm the connection is established. Without a successful database connection, the PostgreSQL PMDA will remain active, but not be able to provide any metrics.
+
----
[Tue Aug 14 09:21:06] pmdapostgresql(25056) Info: PostgreSQL connection established
----
+
If you find errors in `/var/log/pcp/pmcd/postgresql.log`, restart the *pmcd* service.
+
----
# systemctl restart pmcd
----

. Enable telemetry functionality in {Project}.
+
To enable collection of metrics from {Project}, you must send metrics via the `statsd` protocol into the `pcp-mmvstatsd` daemon. The metrics are aggregated and available via the PCP MMV API.

.. Install the Foreman Telemetry and `pcp-mmvstatsd` packages.
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
# {package-install-project} foreman-telemetry pcp-mmvstatsd
----

.. Enable and start the `pcp-mmvstatsd` service.
+
----
# systemctl enable pcp-mmvstatsd
# systemctl start pcp-mmvstatsd
# {foreman-installer} --foreman-telemetry-prometheus-enabled true
----

.. Enable the {Project} telemetry functionality.
+
Add the following lines to `/etc/foreman/settings.yaml` configuration file:
+
----
:telemetry:
:prefix: 'fm_rails'
:statsd:
:enabled: true
:host: '127.0.0.1:8125'
:protocol: 'statsd'
:prometheus:
:enabled: false
:logger:
:enabled: false
:level: 'INFO'
----

. Schedule daily storage of metrics in archive files:
. Configure PCP to collect data from {Project}.
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
# cat >/etc/cron.daily/refresh_mmv <<EOF
#!/bin/bash
echo "log mandatory on 1 minute mmv" | /usr/bin/pmlc -P
EOF
# chmod +x /etc/cron.daily/refresh_mmv
# cd /var/lib/pcp/pmdas/openmetrics
# echo "https://_{foreman-example-com}_/metrics" > config.d/foreman.url
# ./Install
----

. Restart the Apache HTTP Server and PCP to begin data collection:
. Restart PCP to begin data collection:
+
----
# systemctl restart httpd pmcd pmlogger
# systemctl restart pmcd pmlogger
----
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,14 @@

To confirm data storage usage, enter the following command:

[options="nowrap", subs="verbatim,quotes,attributes"]
----
# less /var/log/pcp/pmlogger/$(hostname)/pmlogger.log
# less /var/log/pcp/pmlogger/_{foreman-example-com}_/pmlogger.log
----

This lists all available metrics, grouped by the frequency at which they are logged. For each group it also lists the storage required to store the listed metrics, per day.

.Example storage statistics
----
logged every 60 sec: 61752 bytes or 84.80 Mbytes/day
----
----
Original file line number Diff line number Diff line change
Expand Up @@ -5,29 +5,22 @@ This procedure describes how to access metrics collected by PCP, via the web UI.

.Procedure

. Enable the Red{nbsp}Hat Enterprise Linux `optional` repository:
+
----
# subscription-manager repos --enable rhel-7-server-optional-rpms
----

. Install the PCP web API and applications:
. Install Grafana and the Grafana PCP plugin:
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
# {package-install-project} pcp-webapi pcp-webapp-grafana pcp-webapp-vector
# {package-install-project} grafana grafana-pcp
----

. Start and enable the PCP web service:
. Start and enable the Grafana web service and the PCP proxy service:
+
----
# systemctl start pmwebd
# systemctl enable pmwebd
# systemctl enable --now pmproxy grafana-server
----

. Open firewall port to allow access to the PCP web service:
. Open firewall port to allow access to the Grafana web interface:
+
----
# firewall-cmd --add-port=44323/tcp
# firewall-cmd --permanent --add-port=44323/tcp
# firewall-cmd --add-port=3000/tcp
# firewall-cmd --permanent --add-port=3000/tcp
----
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@

* To list the archived metrics, enter the following command:
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
# less /var/log/pcp/pmlogger/$(hostname)/pmlogger.log
# less /var/log/pcp/pmlogger/_{foreman-example-com}_/pmlogger.log
----

* The pmlogger daemon archives data as it is received, according to its configuration. To confirm the active archive file, enter the following command:
Expand All @@ -29,5 +30,5 @@ The output includes the file name of the active archive file, for example:
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
/var/log/pcp/pmlogger/{foreman-example-com}/20180814.00.10
/var/log/pcp/pmlogger/_{foreman-example-com}_/20230814.00.10
----
Original file line number Diff line number Diff line change
Expand Up @@ -9,30 +9,27 @@ This procedure describes how to install the PCP packages.
+
The default PCP data retention policy is to retain only that data collected during the past 14 days. Data storage per day is estimated to use usually between 100 MB and 500 MB of disk space, but may use up to several gigabytes.

* Ensure that the base system on which {ProjectServer} is running is Red{nbsp}Hat Enterprise Linux 7.6. or later. The minimum supported version for the PCP packages is PCP version 4.1.

.Procedure

. Enable the Red{nbsp}Hat Enterprise Linux `optional` repository:
+
----
# subscription-manager repos --enable rhel-7-server-optional-rpms
----

. Install the PCP packages:
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
ifndef::foreman-deb[]
# {package-install-project} pcp \
pcp-pmda-apache \
pcp-pmda-openmetrics \
pcp-pmda-postgresql \
pcp-system-tools \
pcp-webjs
pcp-pmda-redis \
pcp-system-tools
endif::[]
ifdef::foreman-deb[]
# {package-install-project} pcp
endif::[]
----

. Enable and start the Performance Metrics Collector daemon, and the Performance Metrics Logger daemon:
+
----
# systemctl enable pmcd pmlogger
# systemctl start pmcd pmlogger
# systemctl enable --now pmcd pmlogger
----
Loading

0 comments on commit 5532c86

Please sign in to comment.