From 98d49d927a3f7daf3bb46913e4c35b822b60c235 Mon Sep 17 00:00:00 2001 From: Leif Madsen Date: Thu, 17 Jun 2021 17:07:41 -0400 Subject: [PATCH 01/15] Include enable-stf.yaml for simplified config Include the use of a new enable-stf.yaml to simplify and split out the default parameters and values for an OSP installation using STF. This makes the stf-connectors.yaml simpler to configure, and makes both OSP13 and OSP16 overcloud deployment procedure the same. Signed-off-by: Leif Madsen --- .../proc_configuring-gnocchi-metrics.adoc | 22 +- ...-openstack-platform-overcloud-for-stf.adoc | 254 +++++++++++------- 2 files changed, 171 insertions(+), 105 deletions(-) diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc index 7bfc44f7..714c534d 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc @@ -13,7 +13,7 @@ To send metrics to {Project} ({ProjectShort}} and Gnocchi simultaneously, you mu . Create an environment file named `gnocchi-connectors.yaml` in the `/home/stack` directory. + -[source,yaml] +[source,yaml,options="nowrap",subs="none"] ---- resource_registry: OS::TripleO::Services::GnocchiApi: /usr/share/openstack-tripleo-heat-templates/deployment/gnocchi/gnocchi-api-container-puppet.yaml @@ -40,20 +40,20 @@ parameter_defaults: + [source,bash,options="nowrap",subs="+quotes"] ---- -$ openstack overcloud deploy - --templates /usr/share/openstack-tripleo-heat-templates \ +openstack overcloud deploy +--templates /usr/share/openstack-tripleo-heat-templates \ --environment-file <...other-environment-files...> \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/metrics/ceilometer-write-qdr.yaml \ - --environment-file /usr/share/openstack-tripleo-heat-templates/environments/enable-stf.yaml \ + --environment-file /usr/share/openstack-tripleo-heat-templates/environments/metrics/collectd-write-qdr.yaml \ + --environment-file /usr/share/openstack-tripleo-heat-templates/environments/metrics/qdr-edge-only.yaml \ + --environment-file /home/stack/enable-stf.yaml \ --environment-file /home/stack/stf-connectors.yaml \ --environment-file /home/stack/gnocchi-connectors.yaml ---- -. Verification: To verify that the configuration was successful, verify the content -of the file `/var/lib/config-data/puppet-generated/ceilometer/etc/ceilometer/pipeline.yaml` on a Controller -node: - -[source,yaml] +. To verify that the configuration was successful, verify the content of the file `/var/lib/config-data/puppet-generated/ceilometer/etc/ceilometer/pipeline.yaml` on a Controller node. Ensure that the `publishers` section of the file contains information for both notifier and Gnocchi. ++ +[source,yaml,options="nowrap",subs="+quotes"] ---- sources: - name: meter_source @@ -66,6 +66,4 @@ sinks: publishers: - gnocchi://?filter_project=service&archive_policy=high - notifier://172.17.1.35:5666/?driver=amqp&topic=metering ----- -+ -Ensure that the `publishers` section of the file contains information for both notifier and Gnocchi. +---- \ No newline at end of file diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc index 131f0f73..a3334844 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc @@ -29,7 +29,9 @@ To configure the {OpenStack} overcloud, you must configure the data collection a To configure the {OpenStack} overcloud, complete the following tasks: . xref:retrieving-the-qdr-route-address[] +. xref:configuring-the-enable-stf-for-the-overcloud[] . xref:configuring-the-stf-connection-for-the-overcloud[] +. xref:deploying-the-overcloud[] . xref:validating-clientside-installation_assembly-completing-the-stf-configuration[] ifdef::include_when_16_1[] @@ -59,10 +61,145 @@ default-interconnect-5671-service-telemetry.apps.infra.watch [NOTE] If your {ProjectShort} installation differs from the documentation, ensure that you retrieve the correct {MessageBus} route address. +[[configuring-the-enable-stf-for-the-overcloud]] +== Creating the base configuration for {ProjectShort} + +To configure the base parameters to provide a compatible data collection and transport for {ProjectShort}, you must create a file that defines the default data collection values. + +.Procedure + +. Log in to the {OpenStack} undercloud as the `stack` user. + +. Create a configuration file called `enable-stf.yaml` in the `/home/stack` directory. ++ +ifdef::include_when_13[] +[source,yaml,options="nowrap",subs="+quotes"] +---- +parameter_defaults: + # only send to STF, not other publishers + EventPipelinePublishers: [] + PipelinePublishers: [] + + # manage the polling and pipeline configuration files for Ceilometer agents + ManagePolling: true + ManagePipeline: true + + # enable Ceilometer metrics and events + CeilometerQdrPublishMetrics: true + CeilometerQdrPublishEvents: true + + # set collectd overrides for higher telemetry resolution and extra plugins + # to load + CollectdConnectionType: amqp1 + CollectdAmqpInterval: 5 + CollectdDefaultPollingInterval: 5 + CollectdExtraPlugins: + - vmem + + ExtraConfig: + ceilometer::agent::polling::polling_interval: 30 + ceilometer::agent::polling::polling_meters: + - cpu + - disk.* + - ip.* + - image.* + - memory + - memory.* + - network.* + - perf.* + - port + - port.* + - switch + - switch.* + - storage.* + - volume.* + + # to avoid filling the memory buffers if disconnected from the message bus + collectd::plugin::amqp1::send_queue_limit: 50 + + # receive extra information about virtual memory + collectd::plugin::vmem::verbose: true + + # provide the human-friendly name of the virtual instance + collectd::plugin::virt::plugin_instance_format: metadata +---- +endif::include_when_13[] +ifdef::include_when_16[] +[source,yaml,options="nowrap",subs="+quotes"] +---- +parameter_defaults: + # only send to STF, not other publishers + EventPipelinePublishers: [] + PipelinePublishers: [] + + # manage the polling and pipeline configuration files for Ceilometer agents + ManagePolling: true + ManagePipeline: true + + # required to set valid parameter due to typo in ceilometer-write-qdr.yaml + # and will be resolved in a future release + CeilometerQdrPublishMetrics: true + + # enable collection of API status + CollectdEnableSensubility: true + CollectdSensubilityTransport: amqp1 + + # enable collection of containerized service metrics + CollectdEnableLibpodstats: true + + # set collectd overrides for higher telemetry resolution and extra plugins + # to load + CollectdConnectionType: amqp1 + CollectdAmqpInterval: 5 + CollectdDefaultPollingInterval: 5 + CollectdExtraPlugins: + - vmem + + ExtraConfig: + ceilometer::agent::polling::polling_interval: 30 + ceilometer::agent::polling::polling_meters: + - cpu + - disk.* + - ip.* + - image.* + - memory + - memory.* + - network.* + - perf.* + - port + - port.* + - switch + - switch.* + - storage.* + - volume.* + + # to avoid filling the memory buffers if disconnected from the message bus + collectd::plugin::amqp1::send_queue_limit: 50 + + # receive extra information about virtual memory + collectd::plugin::vmem::verbose: true + + # provide name and uuid in addition to hostname for better correlation + # to ceilometer data + collectd::plugin::virt::hostname_format: "name uuid hostname" + + # provide the human-friendly name of the virtual instance + collectd::plugin::virt::plugin_instance_format: metadata + + # set memcached collectd plugin to report its metrics by hostname + # rather than host IP, ensuring metrics in the dashboard remain uniform + collectd::plugin::memcached::instances: + local: + host: "%{hiera('fqdn_canonical')}" + port: 11211 +---- +endif::include_when_16[] + + [[configuring-the-stf-connection-for-the-overcloud]] == Configuring the {ProjectShort} connection for the overcloud -To configure the {ProjectShort} connection, you must create a file that contains the connection configuration of the {MessageBus} for the overcloud to the {ProjectShort} deployment. Enable the collection of events and storage of the events in {ProjectShort} and deploy the overcloud. +To configure the {ProjectShort} connection, you must create a file that contains the connection configuration of the {MessageBus} for the overcloud to the {ProjectShort} deployment. Enable the collection of events and storage of the events in {ProjectShort} and deploy the overcloud. The configuration is set up for a single cloud instance with the default message bus topics. For configuration of multiple cloud deployments, see xref:configuring-multiple-clouds_assembly-advanced-features[]. .Procedure @@ -74,117 +211,59 @@ To configure the {ProjectShort} connection, you must create a file that contains ==== The Service Telemetry Operator simplifies the deployment of all data ingestion and data storage components for single cloud deployments. To share the data storage domain with multiple clouds, see xref:configuring-multiple-clouds_assembly-advanced-features[]. -Additionally, setting `EventPipelinePublishers` and `PipelinePublishers` to empty lists results in no metric or event data passing to {OpenStack} legacy telemetry components, such as Gnocchi or Panko. If you need to send data to additional pipelines, the Ceilometer polling interval of 5 seconds as specified in `ExtraConfig` might overwhelm the legacy components. If you configure a longer polling interval, you must also modify {ProjectShort} to avoid stale metrics, resulting in what appears to be missing data in Prometheus. - -If an adjustment needs to be made to the polling interval, then modify the ServiceTelemetry object `backends.metrics.prometheus.scrapeInterval` parameter from the default value of `10s` to double the polling interval of the data collectors. For example, if `CollectdAmqpInterval` and `ceilometer::agent::polling::polling_interval` are adjusted to `30` then set the `backends.metrics.prometheus.scrapeInterval` to a value of `60s`. +Additionally, setting `EventPipelinePublishers` and `PipelinePublishers` to empty lists results in no metric or event data passing to {OpenStack} legacy telemetry components, such as Gnocchi or Panko. If you need to send data to additional pipelines, the Ceilometer polling interval of 30 seconds as specified in `ExtraConfig` might overwhelm the legacy components. ==== . In the `stf-connectors.yaml` file, configure the `MetricsQdrConnectors` address to connect the {MessageBus} on the overcloud to the {ProjectShort} deployment. -* Add the `CeilometerQdrPublishMetrics: true` parameter to enable collection and transport of Ceilometer metrics to {ProjectShort}. -* Add the `CeilometerQdrPublishEvents: true` parameter to enable collection and transport of Ceilometer events to {ProjectShort}. -* Add the `EventPiplinePublishers: []` and `PipelinePublishers: []` to avoid writing data to Gnocchi and Panko. -* Add the `ManagePolling: true` and `ManagePipeline: true` parameters to allow full control of Ceilometer polling and pipeline configuration. -* Add the `ExtraConfig` parameter `ceilometer::agent::polling::polling_interval` to set the polling interval of Ceilometer to be compatible with the default {ProjectShort} scrape interval. * Replace the `host` parameter with the value of `HOST/PORT` that you retrieved in xref:retrieving-the-qdr-route-address[]: + ifdef::include_when_13[] -[source,yaml] +[source,yaml,options="nowrap",subs="+quotes"] ---- parameter_defaults: - EventPipelinePublishers: [] - PipelinePublishers: [] - CeilometerEnablePanko: false - CeilometerQdrPublishEvents: true - CeilometerQdrPublishMetrics: true - ManagePipeline: true - ManagePolling: true - CollectdAmqpInstances: - notify: - format: JSON - notify: true - presettle: false - telemetry: - format: JSON - presettle: false - CollectdAmqpInterval: 5 - CollectdConnectionType: amqp1 - CollectdDefaultPlugins: - - cpu - - df - - disk - - hugepages - - interface - - load - - memory - - processes - - unixsock - - uptime - - connectivity - - intel_rdt - - ipmi - - procevent - CollectdDefaultPollingInterval: 5 - MetricsQdrAddresses: - - distribution: multicast - prefix: collectd - - distribution: multicast - prefix: anycast/ceilometer MetricsQdrConnectors: - - host: default-interconnect-5671-service-telemetry.apps.infra.watch - port: 443 - role: edge - sslProfile: sslProfile - verifyHostname: false + - host: default-interconnect-5671-service-telemetry.apps.stf.cloudops.psi.redhat.com + port: 443 + role: edge + sslProfile: sslProfile + verifyHostname: false + MetricsQdrSSLProfiles: -ifdef::include_when_13[] - name: sslProfile caCertFileContent: | ----BEGIN CERTIFICATE---- ----END CERTIFICATE---- -endif::include_when_13[] - ExtraConfig: - collectd::plugin::cpu::reportbycpu: true - collectd::plugin::cpu::reportbystate: true - collectd::plugin::cpu::reportnumcpu: false - collectd::plugin::cpu::valuespercentage: true - collectd::plugin::df::ignoreselected: true - collectd::plugin::df::reportbydevice: true - collectd::plugin::df::fstypes: ['xfs'] - collectd::plugin::load::reportrelative: true - collectd::plugin::virt::connection: "qemu:///system" - collectd::plugin::virt::extra_stats: "cpu_util disk disk_err pcpu job_stats_background perf vcpupin" - collectd::plugin::virt::hostname_format: "hostname" - ceilometer::agent::polling::polling_interval: 5 ---- endif::include_when_13[] ifdef::include_when_16[] -[source,yaml] +[source,yaml,options="nowrap",subs="+quotes"] ---- parameter_defaults: - EventPipelinePublishers: [] - PipelinePublishers: [] - CeilometerQdrPublishEvents: true - CeilometerQdrPublishMetrics: true MetricsQdrConnectors: - host: default-interconnect-5671-service-telemetry.apps.infra.watch port: 443 role: edge sslProfile: sslProfile verifyHostname: false - ExtraConfig: - ceilometer::agent::polling::polling_interval: 5 ---- endif::include_when_16[] -. Add the following files to your {OpenStack} {OpenStackInstaller} deployment to setup collectd and {MessageBus}: +[[deploying-the-overcloud]] +== Deploying the overcloud + +Deploy or update the overcloud with the required environment files to result in data being collected and transmitted to {ProjectShort}. + +.Procedure + +. Add the following files to your {OpenStack} {OpenStackInstaller} deployment to setup data collection and {MessageBus}: + -* the `stf-connectors.yaml` environment file -ifdef::include_when_16[* the `enable-stf.yaml` file that ensures that the environment is being used during the overcloud deployment] -ifdef::include_when_13[* the `collectd-write-qdr.yaml` file that ensures that collectd telemetry is sent to {ProjectShort}] -* the `ceilometer-write-qdr.yaml` file that ensures that Ceilometer telemetry is sent to {ProjectShort} +* the `enable-stf.yaml` environment file to ensure defaults are set up correctly +* the `stf-connectors.yaml` environment file to define the connection to {ProjectShort} +* the `collectd-write-qdr.yaml` file that ensures that collectd telemetry and events are sent to {ProjectShort} +* the `ceilometer-write-qdr.yaml` file that ensures that Ceilometer telemetry and events are sent to {ProjectShort} +* the `qdr-edge-only.yaml` file that ensures the message bus is enabled and connected to {ProjectShort} message bus routers + -ifdef::include_when_13[] [source,bash,options="nowrap",subs="+quotes"] ---- openstack overcloud deploy @@ -193,19 +272,8 @@ openstack overcloud deploy --environment-file /usr/share/openstack-tripleo-heat-templates/environments/metrics/ceilometer-write-qdr.yaml \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/metrics/collectd-write-qdr.yaml \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/metrics/qdr-edge-only.yaml \ + --environment-file /home/stack/enable-stf.yaml \ --environment-file /home/stack/stf-connectors.yaml ---- -endif::include_when_13[] -ifdef::include_when_16[] -[source,bash,options="nowrap",subs="+quotes"] ----- -openstack overcloud deploy - --templates /usr/share/openstack-tripleo-heat-templates \ - --environment-file <...other-environment-files...> \ - --environment-file /usr/share/openstack-tripleo-heat-templates/environments/metrics/ceilometer-write-qdr.yaml \ - --environment-file /usr/share/openstack-tripleo-heat-templates/environments/enable-stf.yaml \ - --environment-file /home/stack/stf-connectors.yaml ----- -endif::include_when_16[] . Deploy the {OpenStack} overcloud. From cc8018f5c6a9c3ed6731e10347f09d437c7cd79c Mon Sep 17 00:00:00 2001 From: Leif Madsen Date: Thu, 17 Jun 2021 17:15:31 -0400 Subject: [PATCH 02/15] Minor fixup for code block wrapping --- .../modules/proc_setting-up-grafana-to-host-the-dashboard.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc b/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc index dfae2fa2..9d65e357 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc @@ -122,6 +122,7 @@ default-datasources 20h $ oc get route grafana-route ---- + +[source,bash,options="nowrap"] ---- NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD grafana-route grafana-route-service-telemetry.apps.infra.watch grafana-service 3000 edge None From 353de90ebe6532f22938575f8dc58f7c01210f17 Mon Sep 17 00:00:00 2001 From: JoanneOFlynn2018 <45287002+JoanneOFlynn2018@users.noreply.github.com> Date: Thu, 17 Jun 2021 22:22:04 +0100 Subject: [PATCH 03/15] Update doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc --- ...onfiguring-red-hat-openstack-platform-overcloud-for-stf.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc index a3334844..5743cc7e 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc @@ -260,7 +260,7 @@ Deploy or update the overcloud with the required environment files to result in + * the `enable-stf.yaml` environment file to ensure defaults are set up correctly * the `stf-connectors.yaml` environment file to define the connection to {ProjectShort} -* the `collectd-write-qdr.yaml` file that ensures that collectd telemetry and events are sent to {ProjectShort} +* the `collectd-write-qdr.yaml` file to ensure that collectd telemetry and events are sent to {ProjectShort} * the `ceilometer-write-qdr.yaml` file that ensures that Ceilometer telemetry and events are sent to {ProjectShort} * the `qdr-edge-only.yaml` file that ensures the message bus is enabled and connected to {ProjectShort} message bus routers + From febc83d11b0f51ee10fbbdc44eabc0f728d216fa Mon Sep 17 00:00:00 2001 From: JoanneOFlynn2018 <45287002+JoanneOFlynn2018@users.noreply.github.com> Date: Thu, 17 Jun 2021 22:23:40 +0100 Subject: [PATCH 04/15] Update doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc --- ...onfiguring-red-hat-openstack-platform-overcloud-for-stf.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc index 5743cc7e..770c03c9 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc @@ -261,7 +261,7 @@ Deploy or update the overcloud with the required environment files to result in * the `enable-stf.yaml` environment file to ensure defaults are set up correctly * the `stf-connectors.yaml` environment file to define the connection to {ProjectShort} * the `collectd-write-qdr.yaml` file to ensure that collectd telemetry and events are sent to {ProjectShort} -* the `ceilometer-write-qdr.yaml` file that ensures that Ceilometer telemetry and events are sent to {ProjectShort} +* the `ceilometer-write-qdr.yaml` file to ensure that Ceilometer telemetry and events are sent to {ProjectShort} * the `qdr-edge-only.yaml` file that ensures the message bus is enabled and connected to {ProjectShort} message bus routers + [source,bash,options="nowrap",subs="+quotes"] From 96465f464d695298eeed57f3d568fcd531e75bca Mon Sep 17 00:00:00 2001 From: JoanneOFlynn2018 <45287002+JoanneOFlynn2018@users.noreply.github.com> Date: Thu, 17 Jun 2021 22:24:01 +0100 Subject: [PATCH 05/15] Update doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc --- ...onfiguring-red-hat-openstack-platform-overcloud-for-stf.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc index 770c03c9..b18997f0 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc @@ -262,7 +262,7 @@ Deploy or update the overcloud with the required environment files to result in * the `stf-connectors.yaml` environment file to define the connection to {ProjectShort} * the `collectd-write-qdr.yaml` file to ensure that collectd telemetry and events are sent to {ProjectShort} * the `ceilometer-write-qdr.yaml` file to ensure that Ceilometer telemetry and events are sent to {ProjectShort} -* the `qdr-edge-only.yaml` file that ensures the message bus is enabled and connected to {ProjectShort} message bus routers +* the `qdr-edge-only.yaml` file to ensure that the message bus is enabled and connected to {ProjectShort} message bus routers + [source,bash,options="nowrap",subs="+quotes"] ---- From b3c2ac6d656db5890bc3f82a9e929529e4bcea69 Mon Sep 17 00:00:00 2001 From: JoanneOFlynn2018 <45287002+JoanneOFlynn2018@users.noreply.github.com> Date: Thu, 17 Jun 2021 22:38:05 +0100 Subject: [PATCH 06/15] Update doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc --- ...onfiguring-red-hat-openstack-platform-overcloud-for-stf.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc index b18997f0..579b13c2 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc @@ -61,7 +61,7 @@ default-interconnect-5671-service-telemetry.apps.infra.watch [NOTE] If your {ProjectShort} installation differs from the documentation, ensure that you retrieve the correct {MessageBus} route address. -[[configuring-the-enable-stf-for-the-overcloud]] +[[creating-the-base-configuration-for-stf]] == Creating the base configuration for {ProjectShort} To configure the base parameters to provide a compatible data collection and transport for {ProjectShort}, you must create a file that defines the default data collection values. From e3b4a0cfa28d8e5069295eedce511f71c21662db Mon Sep 17 00:00:00 2001 From: JoanneOFlynn2018 <45287002+JoanneOFlynn2018@users.noreply.github.com> Date: Thu, 17 Jun 2021 22:38:17 +0100 Subject: [PATCH 07/15] Update doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc --- .../modules/proc_configuring-gnocchi-metrics.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc index 714c534d..662f2dd7 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc @@ -40,7 +40,7 @@ parameter_defaults: + [source,bash,options="nowrap",subs="+quotes"] ---- -openstack overcloud deploy +$ openstack overcloud deploy __ --templates /usr/share/openstack-tripleo-heat-templates \ --environment-file <...other-environment-files...> \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/metrics/ceilometer-write-qdr.yaml \ @@ -66,4 +66,4 @@ sinks: publishers: - gnocchi://?filter_project=service&archive_policy=high - notifier://172.17.1.35:5666/?driver=amqp&topic=metering ----- \ No newline at end of file +---- From 55db83119e767d4a663459782e0845e1bae7dd53 Mon Sep 17 00:00:00 2001 From: JoanneOFlynn2018 <45287002+JoanneOFlynn2018@users.noreply.github.com> Date: Thu, 17 Jun 2021 22:38:38 +0100 Subject: [PATCH 08/15] Update doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc --- ...onfiguring-red-hat-openstack-platform-overcloud-for-stf.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc index 579b13c2..cb23bb84 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc @@ -29,7 +29,7 @@ To configure the {OpenStack} overcloud, you must configure the data collection a To configure the {OpenStack} overcloud, complete the following tasks: . xref:retrieving-the-qdr-route-address[] -. xref:configuring-the-enable-stf-for-the-overcloud[] +. xref:creating-the-base-configuration-for-stf[] . xref:configuring-the-stf-connection-for-the-overcloud[] . xref:deploying-the-overcloud[] . xref:validating-clientside-installation_assembly-completing-the-stf-configuration[] From ea8cd83cba917881f3df8d646ce5bdde3f7a8cdb Mon Sep 17 00:00:00 2001 From: Leif Madsen Date: Thu, 17 Jun 2021 17:43:37 -0400 Subject: [PATCH 09/15] Update doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc Co-authored-by: JoanneOFlynn2018 <45287002+JoanneOFlynn2018@users.noreply.github.com> --- ...onfiguring-red-hat-openstack-platform-overcloud-for-stf.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc index cb23bb84..ae0e1e23 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc @@ -266,7 +266,7 @@ Deploy or update the overcloud with the required environment files to result in + [source,bash,options="nowrap",subs="+quotes"] ---- -openstack overcloud deploy +$ openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates \ --environment-file <...other-environment-files...> \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/metrics/ceilometer-write-qdr.yaml \ From 81c6544d84eff2d290ebab1eec7545c0440fd4b0 Mon Sep 17 00:00:00 2001 From: Leif Madsen Date: Thu, 17 Jun 2021 20:52:53 -0400 Subject: [PATCH 10/15] Add nicer syntax highlighting --- doc-Service-Telemetry-Framework/master.adoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc-Service-Telemetry-Framework/master.adoc b/doc-Service-Telemetry-Framework/master.adoc index 6853c597..4517b372 100644 --- a/doc-Service-Telemetry-Framework/master.adoc +++ b/doc-Service-Telemetry-Framework/master.adoc @@ -1,6 +1,9 @@ = Service Telemetry Framework 1.3 OpenStack Documentation Team :imagesdir: images +:source-highlighter: highlightjs +:highlightjs-languages: yaml,bash +:highlightjs-theme: monokai :vernum: 16.1 :toc: left :icons: font From ef2d590090226aa56a144ab9379931e02821aa90 Mon Sep 17 00:00:00 2001 From: Leif Madsen Date: Thu, 17 Jun 2021 20:53:24 -0400 Subject: [PATCH 11/15] Add id to link for gnocchi config --- .../modules/proc_configuring-gnocchi-metrics.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc index 662f2dd7..6e880c0b 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-gnocchi-metrics.adoc @@ -1,3 +1,4 @@ +[id="sending-metrics-to-gnocchi-and-to-stf_{context}"] = Sending metrics to Gnocchi and to {ProjectShort} [role="_abstract"] From 643de07f684eb40cf61eddd11ac69a17dfbf0758 Mon Sep 17 00:00:00 2001 From: Leif Madsen Date: Thu, 17 Jun 2021 20:53:48 -0400 Subject: [PATCH 12/15] Additional clean up and OSP13 bug fix --- ...t-openstack-platform-overcloud-for-stf.adoc | 18 +++++++++--------- ...oc_creating-openstack-environment-file.adoc | 2 ++ 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc index ae0e1e23..e76577e6 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-red-hat-openstack-platform-overcloud-for-stf.adoc @@ -72,6 +72,13 @@ To configure the base parameters to provide a compatible data collection and tra . Create a configuration file called `enable-stf.yaml` in the `/home/stack` directory. + +[IMPORTANT] +==== +Setting `EventPipelinePublishers` and `PipelinePublishers` to empty lists results in no event or metric data passing to {OpenStack} legacy telemetry components, such as Gnocchi or Panko. If you need to send data to additional pipelines, the Ceilometer polling interval of 30 seconds as specified in `ExtraConfig` might overwhelm the legacy components, and should be increased to a larger value such as `300`. Increasing the value to a longer polling interval will result in less telemetry resolution in {ProjectShort}. + +To enable collection of telemetry with {ProjectShort} and Gnocchi, see xref:sending-metrics-to-gnocchi-and-to-stf_assembly-completing-the-stf-configuration[] +==== ++ ifdef::include_when_13[] [source,yaml,options="nowrap",subs="+quotes"] ---- @@ -206,13 +213,6 @@ To configure the {ProjectShort} connection, you must create a file that contains . Log in to the {OpenStack} undercloud as the `stack` user. . Create a configuration file called `stf-connectors.yaml` in the `/home/stack` directory. -+ -[IMPORTANT] -==== -The Service Telemetry Operator simplifies the deployment of all data ingestion and data storage components for single cloud deployments. To share the data storage domain with multiple clouds, see xref:configuring-multiple-clouds_assembly-advanced-features[]. - -Additionally, setting `EventPipelinePublishers` and `PipelinePublishers` to empty lists results in no metric or event data passing to {OpenStack} legacy telemetry components, such as Gnocchi or Panko. If you need to send data to additional pipelines, the Ceilometer polling interval of 30 seconds as specified in `ExtraConfig` might overwhelm the legacy components. -==== . In the `stf-connectors.yaml` file, configure the `MetricsQdrConnectors` address to connect the {MessageBus} on the overcloud to the {ProjectShort} deployment. * Replace the `host` parameter with the value of `HOST/PORT` that you retrieved in xref:retrieving-the-qdr-route-address[]: @@ -258,11 +258,11 @@ Deploy or update the overcloud with the required environment files to result in . Add the following files to your {OpenStack} {OpenStackInstaller} deployment to setup data collection and {MessageBus}: + -* the `enable-stf.yaml` environment file to ensure defaults are set up correctly -* the `stf-connectors.yaml` environment file to define the connection to {ProjectShort} * the `collectd-write-qdr.yaml` file to ensure that collectd telemetry and events are sent to {ProjectShort} * the `ceilometer-write-qdr.yaml` file to ensure that Ceilometer telemetry and events are sent to {ProjectShort} * the `qdr-edge-only.yaml` file to ensure that the message bus is enabled and connected to {ProjectShort} message bus routers +* the `enable-stf.yaml` environment file to ensure defaults are set up correctly +* the `stf-connectors.yaml` environment file to define the connection to {ProjectShort} + [source,bash,options="nowrap",subs="+quotes"] ---- diff --git a/doc-Service-Telemetry-Framework/modules/proc_creating-openstack-environment-file.adoc b/doc-Service-Telemetry-Framework/modules/proc_creating-openstack-environment-file.adoc index 0cc87abb..9e0e637b 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_creating-openstack-environment-file.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_creating-openstack-environment-file.adoc @@ -204,8 +204,10 @@ endif::include_when_16[] <3> Define the topic for collectd events. This value is the format of `collectd/cloud1-notify`. <4> Define the topic for collectd metrics. This value is the format of `collectd/cloud1-telemetry`. <5> Define the topic for collectd-sensubility events. This should be the exact string format of `collectd/cloud1-notify` +ifdef::include_when_16[] <6> Adjust the `MetricsQdrConnectors` host to the address of the {ProjectShort} route. <7> Enable monitoring of health and API status. +endif::include_when_16[] + . Ensure that the naming convention in the `stf-connectors.yaml` file aligns with the `spec.amqpUrl` field in the Smart Gateway configuration. For example, configure the `CeilometerQdrEventsConfig.topic` field to a value of `cloud1-event`. From 8ffec3c3c3d22402b4eab197dce5d4aa7bd80b53 Mon Sep 17 00:00:00 2001 From: Leif Madsen Date: Thu, 17 Jun 2021 20:57:45 -0400 Subject: [PATCH 13/15] Move added syntax highlighting to attributes in Makefile --- doc-Service-Telemetry-Framework/Makefile | 4 ++-- doc-Service-Telemetry-Framework/master.adoc | 3 --- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/doc-Service-Telemetry-Framework/Makefile b/doc-Service-Telemetry-Framework/Makefile index cd7584bd..6a9864f8 100644 --- a/doc-Service-Telemetry-Framework/Makefile +++ b/doc-Service-Telemetry-Framework/Makefile @@ -45,10 +45,10 @@ $(IMAGES_TS): $(IMAGES) touch $(IMAGES_TS) $(DEST_HTML): $(SOURCES) - asciidoctor --failure-level WARN -a build=$(BUILD) -b xhtml5 -d book -o $@ $< + asciidoctor -a source-highlighter=highlightjs -a highlightjs-languages="yaml,bash" -a highlightjs-theme="monokai" --failure-level WARN -a build=$(BUILD) -b xhtml5 -d book -o $@ $< $(DEST_HTML_13): $(SOURCES) - asciidoctor --failure-level WARN -a build=$(BUILD) -a vernum=13 -b xhtml5 -d book -o $@ $< + asciidoctor -a source-highlighter=highlightjs -a highlightjs-languages="yaml,bash" -a highlightjs-theme="monokai" --failure-level WARN -a build=$(BUILD) -a vernum=13 -b xhtml5 -d book -o $@ $< $(DEST_PDF): $(SOURCES) $(IMAGES) asciidoctor-pdf -a build=$(BUILD) -d book -o $@ $< diff --git a/doc-Service-Telemetry-Framework/master.adoc b/doc-Service-Telemetry-Framework/master.adoc index 4517b372..6853c597 100644 --- a/doc-Service-Telemetry-Framework/master.adoc +++ b/doc-Service-Telemetry-Framework/master.adoc @@ -1,9 +1,6 @@ = Service Telemetry Framework 1.3 OpenStack Documentation Team :imagesdir: images -:source-highlighter: highlightjs -:highlightjs-languages: yaml,bash -:highlightjs-theme: monokai :vernum: 16.1 :toc: left :icons: font From 751a79759b9b83d51d6240988dcf409a29a86087 Mon Sep 17 00:00:00 2001 From: Leif Madsen Date: Thu, 17 Jun 2021 21:04:07 -0400 Subject: [PATCH 14/15] Minor adjustments for multi-cloud --- .../proc_creating-openstack-environment-file.adoc | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/doc-Service-Telemetry-Framework/modules/proc_creating-openstack-environment-file.adoc b/doc-Service-Telemetry-Framework/modules/proc_creating-openstack-environment-file.adoc index 9e0e637b..6068908b 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_creating-openstack-environment-file.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_creating-openstack-environment-file.adoc @@ -29,17 +29,10 @@ To label traffic according to the cloud of origin, you must create a configurati [NOTE] If you enabled container health and API status monitoring, you must also modify the `CollectdSensubilityResultsChannel` parameter. For more information, see xref:monitoring-container-health-and-api-status_assembly-advanced-features[]. - -[WARNING] -Remove -ifdef::include_when_16[] -`enable-stf.yaml` and -endif::include_when_16[] -`ceilometer-write-qdr.yaml` environment file references from your overcloud deployment. This configuration is redundant and results in duplicate information being sent from each cloud node. - .Prerequisites * You have created a unique domain name environment file. For more information, see xref:setting-a-unique-cloud-domain_assembly-advanced-features[]. +* You have created the `enabled-stf.yaml` to set environment defaults for {ProjectShort}. .Procedure From 74d7292d4ac1dea31cc03d38569d9297b0ea3a08 Mon Sep 17 00:00:00 2001 From: Leif Madsen Date: Thu, 17 Jun 2021 21:16:22 -0400 Subject: [PATCH 15/15] Syntax cleanup --- ...c_deleting-the-default-smart-gateways.adoc | 3 +- .../proc_deploying-smart-gateways.adoc | 5 ++ ...oring-container-health-and-api-status.adoc | 2 + ...-resource-usage-of-openstack-services.adoc | 1 + .../proc_setting-a-unique-cloud-domain.adoc | 12 ++--- ...ting-up-grafana-to-host-the-dashboard.adoc | 51 ++++++------------- 6 files changed, 31 insertions(+), 43 deletions(-) diff --git a/doc-Service-Telemetry-Framework/modules/proc_deleting-the-default-smart-gateways.adoc b/doc-Service-Telemetry-Framework/modules/proc_deleting-the-default-smart-gateways.adoc index ed1a52c4..8b48d488 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_deleting-the-default-smart-gateways.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_deleting-the-default-smart-gateways.adoc @@ -32,11 +32,11 @@ WARNING: The `cloudsRemoveOnMissing` parameter is disabled by default. If you en .Procedure - . Define your `clouds` parameter with the list of cloud objects to be managed by the Service Telemetry Operator. For more information, see xref:clouds_assembly-installing-the-core-components-of-stf[]. . Edit the ServiceTelemetry object and add the `cloudsRemoveOnMissing` parameter: + +[source,yaml] ---- apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry @@ -53,6 +53,7 @@ spec: . Verify that the Operator deleted the Smart Gateways. This can take several minutes while the Operators reconcile the changes: + +[source,bash] ---- $ oc get smartgateways ---- diff --git a/doc-Service-Telemetry-Framework/modules/proc_deploying-smart-gateways.adoc b/doc-Service-Telemetry-Framework/modules/proc_deploying-smart-gateways.adoc index f7cd3de2..1ac9f2dc 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_deploying-smart-gateways.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_deploying-smart-gateways.adoc @@ -47,16 +47,19 @@ When you deploy {ProjectShort} for the first time, Smart Gateway manifests are c . Log in to {OpenShift}. . Change to the `service-telemetry` namespace: + +[source,bash] ---- $ oc project service-telemetry ---- . Edit the `default` ServiceTelemetry object and add a `clouds` parameter with your configuration: + +[source,bash] ---- $ oc edit stf default ---- + +[source,yaml,options="nowrap"] ---- apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry @@ -87,10 +90,12 @@ spec: . Verify that each Smart Gateway is running. This can take several minutes depending on the number of Smart Gateways: + +[source,bash] ---- $ oc get po -l app=smart-gateway ---- + +[source,bash] ---- NAME READY STATUS RESTARTS AGE default-cloud1-ceil-event-smartgateway-6cfb65478c-g5q82 1/1 Running 0 13h diff --git a/doc-Service-Telemetry-Framework/modules/proc_monitoring-container-health-and-api-status.adoc b/doc-Service-Telemetry-Framework/modules/proc_monitoring-container-health-and-api-status.adoc index 71fe7991..b171af21 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_monitoring-container-health-and-api-status.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_monitoring-container-health-and-api-status.adoc @@ -38,12 +38,14 @@ To monitor healthchecks in {Project} ({ProjectShort}), you must enable and confi . Open the `stf-connectors.yaml` and edit the `collectd-sensubility` parameter: + +[source,yaml] ---- CollectdEnableSensubility: true CollectdSensubilityTransport: amqp1 ---- . If your environment has multiple clouds, configure the `collectd-sensubility` events channel with the new collectd events address. Edit the `stf-connectors.yaml` file: + +[source,yaml] ---- CollectdSensubilityResultsChannel: collectd/cloudprefix-notify ---- diff --git a/doc-Service-Telemetry-Framework/modules/proc_monitoring-resource-usage-of-openstack-services.adoc b/doc-Service-Telemetry-Framework/modules/proc_monitoring-resource-usage-of-openstack-services.adoc index c2283acb..ebb3da45 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_monitoring-resource-usage-of-openstack-services.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_monitoring-resource-usage-of-openstack-services.adoc @@ -40,6 +40,7 @@ Monitor the resource usage of the {OpenStack} services, such as the APIs and oth . Add the following configuration to `parameter_defaults`: + +[source,yaml] ---- CollectdEnableLibpodstats: true ---- diff --git a/doc-Service-Telemetry-Framework/modules/proc_setting-a-unique-cloud-domain.adoc b/doc-Service-Telemetry-Framework/modules/proc_setting-a-unique-cloud-domain.adoc index 6aff4a5a..feaf85bc 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_setting-a-unique-cloud-domain.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_setting-a-unique-cloud-domain.adoc @@ -32,12 +32,12 @@ To ensure that AMQ router connections from {OpenStack} to {Project} ({ProjectSho . Set the `CloudDomain` parameter in the environment file, as shown in the following example: + +[source,yaml,options="nowrap"] ---- - - parameter_defaults: - CloudDomain: newyork-west-04 - CephStorageHostnameFormat: 'ceph-%index%' - ObjectStorageHostnameFormat: 'swift-%index%' - ComputeHostnameFormat: 'compute-%index%' +parameter_defaults: + CloudDomain: newyork-west-04 + CephStorageHostnameFormat: 'ceph-%index%' + ObjectStorageHostnameFormat: 'swift-%index%' + ComputeHostnameFormat: 'compute-%index%' ---- . Add this environment file to your deployment. For more information, see xref:creating-openstack-environment-file_assembly-advanced-features[] and https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/overcloud_parameters/core-overcloud-parameters[Core overcloud parameters] in the _Overcloud Parameters_ guide. diff --git a/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc b/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc index 9d65e357..73e55c1a 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc @@ -35,13 +35,14 @@ Enable OperatorHub.io catalog source for the Grafana Operator. For more informat . Log in to {OpenShift}. . Change to the `service-telemetry` namespace: + +[source,bash] ---- $ oc project service-telemetry ---- . Deploy the Grafana operator: + -[source,bash] +[source,yaml] ---- $ oc apply -f - < in a web browser. Replace with the value that you retrieved in the previous step. @@ -216,4 +195,4 @@ $ oc project service-telemetry $ oc describe grafana default ---- -. To modify the default values of the Grafana administrator username and password through the ServiceTelemetry object, use the `graphing.grafana.adminUser` and `graphing.grafana.adminPassword` parameters. +. To modify the default values of the Grafana administrator username and password through the ServiceTelemetry object, use the `graphing.grafana.adminUser` and `graphing.grafana.adminPassword` parameters.