prometheus-to-sd is a simple component that can scrape metrics stored in prometheus text format from one or multiple components and push them to the Stackdriver. Main requirement: k8s cluster should run on GCE or GKE.
Look at the following link: https://gcr.io/google-containers/prometheus-to-sd and pick the latest image.
For scraping metrics from the component it's name, host, port and metrics should passed
through the flag
source in the next format:
component-name:http://host:port?whitelisted=a,b,c. If whitelisted part is
omitted, then all metrics that are scraped from the component will be pushed
to the Stackdriver.
To be able to push custom metrics to the Stackdriver flag
has to be specified. In such case metric
bar from the component
foo is going to be pushed to the Stackdriver as
If metric descriptors already exist on the Stackdriver (created manually or by different component)
then autodiscovery feature could be used. In such case prometheus-to-sd will push metrics for
which metric descriptors are available on the Stackdriver. To use this feature a flag
auto-whitelist-metrics=true has to be passed.
Each pushed metric includes monitored resource
descriptor. Fields, such as
zone are filled automatically by
the prometheus-to-sd. Values of the
pod_id can be passed to
the component through the additional flags or omitted.
always empty for now. Field
zone is overridable via flag.
Scrape interval vs. export interval
There are two flags:
export-interval that allow
specifying how often metrics are read from the sources and how often they are
exported to Stackdriver, respectively. By default both are set to 1m. The
scrapes can be more frequent than exports, however, to achieve grater precision
for metrics being exported. For example, if metrics are exported once every
minute and a container dies between scrapes, up to 1 minutes of metrics can be
lost. Frequent scrapes mitigate that, at the cost of elevated resource usage.
Container monitoring agents
Container monitoring agents, such as cAdvisor,
collect metrics about containers other than itself. Monitoring agents use prometheus
labels for the namespace, pod, and container to differentiate metrics from different
containers. To use these prometheus labels as the monitored resource labels in
containerNameLabel in the
source flag in this format:
Note that prometheus labels used for the monitored resource are not included as labels in stackdriver.
For example, if prom-to-sd scraped the prometheus metric:
It would be displayed in stackdriver with the namespace,
my-namespace, the pod id,
abc123, and the container name,
my-app. The only stackdriver label for this metric would be