Merge main into dev.multiple-integrations (grafana#1184)

* Fix typo (grafana#1141) * Traces: Improved pod association in PromSD processor (grafana#1137) * Improve k8s pod association * Add tests * Changelog * typo * Add prom_sd_pod_association * Extend tests for pod associations * Docs for pod association config * Lint fixes * Move to unreleased * Add instrumentation recommendations * Remove uncessary constants * Improve tests * remote config with http(s) provider (grafana#1143) * sample remote config code with http provider * use t.TempDir() in unit test * no need to clean up after T.TempDir() * use NewClientFromConfig and make caller responsible for calling SetDirectory * handle nil HTTPClientConfig * remove blank identifier assignment * pass basic auth command line flags for remote config * address pr nits * add expiremental flag * set loader inline * update changelog * add remote config section in docs * pr comment updates * announce patch releases for cve-2021-41090 (grafana#1152) * Merge patch release to main (grafana#1153) * Add secret type to sensitive values * Break out config tests to their own implementation. Also remove username has a sensitive value. * Update changelog * Fix failing test * Scrub secrets when marshaling instance configs * update for v0.21 * Updated changes from the merge. * Remove changelog * Scrub out receivers has ***receivers_scrubber***:null * obscure etcd/consul credentials * Update pkg/traces/config_test.go Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Update pkg/config/config.go * go fmt * Change to using custom object and return <secret> * Fix bad merge * [v0.21.2] toggle config endpoint (#19) * disable /-/config endpoint by default * disable scraping api get endpoint as well * fix new test * add test and rename flag Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Update version to v0.21.2 * Update defaults.go * fix /-/config endpoint * also fix non-pointer config bug * temporarily disable linting for release * fix lint errors Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> * Fix POSTGRES_EXPORTER_DATA_SOURCE_NAME usage for postgres_exporter (grafana#1162) * Fix POSTGRES_EXPORTER_DATA_SOURCE_NAME usage for postgres_exporter A recent change broke the usage of POSTGRES_EXPORTER_DATA_SOURCE_NAME for the postgres_exporter. As the incorrect variable was checked in the if clause, it always raises an error. * changelog: keep feature -> enhancement -> bugfix order * postgres_exporter: add regression test Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com> * Fix syntax error in Jsonnet logs helper method (grafana#1174) Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * cAdvisor Integration (grafana#1081) * Add cadvisor module * Begin creating common config for cadvisor * Don't export internal state * Finish config options for cadvisor * Set config options, and implement cAdvisor collectors * Linting * Buildflags for cadvisor only in linux * I R LEArN Build Tags * Don't zero value the zero value * Offload sketchy global var manipulation to the integrations Run func * Remove unused collectors * Lint * Create generic stub integration and use it for cadvisor * Lint * Final refactor of cAdvisor config for unsupported platforms. Pared down stub integrations. * Lint * Docs for cadvisor config * Update changelog * Update pkg/integrations/stub_integration.go Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Reorder changelog * Instance key clarity * Inclusive naming * Finish name changes Keep default disable metric list in sync with upstream Idiomatic golang * Hardcode disabled metrics for cadvisor Co-authored-by: Robert Fratto <robert.fratto@grafana.com> * Remove log-level flag from systemd unit file (grafana#1177) * Upgrade to OTel v0.40.0 (grafana#1176) * Upgrade to OTel v0.40.0 * Changelog * Add factories check * go mod tidy * config/features: create package to standardize experimental features (grafana#1170) * config/features: create package to standardize experiemental features This commit introduces a new package, pkg/config/features, which allows defining a set of features and validating whether flags associated with those features are allowed to be set. Closes grafana#1163 * update documentation (also s/enabled-features/enable-features) * Fix typo * Update pkg/config/features/features.go Co-authored-by: Robert Lankford <rlankfo@gmail.com> Co-authored-by: Robert Lankford <rlankfo@gmail.com> * enable cadvisor by default * switch to using real feature flag * fix postgres_exporter Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com> Co-authored-by: Mario <mariorvinas@gmail.com> Co-authored-by: Robert Lankford <robert.lankford@grafana.com> Co-authored-by: Matt Durham <mattdurham@ppog.org> Co-authored-by: f11r <fiete.gruenter@rwth-aachen.de> Co-authored-by: f11r <f11r@users.noreply.github.com> Co-authored-by: Nick Pillitteri <56quarters@users.noreply.github.com> Co-authored-by: Ryan Geyer <me@ryangeyer.com> Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de> Co-authored-by: Robert Lankford <rlankfo@gmail.com>
shamsalmon · Dec 17, 2021 · ad76ec5 · ad76ec5
1 parent 5de23d3
commit ad76ec5
Show file tree

Hide file tree

Showing 74 changed files with 2,229 additions and 346 deletions.
diff --git a/.drone/drone.yml b/.drone/drone.yml
@@ -129,7 +129,7 @@ volumes:
 ---
 kind: pipeline
 type: docker
-name: Deploy-To-Deployment-Tools 
+name: Deploy-To-Deployment-Tools
 platform:
   os: linux
   arch: amd64
@@ -146,10 +146,10 @@ steps:
     commands:
       - apk update && apk add git
       - echo "grafana/agent:$(sh ./tools/image-tag)" > .image-tag
-  - name: Update Deployment Tools 
+  - name: Update Deployment Tools
     image: us.gcr.io/kubernetes-dev/drone/plugins/updater
     settings:
-      config_json: |- 
+      config_json: |-
         {
           "destination_branch": "master",
           "pull_request_branch_prefix": "cd-agent",
@@ -168,9 +168,9 @@ steps:
         }
       github_token:
         from_secret: gh_token
- 
+
 depends_on:
-  - Containerize 
+  - Containerize
 
 volumes:
   - name: docker
@@ -240,6 +240,6 @@ get:
   name: pat
 ---
 kind: signature
-hmac: 53e9986dcd0fdbb78e1c09b6ba363ebcb94bc0801796510923f647b5c7d4ec46
+hmac: c32fd61018bef3703edb694acd0c7312be8cb07898c2cb42eb0e84fee84efb67
 
 ...
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,19 @@
 BREAKING CHANGE: Integrations have changed in this release.
 Please review the [migration guide](./docs/migration-guide.md) for details.
 
+- [FEATURE] (beta) Enable experimental config urls for fetching remote configs. Currently,
+   only HTTP/S is supported. Pass the `-enable-features=remote-configs` flag to turn this on. (@rlankfo)
+
+- [FEATURE] Added [cAdvisor](https://github.com/google/cadvisor) integration. (@rgeyer)
+
+- [ENHANCEMENT] Traces: Improved pod association in PromSD processor (@mapno)
+
+- [ENHANCEMENT] Updated OTel to v0.40.0 (@mapno)
+
+- [BUGFIX] Fix usage of POSTGRES_EXPORTER_DATA_SOURCE_NAME when using postgres_exporter integration (@f11r)
+
+- [CHANGE] Remove log-level flag from systemd unit file (@jpkrohling)
+
 - [CHANGE] Integrations present in the `integrations:` map will now default to
   being enabled. (@rfratto)
 
@@ -13,6 +26,16 @@ Please review the [migration guide](./docs/migration-guide.md) for details.
 - [DEPRECATION] Integrations: The `enabled` field is now deprecated and will be
   removed in a future release. (@rfratto)
 
+# v0.21.2 (2021-12-08)
+
+- [SECURITY] This release contains a fix for
+  [CVE-2021-41090](https://github.com/grafana/agent/security/advisories/GHSA-9c4x-5hgq-q3wh).
+
+- [CHANGE] This release disables the existing `/-/config` and
+  `/agent/api/v1/configs/{name}` endpoitns by default. Pass the
+  `--config.enable-read-api` flag at the command line to opt in to these
+  endpoints.
+
 # v0.21.1 (2021-11-18)
 
 - [BUGFIX] Fix panic when using postgres_exporter integration (@saputradharma)
@@ -55,6 +78,18 @@ Please review the [migration guide](./docs/migration-guide.md) for details.
 
 - [CHANGE] Traces: Changed service graphs store implementation to improve CPU performance (@mapno)
 
+# v0.20.1 (2021-12-08)
+
+*NOTE*: The fixes in this patch are only present in v0.20.1 and >=v0.21.2.
+
+- [SECURITY] This release contains a fix for
+  [CVE-2021-41090](https://github.com/grafana/agent/security/advisories/GHSA-9c4x-5hgq-q3wh).
+
+- [CHANGE] This release disables the existing `/-/config` and
+  `/agent/api/v1/configs/{name}` endpoitns by default. Pass the
+  `--config.enable-read-api` flag at the command line to opt in to these
+  endpoints.
+
 # v0.20.0 (2021-10-28)
 
 - [FEATURE] Operator: The Grafana Agent Operator can now generate a Kubelet

diff --git a/cmd/agent/entrypoint.go b/cmd/agent/entrypoint.go
@@ -202,13 +202,19 @@ func (ep *Entrypoint) wire(mux *mux.Router, grpc *grpc.Server) {
 
 	mux.HandleFunc("/-/config", func(rw http.ResponseWriter, r *http.Request) {
 		ep.mut.Lock()
-		bb, err := yaml.Marshal(ep.cfg)
+		cfg := ep.cfg
 		ep.mut.Unlock()
 
-		if err != nil {
-			http.Error(rw, fmt.Sprintf("failed to marshal config: %s", err), http.StatusInternalServerError)
+		if cfg.EnableConfigEndpoints {
+			bb, err := yaml.Marshal(cfg)
+			if err != nil {
+				http.Error(rw, fmt.Sprintf("failed to marshal config: %s", err), http.StatusInternalServerError)
+			} else {
+				_, _ = rw.Write(bb)
+			}
 		} else {
-			_, _ = rw.Write(bb)
+			rw.WriteHeader(http.StatusNotFound)
+			_, _ = rw.Write([]byte("404 - config endpoint is disabled"))
 		}
 	})
 

diff --git a/docs/_index.md b/docs/_index.md
@@ -1,11 +1,11 @@
 +++
-title = "Grafana Agent Documentation"
+title = "Grafana Agent documentation"
 weight = 1
 +++
 
 # Grafana Agent
 
-Grafana Agent is an telemetry collector for sending metrics, logs,
+Grafana Agent is a telemetry collector for sending metrics, logs,
 and trace data to the opinionated Grafana observability stack. It works best
 with:
 

diff --git a/docs/configuration/_index.md b/docs/configuration/_index.md
@@ -115,3 +115,15 @@ Support contents and default values of `agent.yaml`:
 # Configures integrations for the Agent.
 [integrations: <integrations_config>]
 ```
+
+## Remote Configuration (Beta)
+
+An experimental feature for fetching remote configuration files over HTTP/S can be
+enabled by passing the `-enable-features=remote-configs` flag at the command line.
+With this feature enabled, you may pass an HTTP/S URL to the `-config.file` flag.
+
+The following flags will configure basic auth for requests made to HTTP/S remote config URLs:
+- `-config.url.basic-auth-user <user>`: the basic auth username
+- `-config.url.basic-auth-password-file <file>`: path to a file containing the basic auth password
+
+Note that this beta feature is subject to change in future releases.
diff --git a/docs/configuration/integrations/cadvisor-config.md b/docs/configuration/integrations/cadvisor-config.md
@@ -0,0 +1,116 @@
++++
+title = "cadvisor_config"
++++
+
+# cadvisor_config
+
+The `cadvisor_config` block configures the `cadvisor` integration,
+which is an embedded version of
+[`cadvisor`](https://github.com/google/cadvisor). This allows for the collection of container utilization metrics.
+
+The cAdvisor integration requires some broad privileged permissions to the host. Without these permissions the metrics will not be accessible. This means that the agent must *also* have those elevated permissions.
+
+A good example of the required file, and system permissions can be found in the docker run command published in the [cAdvisor docs](https://github.com/google/cadvisor#quick-start-running-cadvisor-in-a-docker-container).
+
+Full reference of options:
+
+```yaml
+  # Enables the cadvisor integration, allowing the Agent to automatically
+  # collect metrics for the specified github objects.
+  #
+  # Enabled is DEPRECATED and will be removed in a future release. To disable
+  # an integration, comment it out or remove it from your config instead of
+  # setting `enabled: false`.
+  [enabled: <boolean> | default = true]
+
+  # Sets an explicit value for the instance label when the integration is
+  # self-scraped. Overrides inferred values.
+  [instance: <string> | default = <integrations_config.instance>]
+
+  # Automatically collect metrics from this integration. If disabled,
+  # the cadvisor integration will be run but not scraped and thus not
+  # remote-written. Metrics for the integration will be exposed at
+  # /integrations/cadvisor/metrics and can be scraped by an external
+  # process.
+  [scrape_integration: <boolean> | default = <integrations_config.scrape_integrations>]
+
+  # How often should the metrics be collected? Defaults to
+  # prometheus.global.scrape_interval.
+  [scrape_interval: <duration> | default = <global_config.scrape_interval>]
+
+  # The timeout before considering the scrape a failure. Defaults to
+  # prometheus.global.scrape_timeout.
+  [scrape_timeout: <duration> | default = <global_config.scrape_timeout>]
+
+  # Allows for relabeling labels on the target.
+  relabel_configs:
+    [- <relabel_config> ... ]
+
+  # Relabel metrics coming from the integration, allowing to drop series
+  # from the integration that you don't care about.
+  metric_relabel_configs:
+    [ - <relabel_config> ... ]
+
+  # How frequent to truncate the WAL for this integration.
+  [wal_truncate_frequency: <duration> | default = "60m"]
+
+  #
+  # cAdvisor-specific configuration options
+  #
+
+  # Convert container labels and environment variables into labels on prometheus metrics for each container. If false, then only metrics exported are container name, first alias, and image name.
+  [store_container_labels: <boolean> | default = true]
+
+  # List of container labels to be converted to labels on prometheus metrics for each container. store_container_labels must be set to false for this to take effect.
+  allowlisted_container_labels:
+    [ - <string> ]
+
+  # List of environment variable keys matched with specified prefix that needs to be collected for containers, only support containerd and docker runtime for now.
+  env_metadata_allowlist:
+    [ - <string> ]
+
+  # List of cgroup path prefix that needs to be collected even when docker_only is specified.
+  raw_cgroup_prefix_allowlist:
+    [ - <string> ]
+
+  # Path to a JSON file containing configuration of perf events to measure. Empty value disabled perf events measuring.
+  [perf_events_config: <boolean>]
+
+  # resctrl mon groups updating interval. Zero value disables updating mon groups.
+  [resctrl_interval: <int> | default = 0]
+
+  # List of `metrics` to be disabled.
+  disabled_metrics:
+    [ - <string> ]
+
+  # List of `metrics` to be enabled. If set, overrides disabled_metrics
+  enabled_metrics:
+    [ - <string> ]
+
+  # Length of time to keep data stored in memory
+  [storage_duration: <duration> | default = "2m"]
+
+  # Containerd endpoint
+  [containerd: <string> | default = "/run/containerd/containerd.sock"]
+
+  # Containerd namespace
+  [containerd_namespace: <string> | default = "k8s.io"]
+
+  # Docker endpoint
+  [docker: <string> | default = "unix:///var/run/docker.sock"]
+
+  # Use TLS to connect to docker
+  [docker_tls: <boolean> | default = false]
+
+  # Path to client certificate for TLS connection to docker
+  [docker_tls_cert: <string> | default = "cert.pem"]
+
+  # Path to private key for TLS connection to docker
+  [docker_tls_key: <string> | default = "key.pem"]
+
+  # Path to a trusted CA for TLS connection to docker
+  [docker_tls_ca: <string> | default = "ca.pem"]
+
+  # Only report docker containers in addition to root stats
+  [docker_only: <boolean> | default = false]
+```
diff --git a/docs/configuration/integrations/node-exporter-config.md b/docs/configuration/integrations/node-exporter-config.md
@@ -26,7 +26,7 @@ docker run \
   -v "/proc:/host/proc:ro,rslave" \
   -v /tmp/agent:/etc/agent \
   -v /path/to/config.yaml:/etc/agent-config/agent.yaml \
-  grafana/agent:v0.21.1 \
+  grafana/agent:v0.21.2 \
   --config.file=/etc/agent-config/agent.yaml
 ```
 
@@ -65,7 +65,7 @@ metadata:
   name: agent
 spec:
   containers:
-  - image: grafana/agent:v0.21.1
+  - image: grafana/agent:v0.21.2
     name: agent
     args:
     - --config.file=/etc/agent-config/agent.yaml

diff --git a/docs/configuration/integrations/process-exporter-config.md b/docs/configuration/integrations/process-exporter-config.md
@@ -18,7 +18,7 @@ docker run \
   -v "/proc:/proc:ro" \
   -v /tmp/agent:/etc/agent \
   -v /path/to/config.yaml:/etc/agent-config/agent.yaml \
-  grafana/agent:v0.21.1 \
+  grafana/agent:v0.21.2 \
   --config.file=/etc/agent-config/agent.yaml
 ```
 
@@ -35,7 +35,7 @@ metadata:
   name: agent
 spec:
   containers:
-  - image: grafana/agent:v0.21.1
+  - image: grafana/agent:v0.21.2
     name: agent
     args:
     - --config.file=/etc/agent-config/agent.yaml

diff --git a/docs/configuration/traces-config.md b/docs/configuration/traces-config.md
@@ -145,6 +145,26 @@ scrape_configs:
 # `update` only modifies an existing k/v and `insert` only appends if the k/v
 # is not present. `upsert` does both.
 [ prom_sd_operation_type: <string> | default = "upsert" ]
+# Configures what methods to use to do association between spans and pods.
+# PromSD processor matches the IP address of the metadata labels from the k8s API
+# with the IP address obtained from the specified pod association method.
+# If a match is found then the span is labeled.
+#
+# Options are `ip`, `net.host.ip`, `k8s.pod.ip`, `hostname` and `connection`.
+#   - `ip`, `net.host.ip` and `k8s.pod.ip`, `hostname` match spans tags.
+#   - `connection` inspects the context from the incoming requests (gRPC and HTTP).
+#
+# Tracing instrumentation is commonly the responsible for tagging spans
+# with IP address to the labels mentioned above.
+# If running on kubernetes, `k8s.pod.ip` can be automatically attached via the
+# downward API. For example, if you're using OTel instrumentation libraries, set 
+# OTEL_RESOURCE_ATTRIBUTES=k8s.pod.ip=$(POD_IP) to inject spans with the sender
+# pod's IP.
+#
+# By default, all methods are enabled, and evaluated in the order specified above.
+# Order of evaluation is honored when multiple methods are enabled.
+prom_sd_pod_association: 
+  - [ <string>... ]
 
 # spanmetrics supports aggregating Request, Error and Duration (R.E.D) metrics
 # from span data.

diff --git a/docs/getting-started/_index.md b/docs/getting-started/_index.md
@@ -26,7 +26,7 @@ See the list of [Community Projects](#community-projects) for the community-driv
 docker run \
   -v /tmp/agent:/etc/agent/data \
   -v /path/to/config.yaml:/etc/agent/agent.yaml \
-  grafana/agent:v0.21.1
+  grafana/agent:v0.21.2
 ```
 
 Replace `/tmp/agent` with the folder you wish to store WAL data in. WAL data is

diff --git a/docs/operator/custom-resource-quickstart.md b/docs/operator/custom-resource-quickstart.md
@@ -43,7 +43,7 @@ metadata:
   labels:
     app: grafana-agent
 spec:
-  image: grafana/agent:v0.21.1
+  image: grafana/agent:v0.21.2
   logLevel: info
   serviceAccountName: grafana-agent
   metrics:

diff --git a/docs/operator/getting-started.md b/docs/operator/getting-started.md
@@ -72,7 +72,7 @@ spec:
       serviceAccountName: grafana-agent-operator
       containers:
       - name: operator
-        image: grafana/agent-operator:v0.21.1
+        image: grafana/agent-operator:v0.21.2
         args:
         - --kubelet-service=default/kubelet
 ---

diff --git a/docs/upgrade-guide/_index.md b/docs/upgrade-guide/_index.md
@@ -65,6 +65,16 @@ integrations:
   agent: {}
 ```
 
+## v0.21.2, v0.20.1
+
+### Disabling of config retrieval enpoints
+
+These two patch releases, as part of a fix for
+[CVE-2021-41090](https://github.com/grafana/agent/security/advisories/GHSA-9c4x-5hgq-q3wh),
+disable the `/-/config` and `/agent/api/v1/configs/{name}` endpoints by
+default. Pass the `--config.enable-read-api` flag at the command line to
+re-enable them.
+
 ## v0.21.0
 
 ### Integrations: Change in how instance labels are handled (Breaking change)

diff --git a/example/docker-compose/agent/config/agent-local.yaml b/example/docker-compose/agent/config/agent-local.yaml
@@ -72,3 +72,8 @@ integrations:
     - source_labels: [__address__]
       target_label: mongodb_cluster
       replacement: 'mongodb-cluster'
+  cadvisor:
+    disabled_metrics:
+    - disk
+    enabled_metrics:
+    - percpu