7.22.1

Prelude

Release on: 2020-09-17

• Please refer to the 7.22.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Define a default logs file (security-agent.log) for the security-agent.
• Fix segfault when listing Garden containers that are in error state.
• Do not activate security-agent service by default in the Linux packages of the Agent (RPM/DEB).
  The security-agent was already properly starting and exiting if not activated in configuration.

6.22.1

6.22.1 ships the same features as 7.22.1 except for the Python versions it supports.

Please refer to the 7.22.1 changelog.

7.22.0

Release Notes

7.22.0

Prelude

Release on: 2020-08-25

• Please refer to the 7.22.0 tag on
  integrations-core
  for the list of changes on the Core Checks

New Features

• Implements agent-side compliance rule evaluation in security agent
  using expressions.
• Add IO operations monitoring for Docker check
  (docker.io.read/write_operations)
• Track TCP connection churn on system-probe
• The new Runtime Security Agent collects file integrity monitoring
  events. It is disabled by default and only available for Linux for
  now.
• Make security-agent part of automatically started agents in
  RPM/DEB/etc. packages (will do nothing and exit 0 by default)
• Add support for receiving and processing SNMP traps, and forwarding
  them as logs to Datadog.
• APM: A new trace ingestion endpoint was introduced at /v0.5/traces which
  supports a more compact payload format, greatly improving resource usage.
  The spec for the new wire format can be viewed at here.
  Tracers supporting this change, will automatically use the new endpoint.

Enhancement Notes

• Adds a gauge for system.mem.slab_reclaimable. This is part
  of slab memory that might be reclaimed (i.e. caches). Datadog 7.x
  adds SReclaimable memory, if
  available on the system, to the system.mem.cached gauge by default. This
  may lead to inconsistent metrics for clients migrating from Datadog
  5.x, where system.mem.cached didn't
  include SReclaimable memory. Adding a
  gauge for system.mem.slab_reclaimable allows inverse
  calculation to remove this value from the system.mem.cached gauge.

• Expand GCR pause container image filter

• Kubernetes events for pods, replicasets and deployments now have
  tags that match the metrics metadata. Namely, pod_name, kube_deployment, kube_replicas_set.

• Enabled the collection of the kubernetes resource requirements
  (requests and limits) by bumping the agent-payload dep. and
  collecting the resource requirements.

• Implements resource fallbacks for complex compliance check
  assertions.

• Add system.cpu.num_cores metric with the number of CPU cores
  (windows/linux)

• compliance: Add support for Go custom compliance checks and
  implement two for CIS Kubernetes

• Make DSD Mapper also map metrics that already contain tags.

• If the retrieval of the AWS EC2 instance ID or hostname fails,
  previously-retrieved values are now sent, which should mitigate host
  aliases flapping issues in-app.

• Increase default timeout on AWS EC2 metadata endpoints, and make it
  configurable with ec2_metadata_timeout

• Add container incl./excl. lists support for ECS Fargate
  (process-agent)

• Adds support for a heap profile and cpu profile (of configurable
  length) to be created and included in the flare.

• Upgrade embedded Python 3 to 3.8.5. Link to Python 3.8 changelog:
  https://docs.python.org/3/whatsnew/3.8.html

  Note that the Python 2 version shipped in Agent v6 continues to be
  version 2.7.18 (unchanged).

• Upgrade pip to v20.1.1. Link to pip 20.1.1 changelog:
  https://pip.pypa.io/en/stable/news/#id54

• Upgrade pip-tools to v5.3.1. Link to pip-tools 5.3.1 changelog:
  https://github.com/jazzband/pip-tools/blob/master/CHANGELOG.md

• Introduces support for resolving pathFrom from in File and Audit
  checks.

• On Windows, always add the user to the required groups during
  installation.

• APM: A series of changes to internal algorithms were made which reduced
  CPU usage between 20-40% based on throughput.

Bug Fixes

• Allow integration commands to work for pre-release versions.
• [Windows] Ensure PYTHONPATH variable is ignored correctly when
  initializing the Python runtime.
• Enable listening for conntrack info from all namespaces in system
  probe
• Fix cases where the resolution of secrets in integration configs
  would not be performed for autodiscovered containers.
• Fixes submission of containers blkio metrics that may modify array
  after being already used by aggregator. Can cause missing tags on
  containerd.* metrics
• Restore support of JSON-formatted lists for configuration options
  passed as environment variables.
• Don't allow pressing the disable button on checks twice.
• Fix container_include_metrics
  support for all container checks
• Fix a bug where the Agent disables collecting tags when the cluster
  checks advanced dispatching is enabled in the Daemonset Agent.
• Fixes a bug where the ECS metadata endpoint V2 would get queried
  even though it was not configured with the configuration option
  cloud_provider_metadata.
• Fix a bug when a kubernetes job has exited after some time the
  tagger does not update it even if it did change its state.
• Fixes the Agent failing to start on sysvinit on systems with
  dpkg >= 1.19.3
• The agent was collecting docker container logs (metrics) even if
  they are matching DD_CONTAINER_EXCLUDE_LOGS (resp. DD_CONTAINER_EXCLUDE_METRICS) if they
  were started before the agent. This is now fixed.
• Fix a bug where the Agent would not remove tags for pods that no
  longer exist, potentially causing unbounded memory growth.
• Fix pidfile support on security-agent
• Fixed system-probe not working on CentOS/RHEL 8 due to our custom
  SELinux policy. We now install the custom policy only on CentOS/RHEL
  7, where the system-probe is known not to work with the default. On
  other platform the default will be used.
• Stop sending payload for Cloud Foundry applications containers that
  have no container_name tag attached
  to avoid them showing up in the UI with empty name.

Other Notes

• APM: datadog.trace_agent.receiver.* metrics are now also tagged by
  endpoint_version

6.22.0

6.22.0 ships the same features as 7.22.0 except for the Python versions it supports.

Please refer to the 7.22.0 changelog.

7.21.1

Release Notes

7.21.1

Prelude

Release on: 2020-07-22

Bug Fixes

• JMXFetch upgraded to 0.38.2 to fix Java 7 support.
• Fix init of security-agent - exit properly when no feature requiring it is activated and avoid conflicting with core agent port bindings.

6.21.1

6.21.1 ships the same features as 7.21.1 except for the Python versions it supports.

Please refer to the 7.21.1 changelog.

7.21.0

Prelude

Release on: 2020-07-16

• Please refer to the 7.21.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• APM: The maximum allowed payload size by the agent was increased
  from 10MB to 50MB. This could result in traffic increases for users
  which were affected by this issue.
• APM: The maximum connection limit over a 30s period was removed.
  This can result in an increase of tracing data for users that were
  affected by this limitation.

New Features

• Add support of new DatadogMetric CRD in DCA. Allows to autoscale
  based on any valid Datadog query.
• Add packages scripts for dogstatsd that have the same features as
  the agent: create symlink for binary, create dd-agent user and
  group, setup the service and cleanup those when uninstalling.
• Adds OOM Kill probe to ebpf package and corresponding corecheck to
  the agent.
• The Datadog IoT Agent is now available for 32 bit ARM architecture
  (armv7l/armhf).
• Add Compliance agent in Cluster Agent to monitor Kubernetes objects
• Add docker.cpu.limit and containerd.cpu.limit metrics, reporting
  maximum cpu time (hz or ns) available for each container based on
  their limits. (Only supported on Linux)
• Addition of a gRPC server and a hostname resolution endpoint,
  including a grpc-gateway that exposes said endpoint as a REST
  service.
• Adding a 'log_format_rfc3339' option to use the RFC3339 format for
  the log time.
• Compliance Agent implementing scheduling of compliance checks for
  Docker and Kubernetes benchmarks.
• Expose agent's sql obfuscation to python checks via new
  datadog_agent.obfuscate_sql method
• Support installing non-core integrations with the integration
  command, such as those located in the integrations-extras
  repository.

Enhancement Notes

• The Agent status command now includes the flavor of the Agent that
  is running.
• The Agent GUI now includes the flavor of the Agent that is running.
• Adds Tagger information to Datadog Agent flare for support
  investigations.
• Add a static collector in the tagger package for tags that do not
  change after pod start (such as those from an environment variable).
• Add autodiscovery_subnet to available SNMP template extra configs
• When enabling collect_ec2_tags or collect_gce_tags option,
  EC2/GCE tags are now cached to avoid missing tags when user exceed
  his AWS/GCE quotas.
• Chocolatey package can be installed on Domain Controller
• The Agent now collects the Availability Zone a Fargate Task (using
  platform version 1.4 or later) is running in as an
  "availability_zone" tag.
• Enabled the collection of the init-containers by bumping the
  agent-payload dep. and collecting the init-containers.
• The Agent now collects recommended "app.kubernetes.io" Kubernetes
  labels as tags by default, and exposes them under a "kube_app"
  prefix.
• Docker and Containerd checks now support filtering containers by
  kube_namespace.
• Add support for sampling to distribution metrics
• Flare now includes the permission information for parents of config
  and log file directories.
• Collect processes namespaced PID.
• You can now enable or disable the dogstatsd-stats troubleshooting
  feature at runtime using the config set dogstatsd_stats command of
  the Agent.
• API Keys are now sanitized for logs_config and
  additional_endpoints.
• Upgrade gosnmp to support more authentication and privacy protocols
  for v3 connections.
• Use the standard tag 'service' as a log collection attribute for
  container's logs collected from both kubernetes and docker log
  sources.
• agent check returns non zero exit code when trace malloc is enabled
  (tracemalloc_debug: true) when using python 2
• Added the checksum type to the checksum key itself, as it is
  deprecated to have a separate checksum_type key.
• Add lowercase_device_tag option to the system io core check on
  Windows. When enabled, sends metrics with a lowercased device tag,
  which is consistent with the system.io.* metrics of Agent v5 and
  the system.disk.* metrics of all Agent versions.

Bug Fixes

• Fix missing values from cluster-agent status command.

• Add missing device_name tag in iostats_pdh

• Fixes an issue where DD_TAGS were not applied to EKS Fargate pods
  and containers.

• Add freetds linux dep needed for SQL Server to run in Docker
  Agent.

• APM : Fix parsing of non-ASCII numerals in the SQL obfuscator.
  Previously unicode characters for which unicode.IsDigit returns true
  could cause a hang in the SQL obfuscator

• APM: correctly obfuscate AUTH command.

• Dogstatsd standalone: when running on a systemd-based system, do not
  stop Dogstatsd when journald is stopped or restarted.

• Fix missing logs and metrics for docker-labels based autodiscovery
  configs after container restart.

• Fix bugs introduced in 7.20.0/6.20.0 in the Agent 5 configuration
  import command: the command would not import some Agent config
  settings, including api_key, and would write some Docker &
  Kubernetes config settings to wrongly-located files.

• Fixes tag extraction from Kubernetes pod labels when using patterns
  on certain non-alphanumeric label names (e.g.
  app.kubernetes.io/managed-by).

• Fixes the /ready health endpoint on the cluster-agent.

  The /ready health endpoint was reporting 200 at the cluster-agent
  startup and was then, permanently reporting 500 even though the
  cluster-agent was experiencing no problem. In the body of the
  response, we could see that a healthcheck component was failing.
  This change fixes this issue.

• This fix aims to cover the case when the agent is running inside GKE
  with workload identity enabled. If workload identity is enabled,
  access to /instance/name is forbidden, resulting into an empty host
  alias.

• Fix hostname resolution issue preventing the Process and APM agents
  from picking up a valid hostname on some containerized environments

• Fix a bug which causes certain configuration options to be ignored
  by the process-agent in the presence of a system-probe.yaml.

• Process agent and system probe now correctly accept multiple API
  keys per endpoint.

• The device_name tag is not used anymore to populate the Device
  field of a series. Only the device tag is considered.

• Fixes problem on Windows where ddagentuser home directory is left
  behind.

• Revert upgrade of GoSNMP and addition of extra authentication
  protocols.

• Add support for examining processes inside Docker containers running
  under systemd cgroups. This also reduces agent logging volume as
  it's able to capture those statistics going forward.

• APM: The agent now exits with code 0 when the API key is not
  specified. This is so to prevent the Windows SCM from restarting the
  process.

Other Notes

• All Agents binaries are now compiled with Go 1.13.11.
• In Debug mode, DogStatsD log a warning message when a burst of
  metrics is detected.
• JMXFetch upgraded to
  0.38.0
• JQuery, used in the web-based agent GUI, has been upgraded to 3.5.1

6.21.0

6.21.0 ships the same features as 7.21.0 except for the Python versions it supports.

Please refer to the 7.21.0 changelog.

7.20.2

Prelude

Release on: 2020-06-16

• Please refer to the 7.20.2 tag on integrations-core for the list of changes on the Core Checks

7.20.1

Prelude

Release on: 2020-06-11

• Please refer to the 7.20.1 tag on integrations-core for the list of changes on the Core Checks