Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DD_CONTAINER_EXCLUDE_LOGS excludes matching like namespaces #471 #10204

Closed
sadok-f opened this issue Dec 13, 2021 · 1 comment
Closed

DD_CONTAINER_EXCLUDE_LOGS excludes matching like namespaces #471 #10204

sadok-f opened this issue Dec 13, 2021 · 1 comment

Comments

@sadok-f
Copy link

sadok-f commented Dec 13, 2021

Output of the info page (if this is a bug)

root@datadog-xt2h9:/# agent status
2021-12-13 20:03:24 UTC | CORE | WARN | (pkg/util/log/log.go:630 in func1) | Deactivating Autoconfig will disable most components. It's recommended to use autoconfig_exclude_features and autoconfig_include_features to activate/deactivate features selectively
Getting the status from the agent.

2021-12-13 20:03:24 UTC | CORE | INFO | (cmd/system-probe/config/config.go:172 in load) | network_config.enabled detected: enabling system-probe with network module running.
===============
Agent (v7.32.1)
===============

  Status date: 2021-12-13 20:03:24.525 UTC (1639425804525)
  Agent start: 2021-12-13 16:22:34.644 UTC (1639412554644)
  Pid: 1
  Go Version: go1.16.7
  Python Version: 3.8.11
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: INFO

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: -400µs
    System time: 2021-12-13 20:03:24.525 UTC (1639425804525)

  Host Info
  =========
    bootTime: 2021-10-20 12:19:31 UTC (1634732371000)
    kernelArch: x86_64
    kernelVersion: 5.8.0-1041-aws
    os: linux
    platform: ubuntu
    platformFamily: debian
    platformVersion: 21.04
    procs: 286
    uptime: 1300h3m11s
    virtualizationRole: guest
    virtualizationSystem: docker

  Hostnames
  =========
    ec2-hostname: xxxx
    host_aliases: xxxx
    hostname: i-xx
    instance-id: i-xx
    socket-fqdn: datadog-xt2h9
    socket-hostname: datadog-xt2h9
    host tags:
      "env:production"
      kube_node_role:node
    hostname provider: aws
    unused hostname providers:
      azure: azure_hostname_style is set to 'os'
      configuration/environment: hostname is empty
      gce: unable to retrieve hostname from GCE: status code 404 trying to GET http://xxx/computeMetadata/v1/instance/hostname

  Metadata
  ========
    cloud_provider: AWS
    hostname_source: aws

=========
Collector
=========

  Running Checks
  ==============

    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 883
      Metric Samples: Last Run: 9, Total: 7,940
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-13 20:03:11 UTC (1639425791000)
      Last Successful Execution Date : 2021-12-13 20:03:11 UTC (1639425791000)


    disk (4.4.0)
    ------------
      Instance ID: disk:exxxx [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml.default
      Total Runs: 883
      Metric Samples: Last Run: 372, Total: 328,476
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 41ms
      Last Execution Date : 2021-12-13 20:03:18 UTC (1639425798000)
      Last Successful Execution Date : 2021-12-13 20:03:18 UTC (1639425798000)


    docker
    ------
      Instance ID: docker [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 882
      Metric Samples: Last Run: 206, Total: 181,692
      Events: Last Run: 1, Total: 1
      Service Checks: Last Run: 1, Total: 882
      Average Execution Time : 38ms
      Last Execution Date : 2021-12-13 20:03:10 UTC (1639425790000)
      Last Successful Execution Date : 2021-12-13 20:03:10 UTC (1639425790000)


    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
      Total Runs: 883
      Metric Samples: Last Run: 5, Total: 4,415
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-13 20:03:17 UTC (1639425797000)
      Last Successful Execution Date : 2021-12-13 20:03:17 UTC (1639425797000)


    io
    --
      Instance ID: io [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
      Total Runs: 882
      Metric Samples: Last Run: 208, Total: 183,312
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-13 20:03:09 UTC (1639425789000)
      Last Successful Execution Date : 2021-12-13 20:03:09 UTC (1639425789000)


    kubelet (7.1.0)
    ---------------
      Instance ID: kubelet:5bbc63f3938c02f4 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/kubelet.d/conf.yaml.default
      Total Runs: 663
      Metric Samples: Last Run: 549, Total: 363,979
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 4, Total: 2,652
      Average Execution Time : 241ms
      Last Execution Date : 2021-12-13 20:03:21 UTC (1639425801000)
      Last Successful Execution Date : 2021-12-13 20:03:21 UTC (1639425801000)


    load
    ----
      Instance ID: load [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
      Total Runs: 883
      Metric Samples: Last Run: 6, Total: 5,298
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-13 20:03:16 UTC (1639425796000)
      Last Successful Execution Date : 2021-12-13 20:03:16 UTC (1639425796000)


    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
      Total Runs: 883
      Metric Samples: Last Run: 18, Total: 15,894
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-13 20:03:23 UTC (1639425803000)
      Last Successful Execution Date : 2021-12-13 20:03:23 UTC (1639425803000)


    network (2.4.0)
    ---------------
      Instance ID: network:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
      Total Runs: 883
      Metric Samples: Last Run: 55, Total: 48,565
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 3ms
      Last Execution Date : 2021-12-13 20:03:15 UTC (1639425795000)
      Last Successful Execution Date : 2021-12-13 20:03:15 UTC (1639425795000)


    ntp
    ---
      Instance ID: ntp:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/ntp.d/conf.yaml.default
      Total Runs: 15
      Metric Samples: Last Run: 1, Total: 15
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 15
      Average Execution Time : 0s
      Last Execution Date : 2021-12-13 19:52:41 UTC (1639425161000)
      Last Successful Execution Date : 2021-12-13 19:52:41 UTC (1639425161000)


    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 883
      Metric Samples: Last Run: 1, Total: 883
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2021-12-13 20:03:22 UTC (1639425802000)
      Last Successful Execution Date : 2021-12-13 20:03:22 UTC (1639425802000)

========
JMXFetch
========

  Information
  ==================
  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    Cluster: 0
    ClusterRole: 0
    ClusterRoleBinding: 0
    CronJob: 0
    DaemonSet: 0
    Deployment: 0
    Dropped: 0
    HighPriorityQueueFull: 0
    Job: 0
    Node: 0
    PersistentVolume: 0
    PersistentVolumeClaim: 0
    Pod: 0
    ReplicaSet: 0
    Requeued: 1
    Retried: 1
    RetryQueueSize: 0
    Role: 0
    RoleBinding: 0
    Service: 0
    ServiceAccount: 0
    StatefulSet: 0

  Transaction Successes
  =====================
    Total number: 1864
    Successes By Endpoint:
      check_run_v1: 883
      intake: 98
      series_v1: 883

  Transaction Errors
  ==================
    Total number: 1
    Errors By Type:

  API Keys status
  ===============
    API key ending with xxxx: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.eu - API Key ending with:
      - xxxx

==========
Logs Agent
==========

    Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.eu on port 443
    BytesSent: 1.3538048e+07
    EncodedBytesSent: 1.778179e+06
    LogsProcessed: 11318
    LogsSent: 11318

  kube-system/kube-proxy-xxxxx.eu-central-1.compute.internal/kube-proxy
  -------------------------------------------------------------------------------
    - Type: file
      Identifier: fb8a5e91607d146b49555cb0a5513dbafe00f21cfe05433bd5c8626e10b56eff
      Path: /var/log/pods/kube-system_kube-proxy-ip-xxxx.eu-central-1.compute.internal_f871400f937e6cb8d8b323b4e1281121/kube-proxy/*.log
      Status: OK
        1 files tailed out of 1 files matching
      Inputs:
        /var/log/pods/kube-system_kube-proxy-ip-xxxx.eu-central-1.compute.internal_f871400f937e6cb8d8b323b4e1281121/kube-proxy/0.log
      BytesRead: 8094
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 11
      24h Peak Latency (ms): 11

  container_collect_all
  ---------------------
    - Type: docker
      Status: Pending
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  datadog/datadog-xt2h9/process-agent
  -----------------------------------
    - Type: file
      Identifier: d7023d1aaaf08ea472e94b11a2c19847156fc20be48ffbb98f7cdd42d5a14a36
      Path: /var/log/pods/datadog_datadog-xt2h9_c33d07b5-e3b9-4f34-8391-ed08d3950617/process-agent/*.log
      Status: Pending
        1 files tailed out of 1 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0



============
System Probe
============
System Probe is running



=========
Aggregator
=========
  Checks Metric Sample: 1,157,715
  Dogstatsd Metric Sample: 208,049
  Event: 2
  Events Flushed: 2
  Number Of Flushes: 883
  Series Flushed: 1,030,670
  Service Check: 12,172
  Service Checks Flushed: 13,052
=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 208,048
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 20,564,669
  Udp Packet Reading Errors: 0
  Udp Packets: 128,750
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 1
  Unterminated Metric Errors: 0

=====================
Datadog Cluster Agent
=====================

  - Datadog Cluster Agent endpoint detected: https://xxxx:5005
  Successfully connected to the Datadog Cluster Agent.
  - Running: 1.16.0+commit.9961689

=============
Autodiscovery
=============
  Enabled Features
  ================
    docker
    kubernetes```

**Describe what happened:**

We're using DD agent in our k8s cluster hosted on AWS, we have the following exclude rules:

DD_CONTAINER_EXCLUDE_LOGS="kube_namespace:logging kube_namespace:monitoring"

We want to exclude logs from logging and monitoring namespaces, the issue appears for other namespaces called processing-logging and processing-monitoring as we want to fetch logs from those namespaces as well, but it seems
DD-agent configuration excludes them because of the DD_CONTAINER_EXCLUDE_LOGS flag.

**Describe what you expected:**
Only logs from namespaces that match exactly the flag DD_CONTAINER_EXCLUDE_LOGS should be excluded, unless there's another workaround for this.

**Steps to reproduce the issue:**

DD_CONTAINER_EXCLUDE_LOGS="kube_namespace:login"
namespace logging : pod test_pod_1
namespace processing-logging: : pod test_pod_2
namespace test : pod test_pod_3

Only logs from the test namespace is appearing in the DD dashboard.


**Additional environment details (Operating System, Cloud provider, etc):**
AWS cloud provider
Agent version: `Agent 7.32.1 - Commit: 52f034f - Serialization version: v4.85.0 - Go version: go1.16.7`
Datadog Helm version: `2.27.5`

Thank you
@sadok-f sadok-f closed this as completed Dec 14, 2021
@sadok-f
Copy link
Author

sadok-f commented Dec 14, 2021

resolved in dd-helm repo comment :
DataDog/helm-charts#471 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sadok-f