pods metadata is missing in the logs

[adil-jeel]

The logging part works but all the meta data is missing. the deployment was perform using default manifest using terraform which supposed to have meta data.

      # Filter to add kubernetes metadata
      <filter oci.oke.containerlogs.**>
         @type kubernetes_metadata
         @id filter_kube_metadata_containerlogs
         verify_ssl "true"
         skip_labels "false"
         skip_container_metadata "false"
         skip_master_url "false"
         skip_namespace_metadata "false"
         watch "true"
         de_dot false
         annotation_match [ ".*" ]
      </filter>

and here is the logs

{
  "datetime": 1689052825419,
  "logContent": {
    "id": "f737c3ce-72ae-45dc-bafa-a156109dd16d",
    "time": "2023-07-11T05:20:25.419Z",
    "oracle": {
      "compartmentid": "example",
      "ingestedtime": "2023-07-11T05:20:26.989752189Z",
      "instanceid": "example",
      "loggroupid": "example",
      "logid": "example",
      "tenantid": "example"
    },
    "source": "oke-cxn7k3krqoa-nd7boamq3oq-se3jtdyyvla-0",
    "specversion": "1.0",
    "subject": "/var/log/pods/fe-appcbd6dfb77-7prtq_7e47791b-8f88-4edf-a8b8-eb7b76e13c10/fe-app/0.log",
    "type": "com.oraclecloud.logging.custom.demo-oke-logs",
    "data": {
      "message": "2023-07-11T05:20:25.274670261+00:00 stdout F 10.0.11.254 - - [11/Jul/2023:05:20:25 +0000] \"GET / HTTP/1.1\" 200 363088 \"-\" \"kube-probe/1.26\" \"-\"",
      "tailed_path": "/var/log/pods/fe-app-cbd6dfb77-7prtq_7e47791b-8f88-4edf-a8b8-eb7b76e13c10/fe-app/0.log"
    }
  },
  "regionId": "me-jeddah-1"
}

[santhoshkvuda]

Based on the log u posted, It looks like you are using a different solution of OCI (OCI Logging) to collect your logs but not the solution offered through this repo. Please check with respective team.

[adil-jeel]

I used the exact repo using terraform, might need some extra configuration ?
here ist he terragrunt file

terraform {
  source = "github.com/oracle-quickstart/oci-kubernetes-monitoring.git//terraform/oke?ref=v3.0.0"
}
locals {
  common-vars =read_terragrunt_config("${get_repo_root()}/common-vars.json")
  environment-vars = read_terragrunt_config("${get_repo_root()}/${local.common-vars.locals.environment}/${ local.common-vars.locals.region}/environment-vars.json")
}

dependency "oke" {
  config_path = "../oke/"
}


inputs = {
    oke_cluster_ocid      = dependency.oke.outputs.cluster_id
    user_ocid             = local.common-vars.locals.user_ocid
    current_user_ocid     = local.common-vars.locals.user_ocid
    tenancy_ocid          = local.common-vars.locals.tenancy_ocid
    region                = local.common-vars.locals.region
    private_key_path      = local.common-vars.locals.private_key_path
    fingerprint           = local.common-vars.locals.fingerprint
    oke_compartment_ocid  = local.common-vars.locals.compartment_id
    kubernetes_namespace  = "oci-onm"
    opt_deploy_metric_server = true
    oci_onm_compartment_ocid = local.common-vars.locals.compartment_id
    opt_create_new_la_logGroup = false
    oci_onm_compartment_ocid  = local.common-vars.locals.compartment_id
    oci_la_logGroup_name = "oke"
    opt_use_existing_la_logGroup=true
    oci_la_logGroup_id = "demo-oke"
    opt_create_dynamicGroup_and_policies = true
    fluentd_baseDir_path = "/var/log"
    enable_dashboard_module  = true
    enable_helm_module       = true
}

Am i missing anything?
thanks

[adil-jeel]

[santhoshkvuda]

hi,
The screenshot that you have shared above for logs is not reflecting the same. The log format is matching to OCI Logging
(When you collect logs to OCI LA using a Service Connector from OCI Logging) and they are not collected using this solution.
Please check in OCI LA Log Explorer for logs corresponding to "Kubernetes Container Generic Logs" Log Source (the default/generic source for all Container logs), you should find the necessary metadata (like Pod, Namespace etc) attached to the logs. The same is true for all the other collected logs through this Solution, which you should be able to find applying a filter on "Kubernetes Cluster Name" field for example.
Thanks

[adil-jeel]

Thank you for your reply. It seems that I am still facing difficulties. Could you kindly provide me with any relevant documentation or resources that can assist me in resolving this issue?
Thank you.