Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cleanup fluentd sumologic filter plugin #310

Merged
merged 3 commits into from
Nov 26, 2019
Merged

cleanup fluentd sumologic filter plugin #310

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
571e056
cleanup some of filter plugin code
samjsong Oct 9, 2019
64ef90b
fix typo
samjsong Nov 22, 2019
740080f
Merge branch 'master' of github.com:SumoLogic/sumologic-kubernetes-co…
samjsong Nov 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 34 additions & 59 deletions fluent-plugin-kubernetes-sumologic/lib/fluent/plugin/filter_kubernetes_sumologic.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,60 +57,51 @@ def to_hash(pod_template_hash)
end

def filter(tag, time, record)
log_fields = {}

# Set the sumo metadata fields
sumo_metadata = record["_sumo_metadata"] || {}
record["_sumo_metadata"] = sumo_metadata
log_fields = {}
sumo_metadata[:log_format] = @log_format
sumo_metadata[:host] = @source_host if @source_host
sumo_metadata[:source] = @source_name if @source_name

unless @source_category.nil?
sumo_metadata[:category] = @source_category.dup
unless @source_category_prefix.nil?
sumo_metadata[:category].prepend(@source_category_prefix)
end
end
sumo_metadata[:category].gsub!("-", @source_category_replace_dash)

# Check systemd exclude filters
if record.key?("_SYSTEMD_UNIT") and not record.fetch("_SYSTEMD_UNIT").nil?
unless @exclude_unit_regex.empty?
if Regexp.compile(@exclude_unit_regex).match(record["_SYSTEMD_UNIT"])
return nil
end
return nil if Regexp.compile(@exclude_unit_regex).match(record["_SYSTEMD_UNIT"])
end

unless @exclude_facility_regex.empty?
if Regexp.compile(@exclude_facility_regex).match(record["SYSLOG_FACILITY"])
return nil
end
return nil if Regexp.compile(@exclude_facility_regex).match(record["SYSLOG_FACILITY"])
end

unless @exclude_priority_regex.empty?
if Regexp.compile(@exclude_priority_regex).match(record["PRIORITY"])
return nil
end
return nil if Regexp.compile(@exclude_priority_regex).match(record["PRIORITY"])
end

unless @exclude_host_regex.empty?
if Regexp.compile(@exclude_host_regex).match(record["_HOSTNAME"])
return nil
end
return nil if Regexp.compile(@exclude_host_regex).match(record["_HOSTNAME"])
end
end

# Allow fields to be overridden by annotations
if record.key?("kubernetes") and not record.fetch("kubernetes").nil?
# Clone kubernetes hash so we don't override the cache
# Note (sam 10/9/19): this is a shallow copy; nested hashes can still be overriden
kubernetes = record["kubernetes"].clone

# Populate k8s_metadata to use later in sumo_metadata
k8s_metadata = {
:namespace => kubernetes["namespace_name"],
:pod => kubernetes["pod_name"],
:pod_id => kubernetes['pod_id'],
:container => kubernetes["container_name"],
:source_host => kubernetes["host"],
}


if kubernetes.has_key? "labels"
kubernetes["labels"].each { |k, v| k8s_metadata["label:#{k}".to_sym] = v }
end
Expand All @@ -119,34 +110,22 @@ def filter(tag, time, record)
end
k8s_metadata.default = "undefined"

# Fetch annotations for config
annotations = kubernetes.fetch("annotations", {})
if annotations["sumologic.com/include"] == "true"
include = true
else
include = false
end

unless @exclude_namespace_regex.empty?
if Regexp.compile(@exclude_namespace_regex).match(k8s_metadata[:namespace]) and not include
return nil
unless annotations["sumologic.com/include"] == "true"
# Check kubernetes exclude filters
unless @exclude_namespace_regex.empty?
return nil if Regexp.compile(@exclude_namespace_regex).match(k8s_metadata[:namespace])
end
end

unless @exclude_pod_regex.empty?
if Regexp.compile(@exclude_pod_regex).match(k8s_metadata[:pod]) and not include
return nil
unless @exclude_pod_regex.empty?
return nil if Regexp.compile(@exclude_pod_regex).match(k8s_metadata[:pod])
end
end

unless @exclude_container_regex.empty?
if Regexp.compile(@exclude_container_regex).match(k8s_metadata[:container]) and not include
return nil
unless @exclude_container_regex.empty?
return nil if Regexp.compile(@exclude_container_regex).match(k8s_metadata[:container])
end
end

unless @exclude_host_regex.empty?
if Regexp.compile(@exclude_host_regex).match(k8s_metadata[:source_host]) and not include
return nil
unless @exclude_host_regex.empty?
return nil if Regexp.compile(@exclude_host_regex).match(k8s_metadata[:source_host])
end
end

Expand All @@ -158,23 +137,18 @@ def filter(tag, time, record)

sumo_metadata[:log_format] = annotations["sumologic.com/format"] if annotations["sumologic.com/format"]

if annotations["sumologic.com/sourceHost"].nil?
sumo_metadata[:host] = sumo_metadata[:host] % k8s_metadata
else
sumo_metadata[:host] = annotations["sumologic.com/sourceHost"] % k8s_metadata
unless annotations["sumologic.com/sourceHost"].nil?
sumo_metadata[:host] = annotations["sumologic.com/sourceHost"]
end

if annotations["sumologic.com/sourceName"].nil?
sumo_metadata[:source] = sumo_metadata[:source] % k8s_metadata
else
sumo_metadata[:source] = annotations["sumologic.com/sourceName"] % k8s_metadata
unless annotations["sumologic.com/sourceName"].nil?
sumo_metadata[:source] = annotations["sumologic.com/sourceName"]
end

if annotations["sumologic.com/sourceCategory"].nil?
sumo_metadata[:category] = sumo_metadata[:category] % k8s_metadata
else
sumo_metadata[:category] = (annotations["sumologic.com/sourceCategory"] % k8s_metadata).prepend(@source_category_prefix)
unless annotations["sumologic.com/sourceCategory"].nil?
sumo_metadata[:category] = annotations["sumologic.com/sourceCategory"].dup.prepend(@source_category_prefix)
end
sumo_metadata[:host] = sumo_metadata[:host] % k8s_metadata
sumo_metadata[:source] = sumo_metadata[:source] % k8s_metadata
sumo_metadata[:category] = sumo_metadata[:category] % k8s_metadata
sumo_metadata[:category].gsub!("-", @source_category_replace_dash)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this be removed due to L74

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the reason I left it is that sumo_metadata[:category] is modified above, so that if any of k8s_metadata includes a dash we replace it.

I believe this would be consistent with current behavior, but I'm not actually sure this is the behavior we want to enforce. For instance, if pod name is

kube-apiserver-ip-172-20-51-167.us-west-1.compute.internal

then the source category given our default of sourceCategory: "%{namespace}/%{pod_name}" would end up like this:

kubernetes/kube/system/kube/apiserver/ip/172/20/51/167.us/west

(as a side note, I'm not actually sure what happened to -1.compute.internal, will need to look into that)

For this PR I vote for keeping this behavior to be backwards compatible, but @frankreno can you confirm what behavior we want to enforce for source_category_replace_dash, and what the usecase is/was?


# Strip kubernetes metadata from json if disabled
Expand All @@ -198,10 +172,12 @@ def filter(tag, time, record)
record.delete("time")
end
# Strip sumologic.com annotations
# Note (sam 10/9/19): we're stripping from the copy, so this has no effect on output
kubernetes.delete("annotations") if annotations

if @log_format == "fields" and record.key?("docker") and not record.fetch("docker").nil?
record["docker"].each {|k, v| log_fields[k] = v}
record.delete("docker")
end

if @log_format == "fields" and record.key?("kubernetes") and not record.fetch("kubernetes").nil?
Expand All @@ -218,13 +194,12 @@ def filter(tag, time, record)
log_fields[key] = kubernetes[key] unless kubernetes[key].nil?
end
log_fields["node"] = kubernetes["host"] unless kubernetes["host"].nil?
record.delete("kubernetes")
end
end

if @log_format == "fields" and not log_fields.nil?
sumo_metadata[:fields] = log_fields.select{|k,v| !(v.nil? || v.empty?)}.map{|k,v| "#{k}=#{v}"}.join(',')
record.delete("docker")
record.delete("kubernetes")
end
record
end
Expand Down