Skip to content

dynamic templates #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

dynamic templates #25

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a4aabc9
Added the possiblity to use dinamyc templates. For example:
Nov 27, 2014
e3d7f30
Removed unnecessary parsing
Nov 28, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 16 additions & 15 deletions lib/logstash/outputs/elasticsearch.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ def register
client_settings["cluster.name"] = @cluster if @cluster
client_settings["network.host"] = @bind_host if @bind_host
client_settings["transport.tcp.port"] = @bind_port if @bind_port

if @node_name
client_settings["node.name"] = @node_name
else
Expand Down Expand Up @@ -301,18 +301,6 @@ def register
end # @host.each
end

if @manage_template
for client in @client
begin
@logger.info("Automatic template management enabled", :manage_template => @manage_template.to_s)
client.template_install(@template_name, get_template, @template_overwrite)
break
rescue => e
@logger.error("Failed to install template: #{e.message}")
end
end # for @client loop
end # if @manage_templates

@logger.info("New Elasticsearch output", :cluster => @cluster,
:host => @host, :port => @port, :embedded => @embedded,
:protocol => @protocol)
Expand Down Expand Up @@ -369,14 +357,15 @@ def setup_basic_auth
end

public
def get_template
def get_template(event)
if @template.nil?
@template = ::File.expand_path('elasticsearch/elasticsearch-template.json', ::File.dirname(__FILE__))
if !File.exists?(@template)
raise "You must specify 'template => ...' in your elasticsearch output (I looked for '#{@template}')"
end
end
template_json = IO.read(@template).gsub(/\n/,'')
template_json = IO.read(event.sprintf(@template)).gsub(/\n/,'')
template_json = event.sprintf(@template_json)
template = LogStash::Json.load(template_json)
template['template'] = wildcard_substitute(@index)
@logger.info("Using mapping template", :template => template)
Expand Down Expand Up @@ -425,6 +414,18 @@ def generate_jks cert_path
def receive(event)
return unless output?(event)

if @manage_template
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking if we should create a template every time there's an event is going to really hurt performance. I believe we'll have to solve this another way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or it's going to have to be semi-manual, e.g. using conditionals and %{type} to have separate outputs which each reference their own template file.

Even if we try to tie in the ES mapping API you'd still end up with a test for each event. This is a tricky edge case to handle dynamically.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using conditionals like:

  if  [type] == "apache" {
            elasticsearch {
                    .........
                    template => "/etc/logstash/conf.d/apache-template.json" 
                    template_name => "apache" 
            }
  }

Requires to do a reload of logstash config. Which might not be optimal.
Maybe we could add something to check if an event has already occured, therefore it does not need to be installed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since there's been no activity here I'm going to close this issue. Thanks for he patch @Bradox , if you'd like to try to revive this idea I'd recommend opening an issue as a design discussion to suss out the pitfalls here!

for client in @client
begin
@logger.info("Automatic template management enabled", :manage_template => @manage_template.to_s)
client.template_install(event.sprintf(@template_name), get_template(event), @template_overwrite)
break
rescue => e
@logger.error("Failed to install template: #{e.message}")
end
end # for @client loop
end # if @manage_templates

# Set the 'type' value for the index.
if @index_type
type = event.sprintf(@index_type)
Expand Down