New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hello World example of python script? #2

Closed
ChristianKniep opened this Issue Mar 2, 2015 · 24 comments

Comments

Projects
None yet
4 participants
@ChristianKniep

ChristianKniep commented Mar 2, 2015

Hey there,

I have trouble putting the filter to work.
My little python script looks like this:

context = zmq.Context()
consumer_receiver = context.socket(zmq.REP)
consumer_receiver.bind("tcp://0.0.0.0:5557")
while True:
    work = consumer_receiver.recv()
    consumer_receiver.send(work)

And my logstash config:

input {
    tcp { port => 5000 }
}

filter {
   zeromq {
        address => "tcp://192.168.59.3:5557"
   }
}

output {
    stdout { codec => rubydebug }
}

This should send the event to the script and get the same event in return, but I got the following message:

{:timestamp=>"2015-03-02T18:05:33.390000+0100", :message=>"0mq filter exception", :address=>"tcp://192.168.59.3:5557", :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/opt/logstash/lib/logstash/event.rb:163:in `overwrite'", "/opt/logstash/lib/logstash/filters/zeromq.rb:197:in `filter'", "(eval):23:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/lib/logstash/pipeline.rb:262:in `filter'", "/opt/logstash/lib/logstash/pipeline.rb:203:in `filterworker'", "/opt/logstash/lib/logstash/pipeline.rb:143:in `start_filters'"], :level=>:warn}
nil

What do I miss?

EDIT: Forgot to mention... I am using logstash 1.4.2 on fedora20

logstash.noarch             1.4.2-1_2c0f5a1          @logstash-1.4
logstash-contrib.noarch     1.4.2-1_efd53ef          @logstash-1.4
@ph

This comment has been minimized.

Show comment
Hide comment
@ph

ph Mar 2, 2015

Contributor

Looking at the code the consumer should receive a json string and the filter also expect a json string back. The NIL error error probably mean there is something wrong in the parsing. Can you add some debug statements on the python side?

Contributor

ph commented Mar 2, 2015

Looking at the code the consumer should receive a json string and the filter also expect a json string back. The NIL error error probably mean there is something wrong in the parsing. Can you add some debug statements on the python side?

@ChristianKniep

This comment has been minimized.

Show comment
Hide comment
@ChristianKniep

ChristianKniep Mar 3, 2015

Hey @ph,

my python script:

#!/usr/bin/python
import zmq
import json
zcontext = zmq.Context()
zsocket = zcontext.socket(zmq.REP)
zsocket.bind('tcp://0.0.0.0:5557')
while True:
    msg = zsocket.recv()
    print "GOT (%s): %s" % (type(msg), msg)
    zsocket.send(msg)

And the output.

$ python test.py
GOT (<type 'str'>): {"message":"huhu","@version":"1","@timestamp":"2015-03-03T07:06:10.435Z","host":"0:0:0:0:0:0:0:1:53112","type":"syslog"}

Logstash still complains like this:

{:timestamp=>"2015-03-03T08:06:10.444000+0100", :message=>"0mq filter exception", :address=>"tcp://192.168.59.3:5557", :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/opt/logstash/lib/logstash/event.rb:163:in `overwrite'", "/opt/logstash/lib/logstash/filters/zeromq.rb:197:in `filter'", "(eval):23:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/lib/logstash/pipeline.rb:262:in `filter'", "/opt/logstash/lib/logstash/pipeline.rb:203:in `filterworker'", "/opt/logstash/lib/logstash/pipeline.rb:143:in `start_filters'"], :level=>:warn}
nil

If we can figure it out I will add an example to the README. Should prevent others from wondering. :)

ChristianKniep commented Mar 3, 2015

Hey @ph,

my python script:

#!/usr/bin/python
import zmq
import json
zcontext = zmq.Context()
zsocket = zcontext.socket(zmq.REP)
zsocket.bind('tcp://0.0.0.0:5557')
while True:
    msg = zsocket.recv()
    print "GOT (%s): %s" % (type(msg), msg)
    zsocket.send(msg)

And the output.

$ python test.py
GOT (<type 'str'>): {"message":"huhu","@version":"1","@timestamp":"2015-03-03T07:06:10.435Z","host":"0:0:0:0:0:0:0:1:53112","type":"syslog"}

Logstash still complains like this:

{:timestamp=>"2015-03-03T08:06:10.444000+0100", :message=>"0mq filter exception", :address=>"tcp://192.168.59.3:5557", :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/opt/logstash/lib/logstash/event.rb:163:in `overwrite'", "/opt/logstash/lib/logstash/filters/zeromq.rb:197:in `filter'", "(eval):23:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/lib/logstash/pipeline.rb:262:in `filter'", "/opt/logstash/lib/logstash/pipeline.rb:203:in `filterworker'", "/opt/logstash/lib/logstash/pipeline.rb:143:in `start_filters'"], :level=>:warn}
nil

If we can figure it out I will add an example to the README. Should prevent others from wondering. :)

@ChristianKniep

This comment has been minimized.

Show comment
Hide comment
@ChristianKniep

ChristianKniep Mar 3, 2015

I got the same result with a ruby script I found here: http://blog.lusis.org/blog/2012/02/08/zeromq-and-logstash-part-2/

$ cat test.rb
require 'rubygems'
require 'ffi-rzmq'
require "json"

context = ZMQ::Context.new
socket = context.socket(ZMQ::REP)
socket.bind("tcp://*:5557")
msg = ''
puts "starting up"
while true do
  socket.recv_string(msg)
  modified_message = JSON.parse(msg)
  puts "Message received: #{msg}"
  # Simulate using an external data source to
  # to something that you need
  case modified_message["@source"]
  when "stdin://jvstratusmbp.lusis.org/"
    puts "Doing db lookup"
    sleep 10
    modified_message["@source"] = "john's laptop"
  end
  puts "Message responded: #{modified_message.to_json}"
  socket.send_string(modified_message.to_json)
end
$ ruby test.rb
starting up
Message received: {"message":"huhu","@version":"1","@timestamp":"2015-03-03T07:32:14.459Z","host":"0:0:0:0:0:0:0:1:38971","type":"syslog"}
Message responded: {"message":"huhu","@version":"1","@timestamp":"2015-03-03T07:32:14.459Z","host":"0:0:0:0:0:0:0:1:38971","type":"syslog"}

Logstash log:

{:timestamp=>"2015-03-03T08:32:14.464000+0100", :message=>"0mq filter exception", :address=>"tcp://192.168.59.3:5557", :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/opt/logstash/lib/logstash/event.rb:163:in `overwrite'", "/opt/logstash/lib/logstash/filters/zeromq.rb:197:in `filter'", "(eval):23:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/lib/logstash/pipeline.rb:262:in `filter'", "/opt/logstash/lib/logstash/pipeline.rb:203:in `filterworker'", "/opt/logstash/lib/logstash/pipeline.rb:143:in `start_filters'"], :level=>:warn}
nil

ChristianKniep commented Mar 3, 2015

I got the same result with a ruby script I found here: http://blog.lusis.org/blog/2012/02/08/zeromq-and-logstash-part-2/

$ cat test.rb
require 'rubygems'
require 'ffi-rzmq'
require "json"

context = ZMQ::Context.new
socket = context.socket(ZMQ::REP)
socket.bind("tcp://*:5557")
msg = ''
puts "starting up"
while true do
  socket.recv_string(msg)
  modified_message = JSON.parse(msg)
  puts "Message received: #{msg}"
  # Simulate using an external data source to
  # to something that you need
  case modified_message["@source"]
  when "stdin://jvstratusmbp.lusis.org/"
    puts "Doing db lookup"
    sleep 10
    modified_message["@source"] = "john's laptop"
  end
  puts "Message responded: #{modified_message.to_json}"
  socket.send_string(modified_message.to_json)
end
$ ruby test.rb
starting up
Message received: {"message":"huhu","@version":"1","@timestamp":"2015-03-03T07:32:14.459Z","host":"0:0:0:0:0:0:0:1:38971","type":"syslog"}
Message responded: {"message":"huhu","@version":"1","@timestamp":"2015-03-03T07:32:14.459Z","host":"0:0:0:0:0:0:0:1:38971","type":"syslog"}

Logstash log:

{:timestamp=>"2015-03-03T08:32:14.464000+0100", :message=>"0mq filter exception", :address=>"tcp://192.168.59.3:5557", :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/opt/logstash/lib/logstash/event.rb:163:in `overwrite'", "/opt/logstash/lib/logstash/filters/zeromq.rb:197:in `filter'", "(eval):23:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/lib/logstash/pipeline.rb:262:in `filter'", "/opt/logstash/lib/logstash/pipeline.rb:203:in `filterworker'", "/opt/logstash/lib/logstash/pipeline.rb:143:in `start_filters'"], :level=>:warn}
nil
@ph

This comment has been minimized.

Show comment
Hide comment
@ph

ph Mar 3, 2015

Contributor

This look like a bug to me on our side. :(

I've looked at the Event#overwrite's code and I think we changed how it work, It expect to receive a concrete Event instance and the filter is sending a Hash. Sadly this filter doesn't have any test to detect this regression..

  def overwrite(event)
    # pickup new event @data and also pickup @accessors
    # otherwise it will be pointing on previous data
    @data = event.instance_variable_get(:@data)
    @accessors = event.instance_variable_get(:@accessors)

    #convert timestamp if it is a String
    if @data[TIMESTAMP].is_a?(String)
      @data[TIMESTAMP] = LogStash::Time.parse_iso8601(@data[TIMESTAMP])
    end
  end
Contributor

ph commented Mar 3, 2015

This look like a bug to me on our side. :(

I've looked at the Event#overwrite's code and I think we changed how it work, It expect to receive a concrete Event instance and the filter is sending a Hash. Sadly this filter doesn't have any test to detect this regression..

  def overwrite(event)
    # pickup new event @data and also pickup @accessors
    # otherwise it will be pointing on previous data
    @data = event.instance_variable_get(:@data)
    @accessors = event.instance_variable_get(:@accessors)

    #convert timestamp if it is a String
    if @data[TIMESTAMP].is_a?(String)
      @data[TIMESTAMP] = LogStash::Time.parse_iso8601(@data[TIMESTAMP])
    end
  end

@ph ph added the bug label Mar 3, 2015

@ChristianKniep

This comment has been minimized.

Show comment
Hide comment
@ChristianKniep

ChristianKniep Mar 4, 2015

do I have an option to create a workaround?

ChristianKniep commented Mar 4, 2015

do I have an option to create a workaround?

@ChristianKniep

This comment has been minimized.

Show comment
Hide comment
@ChristianKniep

ChristianKniep Mar 13, 2015

Hey there, any progress on this one?
I would like to integrate with an external script I would like to use zmq.
I might use the ruby-filter to push the json over zmq, but that seems ugly.

ChristianKniep commented Mar 13, 2015

Hey there, any progress on this one?
I would like to integrate with an external script I would like to use zmq.
I might use the ruby-filter to push the json over zmq, but that seems ugly.

@barravi

This comment has been minimized.

Show comment
Hide comment
@barravi

barravi Apr 16, 2015

Filed a pull request to fix the issue here: #3

barravi commented Apr 16, 2015

Filed a pull request to fix the issue here: #3

@ph

This comment has been minimized.

Show comment
Hide comment
@ph

ph Apr 16, 2015

Contributor

@ChristianKniep can you give it a try? if it solve your issue I'll gladly merge it.

Contributor

ph commented Apr 16, 2015

@ChristianKniep can you give it a try? if it solve your issue I'll gladly merge it.

@ChristianKniep

This comment has been minimized.

Show comment
Hide comment
@ChristianKniep

ChristianKniep Apr 19, 2015

Works for me! +1
I wrote a blog post about how I tested it: http://qnib.org/2015/04/19/logstash-dev/

ChristianKniep commented Apr 19, 2015

Works for me! +1
I wrote a blog post about how I tested it: http://qnib.org/2015/04/19/logstash-dev/

@ph

This comment has been minimized.

Show comment
Hide comment
@ph

ph Apr 20, 2015

Contributor

@ChristianKniep as soon as @barravi sign the cla, I'll be able to merge it and deploy a new version.

Contributor

ph commented Apr 20, 2015

@ChristianKniep as soon as @barravi sign the cla, I'll be able to merge it and deploy a new version.

@ChristianKniep

This comment has been minimized.

Show comment
Hide comment
@ChristianKniep

ChristianKniep May 28, 2015

Hey @ph,@barravi,

how is the state-of-the-pugin? Installing the upstream packages I still got the error and my workaround is quite messy. I would like to use the plugin in logstash 1.4... :)

Cheers and thx for the plugin
Christian

ChristianKniep commented May 28, 2015

Hey @ph,@barravi,

how is the state-of-the-pugin? Installing the upstream packages I still got the error and my workaround is quite messy. I would like to use the plugin in logstash 1.4... :)

Cheers and thx for the plugin
Christian

@ph

This comment has been minimized.

Show comment
Hide comment
@ph

ph May 28, 2015

Contributor

@ChristianKniep I've asked @barravi to sign the CLA with the same email the patch was made, I am waiting on this. We enforce this when we merge pull request :(

Contributor

ph commented May 28, 2015

@ChristianKniep I've asked @barravi to sign the CLA with the same email the patch was made, I am waiting on this. We enforce this when we merge pull request :(

@ChristianKniep

This comment has been minimized.

Show comment
Hide comment
@ChristianKniep

ChristianKniep May 28, 2015

@barravi (google suggests your name is Vincenzo)! I would like to hunt you down and serve you a cold beer!
Please make this happen... :)

ChristianKniep commented May 28, 2015

@barravi (google suggests your name is Vincenzo)! I would like to hunt you down and serve you a cold beer!
Please make this happen... :)

@barravi

This comment has been minimized.

Show comment
Hide comment
@barravi

barravi May 29, 2015

So sorry for the delay, guys. Signing it right now...

barravi commented May 29, 2015

So sorry for the delay, guys. Signing it right now...

@barravi

This comment has been minimized.

Show comment
Hide comment
@barravi

barravi May 29, 2015

It is signed. Try it out and let me know.

barravi commented May 29, 2015

It is signed. Try it out and let me know.

@ph

This comment has been minimized.

Show comment
Hide comment
@ph

ph Jun 5, 2015

Contributor

@barravi weird, on the patch is still with the trendmicro email but in our database we have slashee@ ? did you force push your commit to the branch?

Contributor

ph commented Jun 5, 2015

@barravi weird, on the patch is still with the trendmicro email but in our database we have slashee@ ? did you force push your commit to the branch?

@barravi

This comment has been minimized.

Show comment
Hide comment
@barravi

barravi Jun 5, 2015

not that I know of. In the end I signed the agreement with both my emails, isn't it ok?

barravi commented Jun 5, 2015

not that I know of. In the end I signed the agreement with both my emails, isn't it ok?

@ph

This comment has been minimized.

Show comment
Hide comment
@ph

ph Jun 5, 2015

Contributor

@barravi did you just signed it with the trendmicro? I think I have a 30 minutes delay for the sync

Contributor

ph commented Jun 5, 2015

@barravi did you just signed it with the trendmicro? I think I have a 30 minutes delay for the sync

@barravi

This comment has been minimized.

Show comment
Hide comment
@barravi

barravi Jun 5, 2015

No I signed it 7 days ago when I notified you.

The first time I signed was with my slashvee@gmail.com account. then when you notified me that you needed it form trendmicro, I signed it again (7 days ago) with my corporate email account.

Let me do it one more time just to make sure.

barravi commented Jun 5, 2015

No I signed it 7 days ago when I notified you.

The first time I signed was with my slashvee@gmail.com account. then when you notified me that you needed it form trendmicro, I signed it again (7 days ago) with my corporate email account.

Let me do it one more time just to make sure.

@barravi

This comment has been minimized.

Show comment
Hide comment
@barravi

barravi Jun 5, 2015

(you know, if everything fails just copy that 1 line of code into a new commit from someone allowed to do pull requests, I don't really mind as long as it's fixed :-) and I promise I will pick just one email for the time being)

barravi commented Jun 5, 2015

(you know, if everything fails just copy that 1 line of code into a new commit from someone allowed to do pull requests, I don't really mind as long as it's fixed :-) and I promise I will pick just one email for the time being)

@ph

This comment has been minimized.

Show comment
Hide comment
@ph

ph Jun 5, 2015

Contributor

@barravi I'll do another thing to make sure you have credits, we also care about that. :)

Contributor

ph commented Jun 5, 2015

@barravi I'll do another thing to make sure you have credits, we also care about that. :)

@barravi

This comment has been minimized.

Show comment
Hide comment
@barravi

barravi Jun 5, 2015

Thanks, anyway I just resigned it with my corporate email account (@trendmicro.it).

barravi commented Jun 5, 2015

Thanks, anyway I just resigned it with my corporate email account (@trendmicro.it).

@ChristianKniep

This comment has been minimized.

Show comment
Hide comment
@ChristianKniep

ChristianKniep Dec 11, 2015

Hey there... How about this issue? I updated to logstash-2.1 and still got the error.

{:timestamp=>"2015-12-11T12:13:53.603000+0100", :message=>"0mq filter exception", :address=>"tcp://inventory.service.consul:5557", :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/event.rb:174:in `overwrite'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-zeromq-2.0.2/lib/logstash/filters/zeromq.rb:196:in `filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/filters/base.rb:151:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/filters/base.rb:148:in `multi_filter'", "(eval):283:in `filter_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:244:in `filterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:178:in `start_filters'"], :level=>:warn}

ChristianKniep commented Dec 11, 2015

Hey there... How about this issue? I updated to logstash-2.1 and still got the error.

{:timestamp=>"2015-12-11T12:13:53.603000+0100", :message=>"0mq filter exception", :address=>"tcp://inventory.service.consul:5557", :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/event.rb:174:in `overwrite'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-zeromq-2.0.2/lib/logstash/filters/zeromq.rb:196:in `filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/filters/base.rb:151:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/filters/base.rb:148:in `multi_filter'", "(eval):283:in `filter_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:244:in `filterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:178:in `start_filters'"], :level=>:warn}
@suyograo

This comment has been minimized.

Show comment
Hide comment
@suyograo

suyograo commented Apr 26, 2016

Merged in #7

@suyograo suyograo closed this Apr 26, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment