Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hello World example of python script? #2

Closed
ChristianKniep opened this issue Mar 2, 2015 · 24 comments
Closed

Hello World example of python script? #2

ChristianKniep opened this issue Mar 2, 2015 · 24 comments
Labels
bug

Comments

@ChristianKniep
Copy link

@ChristianKniep ChristianKniep commented Mar 2, 2015

Hey there,

I have trouble putting the filter to work.
My little python script looks like this:

context = zmq.Context()
consumer_receiver = context.socket(zmq.REP)
consumer_receiver.bind("tcp://0.0.0.0:5557")
while True:
    work = consumer_receiver.recv()
    consumer_receiver.send(work)

And my logstash config:

input {
    tcp { port => 5000 }
}

filter {
   zeromq {
        address => "tcp://192.168.59.3:5557"
   }
}

output {
    stdout { codec => rubydebug }
}

This should send the event to the script and get the same event in return, but I got the following message:

{:timestamp=>"2015-03-02T18:05:33.390000+0100", :message=>"0mq filter exception", :address=>"tcp://192.168.59.3:5557", :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/opt/logstash/lib/logstash/event.rb:163:in `overwrite'", "/opt/logstash/lib/logstash/filters/zeromq.rb:197:in `filter'", "(eval):23:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/lib/logstash/pipeline.rb:262:in `filter'", "/opt/logstash/lib/logstash/pipeline.rb:203:in `filterworker'", "/opt/logstash/lib/logstash/pipeline.rb:143:in `start_filters'"], :level=>:warn}
nil

What do I miss?

EDIT: Forgot to mention... I am using logstash 1.4.2 on fedora20

logstash.noarch             1.4.2-1_2c0f5a1          @logstash-1.4
logstash-contrib.noarch     1.4.2-1_efd53ef          @logstash-1.4
@ph
Copy link
Contributor

@ph ph commented Mar 2, 2015

Looking at the code the consumer should receive a json string and the filter also expect a json string back. The NIL error error probably mean there is something wrong in the parsing. Can you add some debug statements on the python side?

@ChristianKniep
Copy link
Author

@ChristianKniep ChristianKniep commented Mar 3, 2015

Hey @ph,

my python script:

#!/usr/bin/python
import zmq
import json
zcontext = zmq.Context()
zsocket = zcontext.socket(zmq.REP)
zsocket.bind('tcp://0.0.0.0:5557')
while True:
    msg = zsocket.recv()
    print "GOT (%s): %s" % (type(msg), msg)
    zsocket.send(msg)

And the output.

$ python test.py
GOT (<type 'str'>): {"message":"huhu","@version":"1","@timestamp":"2015-03-03T07:06:10.435Z","host":"0:0:0:0:0:0:0:1:53112","type":"syslog"}

Logstash still complains like this:

{:timestamp=>"2015-03-03T08:06:10.444000+0100", :message=>"0mq filter exception", :address=>"tcp://192.168.59.3:5557", :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/opt/logstash/lib/logstash/event.rb:163:in `overwrite'", "/opt/logstash/lib/logstash/filters/zeromq.rb:197:in `filter'", "(eval):23:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/lib/logstash/pipeline.rb:262:in `filter'", "/opt/logstash/lib/logstash/pipeline.rb:203:in `filterworker'", "/opt/logstash/lib/logstash/pipeline.rb:143:in `start_filters'"], :level=>:warn}
nil

If we can figure it out I will add an example to the README. Should prevent others from wondering. :)

@ChristianKniep
Copy link
Author

@ChristianKniep ChristianKniep commented Mar 3, 2015

I got the same result with a ruby script I found here: http://blog.lusis.org/blog/2012/02/08/zeromq-and-logstash-part-2/

$ cat test.rb
require 'rubygems'
require 'ffi-rzmq'
require "json"

context = ZMQ::Context.new
socket = context.socket(ZMQ::REP)
socket.bind("tcp://*:5557")
msg = ''
puts "starting up"
while true do
  socket.recv_string(msg)
  modified_message = JSON.parse(msg)
  puts "Message received: #{msg}"
  # Simulate using an external data source to
  # to something that you need
  case modified_message["@source"]
  when "stdin://jvstratusmbp.lusis.org/"
    puts "Doing db lookup"
    sleep 10
    modified_message["@source"] = "john's laptop"
  end
  puts "Message responded: #{modified_message.to_json}"
  socket.send_string(modified_message.to_json)
end
$ ruby test.rb
starting up
Message received: {"message":"huhu","@version":"1","@timestamp":"2015-03-03T07:32:14.459Z","host":"0:0:0:0:0:0:0:1:38971","type":"syslog"}
Message responded: {"message":"huhu","@version":"1","@timestamp":"2015-03-03T07:32:14.459Z","host":"0:0:0:0:0:0:0:1:38971","type":"syslog"}

Logstash log:

{:timestamp=>"2015-03-03T08:32:14.464000+0100", :message=>"0mq filter exception", :address=>"tcp://192.168.59.3:5557", :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/opt/logstash/lib/logstash/event.rb:163:in `overwrite'", "/opt/logstash/lib/logstash/filters/zeromq.rb:197:in `filter'", "(eval):23:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/lib/logstash/pipeline.rb:262:in `filter'", "/opt/logstash/lib/logstash/pipeline.rb:203:in `filterworker'", "/opt/logstash/lib/logstash/pipeline.rb:143:in `start_filters'"], :level=>:warn}
nil
@ph
Copy link
Contributor

@ph ph commented Mar 3, 2015

This look like a bug to me on our side. :(

I've looked at the Event#overwrite's code and I think we changed how it work, It expect to receive a concrete Event instance and the filter is sending a Hash. Sadly this filter doesn't have any test to detect this regression..

  def overwrite(event)
    # pickup new event @data and also pickup @accessors
    # otherwise it will be pointing on previous data
    @data = event.instance_variable_get(:@data)
    @accessors = event.instance_variable_get(:@accessors)

    #convert timestamp if it is a String
    if @data[TIMESTAMP].is_a?(String)
      @data[TIMESTAMP] = LogStash::Time.parse_iso8601(@data[TIMESTAMP])
    end
  end
@ph ph added the bug label Mar 3, 2015
@ChristianKniep
Copy link
Author

@ChristianKniep ChristianKniep commented Mar 4, 2015

do I have an option to create a workaround?

@ChristianKniep
Copy link
Author

@ChristianKniep ChristianKniep commented Mar 13, 2015

Hey there, any progress on this one?
I would like to integrate with an external script I would like to use zmq.
I might use the ruby-filter to push the json over zmq, but that seems ugly.

@barravi
Copy link

@barravi barravi commented Apr 16, 2015

Filed a pull request to fix the issue here: #3

@ph
Copy link
Contributor

@ph ph commented Apr 16, 2015

@ChristianKniep can you give it a try? if it solve your issue I'll gladly merge it.

@ChristianKniep
Copy link
Author

@ChristianKniep ChristianKniep commented Apr 19, 2015

Works for me! +1
I wrote a blog post about how I tested it: http://qnib.org/2015/04/19/logstash-dev/

@ph
Copy link
Contributor

@ph ph commented Apr 20, 2015

@ChristianKniep as soon as @barravi sign the cla, I'll be able to merge it and deploy a new version.

@ChristianKniep
Copy link
Author

@ChristianKniep ChristianKniep commented May 28, 2015

Hey @ph,@barravi,

how is the state-of-the-pugin? Installing the upstream packages I still got the error and my workaround is quite messy. I would like to use the plugin in logstash 1.4... :)

Cheers and thx for the plugin
Christian

@ph
Copy link
Contributor

@ph ph commented May 28, 2015

@ChristianKniep I've asked @barravi to sign the CLA with the same email the patch was made, I am waiting on this. We enforce this when we merge pull request :(

@ChristianKniep
Copy link
Author

@ChristianKniep ChristianKniep commented May 28, 2015

@barravi (google suggests your name is Vincenzo)! I would like to hunt you down and serve you a cold beer!
Please make this happen... :)

@barravi
Copy link

@barravi barravi commented May 29, 2015

So sorry for the delay, guys. Signing it right now...

@barravi
Copy link

@barravi barravi commented May 29, 2015

It is signed. Try it out and let me know.

@ph
Copy link
Contributor

@ph ph commented Jun 5, 2015

@barravi weird, on the patch is still with the trendmicro email but in our database we have slashee@ ? did you force push your commit to the branch?

@barravi
Copy link

@barravi barravi commented Jun 5, 2015

not that I know of. In the end I signed the agreement with both my emails, isn't it ok?

@ph
Copy link
Contributor

@ph ph commented Jun 5, 2015

@barravi did you just signed it with the trendmicro? I think I have a 30 minutes delay for the sync

@barravi
Copy link

@barravi barravi commented Jun 5, 2015

No I signed it 7 days ago when I notified you.

The first time I signed was with my slashvee@gmail.com account. then when you notified me that you needed it form trendmicro, I signed it again (7 days ago) with my corporate email account.

Let me do it one more time just to make sure.

@barravi
Copy link

@barravi barravi commented Jun 5, 2015

(you know, if everything fails just copy that 1 line of code into a new commit from someone allowed to do pull requests, I don't really mind as long as it's fixed :-) and I promise I will pick just one email for the time being)

@ph
Copy link
Contributor

@ph ph commented Jun 5, 2015

@barravi I'll do another thing to make sure you have credits, we also care about that. :)

@barravi
Copy link

@barravi barravi commented Jun 5, 2015

Thanks, anyway I just resigned it with my corporate email account (@trendmicro.it).

@ChristianKniep
Copy link
Author

@ChristianKniep ChristianKniep commented Dec 11, 2015

Hey there... How about this issue? I updated to logstash-2.1 and still got the error.

{:timestamp=>"2015-12-11T12:13:53.603000+0100", :message=>"0mq filter exception", :address=>"tcp://inventory.service.consul:5557", :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/event.rb:174:in `overwrite'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-zeromq-2.0.2/lib/logstash/filters/zeromq.rb:196:in `filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/filters/base.rb:151:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/filters/base.rb:148:in `multi_filter'", "(eval):283:in `filter_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:244:in `filterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:178:in `start_filters'"], :level=>:warn}
@suyograo
Copy link

@suyograo suyograo commented Apr 26, 2016

Merged in #7

@suyograo suyograo closed this Apr 26, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.