undefined method 'Proxy_Address'

[Grunticus03]

Running Logstash 6.3.2 with Twitter Plugin version 3.0.8 on Windows Server 2012 R2, getting the below error when attempting to traverse a proxy.

Input Config:

input {
  twitter {
    consumer_key => "xxxx"
    consumer_secret => "xxx"
    full_tweet => true
    ignore_retweets => true
    oauth_token => "xxxx"
    oauth_token_secret => "xxxx"
    keywords => ["infosec","cybersecurity","cyberattack","CVE-20"]
    follows => ["821806287461740544","77346069,1479864158","3248301079","721158785281044480","718473978"]
    languages => ["en"]
    use_proxy => true
    proxy_address => "1.2.3.4"
    proxy_port => 9098
  }
}

[2018-09-13T15:21:25,591][ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"Twitter", :plugin=>"<LogStash::Inputs::Twitter consumer_key=>\"xxxxx\", consumer_secret=><password>, full_tweet=>true, ignore_retweets=>true, oauth_token=>\"xxxxxx\", oauth_token_secret=><password>, keywords=>[\"infosec\", \"cybersecurity\", \"cyberattack\", \"CVE-20\"], follows=>[\"821806287461740544\", \"77346069,1479864158\", \"3248301079\", \"721158785281044480\", \"718473978\"], languages=>[\"en\"], use_proxy=>true, proxy_address=>\"1.2.3.4\", proxy_port=>9098, id=>\"6f4223770e54af836d6518381117feaa0090de4d899fa863df58c1e54146ca2a\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_53d0a0b9-0bd8-431e-9944-bd3901882f12\", enable_metric=>true, charset=>\"UTF-8\">, use_samples=>false, rate_limit_reset_in=>300>", :error=>"undefined method `proxy_address=' for #<Faraday::ProxyOptions:0x35fad6ce>", :thread=>"#<Thread:0x1c7d577f@D:/ElasticStack/Apps/Logstash/6.3.2/logstash-core/lib/logstash/pipeline_action/create.rb:44 run>"}
[2018-09-13T15:21:27,880][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"Twitter", :exception=>#<NoMethodError: undefined method `proxy_address=' for #<Faraday::ProxyOptions:0x35fad6ce>>, :backtrace=>["D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/options.rb:31:in `block in update'", "org/jruby/RubyHash.java:1343:in `each'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/options.rb:20:in `update'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/options.rb:7:in `from'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/options.rb:231:in `from'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/connection.rb:286:in `proxy'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/connection.rb:83:in `initialize'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday.rb:70:in `new'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/twitter-5.15.0/lib/twitter/rest/client.rb:97:in `connection'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/twitter-5.15.0/lib/twitter/rest/request.rb:25:in `initialize'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/twitter-5.15.0/lib/twitter/rest/utils.rb:50:in `perform_request'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/twitter-5.15.0/lib/twitter/rest/utils.rb:72:in `perform_request_with_object'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/twitter-5.15.0/lib/twitter/rest/utils.rb:57:in `perform_get_with_object'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/twitter-5.15.0/lib/twitter/rest/users.rb:51:in `verify_credentials'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/twitter-5.15.0/lib/twitter/rest/users.rb:258:in `user'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-twitter-3.0.8/lib/logstash/inputs/twitter.rb:248:in `find_user'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-twitter-3.0.8/lib/logstash/inputs/twitter.rb:241:in `block in build_options'", "org/jruby/RubyArray.java:2486:in `map'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-twitter-3.0.8/lib/logstash/inputs/twitter.rb:240:in `build_options'", "D:/ElasticStack/Apps/Logstash/6.3.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-twitter-3.0.8/lib/logstash/inputs/twitter.rb:121:in `register'", "D:/ElasticStack/Apps/Logstash/6.3.2/logstash-core/lib/logstash/pipeline.rb:340:in `register_plugin'", "D:/ElasticStack/Apps/Logstash/6.3.2/logstash-core/lib/logstash/pipeline.rb:351:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "D:/ElasticStack/Apps/Logstash/6.3.2/logstash-core/lib/logstash/pipeline.rb:351:in `register_plugins'", "D:/ElasticStack/Apps/Logstash/6.3.2/logstash-core/lib/logstash/pipeline.rb:498:in `start_inputs'", "D:/ElasticStack/Apps/Logstash/6.3.2/logstash-core/lib/logstash/pipeline.rb:392:in `start_workers'", "D:/ElasticStack/Apps/Logstash/6.3.2/logstash-core/lib/logstash/pipeline.rb:288:in `run'", "D:/ElasticStack/Apps/Logstash/6.3.2/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"], :thread=>"#<Thread:0x1c7d577f@D:/ElasticStack/Apps/Logstash/6.3.2/logstash-core/lib/logstash/pipeline_action/create.rb:44 run>"}
[2018-09-13T15:21:27,895][ERROR][logstash.agent           ] Failed to execute action {:id=>:Twitter, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<Twitter>, action_result: false", :backtrace=>nil}

[guyboertje]

Confirmed, this is a bug. I am going to fix this in a PR that include proxy auth and EOF exception handling too.

[Grunticus03]

Awesome, do you know when this will be published and available to pull via logstash's plugin manager?

[guyboertje]

@wwalker0307
Unfortunately, I had to abandon that PR. After fixing the code, I could not get the other code (that I did not initially touch) to actually work with either mitmproxy, squid or OWASP Zap. I did get it to work experimentally with the most popular Java for Twitter streaming but as it is an almost complete rewrite - making this work with the Java client will be done when we get some time to do it.

[kares]

the issue is defunct as of now, recent plugin library stack does no longer use faraday (part of the reported trace)
and we thoroughly checked proxy to be working in latest (4.0.3) plugin version. there were some issues previously which are summarized in #67 and #68