Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support proxy #44

Merged
merged 2 commits into from Apr 27, 2017
Merged

Support proxy #44

merged 2 commits into from Apr 27, 2017

Conversation

okkez
Copy link
Contributor

@okkez okkez commented Apr 27, 2017

@y-ken
Copy link
Owner

y-ken commented Apr 27, 2017

Thank you very much.
I'm afraid could you update README please?
Thank you.

@y-ken
Copy link
Owner

y-ken commented Apr 27, 2017

Thank you very much!

@y-ken y-ken merged commit 0305863 into y-ken:master Apr 27, 2017
@okkez okkez deleted the support-proxy branch April 27, 2017 15:16
@jerry924
Copy link

I am not having luck getting this to work.

I tried the following in my config:

<source>
  @type twitter
  consumer_key           SECRET    #YOUR_CONSUMER_KEY # Required
  consumer_secret       SECRET   #YOUR_CONSUMER_SECRET # Required
  access_token             SECRET     #YOUR_OAUTH_TOKEN # Required
  access_token_secret  SECRET #YOUR_OAUTH_TOKEN_SECRET # Required
  tag                 input.twitter  # Required
  keyword             baseball
  timeline            sampling        #userstream # Required (sampling or userstream)
  <proxy>
     host myproxyhostname
     port 80
  </proxy>
</source>

But I get this warning in my td-agent.log file:

2017-11-20 18:56:31 -0800 [warn]: section <proxy> is not used in <source> of twitter plugin

I also tried with the following four environment variables both set to the proper value and also clear.
http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY.

What ends up happening with environment variables cleared and without the proxy taking affect above is the request to twitter IP/port times out.

Here is a fragment of my log with all http_proxy env variables cleared and tag in the configuration file:

2017-11-20 18:56:31 -0800 [info]: #0 starting fluentd worker pid=17726 ppid=14103 worker=0
2017-11-20 18:56:31 -0800 [info]: #0 listening dRuby uri="druby://127.0.0.1:24230" object="Fluent::Engine"
2017-11-20 18:56:31 -0800 [info]: #0 twitter: starting Twitter Streaming API for sampling. tag:input.twitter keyword:baseball
2017-11-20 18:56:31 -0800 [info]: #0 listening port port=24224 bind="0.0.0.0"
2017-11-20 18:56:31 -0800 [info]: #0 fluentd worker is now running worker=0
2017-11-20 18:58:38 -0800 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::TwitterInput title=:in_twitter error_class=Errno::ETIMEDOUT error="Connection timed out - connect(2) for \"199.59.148.229\" port 443"
2017-11-20 18:58:38 -0800 [error]: #0 unexpected error error_class=Errno::ETIMEDOUT error="Connection timed out - connect(2) for \"199.59.148.229\" port 443"
  2017-11-20 18:58:38 -0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/twitter-6.1.0/lib/twitter/streaming/connection.rb:16:in `initialize'
  2017-11-20 18:58:38 -0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/twitter-6.1.0/lib/twitter/streaming/connection.rb:16:in `new'
  2017-11-20 18:58:38 -0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/twitter-6.1.0/lib/twitter/streaming/connection.rb:16:in `stream'
  2017-11-20 18:58:38 -0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/twitter-6.1.0/lib/twitter/streaming/client.rb:119:in `request'
  2017-11-20 18:58:38 -0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/twitter-6.1.0/lib/twitter/streaming/client.rb:37:in `filter'
  2017-11-20 18:58:38 -0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-twitter-0.6.1/lib/fluent/plugin/in_twitter.rb:67:in `run'
  2017-11-20 18:58:38 -0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-twitter-0.6.1/lib/fluent/plugin/in_twitter.rb:49:in `block in start'
  2017-11-20 18:58:38 -0800 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.23/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2017-11-20 18:58:38 -0800 [error]: #0 unexpected error error_class=Errno::ETIMEDOUT error="Connection timed out - connect(2) for \"199.59.148.229\" port 443"
  2017-11-20 18:58:38 -0800 [error]: #0 suppressed same stacktrace
2017-11-20 18:58:38 -0800 [info]: Worker 0 finished unexpectedly with status 1
2017-11-20 18:58:39 -0800 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2017-11-20 18:58:39 -0800 [info]: gem 'fluent-plugin-kafka' version '0.6.1'
2017-11-20 18:58:39 -0800 [info]: gem 'fluent-plugin-mongo' version '0.8.1'
2017-11-20 18:58:39 -0800 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.6'
2017-11-20 18:58:39 -0800 [info]: gem 'fluent-plugin-s3' version '0.8.5'
2017-11-20 18:58:39 -0800 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2017-11-20 18:58:39 -0800 [info]: gem 'fluent-plugin-td' version '0.10.29'
2017-11-20 18:58:39 -0800 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3'
2017-11-20 18:58:39 -0800 [info]: gem 'fluent-plugin-twitter' version '0.6.1'
2017-11-20 18:58:39 -0800 [info]: gem 'fluent-plugin-twitter' version '0.6.0'
2017-11-20 18:58:39 -0800 [info]: gem 'fluent-plugin-webhdfs' version '0.7.1'
2017-11-20 18:58:39 -0800 [info]: gem 'fluentd' version '0.14.23'
2017-11-20 18:58:39 -0800 [info]: gem 'fluentd' version '0.12.40'
2017-11-20 18:58:39 -0800 [info]: adding match pattern="td.*.*" type="tdlog"
2017-11-20 18:58:39 -0800 [warn]: #0 secondary type should be same with primary one primary="Fluent::TreasureDataLogOutput" secondary="Fluent::Plugin::FileOutput"
2017-11-20 18:58:39 -0800 [info]: adding match pattern="debug.**" type="stdout"
2017-11-20 18:58:39 -0800 [info]: adding match pattern="input.twitter.sampling" type="stdout"
2017-11-20 18:58:39 -0800 [info]: adding source type="forward"
2017-11-20 18:58:39 -0800 [info]: adding source type="http"
2017-11-20 18:58:39 -0800 [info]: adding source type="debug_agent"
2017-11-20 18:58:39 -0800 [info]: adding source type="twitter"
2017-11-20 18:58:39 -0800 [warn]: section <proxy> is not used in <source> of twitter plugin
2017-11-20 18:58:39 -0800 [warn]: section <proxy> is not used in <source> of twitter plugin

@y-ken
Copy link
Owner

y-ken commented Nov 21, 2017

@okkez
I'm afraid could you check this behavior?
jerry924 using following environment.

  • gem 'fluent-plugin-twitter' version '0.6.1'
  • gem 'fluentd' version '0.14.23'

@okkez
Copy link
Contributor Author

okkez commented Nov 21, 2017

fluent-plugin-twitter 0.6.1 does not support <proxy> section.
Not released yet including <proxy> support.

@jerry924
Copy link

jerry924 commented Nov 21, 2017

I see... since it was merged in April, I thought it would be in latest release. I will try to manually patch my files with the necessary changes then..

Since the docs do mention this, might either want to rev the product or comment in the docs that the version for proxy is not released yet.

Thank You. And thank you for responding so quickly. I've been banging my head on this one for hours.

@jerry924
Copy link

So my not used message went away after patching the changed files in this PR, but I still get timout connecting to Twitter both with and without setting http_proxy/https_proxy. Have not found any ideas via Google search, so may need to just abandon this.

@okkez
Copy link
Contributor Author

okkez commented Nov 21, 2017

I've written some sample code for testing twitter gem's proxy support.
Twitter::REST::Client proxy support works well.
But Twitter::Streaming::Client proxy support does not work properly.

REST API works:

  client = Twitter::REST::Client.new do |config|
    config.consumer_key = "xxx"
    config.consumer_secret = "xxx"
    config.access_token = "xxx"
    config.access_token_secret = "xxx"
    config.proxy = {
      uri: Addressable::URI.parse("http://localhost:8080")
    }
  end

  client.search("Ruby", result_type: "recent").take(3).each do |tweet|
    puts tweet.text
  end

Streaming API doesn't work:

client = Twitter::Streaming::Client.new do |config|
  config.consumer_key = "xxx"
  config.consumer_secret = "xxx"
  config.access_token = "xxx"
  config.access_token_secret = "xxx"
  config.proxy = {
    #uri: Addressable::URI.parse("http://localhost:8080")
    # host: "127.0.0.1",
    # port: 8080,
    proxy_address: "127.0.0.1",
    proxy_port: 8080,
  }
end

client.filter(track: "zipper") do |o|
  puts o.text
end

The reason:

twitter gem's HTTP::Request usage is wrong in streaming API: https://github.com/sferik/twitter/blob/v6.2.0/lib/twitter/streaming/client.rb#L113

HTTP::Request.new expects proxy_address like key, see

@okkez
Copy link
Contributor Author

okkez commented Nov 21, 2017

I got following error:

twitter/streaming/response.rb:24:in `on_headers_complete': Twitter::Error::NotFound
        from lib/twitter/streaming/response.rb:19:in `<<'
        from lib/twitter/streaming/response.rb:19:in `<<'
        from lib/twitter/streaming/connection.rb:25:in `block in stream'
        from lib/twitter/streaming/connection.rb:21:in `loop'
        from lib/twitter/streaming/connection.rb:21:in `stream'
        from lib/twitter/streaming/client.rb:123:in `request'
        from lib/twitter/streaming/client.rb:38:in `filter'
        from t.rb:21:in `<main>'

@okkez
Copy link
Contributor Author

okkez commented Nov 22, 2017

See also sferik/twitter-ruby#639

@okkez
Copy link
Contributor Author

okkez commented Nov 22, 2017

twitter API support only TLS. but twitter gem does not support TLS connection with proxy.
In fact, http gem does not support proxy with HTTPS URI. See here.

@y-ken
Copy link
Owner

y-ken commented Nov 23, 2017

Thank you for investigating the proxy behavior. 🙇
Are there no way to connect to twitter streaming api with proxy for now?

@okkez
Copy link
Contributor Author

okkez commented Nov 25, 2017

Are there no way to connect to twitter streaming api with proxy for now?

No, there isn't, for now.
We need sending patch to http gem or twitter gem.
Or we can replace twitter gem's http backend to other gem... (I'm not sure...)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants