Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Supporting td-agent #32

Closed
ikuwow opened this issue Feb 16, 2017 · 4 comments
Closed

Supporting td-agent #32

ikuwow opened this issue Feb 16, 2017 · 4 comments

Comments

@ikuwow
Copy link

ikuwow commented Feb 16, 2017 •
edited
Loading

This plugin is better than mdoi/fluent-plugin-gcloud-pubsub (e.x. retrying when pulling fails) but does not support td-agent.

https://github.com/mia-0032/fluent-plugin-gcloud-pubsub-custom/blob/master/README.md#installation

I got this error when i tried installing by /usr/sbin/td-agent-gem:

# td-agent-gem install fluent-plugin-gcloud-pubsub-custom
Fetching: fluent-plugin-gcloud-pubsub-custom-0.4.3.gem (100%)
Successfully installed fluent-plugin-gcloud-pubsub-custom-0.4.3
Parsing documentation for fluent-plugin-gcloud-pubsub-custom-0.4.3
Installing ri documentation for fluent-plugin-gcloud-pubsub-custom-0.4.3
Done installing documentation for fluent-plugin-gcloud-pubsub-custom after 0 seconds
1 gem installed
# service td-agent start
Starting td-agent: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/json-1.8.1/lib/json/version.rb:3: warning: already initialized constant JSON::VERSION
/opt/td-agent/embedded/lib/ruby/2.1.0/json/version.rb:3: warning: previous definition of VERSION was here
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/json-1.8.1/lib/json/version.rb:4: warning: already initialized constant JSON::VERSION_ARRAY
/opt/td-agent/embedded/lib/ruby/2.1.0/json/version.rb:4: warning: previous definition of VERSION_ARRAY was here
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/json-1.8.1/lib/json/version.rb:5: warning: already initialized constant JSON::VERSION_MAJOR
/opt/td-agent/embedded/lib/ruby/2.1.0/json/version.rb:5: warning: previous definition of VERSION_MAJOR was here
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/json-1.8.1/lib/json/version.rb:6: warning: already initialized constant JSON::VERSION_MINOR
/opt/td-agent/embedded/lib/ruby/2.1.0/json/version.rb:6: warning: previous definition of VERSION_MINOR was here
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/json-1.8.1/lib/json/version.rb:7: warning: already initialized constant JSON::VERSION_BUILD
/opt/td-agent/embedded/lib/ruby/2.1.0/json/version.rb:7: warning: previous definition of VERSION_BUILD was here
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/json-1.8.1/lib/json/common.rb:99: warning: already initialized constant JSON::NaN
/opt/td-agent/embedded/lib/ruby/2.1.0/json/common.rb:99: warning: previous definition of NaN was here
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/json-1.8.1/lib/json/common.rb:101: warning: already initialized constant JSON::Infinity
/opt/td-agent/embedded/lib/ruby/2.1.0/json/common.rb:101: warning: previous definition of Infinity was here
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/json-1.8.1/lib/json/common.rb:103: warning: already initialized constant JSON::MinusInfinity
/opt/td-agent/embedded/lib/ruby/2.1.0/json/common.rb:103: warning: previous definition of MinusInfinity was here
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/json-1.8.1/lib/json/common.rb:128: warning: already initialized constant JSON::UnparserError
/opt/td-agent/embedded/lib/ruby/2.1.0/json/common.rb:128: warning: previous definition of UnparserError was here
/opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:128:in `require': cannot load such file -- google/gax (LoadError)
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:128:in `rescue in require'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:39:in `require'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.22.0/lib/google/cloud/pubsub/v1/publisher_client.rb:28:in `<top (required)>'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.22.0/lib/google/cloud/pubsub/v1.rb:16:in `<top (required)>'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.22.0/lib/google/cloud/pubsub/service.rb:19:in `<top (required)>'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.22.0/lib/google/cloud/pubsub/project.rb:18:in `<top (required)>'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-cloud-pubsub-0.22.0/lib/google/cloud/pubsub.rb:17:in `<top (required)>'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-gcloud-pubsub-custom-0.4.3/lib/fluent/plugin/gcloud_pubsub/client.rb:1:in `<top (required)>'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:128:in `require'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:128:in `rescue in require'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:39:in `require'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-gcloud-pubsub-custom-0.4.3/lib/fluent/plugin/in_gcloud_pubsub.rb:7:in `<top (required)>'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:172:in `block in try_load_plugin'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:170:in `each'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:170:in `try_load_plugin'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:130:in `new_impl'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:55:in `new_input'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:149:in `add_source'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:95:in `block in configure'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:92:in `each'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:92:in `configure'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/engine.rb:129:in `configure'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/engine.rb:103:in `run_configure'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:489:in `run_configure'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:160:in `block in start'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:366:in `call'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:366:in `main_process'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:339:in `block in supervise'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:338:in `fork'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:338:in `supervise'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:156:in `start'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/bin/fluentd:5:in `<top (required)>'
	from /opt/td-agent/embedded/bin/fluentd:23:in `load'
	from /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
	from /usr/sbin/td-agent:7:in `load'
	from /usr/sbin/td-agent:7:in `<main>'
td-agent                                                   [失敗]
@mia-0032
Copy link
Owner

@ikuwow Hi, thank you for your reporting.

First please fix gem name fluent-plugin-gcloud-pubsub-custom not fluent-plugin-gcloud-pubsub.

And I confirmed why this gem don't work on td-agent. It caused by Google gem's wrong permission.

ref.

If above issues are solved or change permission of both google-gax and googleapis-common-protos, it will work fine on td-agent.

@ikuwow
Copy link
Author

ikuwow commented Feb 16, 2017 •
edited
Loading

First please fix gem name fluent-plugin-gcloud-pubsub-custom not fluent-plugin-gcloud-pubsub

Oh, sorry, I pasted wrong logs. Edited.

If above issues are solved or change permission of both google-gax and googleapis-common-protos, it will work fine on td-agent.

Ok, I understand.

Thank you for fast reply.

@LukeHong
Copy link

LukeHong commented Nov 3, 2017

It seems that this issue were fixed.

@ikuwow
Copy link
Author

ikuwow commented Jan 26, 2018

I checked mia-0032/fluent-plugin-gcloud-pubsub-custom was successfully installed by td-agent-gem (td-agent 1.0.2)
There's no problem now! Close.

@ikuwow ikuwow closed this as completed Jan 26, 2018
taku333 pushed a commit to taku333/fluent-plugin-gcloud-pubsub-custom that referenced this issue Dec 10, 2019
Supported　td-agent
8d6df55 
The following URL was confirmed and corrected.
mia-0032#32

Please confirm whether it really supports.
@taku333 taku333 mentioned this issue Dec 10, 2019
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mia-0032 @ikuwow @LukeHong