New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ArgumentError: invalid byte sequence in UTF-8 #12

Merged
merged 2 commits into from Mar 7, 2014

Conversation

Projects
None yet
2 participants
@y-ken
Member

y-ken commented Feb 26, 2014

It would be get error when evaluate variables has broken byte.

error detail

test_emit9_invalid_byte(RewriteTagFilterOutputTest):
ArgumentError: invalid byte sequence in UTF-8
  ...snip.../fluent-plugin-rewrite-tag-filter/lib/fluent/plugin/out_rewrite_tag_filter.rb:60:in `match'
  ...snip.../fluent-plugin-rewrite-tag-filter/lib/fluent/plugin/out_rewrite_tag_filter.rb:60:in `block in rewrite_tag'
  ...snip.../fluent-plugin-rewrite-tag-filter/lib/fluent/plugin/out_rewrite_tag_filter.rb:57:in `each'
  ...snip.../fluent-plugin-rewrite-tag-filter/lib/fluent/plugin/out_rewrite_tag_filter.rb:57:in `rewrite_tag'
  ...snip.../fluent-plugin-rewrite-tag-filter/lib/fluent/plugin/out_rewrite_tag_filter.rb:47:in `block in emit'
  ...snip.../fluent-plugin-rewrite-tag-filter/vendor/bundle/ruby/1.9.1/gems/fluentd-0.10.43/lib/fluent/event.rb:54:in `call'
  ...snip.../fluent-plugin-rewrite-tag-filter/vendor/bundle/ruby/1.9.1/gems/fluentd-0.10.43/lib/fluent/event.rb:54:in `each'
  ...snip.../fluent-plugin-rewrite-tag-filter/lib/fluent/plugin/out_rewrite_tag_filter.rb:46:in `emit'
  ...snip.../fluent-plugin-rewrite-tag-filter/vendor/bundle/ruby/1.9.1/gems/fluentd-0.10.43/lib/fluent/test/output_test.rb:44:in `emit'
  ...snip.../fluent-plugin-rewrite-tag-filter/test/plugin/test_out_rewrite_tag_filter.rb:239:in `block in test_emit9_invalid_byte'
  ...snip.../fluent-plugin-rewrite-tag-filter/vendor/bundle/ruby/1.9.1/gems/fluentd-0.10.43/lib/fluent/test/input_test.rb:80:in `call'
  ...snip.../fluent-plugin-rewrite-tag-filter/vendor/bundle/ruby/1.9.1/gems/fluentd-0.10.43/lib/fluent/test/input_test.rb:80:in `block in run'
  ...snip.../fluent-plugin-rewrite-tag-filter/vendor/bundle/ruby/1.9.1/gems/fluentd-0.10.43/lib/fluent/test/base.rb:53:in `run'
  ...snip.../fluent-plugin-rewrite-tag-filter/vendor/bundle/ruby/1.9.1/gems/fluentd-0.10.43/lib/fluent/test/input_test.rb:75:in `run'
  ...snip.../fluent-plugin-rewrite-tag-filter/test/plugin/test_out_rewrite_tag_filter.rb:237:in `test_emit9_invalid_byte'

It has reported by bluewhale07, thank you!
https://twitter.com/bluewhale07/status/435673677309292544

Related issue

tagomoris/fluent-plugin-parser#3

@y-ken

This comment has been minimized.

Show comment
Hide comment
@y-ken

y-ken Feb 26, 2014

Member

Hi @repeatedly

string-scrub gem now supports Ruby 1.9.3.
I think string-scrub is better than this ad-hoc routine.

Thank you for giving me good advice.
I have updated the code now.

I feel scrub character is not best for this case because '?' character will be used in tag for this case.
How do you think?

Member

y-ken commented Feb 26, 2014

Hi @repeatedly

string-scrub gem now supports Ruby 1.9.3.
I think string-scrub is better than this ad-hoc routine.

Thank you for giving me good advice.
I have updated the code now.

I feel scrub character is not best for this case because '?' character will be used in tag for this case.
How do you think?

d1.emit({'client_name' => invalid_utf8})
end
emits = d1.emits
p emits

This comment has been minimized.

@repeatedly

repeatedly Feb 26, 2014

Member

These p are should be removed?

@repeatedly

repeatedly Feb 26, 2014

Member

These p are should be removed?

This comment has been minimized.

@y-ken

y-ken Feb 26, 2014

Member

Yes. it should be.
I'm gonna remove them after this PR has merged.

@y-ken

y-ken Feb 26, 2014

Member

Yes. it should be.
I'm gonna remove them after this PR has merged.

@y-ken

This comment has been minimized.

Show comment
Hide comment
@y-ken
Member

y-ken commented Mar 7, 2014

y-ken added a commit that referenced this pull request Mar 7, 2014

Merge pull request #12 from fluent/invalid_byte_sequence
ArgumentError: invalid byte sequence in UTF-8

@y-ken y-ken merged commit a5476ac into master Mar 7, 2014

1 check passed

default The Travis CI build passed
Details

@y-ken y-ken deleted the invalid_byte_sequence branch Mar 7, 2014

y-ken added a commit that referenced this pull request Mar 10, 2014

y-ken added a commit that referenced this pull request Mar 10, 2014

y-ken added a commit that referenced this pull request Oct 31, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment