Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Invalid byte sequence in UTF-8 #70

Closed
Ruben-E opened this Issue May 15, 2013 · 6 comments

Comments

Projects
None yet
3 participants

Ruben-E commented May 15, 2013

Hello,

Sorry for the delay between my question on Stackoverflow and this issue on Github.

I had an issue running Metric_fu in my Ruby on Rails 3.2.13 project.

This was the output last friday:

$ bundle exec metric_fu -r
F, [2013-05-10T17:11:45.980014 #26798] FATAL -- : invalid byte sequence in UTF-8 (ArgumentError)
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/lib/churn/git_analyzer.rb:16:in `split'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/lib/churn/git_analyzer.rb:16:in `get_diff'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/lib/churn/source_control.rb:37:in `get_updated_files_from_log'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/lib/churn/source_control.rb:11:in `get_updated_files_change_info'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/lib/churn/churn_calculator.rb:250:in `parse_logs_for_updated_files'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/lib/churn/churn_calculator.rb:177:in `calculate_revision_data'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/lib/churn/churn_calculator.rb:165:in `block in calculate_revision_changes'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/lib/churn/churn_calculator.rb:160:in `each'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/lib/churn/churn_calculator.rb:160:in `calculate_revision_changes'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/lib/churn/churn_calculator.rb:59:in `analyze'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/lib/churn/churn_calculator.rb:44:in `report'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/bin/churn:32:in `report_churn'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/bin/churn:40:in `run'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/main-5.2.0/lib/main/program/class_methods.rb:155:in `block in run'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/main-5.2.0/lib/main/program/class_methods.rb:144:in `catch'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/main-5.2.0/lib/main/program/class_methods.rb:144:in `run'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/main-5.2.0/lib/main/factories.rb:18:in `run'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/main-5.2.0/lib/main/factories.rb:25:in `Main'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/churn-0.0.28/bin/churn:6:in `<top (required)>'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/bin/mf-churn:11:in `load'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/bin/mf-churn:11:in `<top (required)>'
/usr/local/rvm/gems/ruby-1.9.3-p194/bin/mf-churn:19:in `load'
/usr/local/rvm/gems/ruby-1.9.3-p194/bin/mf-churn:19:in `<main>'
/usr/local/rvm/gems/ruby-1.9.3-p194/bin/ruby_noexec_wrapper:14:in `eval'
/usr/local/rvm/gems/ruby-1.9.3-p194/bin/ruby_noexec_wrapper:14:in `<main>'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/lib/metric_fu/metrics/churn/churn.rb:10:in `analyze': undefined method `match' for nil:NilClass (NoMethodError)
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/lib/metric_fu/metrics/generator.rb:129:in `block in generate_report'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/lib/metric_fu/metrics/generator.rb:127:in `each'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/lib/metric_fu/metrics/generator.rb:127:in `generate_report'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/lib/metric_fu/reporting/report.rb:63:in `add'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/lib/metric_fu/run.rb:25:in `block in run_reports'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/lib/metric_fu/run.rb:23:in `each'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/lib/metric_fu/run.rb:23:in `run_reports'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/lib/metric_fu/run.rb:10:in `run'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/lib/metric_fu/cli/helper.rb:11:in `run'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/lib/metric_fu/cli/client.rb:18:in `run'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/metric_fu-4.1.2/bin/metric_fu:9:in `<top (required)>'
from /usr/local/rvm/gems/ruby-1.9.3-p194/bin/metric_fu:19:in `load'
from /usr/local/rvm/gems/ruby-1.9.3-p194/bin/metric_fu:19:in `<main>'
from /usr/local/rvm/gems/ruby-1.9.3-p194/bin/ruby_noexec_wrapper:14:in `eval'
from /usr/local/rvm/gems/ruby-1.9.3-p194/bin/ruby_noexec_wrapper:14:in `<main>'

I found an similar question here: https://github.com/jscruggs/metric_fu/issues/61. As a result of the question, I created a file named '.metrics' in the root directory of my project, with the following content:

MetricFu::Configuration.run do |config|   
  config.syntax_highlighting = false
end

Unfortunately this didn't work.

However...

Metric_fu is working again. I disabled churn once, by adding --no-churn. Since then Metric_fu is fully operational, even with Churn activated in the next reports (same code was analyzed)… Somehow something seems to be resetted, what suggest that it might be an issue in one of the Metric_fu / Churn files, since the code wasn't changed.

Currently I am not able to reproduce the issue. Sadly I cannot share the source code because it is property of the company I work for.

My Ruby version is / was: ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.4.0]

Metric_fu version: 4.1.2

Although it has been fixed for me, I still decided to report this issue.

Thanks,
Ruben

@danmayer danmayer was assigned May 15, 2013

Owner

bf4 commented May 15, 2013

Thanks. Which versio of metric_fu are you on? metric_fu --version

Ruben-E commented May 15, 2013

Sorry, forgot to add. 4.1.2.

So looks like it isn't while parsing a file but actually on the diff trying to find which files have changed. I found this post

http://stackoverflow.com/questions/2982677/ruby-1-9-invalid-byte-sequence-in-utf-8

which makes me think I could try to force fix the encoding. I don't think I could really capture the file causing the issue since it is actually on a large git diff output.

Anyone have thoughts if forcing encoding sounds like a good solution?

Owner

bf4 commented May 19, 2013

@danmayer We use this on one of our projects, might help

# language support
  # convert non-ascii (e.g. accented characters and ligatures)
  # to ascii equivalents for url slugs and filenames
gem 'talentbox-unidecoder', :require => 'unidecoder'

Alternatively, I have done something like file_contents.encode!('UTF-8', 'ASCII', :invalid => :replace, :replace => '') before which was good enough. It actually removes unconvertable characters.

I know anecdotally Rails deals with this in the codebase but haven't seen the code myself, yet

I do think the canonical exception for this is ArgumentError so you could at least check for that

Owner

bf4 commented Jun 3, 2013

@danmayer any updates, thoughts? should we close this as a churn issue, or leave it open till churn handles the error?

danmayer commented Jun 4, 2013

Sorry I have been traveling a bit and I am out of town one more weekend so I haven't looked at it to much more. We could close this here and I can follow it in churn.

I will try to provide some better errors, and perhaps some error output that metric_fu could parse and display

@bf4 bf4 closed this Jun 4, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment