0.3.45 invalid byte sequence in US-ASCII (ArgumentError) #41

Closed
andoke opened this Issue Nov 30, 2015 · 5 comments

Comments

Projects
None yet
2 participants
@andoke

andoke commented Nov 30, 2015

I got that error by running tzinfo on 1.9.3:

/home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/formatters/base_text_formatter.rb:301:in `split': invalid byte sequence in US-ASCII (ArgumentError)
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/formatters/base_text_formatter.rb:301:in `dump_failure_info'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/formatters/base_text_formatter.rb:293:in `dump_failure'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/formatters/base_text_formatter.rb:24:in `block in dump_failures'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/formatters/base_text_formatter.rb:22:in `each'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/formatters/base_text_formatter.rb:22:in `each_with_index'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/formatters/base_text_formatter.rb:22:in `dump_failures'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/reporter.rb:127:in `block in notify'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/reporter.rb:126:in `each'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/reporter.rb:126:in `notify'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/reporter.rb:109:in `finish'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/reporter.rb:60:in `report'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/command_line.rb:25:in `run'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/runner.rb:80:in `run'
    from /home/fnguyen/.rvm/gems/ruby-1.9.3-p551/gems/rspec-core-2.14.8/lib/rspec/core/runner.rb:17:in `block in autorun'

It probably has to do with that 1.9.3 convert string by default in ASCII and file in utf-8

@philr

This comment has been minimized.

Show comment
Hide comment
@philr

philr Nov 30, 2015

Member

I can't see anything in your stack trace that relates to TZInfo. Can you post some code that reproduces the exception? Why do you think that TZInfo is the cause of the problem?

You mention version 0.3.45. Does the problem also occur with the latest version of TZInfo (currently v1.2.2)?

Member

philr commented Nov 30, 2015

I can't see anything in your stack trace that relates to TZInfo. Can you post some code that reproduces the exception? Why do you think that TZInfo is the cause of the problem?

You mention version 0.3.45. Does the problem also occur with the latest version of TZInfo (currently v1.2.2)?

@andoke

This comment has been minimized.

Show comment
Hide comment
@andoke

andoke Nov 30, 2015

It's one of my test:

class Partner
...
...
    def timezone
      validate_timezone || timezone_from_country
    end

    private

    def timezone_from_country
      TZInfo::Country.get(country_code).zone_identifiers.first
    rescue TZInfo::InvalidCountryCode
      'Europe/Paris'
    end
end

it "return a default time zone it cannot be retrieved from country code" do
    partner = Partner.new(:timezone => 'BadZone', :country_code => '')
    partner.timezone.should == 'Europe/Paris'
end

I don't know if this occurs on 1.2.2, but that code works fine on 0.3.44.

andoke commented Nov 30, 2015

It's one of my test:

class Partner
...
...
    def timezone
      validate_timezone || timezone_from_country
    end

    private

    def timezone_from_country
      TZInfo::Country.get(country_code).zone_identifiers.first
    rescue TZInfo::InvalidCountryCode
      'Europe/Paris'
    end
end

it "return a default time zone it cannot be retrieved from country code" do
    partner = Partner.new(:timezone => 'BadZone', :country_code => '')
    partner.timezone.should == 'Europe/Paris'
end

I don't know if this occurs on 1.2.2, but that code works fine on 0.3.44.

@philr

This comment has been minimized.

Show comment
Hide comment
@philr

philr Nov 30, 2015

Member

What does your Partner class do? Can you post some code that can be run independently to reproduce the problem?

The v0.3.x branch of TZInfo was built with Ruby 1.8.x and pre-dates Ruby's encoding support. I'm maintaining it with data updates only mainly for people who are unable to upgrade to Ruby on Rails 4.1 and later. I'd recommend upgrading to the latest version of TZInfo if you can.

It looks like the IANA tz database has changed its iso3166.tab file so that it no longer contains solely ASCII characters (eggert/tz@30a8c54). This change is included in the lib/tzinfo/indexes/countries.rb file in the tzinfo v0.3.45 gem. You may be able to prevent the exception by adding the following line to the top of this file:

# encoding: UTF-8
Member

philr commented Nov 30, 2015

What does your Partner class do? Can you post some code that can be run independently to reproduce the problem?

The v0.3.x branch of TZInfo was built with Ruby 1.8.x and pre-dates Ruby's encoding support. I'm maintaining it with data updates only mainly for people who are unable to upgrade to Ruby on Rails 4.1 and later. I'd recommend upgrading to the latest version of TZInfo if you can.

It looks like the IANA tz database has changed its iso3166.tab file so that it no longer contains solely ASCII characters (eggert/tz@30a8c54). This change is included in the lib/tzinfo/indexes/countries.rb file in the tzinfo v0.3.45 gem. You may be able to prevent the exception by adding the following line to the top of this file:

# encoding: UTF-8
@andoke

This comment has been minimized.

Show comment
Hide comment
@andoke

andoke Nov 30, 2015

Yeah you were right so countries.rb needs encoding: UTF-8 for those who uses ruby =< 1.9.x
Adding # encoding: UTF-8 to countries.rb solved the problem

andoke commented Nov 30, 2015

Yeah you were right so countries.rb needs encoding: UTF-8 for those who uses ruby =< 1.9.x
Adding # encoding: UTF-8 to countries.rb solved the problem

philr added a commit that referenced this issue Dec 2, 2015

Support UTF-8 encoded files in the IANA Time Zone Database distribution.
From version 2015e, the iso3166.tab file contains UTF-8 characters.
Backport the encoding handling code in TZDataParser from TZInfo::Data.
Resolves #41.

@philr philr added the bug label Dec 2, 2015

@philr

This comment has been minimized.

Show comment
Hide comment
@philr

philr Dec 2, 2015

Member

This issue would have been encountered when the countries index was loaded (e.g. when calling TZInfo::Country.all). It would have only occurred with Ruby 1.9.x, which has a default script encoding of US-ASCII. Ruby 2.0 and later releases have a default script encoding of UTF-8, so would have been able to load the UTF-8 encoded strings from countries.rb in TZInfo 0.3.45.

TZInfo version 0.3.46 includes backported encoding handling changes that were made in TZInfo::Data. All of the definition and index files now include an # encoding: UTF-8 comment to allow them to be loaded with Ruby 1.9.

Member

philr commented Dec 2, 2015

This issue would have been encountered when the countries index was loaded (e.g. when calling TZInfo::Country.all). It would have only occurred with Ruby 1.9.x, which has a default script encoding of US-ASCII. Ruby 2.0 and later releases have a default script encoding of UTF-8, so would have been able to load the UTF-8 encoded strings from countries.rb in TZInfo 0.3.45.

TZInfo version 0.3.46 includes backported encoding handling changes that were made in TZInfo::Data. All of the definition and index files now include an # encoding: UTF-8 comment to allow them to be loaded with Ruby 1.9.

@philr philr closed this Dec 2, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment