-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
encoding issue #49
Comments
The encoding of the database file has not changed. The strings are stored internally as ISO-8859-1. |
I think Gregory is right, which means I need to add conversions to all strings. That won't happen for a while, unless I get a solid PR from someone else. |
Thanks for the replies and sorry for misinterpreting IRB's output. If the db encoding is not changed than I think the problem is limited to changing https://github.com/cjheath/geoip/blob/master/lib/geoip.rb#L792, i.e. almost all fields are already in UTF-8 except for E.g. notice the difference between "Sao Paulo" as timezone, city_name and region_name.
(I am not sure why So the only change would be
to
I can add a couple tests and make the change and send a pull request, but I couldn't find tests for the city database, are there any? |
I just ran into this issue today as well. The one-line fix that the previous poster suggested looks good to me: city.force_encoding('iso-8859-1').encode('utf-8') if city.respond_to?(:force_encoding) |
Apologies for the delay releasing this fix. I had introduced a breaking change and not had time to diagnose it. The gem is now released as 1.5.0 |
Thanks! |
with recent GeoLiteCity.dat files I think the text encoding has changed, i.e.
notice that city_name is badly encoded. Calling
.encode('utf-8')
on it correctly converts it.in #6 the code was updated to force encoding of the strings to
iso-8859-1
, so that seems to cause this issue.This is a problem because the string is in fact utf-8 but considered latin-1 and when concatenated with a string in utf8 it causes a runtime error. I can fix this in user code by re-encoding in utf-8 but it seems wasteful.
I don't know if there is a way to detect the encoding of the database file, but I think we might want to provide a flag to the db constructor or default to utf8 if necessary.
The text was updated successfully, but these errors were encountered: