Encoding::UndefinedConversionError when I tried to save string to mysql dababase #7828

Closed
tborisova opened this Issue Oct 3, 2012 · 2 comments

Projects

None yet

3 participants

@tborisova

In my app the user uploads file and he can choose the encoding of the uploaded file. When I choose correct encodng the code that executes is:
File.open(path, "wb") { |f| f.write(@temp_file['datafile'].read) }
@uploaded_file = File.new @path, "r:#{encoding}"

where encoding is the choosen encoding from the user. Then before string from the file is saved I have:
CSV.read(@uploaded_file, {:headers=>true})

and when I save it to mysql in rails1.9.2-p290 I get this error: Encoding::UndefinedConversionError: "\xD4" from ASCII-8BIT to UTF-8: INSERT INTO "fields" ("created_at", "key", "updated_at", "value") VALUES (?, ?, ?, ?)
in rails1.9.3-p194 I don't get the error but I get error for invalid byte sequence in UTF-8 (ArgumentError) when I only have: CSV.read(@uploaded_file, {:headers=>true}) so I changed it to :
CSV.read(@uploaded_file, {:headers=>true,:encoding=>encoding}) and everything was fine.
The problem is that on the server where my app is hosted the version of rails is 1.9.2-p290 and when we tried to install rails1.9.3-p194 it didn't work because of some permission issues. So can anyone help me with solution of this error ( Encoding::UndefinedConversionError: "\xD4" from ASCII-8BIT to UTF-8: INSERT INTO "fields" ("created_at", "key", "updated_at", "value") VALUES (?, ?, ?, ?)) in rails1.9.2-p290 ? Thanks in advance

For rails 1.9.2-p290 I tried adding this to enviroment.rb
Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8

but it didn't worked.

I tried value.force_encoding('UTF-8')
also value.force_encoding(value.encoding).encode('UTF-8')

I tried with iconv again it didn't worked.

@senny
Member
senny commented Oct 12, 2012

first of all, I don't think this issue is related to rails. It looks like a mess in the encodings. The error:

Encoding::UndefinedConversionError: "\xD4" from ASCII-8BIT to UTF-8 tells me, that your source string is 8-BIT ASCII, which is binary data. This string contains bytes, which can't be mapped to UTF-8.

There could be several problems why you see that error.

  • You need to apply the correct encoding when reading in the CSV file
  • You need to make sure, that the date in your CSV file does not contain corrupted byte sequences

you should read http://yehudakatz.com/2010/05/05/ruby-1-9-encodings-a-primer-and-the-solution-for-rails/ and then try to figure out where the root of your problem lies.

@steveklabnik @rafaelfranca should we close this? I don't think there is a connection to rails.

@rafaelfranca
Member

I have to agree with @senny. Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment