-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problem when converting from byte to UTF8 #35
Comments
Hi, as for the Quarantine.mail_text field, we could try to use a BinaryField and to remove all calls to convert_from. Do you think you could try it ? |
@carragom ping |
Hi @carragom, your problem seems to be related to the email field: |
From @carragom on December 18, 2015 22:25 @tonioo Yes that's the field causing the problems. Like I said, I see two options to fix this: 1- Keep the conversion in the database as it's now, but ask the database to convert to 2- Switch to using I have been looking around here to see if there is any indication on what encoding it's actually used without luck. But it does mention that the option If you ask me I would just go option number 1 which is simple to implement and have no performance issues. I can provide a quick PR for option 1 if you decide to go with it. |
@carragom Using LATIN1 as encoding won't cover all cases. I guess we will encounter the same issues with another encoding soon. I still think a BinaryField is the right answer and I do hope Django uses the appropriate field when it generates queries. The manual conversion you see in the current code would also disappear. |
@carragom BTW, the right place for this issue is into the https://github.com/modoboa/modoboa-amavis repository. |
From @carragom on January 27, 2016 23:40 @tonioo I agree that LATIN1 does not cover all cases and it's far from ideal. The one thing for sure is that this problem renders Modoboa unusable, all of it, not just the amavis module. So this should be fixed in any way necessary. It might be possible that amavis does not intent for these fields to be used as text, from the amavis README.sql-pg.txt:
Thanks a lot for your time. |
Please look at this thread (the end of the page is interesting): And tell me what do you think :) |
From @carragom on January 29, 2016 19:2 Yes using a In any case, it does not matter what type of field is used or where the conversion happens (db or app) at some point those bytes on the database will have to be converted to text in order to be useful and the conversion will require a character set. Again I see two options: 1- Find a way to handle the conversion gracefully at the database level (maybe a stored procedure would help here or just use Again thanks for your time, I hope I was a bit more clear this time. |
@carragom So, BinaryField is definitely not the answer because we can't filter on it... I finally added a new setting to configure the encoding to use (default value is now LATIN1) and I removed some useless convert_from calls. |
From @carragom on November 13, 2015 5:51
Hi there,
I have some data in my amavis database that's rendering modoboa unusable. All parts of the system throw an internal server error. I have traced the issue to the file
modoboa/extensions/amavis/sql_conector.py
. The problem happens with all queries that use the functionconvert_from(maddr.email, 'UTF8')
. Turns out if I manually run the query directly in PostgreSQL it throws the error:There seems to be data in there that's not UTF8 compliant. At first I saw #677 and upgraded to 1.2.2 thinking it was the same problem. But this did not fixed the issue. In the end I replaced all
convert_from
functions inmodoboa/extensions/amavis/sql_conector.py
to useLATIN1
instead ofUTF8
and the issue is now fixed.In the end, the problematic character was a
ñ
. I know RFC6531 states that email addresses could be encoded inUTF8
but amavis does not seem to be playing by those rules; yet?. An immediate solution is to switch toLATIN1
and hope it's the right encoding.As an alternate solution the data could be retrieved in binary form from the database and converted in python maybe even a django filter that would fallback to something like "invalid address" if the conversion fails. But this would force modoboa to retrive all record from the database and in my case that would be around 7K and counting.
Hope this helps.
Cheers.
Copied from original issue: modoboa/modoboa#782
The text was updated successfully, but these errors were encountered: