Encoded txt files as UTF-8 #4
Conversation
|
Thanks for doing that. Did you do this manually or programmatically? Reason I ask, is that our vendor will be pushing updates to FTP that we then automatically add and commit to the repo. I'm seeing if they can just make sure to save with UTF-8 encoding on their end, but just in case, I may need to script this so it doesn't have to be done manually. |
|
Programmatically. Here's my quick hack: require 'charlock_holmes'
detector = CharlockHolmes::EncodingDetector.new
ARGV.each do|f|
content = File.read(f)
detection = detector.detect(content)
puts "#{f} encoding: #{detection[:encoding]}"
utf8_encoded_content = CharlockHolmes::Converter.convert content, detection[:encoding], 'UTF-8'
File.write(f, utf8_encoded_content)
end |
| (J) Petitioning�Requesting persons to sign a petition. | ||
| (K) Publicize�To inform the public of a planned event by means of newspaper articles or notices, radio or television stories or notices, announcements in public places, leafletting, posting signs or written notices in places viewed by the public, or by other means calculated to notify the public of an event. | ||
| (L) Soliciting�Requesting persons to contribute money or anything else of value for charitable, religious or political cause. | ||
| (A) Amusement Park RidesRides of the type normally found in amusement parks or carnivals, such as ferris wheels. |
There was a problem hiding this comment.
#4 In the original Municiap Code, these are em dashes, I believe, encoding them removes them. Thoughts on why this is happening?
|
That's possible. I'm not sure there's a way to have the transcoder be a bit smarter about characters like that, but I'll look. Think the best way forward is to have the vendor encode as UTF-8 directly. |
|
#2 addressed |
No description provided.