fix #46 (make sure symbol-like codepoints have nonzero width)

[stevengj]

This uses width 1 for symbols that are missing from Unifont but have categories indicating that they have nonzero width. See #46.

It also corrects for a few apparent bugs in Unifont's widths (https://savannah.gnu.org/bugs/index.php?45395).

[ScottPJones]

👍 LGTM

[stevengj]

@jiahao, is this the same failure you were seeing in #45?

[jiahao]

It's very similar. The mdf5 hashes agree for both the cached and original versions of UnicodeData.txt - 3a83069e69e2a9101dc4749593cd3268. Cannot reproduce locally.

[jiahao]

My ruby version is ruby 2.1.2p95 (2014-05-08) [x86_64-linux-gnu]

[jiahao]

Looks like we are seeing ruby version-specific behavior. I get a different utf8proc_data.c output on ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14] with the same UnicodeData.txt. Our Travis build uses ruby-1.9.3-p551.

[stevengj]

I'm using Ruby 2.0.0p481.

Perhaps the culprit is the last few lines of data_generator.rb:

$stdout << "const utf8proc_int32_t utf8proc_combinations[] = {\n  "
i = 0
comb1st_indicies.keys.each_index do |a|
  comb2nd_indicies.keys.each_index do |b|
    i += 1
    if i == 8
      i = 0
      $stdout << "\n  "
    end
    $stdout << ( comb_array[a][b] or -1 ) << ", "
  end
end
$stdout << "};\n\n"

It looks like the output order could depend on the order of the keys in a hash table. Probably we should just sort them.

[ScottPJones]

Maybe we should just rewrite that in a better language? 😀 I happen to know a very nice one!

[stevengj]

Nope, that wasn't it.

[jiahao]

I don't think the current Unicode data file is sorted

[stevengj]

I regenerated the unicode_data.c file and it didn't change for me...

[stevengj]

Okay, whatever @jiahao did seems to have worked.