Smaller tables #68

benibela · 2016-05-31T23:52:00Z

…es 10kb)

…tf8proc_totitle, as title_mapping cannot be used to get the title codepoint anymore. Rename xxx_mapping to xxx_seqindex, so programs assuming a value with the old meaning fail at compile time

ivarne · 2016-06-01T08:56:33Z

Seems like travis is failing because of an expired cert

# use JuliaLang caching (https://github.com/staticfloat/cache.julialang.org)
# so that Travis builds do not depend on anyone's flaky servers but our own
    - URLCACHE=https://cache.e.ip.saba.us/

cc: @staticfloat

ivarne · 2016-06-01T10:44:07Z

The travis failiure will probably be fixed by #69

tkelman · 2016-06-01T12:59:06Z

It might need a rebase and force push though. Changes to .travis.yml might not propagate automatically by just restarting the build.

stevengj · 2016-06-01T13:10:11Z

Needs a version bump. Since this is a technically a backwards incompatible change, we'll need to change the major version?

Maybe we should take the opportunity to export other accessor functions, like utf8proc_casefold(codepoint), utf8proc_bidi(codepoint), ...?

We could unexport the utf8proc_property_struct entirely, replacing it by utf8proc_property(codepoint) functions. Theoretically, this would make it less efficient to extract multiple properties of a codepoint at once, but I'm skeptical that this is common in practice.

…be merged in a single array

stevengj · 2016-06-24T13:39:43Z

is this ready to merge?

benibela · 2016-06-24T14:43:40Z

Unless you want to increase the version number first for people using the property struct directly.

And Unicode 9 was just released. Might be worth checking if it still fits in the tables. Some are almost full.

stevengj · 2016-06-28T20:04:55Z

Needs a rebase and update now that Unicode 9 support has been merged.

stevengj · 2016-07-02T15:14:36Z

utf8proc.h

+ * title-case character, if any; otherwise (if there is no title-case
+ * variant, or if `c` is not a valid codepoint) return `c`.
+ */
+UTF8PROC_DLLEXPORT utf8proc_int32_t utf8proc_totitle(utf8proc_int32_t c);


Should this be equivalent to toupper if there is no titlecase?

stevengj · 2016-07-02T15:15:15Z

Awesome, thanks for updating this. Did you have any problems with the table size?

stevengj · 2016-07-02T15:18:06Z

What's the overall reduction in (compiled) table size from this patch?

benibela · 2016-07-02T17:07:35Z

200 k (42%)

7975 elements in the sequence array now (from 7834). At 8192 it gets troublesome

stevengj · 2021-11-28T14:46:40Z

data/data_generator.rb

+    lencode = 7
+  end  
+  idx = pushary(array) 
+  raise "Array index out of bound" if idx > 0x1FFF


This exception triggered in Unicode 14. @benibela, what do you suggest?

@stevengj one bit needs to be taken from lencode (7 -> 3) and given to idx (0x1FFF -> 0x3FFF, 13 -> 14)

and the same in seqindex_write_char_decomposed

Could you make a PR?

benibela added 7 commits May 31, 2016 00:25

convert sequences to utf-16 (saves 25kb)

ae05923

store sequence length in properties instead using -1 termination (sav…

a1fc417

…es 10kb)

cache index for slightly faster data creation

462c8d1

store lower/upper/title mapping in sequence array (saves 25kb). Add u…

d290275

…tf8proc_totitle, as title_mapping cannot be used to get the title codepoint anymore. Rename xxx_mapping to xxx_seqindex, so programs assuming a value with the old meaning fail at compile time

change combination array data type to uint16 (saves 40kb)

0b31fed

merge 1st and 2nd comb index (saves 50kb)

76a4a0f

kill empty prefix/suffix in combination array (saves 50kb)

14a7c7d

benibela added 2 commits June 1, 2016 15:49

Merge https://github.com/JuliaLang/utf8proc into smallertables

fa48b92

there was no need to have a separate combination start array, it can …

d25e485

…be merged in a single array

Keno mentioned this pull request Jun 25, 2016

Unicode 9 updates #70

Merged

benibela added 2 commits July 2, 2016 14:50

merge unicode 9

bd5e0df

some fixes

6d9fd75

stevengj reviewed Jul 2, 2016
View reviewed changes

benibela added 2 commits July 2, 2016 19:13

mark the table as const again

bac56b7

and regen

ad58d09

stevengj merged commit eeebf70 into JuliaStrings:master Jul 12, 2016

stevengj mentioned this pull request Jul 12, 2016

Walk back ABI breaking changes #76

Merged

Keno mentioned this pull request Jul 12, 2016

Minimizing tables for (de)composition #67

Closed

stevengj reviewed Nov 28, 2021

View reviewed changes

benibela mentioned this pull request Dec 16, 2021

reduce lenencode bits #232

Merged

inkydragon mentioned this pull request Oct 9, 2024

update for Unicode 16.0.0 #271

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Smaller tables #68

Smaller tables #68

benibela commented May 31, 2016

ivarne commented Jun 1, 2016

ivarne commented Jun 1, 2016

tkelman commented Jun 1, 2016

stevengj commented Jun 1, 2016 •

edited

Loading

stevengj commented Jun 24, 2016

benibela commented Jun 24, 2016

stevengj commented Jun 28, 2016

stevengj Jul 2, 2016

stevengj commented Jul 2, 2016

stevengj commented Jul 2, 2016

benibela commented Jul 2, 2016 •

edited

Loading

stevengj Nov 28, 2021

benibela Dec 13, 2021

stevengj Dec 14, 2021

Smaller tables #68

Smaller tables #68

Conversation

benibela commented May 31, 2016

ivarne commented Jun 1, 2016

ivarne commented Jun 1, 2016

tkelman commented Jun 1, 2016

stevengj commented Jun 1, 2016 • edited Loading

stevengj commented Jun 24, 2016

benibela commented Jun 24, 2016

stevengj commented Jun 28, 2016

stevengj Jul 2, 2016

Choose a reason for hiding this comment

stevengj commented Jul 2, 2016

stevengj commented Jul 2, 2016

benibela commented Jul 2, 2016 • edited Loading

stevengj Nov 28, 2021

Choose a reason for hiding this comment

benibela Dec 13, 2021

Choose a reason for hiding this comment

stevengj Dec 14, 2021

Choose a reason for hiding this comment

stevengj commented Jun 1, 2016 •

edited

Loading

benibela commented Jul 2, 2016 •

edited

Loading