Hyphenation patterns update (cleaned up #376) #378

poire-z · 2020-09-17T08:40:24Z

See #376 for the whole discussion and work steps.
This is just #376 with the 33 work commits squashed into 17 logical commits.

This change is

poire-z · 2020-09-17T08:42:13Z

Pinging @roshavagarga and @cramoisi (I can't add you as reviewers...) for a quick look and confirmation that I didn't mess anything, before merging.

cramoisi · 2020-09-17T08:46:40Z

I will look into it after work :)

roshavagarga · 2020-09-17T08:48:37Z

cr3gui/data/hyph/languages.json

+        "filename": "Latvian.pattern",
+        "language": "lv",
+        "left_hyphen_min": 2,
+        "right_hyphen_min": 2


I missed adding a comma after this line (157)

Yup, just saw that, going to fix it.

roshavagarga · 2020-09-17T08:53:08Z

I did my once-over and it seems fine to me, though I did notice this has 1 less new line compared to the original, but no idea where that 1 missing line is from, probably something I missed and you fixed?

poire-z · 2020-09-17T08:54:50Z

Yes, a last trailing blank line in Esperanto.pattern that I removed (to avoid github highlighting it as "not clean").

roshavagarga · 2020-09-17T08:57:03Z

Ah, those always annoyed me, good to know I can get rid of them 👍

poire-z · 2020-09-17T09:35:10Z

Added 2 commits to rename badly named Romanian and Ukrainian pattern files, and some methods to help cleaning up frontend readertypography.lua code.

roshavagarga · 2020-09-17T10:18:55Z

@poire-z Not sure how to add changes to this PR, so I've attached the latest Spanish pattern :)

Spanish.zip

poire-z · 2020-09-17T10:45:56Z

OK, added.

I tried learning git some time ago and just gave up out of frustration and stuck with using github's web ui.

If you plan to do more work/fixes like these, and you have a linux machine / can do git command line stuff, I can give you a small cheatsheet on how to proceed to make simple PRs on new branches (that I could more easily reword, but even you, and we'll be here to help if you get stuck).

roshavagarga · 2020-09-17T10:49:34Z

Yeah, the one Linux machine I have is a decrepit single-core tiny notebook from like 7-8 years ago that's running Lubuntu, because nothing else runs well on it :/
I generally stick to Windows 7, because gaming, but you can still share the cheatsheet and if I do resort to it I can always boot from a usb drive or something.

cramoisi · 2020-09-17T10:37:00Z

cr3gui/data/hyph/Bulgarian.pattern

line 7 must be deleted

cramoisi · 2020-09-17T10:47:42Z

cr3gui/data/hyph/Esperanto.pattern

+<pattern>unu3a2nim</pattern>
+<pattern>uo2</pattern>
+<pattern> uv2u3l</pattern>
+<pattern>uzulinterfaco</pattern>


please move the line 1964 at the end of the file, and comment it :)

cramoisi · 2020-09-17T10:50:08Z

cr3gui/data/hyph/Irish.pattern

+<pattern>dtiom5áintí</pattern>
+<pattern>thiom5áintí</pattern>


could you move these lines 6085-6086 before your block of commented exceptions ?

cramoisi · 2020-09-17T10:52:55Z

cr3gui/data/hyph/Norwegian.pattern

+  <pattern>upp5yver</pattern>
+  <pattern>ut5ørk</pattern>
+  <pattern>ut5ørken</pattern>
+<!--  <pattern>velan</pattern>


could you gather all your commented lines in one block at the end of the file ?

cramoisi · 2020-09-17T10:54:05Z

cr3gui/data/hyph/Slovak.pattern

+<!-- Exceptions, see https://github.com/koreader/crengine/pull/376
+<pattern>dosť</pattern>
+-->


commented lines at the end of file ? ;)

cramoisi · 2020-09-17T10:55:49Z

cr3gui/data/hyph/Spanish.pattern

+  <pattern> re3aprend</pattern>
+  <pattern> re3aprénd</pattern>
+  <pattern> re3apret</pattern>
+  <pattern> reapríet</pattern>


to be commented and moved at the end of the file

All 4 ? Or just the last one without any number?

I'll assume it's just the last one.

Of, right, you answered that below :)

yup. only the non-numbered one. I messed with my line selection, sorry :)

cramoisi · 2020-09-17T10:56:17Z

cr3gui/data/hyph/Spanish.pattern

+  <pattern> reu3nia</pattern>
+  <pattern> reu3ní</pattern>
+  <pattern> reu3nis</pattern>
+  <pattern> reunim</pattern>


to be commented and move at the end of the file

Only the ones without numbers @ 4098, right?

cramoisi · 2020-09-17T10:59:11Z

crengine/src/textlang.cpp

+    { "en-GB", "English_GB",    "English_GB.pattern",    2, 3 },
+    { "en",    "English_US",    "English_US.pattern",    2, 3 },
+    { "eo",    "Esperanto",     "Esperanto.pattern",     2, 2 },
+    { "et",    "Estonian",      "Estonian.pattern",      2, 3 },
    { "fi",    "Finnish",       "Finnish.pattern",       2, 2 },
    { "fr",    "French",        "French.pattern",        2, 1 },


could you add // see French.pattern file as a comment for french 2,1 ?

poire-z · 2020-09-17T11:22:45Z

(@roshavagarga : I'll do the requested modifications - later this afternoon.)

poire-z · 2020-09-17T11:28:25Z

OK, some quick notes (build from my own notes, removing more complex and rare stuff):

FORK repo on github

CLONE AND SETUP UPSTREAM :
$ git clone https://github.com/yourname/crengine.git
$ git remote -v
$ git remote add upstream https://github.com/koreader/crengine.git
$ git remote -v
    origin  https://github.com/yourname/crengine.git (fetch)
    origin  https://github.com/yourname/crengine.git (push)
    upstream  https://github.com/koreader/crengine.git (fetch)
    upstream  https://github.com/koreader/crengine.git (push)

NEVER ADD STUFF TO YOUR master BRANCH, it will just be kept up to date
with upstream master as a reference.

To sync your local master branch with upstream:
$ git checkout master
$ git fetch upstream
$ git merge upstream/master
To update your remote branch with your now-sync'ed local branch:
$ git push origin master

----
CREATE A NEW BRANCH to be pushed as a PR:
$ git checkout -b my-mod-branch
<< edit files >>
To see your changed stuff
$ git status
$ git diff

To make a commit with your changed stuff:
$ git add path/to/file
$ git add .    (if you want to add them all)
$ git status
$ git commit
<< lauches vi >>
    Your commit title

    Your commit details if needed.
Check all looks good:
$ git log
$ git log -p
Add more commits if needed.

When ready, push for a PR
$ git push -u origin my-mod-branch         (-u to create it)
And follow the url to github to open a PR.

Add more commits (as above) if needed to answer review comments, and push them:
$ git push origin my-mod-branch

Here, you would have a lot of other options, to rebase, squash, remove commits,
I could detail them when needed.

When PR is merged, cleanup and resync everything:
$ git checkout master
$ git branch -D my-mod-branch
$ git fetch --prune
$ git fetch upstream ; git merge upstream/master
$ git push origin master

Ready for next PR :)

Switch from outdated BGOffice pattern to hyphenation.org's newer version. Update quotation marks.

Update right_hyphen_min from 2 to 3. Add commented out exceptions to pattern file, to be investigated.

Update right_hyphen_min from 2 to 3. Add exceptions to pattern file.

Update right_hyphen_min from 2 to 3. Add exceptions to pattern file (some commented out, to be investigated).

Add exceptions to pattern file.

Add exceptions to pattern file (some commented out, to be investigated).

So frontend code can fetch per-language hyphen limits.

poire-z · 2020-09-17T11:50:55Z

Ok, done. Please have a quick look again as I did this in a hurry.

cramoisi

ok good for me :)

virxkane · 2020-09-20T20:46:20Z

This file is sufficient to describe the 'TeX Hyphenation patterns' format?
https://github.com/hyphenation/tex-hyphen/blob/master/docs/tb87nemeth.pdf

Ok, I can pick hyphenation patterns files from KOReader into CoolReader. Thanks to all contributors.
But I thought, what to do with the "*_hyphen_(Alan) .pdb" files, just delete it? But what if there is something missing in the "*.pattern" files?
So, I wrote a simple converter (dumper) for "*_hyphen_(Alan).pdb" or "*.pattern" files and saw that some of the patterns that are present in "*_hyphen_(Alan).pdb" are missing in "*.pattern".
I guess that "*_hyphen_(Alan).pdb" files transfered from AlReader program:

crengine/crengine/src/hyphman.cpp

Line 2 in a00d1b8

\brief AlReader hyphenation manager

And I don't know where Alan got these patterns and how correct they are.
But if I make dump from "English_US_hyphen_(Alan).pdb" I got 295799 bytes instead of 126141 from file English_US.pattern. And this fact makes you wonder which file is better and more correct.
I found that in KOReader "*_hyphen_(Alan).pdb" files dropped in #209 and #206 and I guess nobody checked contents of the "*_hyphen_(Alan).pdb" files.
Maybe we can combine them somehow? What is the best way to do this? Is it ok to just add omitted patterns in "*.pattern" from "*_hyphen_(Alan).pdb"?

P.S. This tool can be found in my not-yet-merged branch here: https://github.com/virxkane/coolreader/tree/updates-2020.09-1/crengine/Tools/HyphDumper
I would be grateful if someone points out the errors.

cramoisi · 2020-09-20T21:07:37Z

I could check the differences if you provide a human readable dump of English pdb file. Différences can be legit as the TeX files are only used as a base for crengine which can't read all TeX rules by design. So when I changed from pdb to pattern I also removed them as they were useless.

But as now we are following the hyphen patterns from recognised sources, so I don't think we need to worry about trying to make better than linguists already did for libreoffice, etc. I think feedback from native speakers should be enough as it's somehow tricky to edit these files without in dept knowledge of the languages...

virxkane · 2020-09-20T21:18:05Z

How can I make "human readable dump of English pdf file"? I can only convert *_hyphen_(Alan).pdb into *.pattern.

cramoisi · 2020-09-20T21:22:09Z

That will do as I'm human and I can read a pattern file with my eyes :)

Btw : I checked the content of pdb files back then but only for a few of them. As I said, some differences are legit

poire-z · 2020-09-20T21:29:52Z

Maybe we can combine them somehow? What is the best way to do this? Is it ok to just add omitted patterns in ".pattern" from "hyphen(Alan).pdb"?

I don't think combining them makes any sense: it's not really "the more items the better", but having a set that is consistent within itself (some patterns with numbers overriding some others with other numbers - mixing sources could possibly have these numbers relations incoherent).
Given that we don't really know who made the _Alan.pdb from what and when, and that the sources used to build the *.pattern are recent, referenced and updated, our *.pattern are a lot more trustable.

virxkane · 2020-09-21T05:12:47Z

I don't think combining them makes any sense: it's not really "the more items the better", but having a set that is consistent within itself (some patterns with numbers overriding some others with other numbers - mixing sources could possibly have these numbers relations incoherent).
Given that we don't really know who made the _Alan.pdb from what and when, and that the sources used to build the *.pattern are recent, referenced and updated, our *.pattern are a lot more trustable.

Yes, it looks logical.
And finally, for the connoisseurs of English, which I am not, dump 4 files:

The dump for "*.pattern" is the same as original, but all patterns are sorted for ease of comparison.

Difference between "English_GB_Alan_dump.pattern" and "English_US_dump.pattern" is less than between "English_US_Alan_dump.pattern" and "English_US_dump.pattern" :).
Patterns in "English_GB_hyphen_(Alan).pdb" like "a8s7s8o7c8i8a8t8e8s" look like a hack.

Is there something useful about these files "*_hyphen_(Alan).pdb" ?

cramoisi · 2020-09-21T06:36:57Z

Ok, I checked and actual patterns are more developed and consistent than the old one. A good example is that exceptions introduced by the hacks you mentioned in the pdb files are all processed correctly in current pattern file. So I guess you could get rid of these _(alan).pdb files once for all :-)

virxkane · 2020-09-21T07:13:03Z

@cramoisi Thank you.

roshavagarga added 2 commits September 17, 2020 10:36

TextLang, hyphenation: add Basque (eu)

d07c459

TextLang, hyphenation: add Croatian (hr)

d8b0f0c

poire-z mentioned this pull request Sep 17, 2020

Hyphenation patterns update #376

Closed

28 tasks

roshavagarga reviewed Sep 17, 2020

View reviewed changes

poire-z force-pushed the roshavagarga_hyph_works branch from c4c57cd to 3c82047 Compare September 17, 2020 08:51

poire-z force-pushed the roshavagarga_hyph_works branch 2 times, most recently from 61ebe5b to 86432e4 Compare September 17, 2020 10:41

cramoisi suggested changes Sep 17, 2020

View reviewed changes

roshavagarga added 11 commits September 17, 2020 13:48

TextLang, hyphenation: add Esperanto (eo)

0dc19ac

TextLang, hyphenation: add Estonian (et)

b6bb684

TextLang, hyphenation: add Georgian (ka)

905f243

TextLang, hyphenation: add Latvian (lv)

db4a8e2

TextLang, hyphenation: add Lithuanian (lt)

6c84c93

TextLang, hyphenation: add Macedonian (mk)

7ff6be4

TextLang, hyphenation: add Occitan (oc)

c0a73d7

TextLang, hyphenation: add Welsh (cy)

069ab18

TextLang, hyphenation: update Bulgarian (bg)

b0f4c1e

Switch from outdated BGOffice pattern to hyphenation.org's newer version. Update quotation marks.

TextLang, hyphenation: update Irish (ga)

f55e4f4

Update right_hyphen_min from 2 to 3. Add commented out exceptions to pattern file, to be investigated.

TextLang, hyphenation: update Portuguese (pt)

7da5309

Update right_hyphen_min from 2 to 3. Add exceptions to pattern file.

roshavagarga and others added 7 commits September 17, 2020 13:48

TextLang, hyphenation: update Slovak (sk)

308ef33

Update right_hyphen_min from 2 to 3. Add exceptions to pattern file (some commented out, to be investigated).

TextLang: update hyphen min for Czech, English, Greek

0b94e4c

Hyphenation: update Dutch (nl)

ae648d6

Add exceptions to pattern file.

Hyphenation: update Norwegian (no)

a052d08

Add exceptions to pattern file (some commented out, to be investigated).

Hyphenation: update Spanish (es)

c4e88d9

Hyphenation: fix Romanian and Ukrainian pattern file names

98021d5

HyphMan: adds HyphMethod::getLeft/RightHyphenMin()

1ebcd14

So frontend code can fetch per-language hyphen limits.

poire-z force-pushed the roshavagarga_hyph_works branch from 86432e4 to 1ebcd14 Compare September 17, 2020 11:49

cramoisi approved these changes Sep 17, 2020

View reviewed changes

roshavagarga approved these changes Sep 17, 2020

View reviewed changes

poire-z merged commit da9cb81 into koreader:master Sep 17, 2020

poire-z deleted the roshavagarga_hyph_works branch September 17, 2020 13:13

This was referenced Sep 18, 2020

bump crengine: ODT support, added/updated hyphenation koreader/koreader-base#1190

Merged

bump crengine: ODT support, more typography languages koreader/koreader#6675

Merged

poire-z mentioned this pull request Sep 22, 2020

Unable to build macos desktop: error during CC dnabasic.lo koreader/koreader#6687

Closed

virxkane mentioned this pull request Sep 30, 2020

Update hyphenations. buggins/coolreader#175

Merged

NiLuJe mentioned this pull request Nov 24, 2020

Add Latin hyphenation koreader/koreader#6910

Merged

		<pattern>dtiom5áintí</pattern>
		<pattern>thiom5áintí</pattern>

Hyphenation patterns update (cleaned up #376) #378

Hyphenation patterns update (cleaned up #376) #378

Conversation

poire-z commented Sep 17, 2020 • edited by Frenzie

poire-z commented Sep 17, 2020

cramoisi commented Sep 17, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roshavagarga commented Sep 17, 2020

poire-z commented Sep 17, 2020

roshavagarga commented Sep 17, 2020

poire-z commented Sep 17, 2020

roshavagarga commented Sep 17, 2020

poire-z commented Sep 17, 2020

roshavagarga commented Sep 17, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

poire-z commented Sep 17, 2020

poire-z commented Sep 17, 2020

poire-z commented Sep 17, 2020

cramoisi left a comment

Choose a reason for hiding this comment

virxkane commented Sep 20, 2020

cramoisi commented Sep 20, 2020 • edited

virxkane commented Sep 20, 2020

cramoisi commented Sep 20, 2020

poire-z commented Sep 20, 2020

virxkane commented Sep 21, 2020

cramoisi commented Sep 21, 2020

virxkane commented Sep 21, 2020

poire-z commented Sep 17, 2020 •

edited by Frenzie

cramoisi commented Sep 20, 2020 •

edited