Skip to content

Commit

Permalink
Avoid case changing high chars in (u)pTeX (issue #939)
Browse files Browse the repository at this point in the history
  • Loading branch information
@josephwright
josephwright committed Jun 14, 2021
1 parent 01c796e commit 0e42f6c
Show file tree
Hide file tree
Showing 8 changed files with 157 additions and 90 deletions.
1 change: 1 addition & 0 deletions l3kernel/CHANGELOG.md
View file
Expand Up @@ -9,6 +9,7 @@ this project uses date-based 'snapshot' version identifiers.

### Fixed
- Incorrect internal function name (issue #939)
- Case-changing exceptions for (u)pTeX (issue #939)

## [2021-06-01]

Expand Down
37 changes: 28 additions & 9 deletions l3kernel/l3text-case.dtx
View file
Expand Up @@ -163,7 +163,12 @@
% \begin{macro}[EXP]
% {\@@_change_case_char_title:nN, \@@_change_case_char_titleonly:nN}
% \begin{macro}[EXP]{\@@_change_case_char_title:nnnN}
% \begin{macro}[EXP]{\@@_change_case_char:nnnN, \@@_change_case_char_aux:nnnN}
% \begin{macro}[EXP]
% {
% \@@_change_case_char:nnnN ,
% \@@_change_case_char_auxi:nnnN ,
% \@@_change_case_char_auxii:nnnN
% }
% \begin{macro}[EXP]{\@@_change_case_char_UTFviii:nnnNN}
% \begin{macro}[EXP]{\@@_change_case_char_UTFviii:nnnNNN}
% \begin{macro}[EXP]{\@@_change_case_char_UTFviii:nnnNNNN}
Expand Down Expand Up @@ -608,6 +613,7 @@
% are to deal with. Those can then be grabbed to reconstruct the full
% Unicode character, which is then used in a lookup. (As will become
% obvious below, there is no intention here of covering all of Unicode.)
% For (u)p-\TeX{} there are a limited number of tokens we can touch.
% \begin{macrocode}
\cs_new:Npn \@@_change_case_char:nnnN #1#2#3#4
{
Expand All @@ -623,23 +629,36 @@
{ \sys_if_engine_luatex_p: }
{ \sys_if_engine_xetex_p: }
{
\cs_new_eq:NN \@@_change_case_char_aux:nnnN
\cs_new_eq:NN \@@_change_case_char_auxi:nnnN
\@@_change_case_char:nnnN
\cs_gset:Npn \@@_change_case_char:nnnN #1#2#3#4
{
\int_compare:nNnTF { `#4 } > { "80 }
{
\int_compare:nNnTF { `#4 } < { "E0 }
{ \@@_change_case_char_UTFviii:nnnNN }
{
\int_compare:nNnTF { `#4 } < { "F0 }
{ \@@_change_case_char_UTFviii:nnnNNN }
{ \@@_change_case_char_UTFviii:nnnNNNN }
}
{#1} {#2} {#3} #4
{ \@@_change_case_char_auxii:nnnN }
}
{ \@@_change_case_char_aux:nnnN {#1} {#2} {#3} #4 }
{ \@@_change_case_char_auxi:nnnN }
{#1} {#2} {#3} #4
}
\sys_if_engine_pdftex:TF
{
\cs_new:Npn \@@_change_case_char_auxii:nnnN #1#2#3#4
{
\int_compare:nNnTF { `#4 } < { "F0 }
{ \@@_change_case_char_UTFviii:nnnNNN }
{ \@@_change_case_char_UTFviii:nnnNNNN }
{#1} {#2} {#3} #4
}
}
{
\cs_new:Npn \@@_change_case_char_auxii:nnnN #1#2#3#4
{
\@@_change_case_store:n {#4}
\use:c { @@_change_case_char_next_ #2 :nn } {#2} {#3}
}
}
\cs_new:Npn \@@_change_case_char_UTFviii:nnnNN #1#2#3#4#5
{ \@@_change_case_char_UTFviii:nnnn {#1} {#2} {#3} {#4#5} }
\cs_new:Npn \@@_change_case_char_UTFviii:nnnNNN #1#2#3#4#5#6
Expand Down
44 changes: 26 additions & 18 deletions l3kernel/testfiles/m3text002.luatex.tlg
View file
Expand Up @@ -88,23 +88,31 @@ No problems
No problems
============================================================
============================================================
TEST 9: Unicode case changing
TEST 9: (u)pTeX-based tests
============================================================
abc^^e8日本語
ABC^^c8日本語
Abc^^e8日本語
ABC^^c8日本語
============================================================
============================================================
TEST 10: Unicode case changing
============================================================
^^e5^^e9^^ee^^f8ὥдαɛ
^^c5^^c9^^ce^^d8ὭДΑƐ
^^c5^^e9^^ee^^f8ὥдαɛ
^^c5^^e9^^ee^^f8ὭдαƐ
============================================================
============================================================
TEST 10: Unicode case changing exceptions
TEST 11: Unicode case changing exceptions
============================================================
fu^^dfball
FUSSBALL
Fu^^dfball
Fu^^dfball
============================================================
============================================================
TEST 11: The final sigma rule
TEST 12: The final sigma rule
============================================================
ὀδυσσεύς (ὀδυσσεύς) ὀδυσσεύς, ὀδυσσεύς{} ὀδυσσεύς\noop
ὈΔΥΣΣΕΎΣ (ὈΔΥΣΣΕΎΣ) ὈΔΥΣΣΕΎΣ, ὈΔΥΣΣΕΎΣ{} ὈΔΥΣΣΕΎΣ\noop
Expand All @@ -116,21 +124,21 @@ TEST 11: The final sigma rule
ὈΔΥΣΣΕΎΣ
============================================================
============================================================
TEST 12: Cyrillic
TEST 13: Cyrillic
============================================================
доклады академии наук
ДОКЛАДЫ АКАДЕМИИ НАУК
Доклады академии наук
Доклады Академии наук
============================================================
============================================================
TEST 13: German-alternative
TEST 14: German-alternative
============================================================
FUSSBALL
FUẞBALL
============================================================
============================================================
TEST 14: Greek
TEST 15: Greek
============================================================
ΆΔΙΚΟΣ, ΚΕΊΜΕΝΟ, ΊΡΙΔΑ
ΑΔΙΚΟΣ, ΚΕΙΜΕΝΟ, ΙΡΙΔΑ
Expand Down Expand Up @@ -170,7 +178,7 @@ TEST 14: Greek
Ὀδυσσεύς
============================================================
============================================================
TEST 15: Turkish/Azeri
TEST 16: Turkish/Azeri
============================================================
ragıp hul^^fbsi ^^f6zdem
ragip hul^^fbsi̇ ^^f6zdem
Expand All @@ -182,7 +190,7 @@ Ip hul^^fbsi ^^f6zdem
Ip hul^^fbsi ^^f6zdem
============================================================
============================================================
TEST 16: Lithuanian
TEST 17: Lithuanian
============================================================
i̇̀i̇́i̇̃i̇̀i̇́i̇̃j̇̀j̇́j̇̃į̇̀į̇́į̇̃
^^ec^^edĩìíĩj̀j́j̃į̀į́į̃
Expand All @@ -194,7 +202,7 @@ Ìi̇̀i̇̃i̇̀i̇́i̇̃j̇̀j̇́j̇̃į̇̀į̇́į̇̃
İ̀i̇̀i̇̃i̇̀i̇́i̇̃j̇̀j̇́j̇̃į̇̀į̇́į̇̃
============================================================
============================================================
TEST 17: Dutch
TEST 18: Dutch
============================================================
ijsselmeer
ijsselmeer
Expand All @@ -210,13 +218,13 @@ Im
Im
============================================================
============================================================
TEST 18: Titlecase exceptions
TEST 19: Titlecase exceptions
============================================================
Ssoo
Dž!
============================================================
============================================================
TEST 19: Case changing braced arguments
TEST 20: Case changing braced arguments
============================================================
foo \emph {BAR} {baz}
FOO \emph {BAR} {BAZ}
Expand All @@ -228,7 +236,7 @@ FOO \emph {BAR} {BAZ}
\emph {BAR} {BAZ}
============================================================
============================================================
TEST 20: Expanding content
TEST 21: Expanding content
============================================================
some text hello
SOME TEXT HELLO
Expand Down Expand Up @@ -256,7 +264,7 @@ Some text \cs_tmp:w
\cs_tmp:w Sometext
============================================================
============================================================
TEST 21: Math-mode escape
TEST 22: Math-mode escape
============================================================
some text $y = mx + c$
SOME TEXT $y = mx + c$
Expand All @@ -272,39 +280,39 @@ Opps not close token in $y = mx + c
Opps not close token in $y = mx + c
============================================================
============================================================
TEST 22: Nesting
TEST 23: Nesting
============================================================
HELLO
hello
FUSSBALL
ὈΔΥΣΣΕΎΣ
============================================================
============================================================
TEST 23: Letter-like commands
TEST 24: Letter-like commands
============================================================
\aa \aa \J \ae \dh \ss \l \o
\AA \AA \J \AE \DH \SS \L \O
\AA \aa \J \ae \dh \ss \l \o
\AA \aa \J \ae \dh \ss \l \O
============================================================
============================================================
TEST 24: Accents
TEST 25: Accents
============================================================
\"{a}\u {e}\H {i}\v {o}\.{u}
\"{A}\u {E}\H {I}\v {O}\.{U}
\"{A}\u {e}\H {i}\v {o}\.{u}
\"{A}\u {e}\H {i}\v {o}\.{u}
============================================================
============================================================
TEST 25: Active chars
TEST 26: Active chars
============================================================
"`lo"=rem"'
"`LO"=REM"'
"`Lo"=rem"'
"`Lo"=rem"'
============================================================
============================================================
TEST 26: Case changing exclusions
TEST 27: Case changing exclusions
============================================================
Variant \tl_show:x already defined; not changing it on line ...
> foo\NoCaseChange {BAR}.
Expand Down
7 changes: 7 additions & 0 deletions l3kernel/testfiles/m3text002.lvt
View file
Expand Up @@ -119,7 +119,14 @@
\NEWLINE
\test:nn { tr } { No~problems }
}

\TESTEXP { (u)pTeX-based~tests }
{
\test:n { ABCÈ日本語}
}

\sys_if_engine_ptex:T { \END }

\TESTEXP { Unicode~case~changing }
{
\test:n { åéîøὭдαƐ }
Expand Down
8 changes: 8 additions & 0 deletions l3kernel/testfiles/m3text002.ptex.tlg
View file
Expand Up @@ -87,3 +87,11 @@ NO PROBLEMS
No problems
No problems
============================================================
============================================================
TEST 9: (u)pTeX-based tests
============================================================
abc竪日本語
ABC�^^88日本語
Abc竪日本語
ABC�^^88日本語
============================================================

0 comments on commit 0e42f6c

Please sign in to comment.