IDLE font settings: use multiple character sets in examples

[terryjreedy]

BPO	13802
Nosy	@terryjreedy, @serwy, @rovitotv, @serhiy-storchaka, @mlouielu, @corona10
PRs	bpo-13802: IDLE: Add pangram and chinese sentence to font sample box #2616 bpo-13802: Use non-Latin characters in IDLE's Font settings sample. #3960 [3.6] bpo-13802: Use non-Latin characters in IDLE's Font settings sample. (GH-3960) #4027
Files	issue13802.patch: Some more example chars

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2017-10-17.23:52:32.369>
created_at = <Date 2012-01-16.23:22:19.711>
labels = ['expert-IDLE', 'type-feature', '3.7']
title = 'IDLE font settings: use multiple character sets in examples'
updated_at = <Date 2017-10-17.23:52:32.368>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2017-10-17.23:52:32.368>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2017-10-17.23:52:32.369>
closer = 'terry.reedy'
components = ['IDLE']
creation = <Date 2012-01-16.23:22:19.711>
creator = 'terry.reedy'
dependencies = []
files = ['29553']
hgrepos = []
issue_num = 13802
keywords = ['patch']
message_count = 19.0
messages = ['151415', '185080', '297868', '297869', '297870', '297878', '297883', '298408', '298413', '298762', '304205', '304210', '304212', '304219', '304222', '304223', '304293', '304543', '304545']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'roger.serwy', 'Todd.Rovito', 'francismb', 'serhiy.storchaka', 'louielu', 'corona10']
pr_nums = ['2616', '3960', '4027']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13802'
versions = ['Python 3.6', 'Python 3.7']

[terryjreedy]

In the Fonts/Tabs tab of the IDLE Preference dialog, the large box for examples of the font selected shows a small square of ascii chars. I think the box should also show 1 char for each of several alphabets so the consequence of choosing various fonts will be more evident. I am thinking of adding several line with the format
Alphabet \uXXXX <char>

[francismb]

Hi Terry,
just take/put away some ... (they're not in a special order nor preference, just some that could 'see' in the browser).

[mlouielu]

Add the string with pangram and chinese, now sample text shows:

AaBbCcDdEe
FfGgHhIiJjK
1234567890
#:+=(){}[]

The quick brown fox jumps over the lazy dog. 1

南去經三國，東來過五湖 2

----------------------

[serhiy-storchaka]

There are many different alphabets in the world. Why Chines but not Cyrillic or Devanagari? An example that includes all scripts would be too large (and likely most characters would not be rendered correctly with an arbitrary font).

I suggest just make the box for examples editable and save entered examples in the configuration files.

[mlouielu]

Serhiy: Or would it be to detect the user language environment, and come up with user language example sentence? What I remember that Windows font preview will change the sentence in the different language.

[serhiy-storchaka]

I don't think that it is worth to include in Python distribution examples for a hundred of languages. If there are some system-wide collections of examples we can use them, but this should be platform specific and not always available.

[serhiy-storchaka]

On other hand, we can use the standard font chooser dialog. But it doesn't allow to configure the sample text at all (at least on X Window).

Terry, try please the following commands on Windows:

import tkinter
root = tkinter.Tk()
root.tk.call('tk', 'fontchooser', 'show')

How it looks?

[terryjreedy]

I made this issue a dependency of bpo-24776, which is about redoing the whole font page.

The fontchooser has good, inappropriate for IDLE, and bad points. I consider it an alternate mockup proposal for the font page. For this issue: displaying each font name in the font is cute, but I am not sure I want to imitate. I address the sample selection below.

This issue is about making it evident that IDLE is a BMP unicode editor, not just an ASCII editor; and about showing the consequence of font choices on a particular OS and machine. Expanding the static sample with example of the top N scripts, with N about 10, will do this.

When I looked at Francis's patch, I thought it deficient in that people would not know what chars were being replaced by boxes. Then I tried the patch and none of them were. At least not on Windows. Since I opened this, we added back the Help button. Added help text for this tab can list the scripts represented in the sample. If nothing else, I will use this patch, as it improves of the status quo. I should have done this years ago. An immediate improvement would be all chars from a script on one line and some (more?) hanji/kanji CJK chars.

bpo-24776 suggests putting the sample beside the font selection box. The font box only needs 75% of the width it has. With the frame around the sample label removed, there may be more width available. There will certainly be more lines.

Louie, I understand your PR to be a suggestion about pangrams, expressed in code, to be 'pulled' into my mind, and possibly into my clone, and not a request to merge as is with only Chinese. Others have noted that submitting work-in-progess patches as PRs, rather than as diffs to the tracker, can be confusing. Yet it makes review easy.

In considering the idea, I looked at https://en.wikipedia.org/wiki/Pangram and also found http://clagnut.com/blog/2380/, which lists pangrams in multiple languages that were once on wiki/List_of_Pangrams.

I am rejecting the idea as is for multiple related reasons. The sample is about scripts, not languages. Long phrases mean fewer scripts. Many scripts are used for multiple dialects or even languages. Which one to choose? An innocent or poetic phrase in one language may be less innocent in another dialect/language, or if interpreted metaphorically, or one consider possible alternate meanings of words.

One thing I would consider is script names written in the script. This would replace an arbitrary sample from the same script and could be done on a script-by-script basis. I believe 'devanagari' in Devanagari should be intelligible in all or most north Indian languages that use Devanagari or derivatives thereof, and offensive in none. But I would enquire.

What do others think of this?

Louie, I presume 'hanji' is a single char. How about using the stroke pangram 永 (all basic strokes) in a phrase such as 'hanji character 永'? (I like the look of the char as well).

Serhiy, is there one non-controversial pan-slavic way to write 'Cyrillic' in Cyrillic? Or controversy-engendering national variations?

A separate issue could add a button and dropdown multiple selection list to print to an output window all or a selection of the 256 blocks of 256 codepoints in the BMP. This should eliminate any need for a configuration option.

likely most characters would not be rendered correctly with an arbitrary font

A goal of this issue is to let people see such problems where they exist.

The fontchooser sample awful. Only couple of ascii and script chars and a small sample of script that changes with each script. It has no knowledge of default characters.

[serhiy-storchaka]

Serhiy, is there one non-controversial pan-slavic way to write 'Cyrillic' in Cyrillic?

No. Even in very close east-slavic languages it is written differently: Ukrainian "Кирилиця", Russian "Кириллица", Belarusian "Кірыліца".

[terryjreedy]

I decided to rearrange page bpo-24776 before changing sample bpo-13802.

[terryjreedy]

I decided to do the needed rearrangement of frames in this issue, leaving revision of widgets other than the sample to bpo-24776.

PR3960 initially puts the following in the sample box. I believe it should cover most Python users.

<ASCII/Latin1>
AaBbCcDdEeFfGgHhIiJj
1234567890#:+=(){}[]
¡¢£¥§©«®¶½ÀÁÂÃÄÅÇÐØß

<IPA,Greek,Cyrillic>
ɐɕɘɞɟɤɫɮɰɷɻʁʃʆʎʞʢʫʭʯ
ΑαΒβΓγΔδΕεΖζΗηΘθΙιΚκ
БбДдЖжПпФфЧчЪъЭэѠѤѬӜ

<Hebrew, Arabic>
אבגדהוזחטיךכלםמןנסעף
ابجدهوزحطي٠١٢٣٤٥٦٧٨٩

<Devanagari>
०१२३४५६७८९अआइईउऊएऐओऔ
कगङचजञतदनपबमयरलवशसह॥

<East Asian>
〇一二三四五六七八九
汉字漢字人木火土金水
ᅡᅦᅩᆨᆫ가결걵곴극
あいうえおかさたなま

The new help message explains

Font sample: This shows what a selection of BMP unicode characters look
like for the current font selection. If a font face does not define a
character, Tk attempts to find another font that does. Substitute
glyphs will not necessarily have the same size as the font selected.
Hebrew and Arabic letters should display right to left, starting with
alef, \u05d0 and \u0627. Arabic numerals display left to right. The
East Asian samples are Chinese digits followed by Chinese Hanzi, Korean
Hangul, and Japanese Hiragana.

Except for Chinese, I intend for the samples to be meaningless subsequences of the respective character set. I will try to get comments from others for the Indian, Korean, and Japanese samples.

The Chinese reads 'hanzi hanzi person wood fire earth metal water' with 'hanzi' repeated in simplified and traditional character variations. Louie, could this be controversial in any way?

[serhiy-storchaka]

I think empty lines and headers are not needed. They just increase the size of the dialog window, it may be too large for low resolution screens.

The standard font chooser on Windows uses the following sets for Unicode-aware fonts: Western, Cyrillic, Hebrew, Arabic, Greek, Turkish, Baltic (doesn't differ from Western), Central-European, Vietnamese. I think it is worth to decrease the Devanagari sample and add few Central-European (ÁáÔô), Turkish (ĞğŞş) and Vietnamese (ƠơƯư) characters. Maybe decrease the size of all non-ASCII samples? 2-3 letters (in upper and lower cases) should be enough.

[mlouielu]

PR 3960 display on my Linux works well, and the combination of the Chinese characters has no political controversial.

But it is lack of simplify characters, the only one is '汉', others are all traditional characters (or they are same in simp. and trad.).

For Japanese, there are katakana (カタカナ), hiragana(ひらがな) and kanji (漢字), the characters you provide only contain hiragana. Maybe a sentence that contains all kinds of characters will be better.

For Korean, the characters are compound to one, but I'm not sure will they use the first four characters to test the font. CC for corona10.

For Chinese, Japanese, Korean, I'll still prefer and recommend to use a sentence to give user the perspective of the font.

For other character set, maybe we can reference from Google fonts:
https://fonts.google.com/
they got a list of Unicode-aware language:

Arbic, Bengali, Cyrillic, Cyrillic Extended, Devanagari, Greek,
Greek Extended, Gujarati, Gurmukhi, Hebrew, Kannada, Khmer,
Latin, Latin Extended, Malayalam, Myanmar, Oriya, Tamil, Telugu,
Thai, Vietnamese.

[serhiy-storchaka]

I suggest to make the sample text editable:

    self.font_sample = Text(frame_sample, font=temp_font, width=20)
    self.font_sample.insert(END, sample)

This will allow a user to test fonts in any perspective.

[corona10]

Louie Lu

Thanks for the cc.
In Korea, 'ᅡᅦᅩᆨᆫ가결걵곴극' are not recommended characters sequence.
This sequence is not fully cover Korean characters.

FYI,
MS Windows default font preview sequence is
'다람쥐 헌 쳇바퀴에 타고파'

Google chrome preview sequence is
'정 참판 양반댁 규수 큰 교자 타고 혼례 치른 날'

Those two sequences are the Korean version of 'The quick brown fox jumps over the lazy dog'.

If you want to use Korean version of 'AaBbCcDdEeFfGgHhIiJj'
I recommend '가나다라마바사아자차카파타하'

[corona10]

'가냐더려모뵤수유즈치캐턔페혜' is better than '가나다라마바사아차카파타하'

[terryjreedy]

I pushed new commits to the PR that changed the Korean, Japanese, and Indian samples and the help message.

Korea: The line is now the first 10 chars of Dong-hee Na's suggestion. Thank you for helping (and Louie for the cc).

Japan: The line is now the hiragana and katakana versions of the 5 vowels. As far as I know, 'kanji' are a subset of CJK 'hanzi', with different pronunciations and possibly different meanings, so I see no need for any here.

India: The second line is now Tamil digits and vowels.

China: I found the chars, with translation, including the one simplified/traditional pair, on Wikipedia. I did not know if more pairs would be a good idea or not, and I do not know of others. Louie, if you have a better idea, please post it (with translation ;-).

Dialog size: The height is about 710 pixels, 10% larger than before the additions to the General page a month ago. So it was previously too tall for 800 x 600 and still fits 1024 x 768. Further expansion should mostly be in width, but there is some vertical room.

Spaces and labels: I initially had neither. I could hardly stand to look. The label are needed for people who do not recognize the character sets, and I use them to make three general points in the help entry: tk uses whatever font it can to cover most of the BMP; fixed font sizes for Latin font only apply to Latin chars; and right-to-left is handled correctly.

Latin1: I consider the Windows fontchooser to be an anti-model for this issue in that it is limited to alphabetic characters and confuses 'language' with 'character set'. Western, Central European, Baltic, Turkish, and Vietnamese languages all use latin characters. The non-ascii, decorated versions are 'covered' by the non-ascii Latin1 line. Á is already present. Â is intended to represent all circumflexed characters, including Ô. I intended the repeated use of A as base character to imply that. I not sure if using a different base for each decoration would be better. Similarly, Ç covers Ş. I am thinking about using fewer symbols and more alphabetic characters. It might be a good idea to add something, such as Ğ, from the Extended block, beyond \u00ff.

User additions: This would be a separate issue, see bpo-31777.

[terryjreedy]

New changeset e2e4227 by Terry Jan Reedy in branch 'master':
bpo-13802: Use non-Latin characters in IDLE's Font settings sample. (bpo-3960)
e2e4227

[terryjreedy]

New changeset ecacbb4 by Terry Jan Reedy (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-13802: Use non-Latin characters in IDLE's Font settings sample. (GH-3960) (bpo-4027)
ecacbb4