/
nonascii.texi
1925 lines (1643 loc) · 82.7 KB
/
nonascii.texi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
@c -*-texinfo-*-
@c This is part of the GNU Emacs Lisp Reference Manual.
@c Copyright (C) 1998-1999, 2001-2011 Free Software Foundation, Inc.
@c See the file elisp.texi for copying conditions.
@setfilename ../../info/characters
@node Non-ASCII Characters, Searching and Matching, Text, Top
@chapter Non-@acronym{ASCII} Characters
@cindex multibyte characters
@cindex characters, multi-byte
@cindex non-@acronym{ASCII} characters
This chapter covers the special issues relating to characters and
how they are stored in strings and buffers.
@menu
* Text Representations:: How Emacs represents text.
* Converting Representations:: Converting unibyte to multibyte and vice versa.
* Selecting a Representation:: Treating a byte sequence as unibyte or multi.
* Character Codes:: How unibyte and multibyte relate to
codes of individual characters.
* Character Properties:: Character attributes that define their
behavior and handling.
* Character Sets:: The space of possible character codes
is divided into various character sets.
* Scanning Charsets:: Which character sets are used in a buffer?
* Translation of Characters:: Translation tables are used for conversion.
* Coding Systems:: Coding systems are conversions for saving files.
* Input Methods:: Input methods allow users to enter various
non-ASCII characters without special keyboards.
* Locales:: Interacting with the POSIX locale.
@end menu
@node Text Representations
@section Text Representations
@cindex text representation
Emacs buffers and strings support a large repertoire of characters
from many different scripts, allowing users to type and display text
in almost any known written language.
@cindex character codepoint
@cindex codespace
@cindex Unicode
To support this multitude of characters and scripts, Emacs closely
follows the @dfn{Unicode Standard}. The Unicode Standard assigns a
unique number, called a @dfn{codepoint}, to each and every character.
The range of codepoints defined by Unicode, or the Unicode
@dfn{codespace}, is @code{0..#x10FFFF} (in hexadecimal notation),
inclusive. Emacs extends this range with codepoints in the range
@code{#x110000..#x3FFFFF}, which it uses for representing characters
that are not unified with Unicode and @dfn{raw 8-bit bytes} that
cannot be interpreted as characters. Thus, a character codepoint in
Emacs is a 22-bit integer number.
@cindex internal representation of characters
@cindex characters, representation in buffers and strings
@cindex multibyte text
To conserve memory, Emacs does not hold fixed-length 22-bit numbers
that are codepoints of text characters within buffers and strings.
Rather, Emacs uses a variable-length internal representation of
characters, that stores each character as a sequence of 1 to 5 8-bit
bytes, depending on the magnitude of its codepoint@footnote{
This internal representation is based on one of the encodings defined
by the Unicode Standard, called @dfn{UTF-8}, for representing any
Unicode codepoint, but Emacs extends UTF-8 to represent the additional
codepoints it uses for raw 8-bit bytes and characters not unified with
Unicode.}. For example, any @acronym{ASCII} character takes up only 1
byte, a Latin-1 character takes up 2 bytes, etc. We call this
representation of text @dfn{multibyte}.
Outside Emacs, characters can be represented in many different
encodings, such as ISO-8859-1, GB-2312, Big-5, etc. Emacs converts
between these external encodings and its internal representation, as
appropriate, when it reads text into a buffer or a string, or when it
writes text to a disk file or passes it to some other process.
Occasionally, Emacs needs to hold and manipulate encoded text or
binary non-text data in its buffers or strings. For example, when
Emacs visits a file, it first reads the file's text verbatim into a
buffer, and only then converts it to the internal representation.
Before the conversion, the buffer holds encoded text.
@cindex unibyte text
Encoded text is not really text, as far as Emacs is concerned, but
rather a sequence of raw 8-bit bytes. We call buffers and strings
that hold encoded text @dfn{unibyte} buffers and strings, because
Emacs treats them as a sequence of individual bytes. Usually, Emacs
displays unibyte buffers and strings as octal codes such as
@code{\237}. We recommend that you never use unibyte buffers and
strings except for manipulating encoded text or binary non-text data.
In a buffer, the buffer-local value of the variable
@code{enable-multibyte-characters} specifies the representation used.
The representation for a string is determined and recorded in the string
when the string is constructed.
@defvar enable-multibyte-characters
This variable specifies the current buffer's text representation.
If it is non-@code{nil}, the buffer contains multibyte text; otherwise,
it contains unibyte encoded text or binary non-text data.
You cannot set this variable directly; instead, use the function
@code{set-buffer-multibyte} to change a buffer's representation.
@end defvar
@defun position-bytes position
Buffer positions are measured in character units. This function
returns the byte-position corresponding to buffer position
@var{position} in the current buffer. This is 1 at the start of the
buffer, and counts upward in bytes. If @var{position} is out of
range, the value is @code{nil}.
@end defun
@defun byte-to-position byte-position
Return the buffer position, in character units, corresponding to given
@var{byte-position} in the current buffer. If @var{byte-position} is
out of range, the value is @code{nil}. In a multibyte buffer, an
arbitrary value of @var{byte-position} can be not at character
boundary, but inside a multibyte sequence representing a single
character; in this case, this function returns the buffer position of
the character whose multibyte sequence includes @var{byte-position}.
In other words, the value does not change for all byte positions that
belong to the same character.
@end defun
@defun multibyte-string-p string
Return @code{t} if @var{string} is a multibyte string, @code{nil}
otherwise.
@end defun
@defun string-bytes string
@cindex string, number of bytes
This function returns the number of bytes in @var{string}.
If @var{string} is a multibyte string, this can be greater than
@code{(length @var{string})}.
@end defun
@defun unibyte-string &rest bytes
This function concatenates all its argument @var{bytes} and makes the
result a unibyte string.
@end defun
@node Converting Representations
@section Converting Text Representations
Emacs can convert unibyte text to multibyte; it can also convert
multibyte text to unibyte, provided that the multibyte text contains
only @acronym{ASCII} and 8-bit raw bytes. In general, these
conversions happen when inserting text into a buffer, or when putting
text from several strings together in one string. You can also
explicitly convert a string's contents to either representation.
Emacs chooses the representation for a string based on the text from
which it is constructed. The general rule is to convert unibyte text
to multibyte text when combining it with other multibyte text, because
the multibyte representation is more general and can hold whatever
characters the unibyte text has.
When inserting text into a buffer, Emacs converts the text to the
buffer's representation, as specified by
@code{enable-multibyte-characters} in that buffer. In particular, when
you insert multibyte text into a unibyte buffer, Emacs converts the text
to unibyte, even though this conversion cannot in general preserve all
the characters that might be in the multibyte text. The other natural
alternative, to convert the buffer contents to multibyte, is not
acceptable because the buffer's representation is a choice made by the
user that cannot be overridden automatically.
Converting unibyte text to multibyte text leaves @acronym{ASCII}
characters unchanged, and converts bytes with codes 128 through 255 to
the multibyte representation of raw eight-bit bytes.
Converting multibyte text to unibyte converts all @acronym{ASCII}
and eight-bit characters to their single-byte form, but loses
information for non-@acronym{ASCII} characters by discarding all but
the low 8 bits of each character's codepoint. Converting unibyte text
to multibyte and back to unibyte reproduces the original unibyte text.
The next two functions either return the argument @var{string}, or a
newly created string with no text properties.
@defun string-to-multibyte string
This function returns a multibyte string containing the same sequence
of characters as @var{string}. If @var{string} is a multibyte string,
it is returned unchanged. The function assumes that @var{string}
includes only @acronym{ASCII} characters and raw 8-bit bytes; the
latter are converted to their multibyte representation corresponding
to the codepoints @code{#x3FFF80} through @code{#x3FFFFF}, inclusive
(@pxref{Text Representations, codepoints}).
@end defun
@defun string-to-unibyte string
This function returns a unibyte string containing the same sequence of
characters as @var{string}. It signals an error if @var{string}
contains a non-@acronym{ASCII} character. If @var{string} is a
unibyte string, it is returned unchanged. Use this function for
@var{string} arguments that contain only @acronym{ASCII} and eight-bit
characters.
@end defun
@defun byte-to-string byte
@cindex byte to string
This function returns a unibyte string containing a single byte of
character data, @var{character}. It signals an error if
@var{character} is not an integer between 0 and 255.
@end defun
@defun multibyte-char-to-unibyte char
This converts the multibyte character @var{char} to a unibyte
character, and returns that character. If @var{char} is neither
@acronym{ASCII} nor eight-bit, the function returns -1.
@end defun
@defun unibyte-char-to-multibyte char
This convert the unibyte character @var{char} to a multibyte
character, assuming @var{char} is either @acronym{ASCII} or raw 8-bit
byte.
@end defun
@node Selecting a Representation
@section Selecting a Representation
Sometimes it is useful to examine an existing buffer or string as
multibyte when it was unibyte, or vice versa.
@defun set-buffer-multibyte multibyte
Set the representation type of the current buffer. If @var{multibyte}
is non-@code{nil}, the buffer becomes multibyte. If @var{multibyte}
is @code{nil}, the buffer becomes unibyte.
This function leaves the buffer contents unchanged when viewed as a
sequence of bytes. As a consequence, it can change the contents
viewed as characters; for instance, a sequence of three bytes which is
treated as one character in multibyte representation will count as
three characters in unibyte representation. Eight-bit characters
representing raw bytes are an exception. They are represented by one
byte in a unibyte buffer, but when the buffer is set to multibyte,
they are converted to two-byte sequences, and vice versa.
This function sets @code{enable-multibyte-characters} to record which
representation is in use. It also adjusts various data in the buffer
(including overlays, text properties and markers) so that they cover the
same text as they did before.
You cannot use @code{set-buffer-multibyte} on an indirect buffer,
because indirect buffers always inherit the representation of the
base buffer.
@end defun
@defun string-as-unibyte string
If @var{string} is already a unibyte string, this function returns
@var{string} itself. Otherwise, it returns a new string with the same
bytes as @var{string}, but treating each byte as a separate character
(so that the value may have more characters than @var{string}); as an
exception, each eight-bit character representing a raw byte is
converted into a single byte. The newly-created string contains no
text properties.
@end defun
@defun string-as-multibyte string
If @var{string} is a multibyte string, this function returns
@var{string} itself. Otherwise, it returns a new string with the same
bytes as @var{string}, but treating each multibyte sequence as one
character. This means that the value may have fewer characters than
@var{string} has. If a byte sequence in @var{string} is invalid as a
multibyte representation of a single character, each byte in the
sequence is treated as a raw 8-bit byte. The newly-created string
contains no text properties.
@end defun
@node Character Codes
@section Character Codes
@cindex character codes
The unibyte and multibyte text representations use different
character codes. The valid character codes for unibyte representation
range from 0 to @code{#xFF} (255)---the values that can fit in one
byte. The valid character codes for multibyte representation range
from 0 to @code{#x3FFFFF}. In this code space, values 0 through
@code{#x7F} (127) are for @acronym{ASCII} characters, and values
@code{#x80} (128) through @code{#x3FFF7F} (4194175) are for
non-@acronym{ASCII} characters.
Emacs character codes are a superset of the Unicode standard.
Values 0 through @code{#x10FFFF} (1114111) correspond to Unicode
characters of the same codepoint; values @code{#x110000} (1114112)
through @code{#x3FFF7F} (4194175) represent characters that are not
unified with Unicode; and values @code{#x3FFF80} (4194176) through
@code{#x3FFFFF} (4194303) represent eight-bit raw bytes.
@defun characterp charcode
This returns @code{t} if @var{charcode} is a valid character, and
@code{nil} otherwise.
@example
@group
(characterp 65)
@result{} t
@end group
@group
(characterp 4194303)
@result{} t
@end group
@group
(characterp 4194304)
@result{} nil
@end group
@end example
@end defun
@cindex maximum value of character codepoint
@cindex codepoint, largest value
@defun max-char
This function returns the largest value that a valid character
codepoint can have.
@example
@group
(characterp (max-char))
@result{} t
@end group
@group
(characterp (1+ (max-char)))
@result{} nil
@end group
@end example
@end defun
@defun get-byte &optional pos string
This function returns the byte at character position @var{pos} in the
current buffer. If the current buffer is unibyte, this is literally
the byte at that position. If the buffer is multibyte, byte values of
@acronym{ASCII} characters are the same as character codepoints,
whereas eight-bit raw bytes are converted to their 8-bit codes. The
function signals an error if the character at @var{pos} is
non-@acronym{ASCII}.
The optional argument @var{string} means to get a byte value from that
string instead of the current buffer.
@end defun
@node Character Properties
@section Character Properties
@cindex character properties
A @dfn{character property} is a named attribute of a character that
specifies how the character behaves and how it should be handled
during text processing and display. Thus, character properties are an
important part of specifying the character's semantics.
On the whole, Emacs follows the Unicode Standard in its implementation
of character properties. In particular, Emacs supports the
@uref{http://www.unicode.org/reports/tr23/, Unicode Character Property
Model}, and the Emacs character property database is derived from the
Unicode Character Database (@acronym{UCD}). See the
@uref{http://www.unicode.org/versions/Unicode5.0.0/ch04.pdf, Character
Properties chapter of the Unicode Standard}, for a detailed
description of Unicode character properties and their meaning. This
section assumes you are already familiar with that chapter of the
Unicode Standard, and want to apply that knowledge to Emacs Lisp
programs.
In Emacs, each property has a name, which is a symbol, and a set of
possible values, whose types depend on the property; if a character
does not have a certain property, the value is @code{nil}. As a
general rule, the names of character properties in Emacs are produced
from the corresponding Unicode properties by downcasing them and
replacing each @samp{_} character with a dash @samp{-}. For example,
@code{Canonical_Combining_Class} becomes
@code{canonical-combining-class}. However, sometimes we shorten the
names to make their use easier.
Here is the full list of value types for all the character
properties that Emacs knows about:
@table @code
@item name
Corresponds to the @code{Name} Unicode property. The value is a
string consisting of upper-case Latin letters A to Z, digits, spaces,
and hyphen @samp{-} characters.
@cindex unicode general category
@item general-category
Corresponds to the @code{General_Category} Unicode property. The
value is a symbol whose name is a 2-letter abbreviation of the
character's classification.
@item canonical-combining-class
Corresponds to the @code{Canonical_Combining_Class} Unicode property.
The value is an integer number.
@item bidi-class
Corresponds to the Unicode @code{Bidi_Class} property. The value is a
symbol whose name is the Unicode @dfn{directional type} of the
character.
@item decomposition
Corresponds to the Unicode @code{Decomposition_Type} and
@code{Decomposition_Value} properties. The value is a list, whose
first element may be a symbol representing a compatibility formatting
tag, such as @code{small}@footnote{
Note that the Unicode spec writes these tag names inside
@samp{<..>} brackets. The tag names in Emacs do not include the
brackets; e.g., Unicode specifies @samp{<small>} where Emacs uses
@samp{small}.
}; the other elements are characters that give the compatibility
decomposition sequence of this character.
@item decimal-digit-value
Corresponds to the Unicode @code{Numeric_Value} property for
characters whose @code{Numeric_Type} is @samp{Digit}. The value is an
integer number.
@item digit-value
Corresponds to the Unicode @code{Numeric_Value} property for
characters whose @code{Numeric_Type} is @samp{Decimal}. The value is
an integer number. Examples of such characters include compatibility
subscript and superscript digits, for which the value is the
corresponding number.
@item numeric-value
Corresponds to the Unicode @code{Numeric_Value} property for
characters whose @code{Numeric_Type} is @samp{Numeric}. The value of
this property is an integer or a floating-point number. Examples of
characters that have this property include fractions, subscripts,
superscripts, Roman numerals, currency numerators, and encircled
numbers. For example, the value of this property for the character
@code{U+2155} (@sc{vulgar fraction one fifth}) is @code{0.2}.
@item mirrored
Corresponds to the Unicode @code{Bidi_Mirrored} property. The value
of this property is a symbol, either @code{Y} or @code{N}.
@item old-name
Corresponds to the Unicode @code{Unicode_1_Name} property. The value
is a string.
@item iso-10646-comment
Corresponds to the Unicode @code{ISO_Comment} property. The value is
a string.
@item uppercase
Corresponds to the Unicode @code{Simple_Uppercase_Mapping} property.
The value of this property is a single character.
@item lowercase
Corresponds to the Unicode @code{Simple_Lowercase_Mapping} property.
The value of this property is a single character.
@item titlecase
Corresponds to the Unicode @code{Simple_Titlecase_Mapping} property.
@dfn{Title case} is a special form of a character used when the first
character of a word needs to be capitalized. The value of this
property is a single character.
@end table
@defun get-char-code-property char propname
This function returns the value of @var{char}'s @var{propname} property.
@example
@group
(get-char-code-property ? 'general-category)
@result{} Zs
@end group
@group
(get-char-code-property ?1 'general-category)
@result{} Nd
@end group
@group
;; subscript 4
(get-char-code-property ?\u2084 'digit-value)
@result{} 4
@end group
@group
;; one fifth
(get-char-code-property ?\u2155 'numeric-value)
@result{} 0.2
@end group
@group
;; Roman IV
(get-char-code-property ?\u2163 'numeric-value)
@result{} 4
@end group
@end example
@end defun
@defun char-code-property-description prop value
This function returns the description string of property @var{prop}'s
@var{value}, or @code{nil} if @var{value} has no description.
@example
@group
(char-code-property-description 'general-category 'Zs)
@result{} "Separator, Space"
@end group
@group
(char-code-property-description 'general-category 'Nd)
@result{} "Number, Decimal Digit"
@end group
@group
(char-code-property-description 'numeric-value '1/5)
@result{} nil
@end group
@end example
@end defun
@defun put-char-code-property char propname value
This function stores @var{value} as the value of the property
@var{propname} for the character @var{char}.
@end defun
@defvar unicode-category-table
The value of this variable is a char-table (@pxref{Char-Tables}) that
specifies, for each character, its Unicode @code{General_Category}
property as a symbol.
@end defvar
@defvar char-script-table
The value of this variable is a char-table that specifies, for each
character, a symbol whose name is the script to which the character
belongs, according to the Unicode Standard classification of the
Unicode code space into script-specific blocks. This char-table has a
single extra slot whose value is the list of all script symbols.
@end defvar
@defvar char-width-table
The value of this variable is a char-table that specifies the width of
each character in columns that it will occupy on the screen.
@end defvar
@defvar printable-chars
The value of this variable is a char-table that specifies, for each
character, whether it is printable or not. That is, if evaluating
@code{(aref printable-chars char)} results in @code{t}, the character
is printable, and if it results in @code{nil}, it is not.
@end defvar
@node Character Sets
@section Character Sets
@cindex character sets
@cindex charset
@cindex coded character set
An Emacs @dfn{character set}, or @dfn{charset}, is a set of characters
in which each character is assigned a numeric code point. (The
Unicode Standard calls this a @dfn{coded character set}.) Each Emacs
charset has a name which is a symbol. A single character can belong
to any number of different character sets, but it will generally have
a different code point in each charset. Examples of character sets
include @code{ascii}, @code{iso-8859-1}, @code{greek-iso8859-7}, and
@code{windows-1255}. The code point assigned to a character in a
charset is usually different from its code point used in Emacs buffers
and strings.
@cindex @code{emacs}, a charset
@cindex @code{unicode}, a charset
@cindex @code{eight-bit}, a charset
Emacs defines several special character sets. The character set
@code{unicode} includes all the characters whose Emacs code points are
in the range @code{0..#x10FFFF}. The character set @code{emacs}
includes all @acronym{ASCII} and non-@acronym{ASCII} characters.
Finally, the @code{eight-bit} charset includes the 8-bit raw bytes;
Emacs uses it to represent raw bytes encountered in text.
@defun charsetp object
Returns @code{t} if @var{object} is a symbol that names a character set,
@code{nil} otherwise.
@end defun
@defvar charset-list
The value is a list of all defined character set names.
@end defvar
@defun charset-priority-list &optional highestp
This functions returns a list of all defined character sets ordered by
their priority. If @var{highestp} is non-@code{nil}, the function
returns a single character set of the highest priority.
@end defun
@defun set-charset-priority &rest charsets
This function makes @var{charsets} the highest priority character sets.
@end defun
@defun char-charset character &optional restriction
This function returns the name of the character set of highest
priority that @var{character} belongs to. @acronym{ASCII} characters
are an exception: for them, this function always returns @code{ascii}.
If @var{restriction} is non-@code{nil}, it should be a list of
charsets to search. Alternatively, it can be a coding system, in
which case the returned charset must be supported by that coding
system (@pxref{Coding Systems}).
@end defun
@defun charset-plist charset
This function returns the property list of the character set
@var{charset}. Although @var{charset} is a symbol, this is not the
same as the property list of that symbol. Charset properties include
important information about the charset, such as its documentation
string, short name, etc.
@end defun
@defun put-charset-property charset propname value
This function sets the @var{propname} property of @var{charset} to the
given @var{value}.
@end defun
@defun get-charset-property charset propname
This function returns the value of @var{charset}s property
@var{propname}.
@end defun
@deffn Command list-charset-chars charset
This command displays a list of characters in the character set
@var{charset}.
@end deffn
Emacs can convert between its internal representation of a character
and the character's codepoint in a specific charset. The following
two functions support these conversions.
@c FIXME: decode-char and encode-char accept and ignore an additional
@c argument @var{restriction}. When that argument actually makes a
@c difference, it should be documented here.
@defun decode-char charset code-point
This function decodes a character that is assigned a @var{code-point}
in @var{charset}, to the corresponding Emacs character, and returns
it. If @var{charset} doesn't contain a character of that code point,
the value is @code{nil}. If @var{code-point} doesn't fit in a Lisp
integer (@pxref{Integer Basics, most-positive-fixnum}), it can be
specified as a cons cell @code{(@var{high} . @var{low})}, where
@var{low} are the lower 16 bits of the value and @var{high} are the
high 16 bits.
@end defun
@defun encode-char char charset
This function returns the code point assigned to the character
@var{char} in @var{charset}. If the result does not fit in a Lisp
integer, it is returned as a cons cell @code{(@var{high} . @var{low})}
that fits the second argument of @code{decode-char} above. If
@var{charset} doesn't have a codepoint for @var{char}, the value is
@code{nil}.
@end defun
The following function comes in handy for applying a certain
function to all or part of the characters in a charset:
@defun map-charset-chars function charset &optional arg from-code to-code
Call @var{function} for characters in @var{charset}. @var{function}
is called with two arguments. The first one is a cons cell
@code{(@var{from} . @var{to})}, where @var{from} and @var{to}
indicate a range of characters contained in charset. The second
argument passed to @var{function} is @var{arg}.
By default, the range of codepoints passed to @var{function} includes
all the characters in @var{charset}, but optional arguments
@var{from-code} and @var{to-code} limit that to the range of
characters between these two codepoints of @var{charset}. If either
of them is @code{nil}, it defaults to the first or last codepoint of
@var{charset}, respectively.
@end defun
@node Scanning Charsets
@section Scanning for Character Sets
Sometimes it is useful to find out which character set a particular
character belongs to. One use for this is in determining which coding
systems (@pxref{Coding Systems}) are capable of representing all of
the text in question; another is to determine the font(s) for
displaying that text.
@defun charset-after &optional pos
This function returns the charset of highest priority containing the
character at position @var{pos} in the current buffer. If @var{pos}
is omitted or @code{nil}, it defaults to the current value of point.
If @var{pos} is out of range, the value is @code{nil}.
@end defun
@defun find-charset-region beg end &optional translation
This function returns a list of the character sets of highest priority
that contain characters in the current buffer between positions
@var{beg} and @var{end}.
The optional argument @var{translation} specifies a translation table
to use for scanning the text (@pxref{Translation of Characters}). If
it is non-@code{nil}, then each character in the region is translated
through this table, and the value returned describes the translated
characters instead of the characters actually in the buffer.
@end defun
@defun find-charset-string string &optional translation
This function returns a list of character sets of highest priority
that contain characters in @var{string}. It is just like
@code{find-charset-region}, except that it applies to the contents of
@var{string} instead of part of the current buffer.
@end defun
@node Translation of Characters
@section Translation of Characters
@cindex character translation tables
@cindex translation tables
A @dfn{translation table} is a char-table (@pxref{Char-Tables}) that
specifies a mapping of characters into characters. These tables are
used in encoding and decoding, and for other purposes. Some coding
systems specify their own particular translation tables; there are
also default translation tables which apply to all other coding
systems.
A translation table has two extra slots. The first is either
@code{nil} or a translation table that performs the reverse
translation; the second is the maximum number of characters to look up
for translating sequences of characters (see the description of
@code{make-translation-table-from-alist} below).
@defun make-translation-table &rest translations
This function returns a translation table based on the argument
@var{translations}. Each element of @var{translations} should be a
list of elements of the form @code{(@var{from} . @var{to})}; this says
to translate the character @var{from} into @var{to}.
The arguments and the forms in each argument are processed in order,
and if a previous form already translates @var{to} to some other
character, say @var{to-alt}, @var{from} is also translated to
@var{to-alt}.
@end defun
During decoding, the translation table's translations are applied to
the characters that result from ordinary decoding. If a coding system
has the property @code{:decode-translation-table}, that specifies the
translation table to use, or a list of translation tables to apply in
sequence. (This is a property of the coding system, as returned by
@code{coding-system-get}, not a property of the symbol that is the
coding system's name. @xref{Coding System Basics,, Basic Concepts of
Coding Systems}.) Finally, if
@code{standard-translation-table-for-decode} is non-@code{nil}, the
resulting characters are translated by that table.
During encoding, the translation table's translations are applied to
the characters in the buffer, and the result of translation is
actually encoded. If a coding system has property
@code{:encode-translation-table}, that specifies the translation table
to use, or a list of translation tables to apply in sequence. In
addition, if the variable @code{standard-translation-table-for-encode}
is non-@code{nil}, it specifies the translation table to use for
translating the result.
@defvar standard-translation-table-for-decode
This is the default translation table for decoding. If a coding
systems specifies its own translation tables, the table that is the
value of this variable, if non-@code{nil}, is applied after them.
@end defvar
@defvar standard-translation-table-for-encode
This is the default translation table for encoding. If a coding
systems specifies its own translation tables, the table that is the
value of this variable, if non-@code{nil}, is applied after them.
@end defvar
@defvar translation-table-for-input
Self-inserting characters are translated through this translation
table before they are inserted. Search commands also translate their
input through this table, so they can compare more reliably with
what's in the buffer.
This variable automatically becomes buffer-local when set.
@end defvar
@defun make-translation-table-from-vector vec
This function returns a translation table made from @var{vec} that is
an array of 256 elements to map bytes (values 0 through #xFF) to
characters. Elements may be @code{nil} for untranslated bytes. The
returned table has a translation table for reverse mapping in the
first extra slot, and the value @code{1} in the second extra slot.
This function provides an easy way to make a private coding system
that maps each byte to a specific character. You can specify the
returned table and the reverse translation table using the properties
@code{:decode-translation-table} and @code{:encode-translation-table}
respectively in the @var{props} argument to
@code{define-coding-system}.
@end defun
@defun make-translation-table-from-alist alist
This function is similar to @code{make-translation-table} but returns
a complex translation table rather than a simple one-to-one mapping.
Each element of @var{alist} is of the form @code{(@var{from}
. @var{to})}, where @var{from} and @var{to} are either characters or
vectors specifying a sequence of characters. If @var{from} is a
character, that character is translated to @var{to} (i.e.@: to a
character or a character sequence). If @var{from} is a vector of
characters, that sequence is translated to @var{to}. The returned
table has a translation table for reverse mapping in the first extra
slot, and the maximum length of all the @var{from} character sequences
in the second extra slot.
@end defun
@node Coding Systems
@section Coding Systems
@cindex coding system
When Emacs reads or writes a file, and when Emacs sends text to a
subprocess or receives text from a subprocess, it normally performs
character code conversion and end-of-line conversion as specified
by a particular @dfn{coding system}.
How to define a coding system is an arcane matter, and is not
documented here.
@menu
* Coding System Basics:: Basic concepts.
* Encoding and I/O:: How file I/O functions handle coding systems.
* Lisp and Coding Systems:: Functions to operate on coding system names.
* User-Chosen Coding Systems:: Asking the user to choose a coding system.
* Default Coding Systems:: Controlling the default choices.
* Specifying Coding Systems:: Requesting a particular coding system
for a single file operation.
* Explicit Encoding:: Encoding or decoding text without doing I/O.
* Terminal I/O Encoding:: Use of encoding for terminal I/O.
* MS-DOS File Types:: How DOS "text" and "binary" files
relate to coding systems.
@end menu
@node Coding System Basics
@subsection Basic Concepts of Coding Systems
@cindex character code conversion
@dfn{Character code conversion} involves conversion between the
internal representation of characters used inside Emacs and some other
encoding. Emacs supports many different encodings, in that it can
convert to and from them. For example, it can convert text to or from
encodings such as Latin 1, Latin 2, Latin 3, Latin 4, Latin 5, and
several variants of ISO 2022. In some cases, Emacs supports several
alternative encodings for the same characters; for example, there are
three coding systems for the Cyrillic (Russian) alphabet: ISO,
Alternativnyj, and KOI8.
Every coding system specifies a particular set of character code
conversions, but the coding system @code{undecided} is special: it
leaves the choice unspecified, to be chosen heuristically for each
file, based on the file's data.
In general, a coding system doesn't guarantee roundtrip identity:
decoding a byte sequence using coding system, then encoding the
resulting text in the same coding system, can produce a different byte
sequence. But some coding systems do guarantee that the byte sequence
will be the same as what you originally decoded. Here are a few
examples:
@quotation
iso-8859-1, utf-8, big5, shift_jis, euc-jp
@end quotation
Encoding buffer text and then decoding the result can also fail to
reproduce the original text. For instance, if you encode a character
with a coding system which does not support that character, the result
is unpredictable, and thus decoding it using the same coding system
may produce a different text. Currently, Emacs can't report errors
that result from encoding unsupported characters.
@cindex EOL conversion
@cindex end-of-line conversion
@cindex line end conversion
@dfn{End of line conversion} handles three different conventions
used on various systems for representing end of line in files. The
Unix convention, used on GNU and Unix systems, is to use the linefeed
character (also called newline). The DOS convention, used on
MS-Windows and MS-DOS systems, is to use a carriage-return and a
linefeed at the end of a line. The Mac convention is to use just
carriage-return.
@cindex base coding system
@cindex variant coding system
@dfn{Base coding systems} such as @code{latin-1} leave the end-of-line
conversion unspecified, to be chosen based on the data. @dfn{Variant
coding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and
@code{latin-1-mac} specify the end-of-line conversion explicitly as
well. Most base coding systems have three corresponding variants whose
names are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}.
@vindex raw-text@r{ coding system}
The coding system @code{raw-text} is special in that it prevents
character code conversion, and causes the buffer visited with this
coding system to be a unibyte buffer. For historical reasons, you can
save both unibyte and multibyte text with this coding system. When
you use @code{raw-text} to encode multibyte text, it does perform one
character code conversion: it converts eight-bit characters to their
single-byte external representation. @code{raw-text} does not specify
the end-of-line conversion, allowing that to be determined as usual by
the data, and has the usual three variants which specify the
end-of-line conversion.
@vindex no-conversion@r{ coding system}
@vindex binary@r{ coding system}
@code{no-conversion} (and its alias @code{binary}) is equivalent to
@code{raw-text-unix}: it specifies no conversion of either character
codes or end-of-line.
@vindex emacs-internal@r{ coding system}
@vindex utf-8-emacs@r{ coding system}
The coding system @code{utf-8-emacs} specifies that the data is
represented in the internal Emacs encoding (@pxref{Text
Representations}). This is like @code{raw-text} in that no code
conversion happens, but different in that the result is multibyte
data. The name @code{emacs-internal} is an alias for
@code{utf-8-emacs}.
@defun coding-system-get coding-system property
This function returns the specified property of the coding system
@var{coding-system}. Most coding system properties exist for internal
purposes, but one that you might find useful is @code{:mime-charset}.
That property's value is the name used in MIME for the character coding
which this coding system can read and write. Examples:
@example
(coding-system-get 'iso-latin-1 :mime-charset)
@result{} iso-8859-1
(coding-system-get 'iso-2022-cn :mime-charset)
@result{} iso-2022-cn
(coding-system-get 'cyrillic-koi8 :mime-charset)
@result{} koi8-r
@end example
The value of the @code{:mime-charset} property is also defined
as an alias for the coding system.
@end defun
@defun coding-system-aliases coding-system
This function returns the list of aliases of @var{coding-system}.
@end defun
@node Encoding and I/O
@subsection Encoding and I/O
The principal purpose of coding systems is for use in reading and
writing files. The function @code{insert-file-contents} uses a coding
system to decode the file data, and @code{write-region} uses one to
encode the buffer contents.
You can specify the coding system to use either explicitly
(@pxref{Specifying Coding Systems}), or implicitly using a default
mechanism (@pxref{Default Coding Systems}). But these methods may not
completely specify what to do. For example, they may choose a coding
system such as @code{undefined} which leaves the character code
conversion to be determined from the data. In these cases, the I/O
operation finishes the job of choosing a coding system. Very often
you will want to find out afterwards which coding system was chosen.
@defvar buffer-file-coding-system
This buffer-local variable records the coding system used for saving the
buffer and for writing part of the buffer with @code{write-region}. If
the text to be written cannot be safely encoded using the coding system
specified by this variable, these operations select an alternative
encoding by calling the function @code{select-safe-coding-system}
(@pxref{User-Chosen Coding Systems}). If selecting a different encoding
requires to ask the user to specify a coding system,
@code{buffer-file-coding-system} is updated to the newly selected coding
system.
@code{buffer-file-coding-system} does @emph{not} affect sending text
to a subprocess.
@end defvar
@defvar save-buffer-coding-system
This variable specifies the coding system for saving the buffer (by
overriding @code{buffer-file-coding-system}). Note that it is not used
for @code{write-region}.
When a command to save the buffer starts out to use
@code{buffer-file-coding-system} (or @code{save-buffer-coding-system}),
and that coding system cannot handle
the actual text in the buffer, the command asks the user to choose
another coding system (by calling @code{select-safe-coding-system}).
After that happens, the command also updates
@code{buffer-file-coding-system} to represent the coding system that
the user specified.
@end defvar
@defvar last-coding-system-used
I/O operations for files and subprocesses set this variable to the
coding system name that was used. The explicit encoding and decoding
functions (@pxref{Explicit Encoding}) set it too.
@strong{Warning:} Since receiving subprocess output sets this variable,
it can change whenever Emacs waits; therefore, you should copy the
value shortly after the function call that stores the value you are
interested in.
@end defvar
The variable @code{selection-coding-system} specifies how to encode
selections for the window system. @xref{Window System Selections}.
@defvar file-name-coding-system
The variable @code{file-name-coding-system} specifies the coding
system to use for encoding file names. Emacs encodes file names using
that coding system for all file operations. If
@code{file-name-coding-system} is @code{nil}, Emacs uses a default
coding system determined by the selected language environment. In the
default language environment, any non-@acronym{ASCII} characters in
file names are not encoded specially; they appear in the file system
using the internal Emacs representation.