Repository navigation
PDF20 Spec 07 Lexical
Part of: PDF20-Spec-07-Syntax
Navigation: PDF20-Spec-07-General | PDF20-Spec-Index | PDF20-Spec-07-Objects
7.2
Lexical conventions
7.2.1 General At the most fundamental level, a PDF file is a sequence of bytes. These bytes can be grouped into tokens according to the syntax rules described in subclauses 7.2.2, “Representation” through 7.2.4, “Comments”. One or more tokens are assembled to form higher-level syntactic entities, principally objects, which are the basic data values from which a PDF file is constructed. A PDF file is represented as a sequence of 8-bit bytes, some of which are interpreted as character codes in the ASCII character set and some of which are treated as arbitrary binary data depending upon the context. The contents (data) of a string or stream object in some contexts are interpreted as character codes in the PDFDocEncoding, UTF-8 or UTF-16 character set as described in 7.9.2, "String object types" and 7.3.8, "Stream objects" respectively. The contents of a string within a PDF content stream in some situations are interpreted as character codes that select glyphs to be drawn on the page according to a character encoding that is associated with the text font as described in clause 9, "Text".
7.2.2 Representation A non-encrypted PDF file can be entirely represented using byte values corresponding to the visible printable subset of the ASCII character set defined in INCITS 4-1986 (R2017), plus white-space characters. However, a PDF file is not restricted to the ASCII character set; it may contain arbitrary bytes, subject to the following considerations: •
The tokens that delimit objects and that describe the structure of a PDF file shall use the ASCII character set. In addition all the reserved words and the names used as keys in PDF standard dictionaries and certain types of arrays shall be defined using the ASCII character set.
•
The data values of strings and streams objects may be written either entirely using the ASCII character set or entirely in binary data. In actual practice, data that is naturally binary, such as sampled images, is usually represented in binary for compactness and efficiency.
•
A PDF file containing binary data shall be transported as a binary file rather than as a text file to ensure that all bytes of the file are faithfully preserved.
NOTE 1
A binary file is not portable to environments that impose reserved character codes, maximum line lengths, end-of-line conventions, or other restrictions.
NOTE 2
In this clause, the usage of the term character is entirely independent of any logical meaning that the value can have when it is treated as data in specific contexts, such as representing humanreadable text or selecting a glyph from a font.
7.2.3 Character set The PDF character set is divided into three classes referred to as regular, delimiter, and white-space © ISO 2020 – All rights reserved
21
ISO 32000-2:2020(E)
characters. This classification enables the grouping of characters into tokens including separating syntactic constructs such as names and numbers from each other. The rules defined in this subclause apply to all characters in the file except within strings, streams, and comments. NOTE 1
See 7.3.4, "String objects", 7.3.8, "Stream objects" and 7.2.4, "Comments".
PDF syntax treats any sequence of consecutive white-space characters, not inside of a string or stream, as one character. The characters that are considered white-space characters are shown in "Table 1 — White-space characters" and can be used to separate syntactic constructs such as names and numbers from each other. In all other contexts, PDF syntax treats any sequence of consecutive white-space characters as one character. Table 1 — White-space characters Decimal
Hexadecimal
Octal
Name
0
00
000
Null (NUL)
9
09
011
HORIZONTAL TAB (HT)
10
0A
012
LINE FEED (LF)
12
0C
014
FORM FEED (FF)
13
0D
015
CARRIAGE RETURN (CR)
32
20
040
SPACE (SP)
The CARRIAGE RETURN (0Dh) and LINE FEED (0Ah) characters, also called newline characters, shall be treated as end-of-line (EOL) markers. The combination of a CARRIAGE RETURN followed immediately by a LINE FEED shall be treated as one EOL marker. EOL markers may be treated the same as any other white-space characters. However, sometimes an EOL marker is required — that is, preceding a token that shall appear at the beginning of a line. NOTE 2
The examples in this document use a convention that arranges tokens into lines. However, the examples’ use of white-space for indentation is purely for clarity of exposition and need not be included in practical use.
The delimiter characters (, ), <, >, [, ], /, and % are special (LEFT PARENTHESIS (28h), RIGHT PARENTHESIS (29h), LESS-THAN SIGN (3Ch), GREATER-THAN SIGN (3Eh), LEFT SQUARE BRACKET (5Bh), RIGHT SQUARE BRACKET (5Dh), SOLIDUS (2Fh) and PERCENT SIGN (25h), respectively). They delimit syntactic entities such as arrays, names, and comments. The delimiter characters { and } (LEFT CURLY BRACE (7Bh) and RIGHT CURLY BRACE (7Dh)) are additional delimiter characters within Type 4 PostScript calculator functions (see 7.10.5 "Type 4 (PostScript calculator) functions"). In addition, the double character constructs << (3C3Ch) and >> (3E3Eh) are used to delimit dictionary objects (see 7.3.7, "Dictionary objects"). Any of these delimiters terminates the entity preceding it and is not included in the entity. Delimiter characters are allowed within the scope of a string when following the rules for composing strings; see 7.3.4.2, "Literal strings". The leading ( of a string does delimit a preceding entity and the closing ) of a string delimits the string’s end. “Table 2 — Delimiter characters” shows the delimiter characters. 22
© ISO 2020 – All rights reserved
ISO 32000-2:2020(E)
Table 2 — Delimiter characters Glyph
Decimal
Hexadecimal
Octal
Name
(
40
28
050
LEFT PARENTHESIS
)
41
29
051
RIGHT PARENTHESIS
<
60
3C
074
LESS-THAN SIGN
62
3E
076
GREATER-THAN SIGN
[
91
5B
133
LEFT SQUARE BRACKET
]
93
5D
135
RIGHT SQUARE BRACKET
{
123
7B
173
LEFT CURLY BRACKET
}
125
7D
175
RIGHT CURLY BRACKET
/
47
2F
057
SOLIDUS
%
37
25
045
PERCENT SIGN
All characters except the white-space characters and delimiters are referred to as regular characters. These characters include bytes that are outside the ASCII character set. A sequence of consecutive regular characters comprises a single token. PDF syntax is case-sensitive; corresponding uppercase and lowercase letters are distinct.
7.2.4 Comments Any occurrence of the PERCENT SIGN (25h) outside a string or inside a content stream (see 7.8.2, "Content streams") introduces a comment. The comment consists of all characters after the PERCENT SIGN and up to but not including the end-of-the-line marker. A PDF processor should ignore comments. PDF processors shall treat comments as single white-space characters for the purposes of lexical conversion (see 7.2, "Lexical conventions"). That is, a comment separates the token preceding it from the one following it. EXAMPLE
The PDF syntax fragment in this example is syntactically equivalent to just the tokens abc and 123.
abc%comment (/%) blah blah blah 123
Comments (other than the %PDF–n.m and %%EOF comments described in 7.5, "File structure") have no semantics defined by this document. PDF writers do not need to preserve comments. NOTE 1
Using comments to communicate information to PDF processors is discouraged. The preferred and documented mechanism is to use PDF names (see Annex E, "Extending PDF").
NOTE 2
This clause was clarified in this document (2020).
© ISO 2020 – All rights reserved
23
ISO 32000-2:2020(E)
Navigation: PDF20-Spec-07-General | PDF20-Spec-Index | PDF20-Spec-07-Objects