tokenize.NL undocumented

[jhermann]

BPO	410155
Nosy	@gvanrossum, @tim-one, @freddrake

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gvanrossum'
closed_at = <Date 2001-03-22.00:24:16.000>
created_at = <Date 2001-03-20.21:28:13.000>
labels = ['docs']
title = 'tokenize.NL undocumented'
updated_at = <Date 2001-03-22.00:24:16.000>
user = 'https://bugs.python.org/jhermann'

bugs.python.org fields:

activity = <Date 2001-03-22.00:24:16.000>
actor = 'gvanrossum'
assignee = 'gvanrossum'
closed = True
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2001-03-20.21:28:13.000>
creator = 'jhermann'
dependencies = []
files = []
hgrepos = []
issue_num = 410155
keywords = []
message_count = 5.0
messages = ['3975', '3976', '3977', '3978', '3979']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'tim.peters', 'fdrake', 'jhermann']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue410155'
versions = []

[jhermann]

In section 17.5 of the Python 2.0 Library Reference,
the constant "NL" is not documented.

[freddrake]

Logged In: YES
user_id=3066

Ping, should NL be documented?

[tim-one]

Logged In: YES
user_id=31435

Yes, it should be, but Ping doesn't know about NL: Guido
added that one. Python's real tokenizer has no such token
type. It was added because-- unlike Python's real
tokenizer --tokenizer.py passes *everything* on to the
client, including blank lines and comments, so passes on
newlines that aren't NEWLINEs too. Reassigned to Guido in
case he remembers more about the details ...

[jhermann]

Logged In: YES
user_id=39128

tokenize.NL seems to be an empty line, as opposed to
token.NEWLINE which is a logical line end. Insofar NL is
comparable to COMMENT, since it has no semantic content and
is ignorable by a parser.

1 a = \
2 2
3
4 b=4

emits these tokens:

1,0-1,1: NAME 'a'
1,2-1,3: OP '='
2,0-2,1: NUMBER '2'
2,1-2,2: NEWLINE '\012'
3,0-3,1: NL '\012'
4,0-4,1: NAME 'b'
4,1-4,2: OP '='
4,2-4,3: NUMBER '4'
4,3-4,4: NEWLINE '\012'
5,0-5,0: ENDMARKER ''

[gvanrossum]

Logged In: YES
user_id=6380

The time machne strikes again. Skip Montanaro added docs
for NL in February. For proof, see:

http://python.sourceforge.net/devel-docs/lib/module-tokenize.html