-b option undocumented

[merwok]

BPO	11681
Nosy	@malemburg, @birkenfeld, @ncoghlan, @ezio-melotti, @merwok, @bitdancer, @vadmium, @zware, @gbengeult
PRs	bpo-11681: Document the -b and -bb options #1562
Files	issue11681.patch b_option.patch: adds codumentation for -b and -bb to main.c and cmdline.rst

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-05-13.14:46:56.669>
created_at = <Date 2011-03-26.00:53:45.308>
labels = ['easy', 'docs']
title = '-b option undocumented'
updated_at = <Date 2017-05-13.14:46:56.667>
user = 'https://github.com/merwok'

bugs.python.org fields:

activity = <Date 2017-05-13.14:46:56.667>
actor = 'zach.ware'
assignee = 'docs@python'
closed = True
closed_date = <Date 2017-05-13.14:46:56.669>
closer = 'zach.ware'
components = ['Documentation']
creation = <Date 2011-03-26.00:53:45.308>
creator = 'eric.araujo'
dependencies = []
files = ['35528', '46261']
hgrepos = []
issue_num = 11681
keywords = ['patch', 'easy']
message_count = 19.0
messages = ['132195', '132197', '132198', '132200', '132201', '132300', '132354', '164089', '207355', '219982', '219985', '220000', '279704', '285272', '285283', '285284', '285334', '293612', '293613']
nosy_count = 12.0
nosy_names = ['lemburg', 'georg.brandl', 'ncoghlan', 'ezio.melotti', 'eric.araujo', 'Arfrever', 'r.david.murray', 'docs@python', 'martin.panter', 'zach.ware', 'emily.zhao', 'gbengeult']
pr_nums = ['1562']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue11681'
versions = ['Python 2.7']

[merwok]

Python 2 has a -b command-line option which is undocumented.

It is used by the warnings machinery to decide to warn about (-b) or raise an error (-bb) if someone uses str(bytes) or str(buffer) (explanation courtesy of grep and Python/pythonrun.c).

[bitdancer]

Are you sure? (see bpo-10471). I don't see how -b could do anything in python2, since bytes and string are the same there (and, indeed, from command line experimentation it doesn't seem to).

[merwok]

The code definitely is in 2.7.

Python/_warnings.c:861: if (Py_BytesWarningFlag > 1)
Python/_warnings.c:863: else if (Py_BytesWarningFlag)
Python/_warnings.c:867: PyList_SET_ITEM(filters, pos++, create_filter(PyExc_BytesWarning,
Python/sysmodule.c:1257: SetFlag(Py_BytesWarningFlag);
Python/pythonrun.c:81:int Py_BytesWarningFlag; /* Warn on str(bytes) and str(buffer) */
Include/pydebug.h:14:PyAPI_DATA(int) Py_BytesWarningFlag;
Include/pyerrors.h:178:PyAPI_DATA(PyObject *) PyExc_BytesWarning;
Modules/main.c:285: Py_BytesWarningFlag++;
Objects/bytearrayobject.c:1003: if (Py_BytesWarningFlag) {
Objects/bytearrayobject.c:1004: if (PyErr_WarnEx(PyExc_BytesWarning,
Objects/bytearrayobject.c:1028: if (Py_BytesWarningFlag && op == Py_EQ) {
Objects/bytearrayobject.c:1029: if (PyErr_WarnEx(PyExc_BytesWarning,
Objects/exceptions.c:2018: * BytesWarning extends Warning
Objects/exceptions.c:2020:SimpleExtendsException(PyExc_Warning, BytesWarning,
Objects/exceptions.c:2110: PRE_INIT(BytesWarning)
Objects/exceptions.c:2178: POST_INIT(BytesWarning)
Lib/warnings.py:399: simplefilter(bytes_action, category=BytesWarning, append=1)
Lib/test/test_bytes.py:25: with test.test_support.check_warnings(('', BytesWarning)):
Lib/test/exception_hierarchy.txt:50: +-- BytesWarning
Include/pydebug.h:14:PyAPI_DATA(int) Py_BytesWarningFlag;
Include/pyerrors.h:178:PyAPI_DATA(PyObject *) PyExc_BytesWarning;
Doc/library/warnings.rst:164:* :exc:`BytesWarning` is ignored unless the :option:`-b` option is given once or

[bitdancer]

rdmurray@hey:/python/p27>cat temp.py
x = bytes('abc')
print x
print str(x)
rdmurray@hey:/python/p27>./python -bb temp.py
abc
abc

[merwok]

test_bytes has a branch for -b and passes without it, with -b and with -bb. I really don’t know.

[ncoghlan]

Even "from __future__ import unicode_literals" doesn't make it do anything.

Perhaps Christian merged it by mistake in [5341b30b1812]?

[malemburg]

Nick Coghlan wrote:

Nick Coghlan <ncoghlan@gmail.com> added the comment:

Even "from __future__ import unicode_literals" doesn't make it do anything.

Perhaps Christian merged it by mistake in [5341b30b1812]?

It looks more like some parts were left out in the merge
by accident: The 2.7 code only raises the warning for bytearrays,
not for bytes and buffers, whereas 3.2 issues the warning for
all bytes and bytearray (but not buffers; which makes sense,
since Python 3 no longer has a text buffer interface).

[moijes12]

Hi Marc

I tried reproducing this for bytearray using Python 2.7.2 but I can't see a warning.

devel@moses:$ python --version
Python 2.7.2+
devel@moses:$ cat test_py.py
k = range(10)
kb = bytearray(k)
print kb
kb = bytearray("hi")
print kb
devel@moses:~$ python -b test_py.py

hi
devel@moses:~$ python -bb test_py.py

hi

Please correct me if I'm wrong anywhere here. I'd like to work on this.

[vadmium]

Try this to trigger a warning:

python2 -b -c 'bytearray("3") == u"3"'
-c:1: BytesWarning: Comparison between bytearray and string

[emilyzhao]

Might be worth making this addition from 3 (I'm not sure how to add this to 2)
-b : issue warnings about str(bytes_instance), str(bytearray_instance)
and comparing bytes/bytearray with str. (-bb: issue errors)

Building on Martin's example:

On all of these, python2 is
Python 2.7.6 (default, Apr 6 2014, 23:14:26)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

emily-mba:cpython emily$ python2
>>> bytearray("3") == u"3"
False

emily-mba:cpython emily$ python2 -b
>>> bytearray("3") == u"3"
__main__:1: BytesWarning: Comparison between bytearray and string
False

emily-mba:cpython emily$ python2 -bb
>>> bytearray("3") == u"3"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
BytesWarning: Comparison between bytearray and string

[emilyzhao]

Here's an attempt (based on 3's main.c http://hg.python.org/cpython/file/8866ac6f2269/Modules/main.c)

[vadmium]

Trouble is, in Python 2 bytes() and str() are the same thing, and most of those conditions don’t apply. Maybe something like this is more correct:

-b : issue warnings about comparing bytearray with unicode.
(-bb: issue errors)

[vadmium]

As well as the usage string, it is missing from the list of options in the main documentation at Doc/using/cmdline.rst. There are a couple of places (sys.rst and warnings.rst) that reference :option:`-b`, and newer versions of Sphinx complain about this.

[gbengeult]

I have attached a patch for 2.7 that adds -b and -bb to the command line documentation in Modules/main.c and Doc/library/cmdline.rst, following the suggested text provided by Martin.

Interestingly, unicode(), bytearray(), and str() don't seem to be transitive in 2.7.

>>> u"3" == str(3)
True

>>> str(3) == bytearray("3")
True

>>> u"3" == bytearray("3")
False

[ncoghlan]

Right, the lack of transitivity comes from the fact that:

    >>> u"3" == str(3)
    True

Is really shorthand for:

    >>> u"3" == str(3).decode("ascii")
    True

However, the implicit decoding only triggers for *exactly* bytes instances, so in the bytearray case it needs to be explicit:

    >>> u"3" == bytearray(b"3").decode("ascii")
    True

[ncoghlan]

Added some review comments on the patch. The only critical one was changing a :class:`bytes` reference to :class:`unicode`, but I also suggested it may be worth mentioning that it *doesn't* enable warnings about all binary/text comparisons the way the Python 3 one does.

[gbengeult]

Thanks for the :class: markup edits. I'm still learning this stuff.

[zware]

New changeset f6c6d1e by Zachary Ware in branch '2.7':
bpo-11681: Document the -b and -bb options (GH-1562)
f6c6d1e

[zware]

Thanks for the patch, Greg! I implemented Nick's suggestions and merged it.