PyUnicode_EncodeFSDefault win32 inconsistancy.

[ideasman42]

BPO	15859
Nosy	@pitrou, @vstinner, @tiran, @ezio-melotti, @ideasman42, @berkerpeksag
Files	fix_unicode.diff: patch on python 3.3 fix_unicode_v2.diff gns3-errorlog

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-01-26.15:48:24.759>
created_at = <Date 2012-09-04.05:24:42.879>
labels = ['type-bug', 'expert-unicode', 'OS-windows']
title = 'PyUnicode_EncodeFSDefault win32 inconsistancy.'
updated_at = <Date 2016-03-28.11:10:38.356>
user = 'https://github.com/ideasman42'

bugs.python.org fields:

activity = <Date 2016-03-28.11:10:38.356>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2015-01-26.15:48:24.759>
closer = 'vstinner'
components = ['Unicode', 'Windows']
creation = <Date 2012-09-04.05:24:42.879>
creator = 'ideasman42'
dependencies = []
files = ['27114', '36913', '42313']
hgrepos = []
issue_num = 15859
keywords = ['patch']
message_count = 11.0
messages = ['169816', '169834', '192793', '192795', '214543', '214559', '229288', '234751', '234752', '262548', '262567']
nosy_count = 8.0
nosy_names = ['pitrou', 'vstinner', 'christian.heimes', 'ezio.melotti', 'ideasman42', 'python-dev', 'berker.peksag', 'shanzhengcheng']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue15859'
versions = ['Python 3.5']

[ideasman42]

There is an inconsistency in PyUnicode_EncodeFSDefault(), on Linux its argument is checked to be unicode, and NULL is returned when its not. On windows however, this throws an assertion.

The problem with this is, in some CAPI code you may pass an argument and check for NULL as an error case, and allow the python API's exception to be exposed to the script author.

The problem here is a linux developer can use PyUnicode_EncodeFSDefault() this way, but not a windows developer.

A simplified use case below of a case where PyUnicode_EncodeFSDefault would fail.

Attached a fix so windows and posix systems behave the same.

---

const char *PyC_UnicodeAsByte(PyObject *py_str, PyObject **coerce)
{
	if (PyBytes_Check(py_str)) {
		return PyBytes_AS_STRING(py_str);
	}
	else if ((*coerce = PyUnicode_EncodeFSDefault(py_str))) {
		return PyBytes_AS_STRING(*coerce);
	}

---

Also: heres a list to the bug report in blenders tracker, where the bug was found.

https://projects.blender.org/tracker/index.php?func=detail&aid=31856&group_id=9&atid=498

[pitrou]

Sounds reasonable to me.

[tiran]

The code has changed considerable since you have created the patch. Is this still an issue for Python 3.3?

[vstinner]

Is this still an issue for Python 3.3?

Yes.

[BreamoreBoy]

msg192795 says this is still an issue so I'm not sure why it's been set to languishing as it's only 18 months old.

[vstinner]

It's not really a bug, Python behaves badly when you pass invalid parameters.

Campbell was asked to update his patch but he didn't 2 years later. So I just close the issue. Reopen a new issue with an updated patch if you want a nice exception message instead of a patch.

Or just fix your application to use Python C API correctly.

[ideasman42]

Updated the patch for '93049:d9a3d23cf8f0'

Note, the link for the original bug report has changed: See https://developer.blender.org/T31856

[python-dev]

New changeset e124aab5d9a0 by Victor Stinner in branch 'default':
Issue bpo-15859: PyUnicode_EncodeFSDefault(), PyUnicode_EncodeMBCS() and
https://hg.python.org/cpython/rev/e124aab5d9a0

[vstinner]

I applied fix_unicode_v2.diff to Python 3.5.

For older Python versions, it's easy to workaround this issue: ensure explicitly that your object is a Unicode object using PyUnicode_Check().

You may write your function to call PyUnicode_EncodeFSDefault() with a PyUnicode_Check() check.

I prefer to not touch the stable 3.4 branch.

[shanzhengcheng]

this issue occureed when i run gns3-1.4.5 in python3.5.1
help me !
my system is linux 3.6.11 release version is red flag inwise 8.0

[berkerpeksag]

shanzhengcheng: Please don't update closed issues.

gns3-1.4.5 is not part of the Python standard library. You'll get better support by using the gns3 support channels.

If you still think that this is a bug in Python, please file a new issue by using the form at http://bugs.python.org/issue?@template=item (it would be great if you provide a reproducer without using gns3)

Thanks!