Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

on Android python fails to decode/encode command line arguments #71106

Closed
xdegaye mannequin opened this issue May 3, 2016 · 10 comments
Closed

on Android python fails to decode/encode command line arguments #71106

xdegaye mannequin opened this issue May 3, 2016 · 10 comments
Assignees
Labels
3.7 interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented May 3, 2016

BPO 26919
Nosy @vstinner, @xdegaye, @moreati
PRs
  • [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
  • Dependencies
  • bpo-27027: add the 'is_android' attribute to test.support
  • Files
  • retrofit_osx.patch
  • retrofit_osx_2.patch: test description changed
  • retrofit_osx_3.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/xdegaye'
    closed_at = <Date 2016-12-15.20:37:29.112>
    created_at = <Date 2016-05-03.09:18:35.370>
    labels = ['interpreter-core', 'type-bug', '3.7']
    title = 'on Android python fails to decode/encode command line arguments'
    updated_at = <Date 2017-03-31.16:36:20.029>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2017-03-31.16:36:20.029>
    actor = 'dstufft'
    assignee = 'xdegaye'
    closed = True
    closed_date = <Date 2016-12-15.20:37:29.112>
    closer = 'xdegaye'
    components = ['Interpreter Core']
    creation = <Date 2016-05-03.09:18:35.370>
    creator = 'xdegaye'
    dependencies = ['27027']
    files = ['42838', '42839', '42911']
    hgrepos = []
    issue_num = 26919
    keywords = ['patch']
    message_count = 10.0
    messages = ['264703', '264800', '265458', '265462', '265550', '265916', '279803', '283340', '283341', '283342']
    nosy_count = 4.0
    nosy_names = ['vstinner', 'xdegaye', 'python-dev', 'Alex.Willmer']
    pr_nums = ['552']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue26919'
    versions = ['Python 3.6', 'Python 3.7']

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 3, 2016

    test_cmd_line fails on an android emulator running an x86 system image at API level 21.

    ======================================================================
    FAIL: test_non_ascii (test.test_cmd_line.CmdLineTest)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_cmd_line.py", line 141, in test_non_ascii
        assert_python_ok('-c', command)
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/script_helper.py", line 135, in assert_python_ok
        return _assert_python(True, *args, **env_vars)
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/script_helper.py", line 121, in _assert_python
        err))
    AssertionError: Process return code is 1
    command line: ['/data/local/tmp/python/bin/python', '-X', 'faulthandler', '-I', '-c', "assert(ord('\xe6') == 230)"]

    stdout:
    ---

    ---

    stderr:
    ---
    Unable to decode the command from the command line:
    UnicodeEncodeError: 'utf-8' codec can't encode characters in position 12-13: surrogates not allowed
    ---

    ----------------------------------------------------------------------
    Ran 33 tests in 1.943s

    FAILED (failures=1, skipped=1)
    test test_cmd_line failed
    1 test failed:
    test_cmd_line
    Total duration: 0:00:03

    @xdegaye xdegaye mannequin added stdlib Python modules in the Lib dir build The build process and cross-build type-bug An unexpected behavior, bug, or error labels May 3, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 4, 2016

    There is a related issue bpo-4388 although different, and issue bpo-22747 may also be related.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 13, 2016

    This patch follows Victor suggestion in msg230407 and brings the changes made in issue bpo-4388 and issue bpo-16416 to the Android platform. As a consequence, it adds a new test for Android in test_cmd_line named test_osx_android_utf8 (was previously test_osx_utf8). The platform.android_ver() function is defined by issue bpo-26855: "add platform.android_ver() for android".

    With this patch test_cmd_line does not have any failure.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 13, 2016

    Patch updated after vadmium review.

    @vstinner
    Copy link
    Member

    vstinner commented May 14, 2016

    I commented retrofit_osx_2.patch on the review.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 20, 2016

    This new patch adds a dependency to issue bpo-27027.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Oct 31, 2016

    An interactive session confirms that the problem is indeed with the command line arguments of python invoked by subprocess (and the problem is fixed by the patch):

    >>> from test.support import FS_NONASCII
    >>> cmd = "assert(ord(%r) == %s)" % (FS_NONASCII, ord(FS_NONASCII))
    >>> exec(cmd)
    >>> import subprocess, sys
    >>> subprocess.run([sys.executable, '-c', cmd])
    Unable to decode the command from the command line:
    UnicodeEncodeError: 'utf-8' codec can't encode characters in position 12-13: surrogates not allowed
    CompletedProcess(args=['/data/data/org.bitbucket.pyona/python/bin/python', '-c', "assert(ord('\xe6') == 230)"], returncode=1)

    @xdegaye xdegaye mannequin added 3.7 interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) and removed stdlib Python modules in the Lib dir build The build process and cross-build labels Oct 31, 2016
    @xdegaye xdegaye mannequin self-assigned this Oct 31, 2016
    @xdegaye xdegaye mannequin changed the title android: test_cmd_line fails on Android python fails to decode/encode command line arguments Oct 31, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 15, 2016

    The patch fixes also the following errors in test_warnings:

    ======================================================================
    FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_warnings/__init__.py", line 1072, in test_nonascii
        "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
    AssertionError: b"['ignore:Deprecaci\\udcc3\\udcb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

    ======================================================================
    FAIL: test_nonascii (test.test_warnings.PyEnvironmentVariableTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_warnings/__init__.py", line 1072, in test_nonascii
        "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
    AssertionError: b"['ignore:Deprecaci\\udcc3\\udcb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Dec 15, 2016

    New changeset e5360d413ce4 by Xavier de Gaye in branch '3.6':
    Issue bpo-26919: On Android, operating system data is now always encoded/decoded
    https://hg.python.org/cpython/rev/e5360d413ce4

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 15, 2016

    Forgot the issue number in the commit message for the default branch, the corresponding changeset is 80a041d39c20.

    @xdegaye xdegaye mannequin closed this as completed Dec 15, 2016
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant