subprocess popen arguments using double quotes

[rengine]

BPO	23862
Nosy	@pfmoore

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-04-03.22:30:37.595>
created_at = <Date 2015-04-03.19:04:53.903>
labels = ['type-bug', 'invalid']
title = 'subprocess popen arguments using double quotes'
updated_at = <Date 2015-04-03.22:30:37.582>
user = 'https://bugs.python.org/rengine'

bugs.python.org fields:

activity = <Date 2015-04-03.22:30:37.582>
actor = 'paul.moore'
assignee = 'none'
closed = True
closed_date = <Date 2015-04-03.22:30:37.595>
closer = 'paul.moore'
components = []
creation = <Date 2015-04-03.19:04:53.903>
creator = 'rengine'
dependencies = []
files = []
hgrepos = []
issue_num = 23862
keywords = []
message_count = 2.0
messages = ['240026', '240042']
nosy_count = 2.0
nosy_names = ['paul.moore', 'rengine']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23862'
versions = ['Python 2.7']

[rengine]

Using Python 2.7.9

Noticed when I run a subprocess Popen with an argument containing double quotes, it will have a different affect depending on my operating system.

In Linux, if I run ./runme.py, it will call runme.sh which will append someargs.txt correctly without escaped quotations

In Windows, if I run ./runme.py, it will call runme.bat which will append someargs.txt with escaped quotations

Also in Windows, if I run runme.bat with an argument containing quotations it will append someargs.txt correctly without escaped quotations so this problem seems to be stemming from Python.

===================================== runme.py

#!/usr/bin/python
import sys
import subprocess
import shlex

# works on Linux:
#command_line = "./runme.sh --include=\"check\""

# fails on Windows:
command_line = "runme.bat --include=\"check\""
#command_line = "runme.bat --include=\"check\""

arg = shlex.shlex(command_line)
arg.quotes = '"'
arg.whitespace_split = True
arg.commenters = ''
command_line_args = list(arg)
print command_line_args

command_line_process = subprocess.Popen(
    command_line_args,
    universal_newlines=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)

line = ""
while True:
line = command_line_process.stdout.readline()
if line:
print line
else:
break

===================================== runme.bat

echo %*
echo %* >> someargs.txt

===================================== runme.sh

#!/bin/bash
echo $@
echo $@ >> someargs.txt

[pfmoore]

On Windows, subprocess.Popen has to convert an argument list to a command line (because that's what the OS expects for CreateProcess). It does so internally, using subprocess.list2cmdline, which follows the quoting rules for the MS C runtime (because that's what 99% of programs use). Unfortunately batfile argument parsing does *not* use these conventions, and you can find odd corner cases like this.

Specifically, it's not possible on Windows to "correctly" quote all forms of argument without knowing what program you are calling (because programs do their own argument parsing, usually but not always via the C runtime). The subprocess module has to work for unknown executables, so it chooses to do the right thing for programs using the MS C runtime argument parser (which is most programs).