Special Characters Don't Work When Using Virtual Environment

[glGarg]

BPO	38098
Nosy	@pfmoore, @vstinner, @tjguk, @ezio-melotti, @zware, @eryksun, @zooba, @glGarg

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-13.17:06:07.703>
created_at = <Date 2019-09-10.23:30:51.990>
labels = ['type-bug', '3.8', 'OS-windows', '3.7', 'expert-unicode', '3.9']
title = "Special Characters Don't Work When Using Virtual Environment"
updated_at = <Date 2019-09-13.17:06:07.579>
user = 'https://github.com/glGarg'

bugs.python.org fields:

activity = <Date 2019-09-13.17:06:07.579>
actor = 'eryksun'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-13.17:06:07.703>
closer = 'eryksun'
components = ['Unicode', 'Windows']
creation = <Date 2019-09-10.23:30:51.990>
creator = 'glGarg'
dependencies = []
files = []
hgrepos = []
issue_num = 38098
keywords = []
message_count = 4.0
messages = ['351757', '352313', '352348', '352370']
nosy_count = 8.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'ezio.melotti', 'zach.ware', 'eryksun', 'steve.dower', 'glGarg']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38098'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[glGarg]

Recently, I noticed that whenever I execute python from inside a virtual environment, I can't seem to write special characters into the shell. Although, when I exit the virtual environment, this issue no longer occurs. I've confirmed that the versions of python inside and outside the virtual environment are the same.

For example, when I paste the string 'doesn’t' (with the Unicode apostrophe) into the shell without virtual environment, I get the following:
>>> doesn’t     # without virtual environment

When I paste the same string into the shell with a virtual environment active (and the same version of python), I get the following:
>>> doesnt      # with virtual environment

The version of python I am using is python 3.7.4 (though I have reproduced this on python 3.6.5 as well). I am creating the virtual environments using the venv module. Also, I think that this problem only happens on Windows as I couldn’t reproduce it on Linux.

Does anyone know why this may be happening? Please let me know if I should provide any other details.

[zooba]

Thanks Spandan

This doesn't reproduce trivially for me, so I suspect there is some difference in the venv.

Could you run "pip list" in both the venv and with it deactivated and share the output? There are some packages that modify how console input works.

[zooba]

Thanks Spandan

This doesn't reproduce trivially for me, so I suspect there is some difference in the venv.

Could you run "pip list" in both the venv and with it deactivated and share the output? There are some packages that modify how console input works.

[glGarg]

This is the output of pip list with venv active:
(aladdin_windows_env) C:\Users\t-spga\Documents\workspace\aladdin_windows_env>pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
adal (1.2.2)
antlr4-python3-runtime (4.7.2)
applicationinsights (0.11.9)
argcomplete (1.10.0)
asn1crypto (0.24.0)
azure-batch (7.0.0)
azure-cli (2.0.72)
azure-cli-command-modules-nspkg (2.0.3)
azure-cli-core (2.0.72)
azure-cli-nspkg (3.0.4)
azure-cli-telemetry (1.0.3)
azure-common (1.1.23)
azure-cosmos (3.1.1)
azure-datalake-store (0.0.47)
azure-functions-devops-build (0.0.22)
azure-graphrbac (0.60.0)
azure-keyvault (1.1.0)
azure-kusto-data (0.0.33)
azure-mgmt-advisor (2.0.1)
azure-mgmt-apimanagement (0.1.0)
azure-mgmt-appconfiguration (0.1.0)
azure-mgmt-applicationinsights (0.1.1)
azure-mgmt-authorization (0.52.0)
azure-mgmt-batch (6.0.0)
azure-mgmt-batchai (2.0.0)
azure-mgmt-billing (0.2.0)
azure-mgmt-botservice (0.2.0)
azure-mgmt-cdn (3.1.0)
azure-mgmt-cognitiveservices (5.0.0)
azure-mgmt-compute (6.0.0)
azure-mgmt-consumption (2.0.0)
azure-mgmt-containerinstance (1.5.0)
azure-mgmt-containerregistry (3.0.0rc5)
azure-mgmt-containerservice (5.3.0)
azure-mgmt-cosmosdb (0.7.0)
azure-mgmt-datalake-analytics (0.2.1)
azure-mgmt-datalake-nspkg (3.0.1)
azure-mgmt-datalake-store (0.5.0)
azure-mgmt-datamigration (0.1.0)
azure-mgmt-deploymentmanager (0.1.0)
azure-mgmt-devtestlabs (2.2.0)
azure-mgmt-dns (2.1.0)
azure-mgmt-eventgrid (2.2.0)
azure-mgmt-eventhub (2.6.0)
azure-mgmt-hdinsight (1.1.0)
azure-mgmt-imagebuilder (0.2.1)
azure-mgmt-iotcentral (1.0.0)
azure-mgmt-iothub (0.8.2)
azure-mgmt-iothubprovisioningservices (0.2.0)
azure-mgmt-keyvault (1.1.0)
azure-mgmt-kusto (0.3.0)
azure-mgmt-loganalytics (0.2.0)
azure-mgmt-managedservices (1.0.0)
azure-mgmt-managementgroups (0.2.0)
azure-mgmt-maps (0.1.0)
azure-mgmt-marketplaceordering (0.2.1)
azure-mgmt-media (1.1.1)
azure-mgmt-monitor (0.5.2)
azure-mgmt-msi (0.2.0)
azure-mgmt-netapp (0.5.0)
azure-mgmt-network (4.0.0)
azure-mgmt-nspkg (3.0.2)
azure-mgmt-policyinsights (0.3.1)
azure-mgmt-privatedns (0.1.0)
azure-mgmt-rdbms (1.9.0)
azure-mgmt-recoveryservices (0.4.0)
azure-mgmt-recoveryservicesbackup (0.4.0)
azure-mgmt-redis (6.0.0)
azure-mgmt-relay (0.1.0)
azure-mgmt-reservations (0.3.1)
azure-mgmt-resource (2.2.0)
azure-mgmt-search (2.1.0)
azure-mgmt-security (0.1.0)
azure-mgmt-servicebus (0.6.0)
azure-mgmt-servicefabric (0.2.0)
azure-mgmt-signalr (0.3.0)
azure-mgmt-sql (0.13.0)
azure-mgmt-sqlvirtualmachine (0.4.0)
azure-mgmt-storage (4.0.0)
azure-mgmt-trafficmanager (0.51.0)
azure-mgmt-web (0.42.0)
azure-multiapi-storage (0.2.4)
azure-nspkg (3.0.2)
azure-storage-blob (1.5.0)
azure-storage-common (1.4.2)
bcrypt (3.1.7)
certifi (2019.6.16)
cffi (1.12.3)
chardet (3.0.4)
colorama (0.4.1)
cryptography (2.7)
fabric (2.5.0)
gitdb2 (2.0.5)
GitPython (2.1.11)
humanfriendly (4.18)
idna (2.8)
invoke (1.3.0)
isodate (0.6.0)
javaproperties (0.5.1)
Jinja2 (2.10.1)
jmespath (0.9.4)
jsondiff (1.2.0)
knack (0.6.3)
MarkupSafe (1.1.1)
mock (2.0.0)
msrest (0.6.10)
msrestazure (0.6.1)
oauthlib (3.1.0)
paramiko (2.6.0)
pbr (5.4.3)
pip (9.0.3)
portalocker (1.5.1)
psutil (5.6.3)
pycparser (2.19)
pydocumentdb (2.3.3)
Pygments (2.4.2)
PyJWT (1.7.1)
PyNaCl (1.3.0)
pyOpenSSL (19.0.0)
pypiwin32 (223)
pyreadline (2.1)
python-dateutil (2.8.0)
python-git (2018.2.1)
pytz (2019.1)
pywin32 (224)
PyYAML (5.1.2)
requests (2.22.0)
requests-oauthlib (1.2.0)
ruamel.yaml (0.16.5)
ruamel.yaml.clib (0.1.2)
scp (0.13.2)
Send2Trash (1.5.0)
setuptools (39.0.1)
six (1.12.0)
smmap2 (2.0.5)
sshtunnel (0.1.5)
tabulate (0.8.3)
urllib3 (1.25.3)
vsts (0.1.25)
vsts-cd-manager (1.0.2)
websocket-client (0.56.0)
wheel (0.30.0)
xmltodict (0.12.0)
You are using pip version 9.0.3, however version 19.2.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

This is the output of pip list with venv deactivated:
C:\Users\t-spga\Documents\workspace\aladdin_windows_env>pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.3)
setuptools (39.0.1)
You are using pip version 9.0.3, however version 19.2.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Thanks!

Spandan

[eryksun]

pyreadline (2.1)

The problem is pyreadline. It uses a low-level console read, but it fails to account for the Alt+Numpad sequence of input records that the console uses for non-OEM characters. IIRC, the Unicode character is only sent in the final input record of the sequence.