idle 3.1a1 utf8

[geon]

BPO	4815
Nosy	@loewis
Files	cp1250.py conv.diff: Convert from source encoding on opening (v2) idleunicode1.jpg conv.diff: Convert from source encoding on opening (v3) hello.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-01-18.20:26:16.888>
created_at = <Date 2009-01-03.04:56:20.628>
labels = ['expert-IDLE', 'release-blocker']
title = 'idle 3.1a1 utf8'
updated_at = <Date 2009-01-18.20:26:16.886>
user = 'https://bugs.python.org/geon'

bugs.python.org fields:

activity = <Date 2009-01-18.20:26:16.886>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = <Date 2009-01-18.20:26:16.888>
closer = 'loewis'
components = ['IDLE']
creation = <Date 2009-01-03.04:56:20.628>
creator = 'geon'
dependencies = []
files = ['12559', '12566', '12576', '12578', '12582']
hgrepos = []
issue_num = 4815
keywords = ['patch', 'needs review']
message_count = 24.0
messages = ['78932', '78936', '78937', '78938', '78939', '78940', '78941', '78949', '78966', '78967', '78970', '78972', '79004', '79006', '79009', '79011', '79012', '79014', '79016', '79052', '79055', '79060', '79068', '80120']
nosy_count = 2.0
nosy_names = ['loewis', 'geon']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue4815'
versions = ['Python 3.1']

[geon]

When you open file without encoding declaration, make changes and save,
then IDLE changes without any question encodings to utf8. You can try it
on attached file that is cp1250 now.

It could be that at first saving we are asked to use
*utf8
*current one.

[loewis]

IDLE is right to save the file as UTF-8; the file is invalid Python 3.0
code. In Python 3.0, the source encoding *is* UTF-8; nothing else is
allowed unless you have an encoding declaration.

Perhaps IDLE should offer to convert it on opening.

[geon]

You can open script made in python 2.x and it stops immediately working
after saving, if it is coding-aware. You can have bigger project and use
idle for editing config and text files from this project too. It is
"unfair" to change without notification the encodings. Or do you
consider IDLE just for beginners for learning?

[geon]

I forgot about "Perhaps IDLE should offer to convert it on opening."
That would be nice, too.

[loewis]

Here is a patch to provide an explicit message that the file will be
converted when the file is opened (also querying what encoding should be
converted from), answering the complaint that the conversion is without
notice.

If you want to edit Python 2.x scripts with IDLE 3, you need to add
encoding declarations to the files. For bigger projects, it is
reasonable to expect that they do add these declarations.

Again, please only report one issue at the time. You chose to make
Python source files the topic of this issue, so please don't divert into
config or text files.

[geon]

Sorry, where is the patch?

[geon]

OK, I got it.

In my opinion it would nice if user can either convert file to utf8 or
to do nothing and add new encodings declaration or cancel. Current
"Cancel" gives an Decoding error. If you give an encodings that doesn't
exist, it shouldn't destroy IDLE. Hoping its not my mistake, cause I do
not have all files from 3.1a - just those from idlelib.

[loewis]

In my opinion it would nice if user can either convert file to utf8 or
to do nothing and add new encodings declaration or cancel.

Ypu can still add an encoding declaration after the file got converted.
Cancelling is also possible.

If you give an encodings that doesn't exist, it shouldn't destroy IDLE.

Right. Here is a patch that fixes that.

[geon]

seems to be working.

Seems to me now I get it. The file encoding is ruled by the encoding
declaration. When I stated

# -- coding: cp1250 --

then the file would be saved in cp1250.

Now hoping that I would keep this issue, cause it comes with this
patches: when I open file *with* say # -- coding: cp1250 --, I am
asked to change to utf8. This behaviour was not before and is probably
unwanted.....

[loewis]

Now hoping that I would keep this issue, cause it comes with this
patches: when I open file *with* say # -- coding: cp1250 --, I am
asked to change to utf8. This behaviour was not before and is probably
unwanted.....

Actually, the behavior was there before - it's just that conversion was
silent. If you put an empty line after the coding declaration, it should
work well. bpo-4008 contains a patch for that problem.

[geon]

Well, thanks a lot.

(aware this is really off this issue): Now I even get the system of
patches - bpo-4008 solved the inconvenience in print Unicode signs
inside IDLE. Still not sure how works patches for Python versions. I
vote for including this a that patch about IDLE even in some 3.0.1, not
only in branch 3.1. In non-English speaking countries troubles with
encodings are more common, more beginners-frustrating than you can believe.

HNY 2009
Pavel Kosina

[loewis]

I
vote for including this a that patch about IDLE even in some 3.0.1, not
only in branch 3.1.

This is my plan, yes - hence I marked them all release-critical. They
still need review. I agree that IDLE in 3.0 is fairly broken wrt.
non-ASCII characters. I knew that before the release, but there is so
much work and only so little time. FWIW, my own language (German) also
uses (some) non-ASCII characters.

[geon]

I might have another problem with this patch and maybe also that one in
bpo-4008. Having a file with

print ("ěščřžýáíé")
# saved in cp1250

Open - confirm converting to utf8 - F5 - error: see attached file
idleunicode1.jpg

[loewis]

print ("ěščřžýáíé")

saved in cp1250

I can't reproduce the problem. Can you please attach the
exact file that failed to work?

[geon]

Martin v. Löwis napsal(a), dne 3.1.2009 22:24:

I can't reproduce the problem. Can you please attach the
exact file that failed to work?

You can use that one that is already here: cp1250.py. It is the same
error with me.

[loewis]

You can use that one that is already here: cp1250.py. It is the same
error with me.

Ok, then what are the exact steps to reproduce? What code base, what
patches applied, what user interaction in what order?

[geon]

Microsoft Windows XP [Verze 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\prg\Python30\Lib\idlelib>svn update # from
http://svn.python.org/projects/python/branches/py3k/Lib/idlelib
Restored 'AutoCompleteWindow.py'
Restored 'ToolTip.py'
Restored 'UndoDelegator.py'
Restored 'Bindings.py'
Restored '__init__.py'
Restored 'AutoComplete.py'
Restored 'configHandler.py'
Restored 'HyperParser.py'
Restored 'ColorDelegator.py'
Restored 'Delegator.py'
Restored 'ObjectBrowser.py'
Restored 'testcode.py'
Restored 'configSectionNameDialog.py'
Restored 'ZoomHeight.py'
Restored 'PyShell.py'
Restored 'ParenMatch.py'
Restored 'config-keys.def'
Restored 'Debugger.py'
Restored 'CREDITS.txt'
Restored 'configDialog.py'
Restored 'StackViewer.py'
Restored 'HISTORY.txt'
Restored 'SearchEngine.py'
Restored 'ReplaceDialog.py'
Restored 'ScriptBinding.py'
Restored 'ChangeLog'
Restored 'tabbedpages.py'
Restored 'keybindingDialog.py'
Restored 'configHelpSourceEdit.py'
Restored 'WidgetRedirector.py'
Restored 'GrepDialog.py'
Restored 'FormatParagraph.py'
Restored 'EditorWindow.py'
Restored 'help.txt'
Restored 'config-highlight.def'
Restored 'PyParse.py'
Restored 'README.txt'
Restored 'rpc.py'
Restored 'OutputWindow.py'
Restored 'aboutDialog.py'
Restored 'idle.bat'
Restored 'TODO.txt'
Restored 'config-main.def'
Restored 'IdleHistory.py'
Restored 'PathBrowser.py'
Restored 'IOBinding.py'
Restored 'WindowList.py'
Restored 'ScrolledList.py'
Restored 'ClassBrowser.py'
Restored 'FileList.py'
Restored 'CallTips.py'
Restored 'idle.py'
Restored 'CodeContext.py'
Restored 'textView.py'
Restored 'SearchDialogBase.py'
Restored 'CallTipWindow.py'
Restored 'SearchDialog.py'
Restored 'RemoteObjectBrowser.py'
Restored 'idlever.py'
Restored 'RemoteDebugger.py'
Restored 'TreeWidget.py'
Restored 'NEWS.txt'
Restored 'idle.pyw'
Restored 'run.py'
Restored 'config-extensions.def'
Restored 'AutoExpand.py'
Restored 'Percolator.py'
Restored 'dynOptionMenuWidget.py'
Restored 'extend.txt'
Restored 'MultiStatusBar.py'
Restored 'MultiCall.py'
Restored 'macosxSupport.py'
At revision 68230.

C:\prg\Python30\Lib\idlelib>patch < conv.diff
patching file IOBinding.py

C:\prg\Python30\Lib\idlelib>patch < idle_encoding_4.patch
patching file ScriptBinding.py
patching file IOBinding.py

-------------

Run IDLE - Open cp1250.py - confirm converting to utf8 - F5 (immediately, no change in code!) - error: see attached file
idleunicode1.jpg

All the other python code base is clean 3.0.
xpsp3

[loewis]

I see. Here is a revised patch. This sets the modified flag on the
buffer after conversion, so that you get asked to save it before running it.

[geon]

Yes. Goooood jooooob. ;-)

[geon]

With this file - hello.py (attached) - I should be also asked for
converting to utf8. When I open it, nothing changes, after making
changes and saving then the encodings is my windows standard cp1250 ....

[loewis]

With this file - hello.py (attached) - I should be also asked for
converting to utf8.

Why that? This file is already encoded in utf-8 just fine. It is,
simultaneously, also encoded in ASCII, cp1250, cp1252, and nearly
any other encoding in use (as long as it is ASCII-based).

When I open it, nothing changes, after making
changes and saving then the encodings is my windows standard cp1250 ....

What did you do to find that out?

[geon]

Martin v. Löwis napsal(a), dne 4.1.2009 14:39:

Why that? This file is already encoded in utf-8 just fine. It is,
simultaneously, also encoded in ASCII, cp1250, cp1252, and nearly
any other encoding in use (as long as it is ASCII-based).

Well I am not much experienced but this file is not real utf8. It is
encoded in ascii, cp1250, cp1252, and many other but not in utf8. utf8
has a special flag, special bytes inside - as a special mark for
editors. That is what this file doesnt have. Even after making change in
it in IDLE, it does not became real utf8. I always check it in another
editor, that can work with encoding very well - PSPad.

[loewis]

utf8 has a special flag

No, it doesn't.

as a special mark for editors.

That's the BOM, or UTF-8 signature. It's optional, and UTF-8-encoded
files typically do *not* have the UTF-8 signature.

Even after making change in
it in IDLE, it does not became real utf8.

Your understanding of UTF-8 is incorrect.

[loewis]

Committed as r68732, r68733.