Skip to content
This repository

Windows: regression in cd magic handling of paths #1009

Closed
jdmarch opened this Issue November 17, 2011 · 22 comments

4 participants

Jonathan March Thomas Kluyver Min RK Fernando Perez
Jonathan March
Collaborator

A. In 0.10.2, the user could copy a path from elsewhere, and paste it after cd magic, backslashes and all:

cd c:\Users\myname

In 0.11, this fails:
SyntaxError: (unicode error) 'rawunicodeescape' codec can't decode bytes in position 5-6: truncated \uXXXX...

No form of quoting (including raw) seems to help.

For manual path entry, forward slashes or double backslashes work, but for copy/paste this is a regression.

B. In 0.10, tab completion of quoted paths with spaces worked a bit (Tabbing immediately after a space, it failed, but then tabbing after a subsequent nonspace it worked again.) AFAICT in 0.12 there is no way to get tab completion of paths with spaces (quoted or not) working following a space character.

C. In 0.10, %cd could recognize paths which included spaces, quoted or unquoted. Not so in 0.12, which requires quotes. Arguable whether this is a troublesome regression or a logical improvement.

Thomas Kluyver
Collaborator

Can you show the traceback that goes with that SyntaxError?

Thomas Kluyver
Collaborator

(High priority for now because the pasting of paths would be particularly good to have)

Jonathan March
Collaborator

Not much more than that:

File "<ipython-input-3-4d4e46418f5a>", line 1
SyntaxError: (unicode error) 'rawunicodeescape' codec can't decode bytes 
in position 5-6: truncated \uXXXX (<ipython-input-3-4d4e46418f5a>, line 1)
Min RK
Owner

If a Windows person wants to put some time into this, that would be great. I stared at it for a while prior to 0.11, but Windows paths were just too much of a mess.

Thomas Kluyver
Collaborator
Fernando Perez
Owner

@takluyver, good point. In fact, if it's really this that we are producing:

`u'get_ipython().magic(ur"cd c:\\Users\\myname")\n'`

then we're simply doing it wrong; it looks like we're both escaping and making it a raw literal, and that's the bug. By escaping it out we duplicate the \, but since it's raw, the second one gets interpreted as a unicode escape instead of a backslash. We should either escape OR make it raw, but not both.

Thomas Kluyver
Collaborator
Fernando Perez
Owner

Ah, got it. OK, then in that case it seems like we should escape \ instead of making it raw. Do we also need to escape ' ' (space)?

Thomas Kluyver
Collaborator
Fernando Perez
Owner

paths with spaces in them are pure evil, but people in windows use them all the time (and XP made it mandatory, with it's boneheaded 'c:\documents and settings' or whatever that directory was named...)

Min RK
Owner

It's honestly pathetic that spaces cause as much trouble as they do. It should be considered a huge failing of command-line environments in general, if support for perfectly logical names on the filesystem is ever anything more than trivial.

FWIW Windows 7 (or possibly Vista) changed the default path, so now it's C:\Users\username

Fernando Perez
Owner

Well, it's the clash with the years-old convenience of using space as an argument separator. Since the spacebar is the easiest key to hit, cmd line environments made the choice of using spaces into the default, universal argument separator (instead of commas like programming languages use). But that meant having to escape spaces all the time. And it means that string splitting logic must be very careful because it must avoid splitting when escapes are present. That's why it's so easy to get code that works with spaces at one level, but once that code calls something else things break, because the escaping at one level is lost at the next...

In any case, we should make it as easy as possible for people to cd, that's for sure :)

Min RK
Owner

Yes, and this is an inherent advantage of GUIs over command-line interfaces. This is all part of why Windows is so horrible as a CLI environment - they really have no interest in it, and if they don't care about CLIs, spaces don't cause problems, so they are free to use them. That of course causes a great deal of pain for those of us who do want a sensible CLI on Windows.

Fernando Perez
Owner

Agreed. Funnily enough, as much as I despise spaces in filenames, I find myself naturally gravitating towards them when I use a GUI for file operations; it's just a natural thing to want...

In any case, it seems like (trying to get back on topic :) for this bug we need to at least escape \ and test what happens with ' '. Shouldn't be too hard... This seems like a nasty regression, but I'm worried that if we don't draw the line somewhere we'll never get 0.12 out so I'm reluctant to make it critical. If anyone can provide a PR for this soon, that would go a big ways towards getting it fixed :)

Thomas Kluyver
Collaborator

I'll look into it.

Thomas Kluyver
Collaborator

The issue is in IPython.utils.text.make_quoted_expr. I think it's fixable, but I suddenly wondered: is make_quoted_expr(s) actually doing anything that we couldn't achieve just by using repr? Is there any case where eval(repr(s)) != s (with a string s)?

Fernando Perez
Owner

@takluyver, that's a very good point! Looking at that code, which is very old and ugly, I'm not sure we actually need it. If you want to take a stab at just replacing that function for now with a bare return repr(s) and seeing how things go, it would be great. The more of those home-brewed weird/hackish little utilities we get rid of, the better off we'll be.

Thomas Kluyver
Collaborator

@jdmarch: Can you test with PR #1019 ? (I don't have a Windows system handy). It probably doesn't fix issues with tab completion and spaces, but I think cd C:\Users should be working, at least.

Jonathan March
Collaborator

@takluyver: Thank you! PR #1019 fixes regressions A and C. Specifically, cd now recognizes all paths, quoted or unquoted, with or without spaces, using any mix of path separators (forward slashes, backslashes, or double backslashes), specifically including c:\Users.

Tab completion only works when using forward slashes, and even then only on paths which do not include spaces (quoted or unquoted). (Tested on Windows only.)

Thomas Kluyver
Collaborator

Great, that's the key thing I wanted to fix. The completion issues will be trickier, so we might leave them until after 0.12.

Thomas Kluyver
Collaborator

I'm closing this - I'll open a new issue for the tab completion, with priority medium.

Thomas Kluyver takluyver closed this November 20, 2011
Fernando Perez
Owner

Perfect, thanks a lot @takluyver for the great and quick work! We'll track the rest of the problem in #1021. @jdmarch, thanks for the report and testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.