New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot save files in non-UTF8 encoding #997

Closed
vsespb opened this Issue Jan 10, 2016 · 13 comments

Comments

Projects
None yet
4 participants
@vsespb

vsespb commented Jan 10, 2016

Getting the following message when trying to save some files.
screenshot from 2016-01-10 23 18 15

Restart does not help. Have no idea why some files affected, some not. File perms are ok.

Komodo Edit, version 9.3.2, build 16460, platform linux-x86_64. Built on Thu Dec 10 19:37:22 2015.

UPD: Ok, I get it. Everything that is not UTF-8 cannot be saved. I.e. files in Latin1 (which is not maps to valid UTF8) give such message.

@vsespb vsespb changed the title from Cannot save files to Cannot save files in non-UTF8 encoding Jan 10, 2016

@vsespb

This comment has been minimized.

vsespb commented Jan 10, 2016

to reproduce: create empty project, then
echo привет | iconv -t windows-1251 > file.pm open it (it's Perl, encoding correctly detected).
you can try any non-ascii in non-utf8 encoding. press Ctrl-S to save.

@mitchell-as

This comment has been minimized.

Member

mitchell-as commented Jan 11, 2016

That's strange. I am also on 64-bit Linux and I cannot reproduce after following your steps. See my screenshot below. Is this what you're seeing? I am able to Ctrl+S (Save) just fine. I can even type some normal characters and still save. Toggling between Latin1 and UTF8 also allows me to save.

latin1

@Defman21

This comment has been minimized.

Contributor

Defman21 commented Jan 11, 2016

I'm able to reproduce it.
echo привет мир | iconv -t windows-1251 > file.pm
komodo file.pm
Added some random text at a new line, pressed Ctrl+S.
Here's what I got:

/home/defman/Apps/Komodo/latest/lib/python/lib/python2.7/difflib.py:416: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  for j in b2j.get(a[i], nothing):
/home/defman/Apps/Komodo/latest/lib/python/lib/python2.7/difflib.py:437: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  a[besti+bestsize] == b[bestj+bestsize]:
ERROR:xpcom:Unhandled exception calling 'int8 * save(in boolean);'
Traceback (most recent call last):
  File "/home/defman/Apps/Komodo/latest/lib/mozilla/python/xpcom/server/policy.py", line 300, in _CallMethod_
    return 0, func(*params)
  File "/home/defman/Apps/Komodo/latest/lib/mozilla/components/koDocument.py", line 1664, in save
    self._clean(ensureFinalEOL, cleanLineEnds)
  File "/home/defman/Apps/Komodo/latest/lib/mozilla/python/xpcom/components.py", line 268, in wrapperFn
    return fn(*args, **kwargs)
  File "/home/defman/Apps/Komodo/latest/lib/mozilla/components/koDocument.py", line 1428, in _clean
    wsLinesToStrip = self.getChangedLinesWithTrailingWhitespace()
  File "/home/defman/Apps/Komodo/latest/lib/mozilla/components/koDocument.py", line 1386, in getChangedLinesWithTrailingWhitespace
    diff_content = self.getUnsavedChanges()
  File "/home/defman/Apps/Komodo/latest/lib/mozilla/components/koDocument.py", line 2060, in getUnsavedChanges
    return ''.join(difflines)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xef in position 1: invalid continuation byte
@mitchell-as

This comment has been minimized.

Member

mitchell-as commented Jan 11, 2016

@vsespb When you get the chance would you please answer my original question? That may help me diagnose and reproduce the issue. Thanks!

@vsespb

This comment has been minimized.

vsespb commented Jan 11, 2016

@mitchell-as,

See my screenshot below. Is this what you're seeing?

Yes, that's what I am seeing. When I told it's Perl, encoding correctly detected it was a bit misleading.
Encoding detected as Latin1 (as in your screenshot) and I am unable to save. I switch encoding to Eastern Europe/Cyrilic (CP1251) and I see привет and that is correct text,
but I still unable to save it (and if I switch to UTF-8 I still unable to save it). (note that Komodo always autodected CP1251 texts as Latin1; it's probably impossible to detect better).

@vsespb

This comment has been minimized.

vsespb commented Jan 11, 2016

@mitchell-as, also on your screenshot it's Komodo 9.1, but I have 9.3

@Defman21

This comment has been minimized.

Contributor

Defman21 commented Jan 11, 2016

@mitchell-as mitchell-as added this to the 10.0 milestone Jan 11, 2016

@mitchell-as

This comment has been minimized.

Member

mitchell-as commented Jan 11, 2016

Thanks for the feedback. I still cannot reproduce this, but @Defman21's traceback is something to go on.

@vsespb

This comment has been minimized.

vsespb commented Jan 11, 2016

sad that cant be fixed till next major version

@mitchell-as mitchell-as modified the milestones: 9.3.3, 10.0 Jan 11, 2016

@mitchell-as

This comment has been minimized.

Member

mitchell-as commented Jan 11, 2016

Okay, I'll file it earlier, but there's no guarantee, as I cannot reproduce it right now.

@vsespb

This comment has been minimized.

vsespb commented Jan 11, 2016

thanks!
i've got log from Troubleshooting/log file:

/opt/komodo-edit-9/lib/python/lib/python2.7/difflib.py:433: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  a[besti-1] == b[bestj-1]:
/opt/komodo-edit-9/lib/python/lib/python2.7/difflib.py:437: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  a[besti+bestsize] == b[bestj+bestsize]:
ERROR:xpcom:Unhandled exception calling 'int8 * save(in boolean);'
Traceback (most recent call last):
  File "/opt/komodo-edit-9/lib/mozilla/python/xpcom/server/policy.py", line 300, in _CallMethod_
    return 0, func(*params)
  File "/opt/komodo-edit-9/lib/mozilla/components/koDocument.py", line 1697, in save
    self._clean(ensureFinalEOL, cleanLineEnds)
  File "/opt/komodo-edit-9/lib/mozilla/python/xpcom/components.py", line 268, in wrapperFn
    return fn(*args, **kwargs)
  File "/opt/komodo-edit-9/lib/mozilla/components/koDocument.py", line 1461, in _clean
    wsLinesToStrip = self.getChangedLinesWithTrailingWhitespace()
  File "/opt/komodo-edit-9/lib/mozilla/components/koDocument.py", line 1419, in getChangedLinesWithTrailingWhitespace
    diff_content = self.getUnsavedChanges()
  File "/opt/komodo-edit-9/lib/mozilla/components/koDocument.py", line 2089, in getUnsavedChanges
    return ''.join(difflines)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcc in position 1: invalid continuation byte

(tried to save different file, so codec can't decode byte 0xcc - different bytes)

mitchell-as added a commit that referenced this issue Mar 7, 2016

fix: editing: Handle encoding mismatch when attempting to pre-process…
… saved files - fixes #997

Pre-process as in strip trailing spaces.

rn=

(integrated from master branch change 9.2.1-1013-g773d414 by Mitchell <mitchellb@activestate.com>)
@mitchell-as

This comment has been minimized.

Member

mitchell-as commented Mar 7, 2016

I finally managed to reproduce it. It turns out you need to have the "clean trailing whitespace" file-saving preference turned on.

@vsespb

This comment has been minimized.

vsespb commented Mar 9, 2016

Ok, thanks!
Indeed reproducible only with "clean trailing whitespaces".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment