qubes-manager: UnicodeEncodeError when restoring from backup

[pdinoto]

Restoring from a path with accented characters results in

UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 31: ordinal not in range(128)
This is most likely a bug in the Qubes Manager
----
line: st = os.stat(path)
func: isfile
line no.: 29
file: /usr/lib64/python2.7/genericpath.py
----
line: if os.path.isfile(backup_location) or \
func: has_selected_dir
line no.: 321
file: /usr/lib64/python2.7/site-packages/qubesmanager/restore.py

and variations of the same error. This one is triggered by pasting a path with unicode chars into Backup File: text field.

Selecting a valid restore file inside a disk mounted on a path with unicode characters triggers same error later in the process, after selecting which VMs to restore and right after showing the restore progress dialog. In that case, error happens at different code:

----
line no.: 246
file: /usr/lib64/python2.7/site-packages/qubesmanager/restore.py

More info can be found on the original mailing list post

[donob4n]

Maybe it's different issue but is likely related. After doing a properly recover I got an UnicodeEncodeError and after click ok I entered in an infinite loop trying to close the restore window.

The relevant code is:

def reject(self):
    if self.currentPage() is self.commit_page:
        if backup.backup_cancel():
            self.emit(SIGNAL("restore_progress(QString)"),
                      '<font color="red">{0}</font>'
                      .format(self.tr("Aborting the operation...")))
            self.button(self.CancelButton).setDisabled(True)
    else:
        self.done(0)

And the line raising UnicodeEncodeError:
.format(self.tr("Aborting the operation...")))

I guess it's problem with the Spanish translation and accents.

[donob4n]

Maybe it's different issue but is likely related. After doing a properly recover I got an UnicodeEncodeError and after click ok I entered in an infinite loop trying to close the restore window.

The relevant code is:

def reject(self):
    if self.currentPage() is self.commit_page:
        if backup.backup_cancel():
            self.emit(SIGNAL("restore_progress(QString)"),
                      '<font color="red">{0}</font>'
                      .format(self.tr("Aborting the operation...")))
            self.button(self.CancelButton).setDisabled(True)
    else:
        self.done(0)

And the line raising UnicodeEncodeError:
.format(self.tr("Aborting the operation...")))

I guess it's problem with the Spanish translation and accents.

[donob4n]

@marmarek

I fixed this adding 'u' for forcing unicode:

            self.emit(SIGNAL("restore_progress(QString)"),
                      u'<font color="red">{0}</font>'
                      .format(self.tr("Aborting the operation...")))

I was going to fix similar bugs but I see there are already some unicode() calls which do same effect. Do you see fine if I change all of them to this way or prefer unicode() method?

I also see some similar problems on Qubes 4 version.

[donob4n]

@marmarek

I fixed this adding 'u' for forcing unicode:

            self.emit(SIGNAL("restore_progress(QString)"),
                      u'<font color="red">{0}</font>'
                      .format(self.tr("Aborting the operation...")))

I was going to fix similar bugs but I see there are already some unicode() calls which do same effect. Do you see fine if I change all of them to this way or prefer unicode() method?

I also see some similar problems on Qubes 4 version.

[marmarek]

Do you see fine if I change all of them to this way or prefer unicode() method?

u'' is fine on string constants

I also see some similar problems on Qubes 4 version.

It shouldn't be an issue there, because qubes-manager was ported to python 3 there, which use unicode by default.

[marmarek]

Do you see fine if I change all of them to this way or prefer unicode() method?

u'' is fine on string constants

I also see some similar problems on Qubes 4 version.

It shouldn't be an issue there, because qubes-manager was ported to python 3 there, which use unicode by default.

[donob4n]

Ok @marmarek

I've fixed the translations problems with donob4n/qubes-manager@f3a16c4

I will look for more potentially similar problems on qubes-manager.

For this current issue I did some steps forward, but I found a problem in https://github.com/QubesOS/qubes-core-admin/blob/19d55b546352c61b88990a8fc6b38f568955a81d/core/backup.py#L491

It seems that I tries to convert the utf8 string into asci for writing into the pipeline. I was trying to enable utf8 encoding but I'm pretty worried if this could break other things.

UPDATE:

Well it seems working fine with:

    vmproc.stdin.write((base_backup_dir.replace("\r", "").replace("\n", "") + "\n").enconde('utf-8'))

Except I don't know why the "select dir" window doesn't appear after finishing when using a utf8 path.

[donob4n]

Ok @marmarek

I've fixed the translations problems with donob4n/qubes-manager@f3a16c4

I will look for more potentially similar problems on qubes-manager.

For this current issue I did some steps forward, but I found a problem in https://github.com/QubesOS/qubes-core-admin/blob/19d55b546352c61b88990a8fc6b38f568955a81d/core/backup.py#L491

It seems that I tries to convert the utf8 string into asci for writing into the pipeline. I was trying to enable utf8 encoding but I'm pretty worried if this could break other things.

UPDATE:

Well it seems working fine with:

    vmproc.stdin.write((base_backup_dir.replace("\r", "").replace("\n", "") + "\n").enconde('utf-8'))

Except I don't know why the "select dir" window doesn't appear after finishing when using a utf8 path.

[marmarek]

I've fixed the translations problems with donob4n/qubes-manager@f3a16c4

Well it seems working fine with:

Care to create pull request with those?

Except I don't know why the "select dir" window doesn't appear after finishing when using a utf8 path.

Probably similar thing - the window is opened by a call to qubes.SelectDirectory service and the path is piped on its stdin. See here: https://github.com/QubesOS/qubes-manager/blob/release3.2/qubesmanager/backup_utils.py#L59

[marmarek]

I've fixed the translations problems with donob4n/qubes-manager@f3a16c4

Well it seems working fine with:

Care to create pull request with those?

Except I don't know why the "select dir" window doesn't appear after finishing when using a utf8 path.

Probably similar thing - the window is opened by a call to qubes.SelectDirectory service and the path is piped on its stdin. See here: https://github.com/QubesOS/qubes-manager/blob/release3.2/qubesmanager/backup_utils.py#L59

[donob4n]

Care to create pull request with those?

Yes, let me check if there is another translation problem and try to fix the unicode path and I will request a pull with all together.

[donob4n]

Care to create pull request with those?

Yes, let me check if there is another translation problem and try to fix the unicode path and I will request a pull with all together.

[donob4n]

It works fine, just failed because I binded the device to a unicode dir without "media" on the path.

[donob4n]

It works fine, just failed because I binded the device to a unicode dir without "media" on the path.

[donob4n]

@andrewdavidwong I think this is already fixed.

[donob4n]

@andrewdavidwong I think this is already fixed.