Testing 1.9-beta #195

stp-ip · 2015-04-16T19:04:40Z

Any specific tags to check out or should I just use the 1.8.2 branch? Feels counterintuative to test 1.9-beta on a 1.8.2 branch.

Give me a hint and I will work on updating the docker image and run my various backups for a test run.

gaubert · 2015-04-16T19:07:44Z

Yep check 1.8.2. I am fixing few small issues right now. But you can pull
it and try it. Let me know if you spot anything.
As for the bad branch name yeah well for the next version everything will
be back in order :-)

On 16 April 2015 at 21:04, Michael Grosser notifications@github.com wrote:

Any specific tags to check out or should I just use the 1.8.2 branch?
Feels counterintuative to test 1.9-beta on a 1.8.2 branch.

Give me a hint and I will work on updating the docker image and run my
various backups for a test run.

—
Reply to this email directly or view it on GitHub
#195.

stp-ip · 2015-04-16T19:09:32Z

OK checking it out now. Will give feedback, when I have some ;)

gaubert · 2015-04-16T19:10:45Z

Just push right now. Please have try and let me know.
Add support for oauth2 and unicode and utf-8 before saving on files.
The rest I have not touched because I want to release quick. I will fix the minor left issues in 1.9.1 very soon

gaubert · 2015-04-16T19:11:24Z

Thanks for your help :-)

On 16 April 2015 at 21:04, Michael Grosser notifications@github.com wrote:

Any specific tags to check out or should I just use the 1.8.2 branch?
Feels counterintuative to test 1.9-beta on a 1.8.2 branch.

Give me a hint and I will work on updating the docker image and run my
various backups for a test run.

—
Reply to this email directly or view it on GitHub
#195.

stp-ip · 2015-04-16T23:06:13Z

Sorry it took a bit longer, but testing on my bigger test account was a bit slow (200k mails)

general:
- oauth2 clean usage !worked!
- oauth2 switch !worked! (old auth was still available, but not used)
sync:
- --resume !worked!
- updating (full run on already synced db) !worked!
- syncing fresh database with 200k mails !worked!
- unicode in mail body !worked!
- unicode in mail subject !worked!
- unicode in mail headers !not tested!
- unicode in label !not working!
  what I am seeing:
  Perform custom synchronisation with gmail request: label:unicde.
  what I should be seeing
  Perform custom synchronisation with gmail request: label:unicöde.

export:

export !worked!
export growing issue export folders growing after each export #146 !still unfixed!
unicode failure (unicode in body, db and export dir, subject)

Start emails export for default labels.
Error: 'ascii' codec can't decode byte 0xc3 in position 21: ordinal not in range(128). 

=== Exception traceback ===
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmv_cmd.py", line 753, in run
  self._export(args)
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmv_cmd.py", line 602, in _export
  exporter.export()
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmvault_export.py", line 69, in export
  default_folder = self.GM_ALL, use_labels = True)
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmvault_export.py", line 104, in export_ids
  self.mailbox.add(msg, folder, meta[gmvault_db.GmailStorer.FLAGS_K])
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmvault_export.py", line 288, in add
  self.subdir(folder).add(mmsg)
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmvault_export.py", line 275, in subdir
  mbox_path = os.path.join(cur_path, s)
File "/usr/lib/python2.7/posixpath.py", line 78, in join
  path +=  b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 21: ordinal not in range(128)

=== End of Exception traceback ===

gaubert · 2015-04-17T07:17:19Z

A million time thank you for your testings. Extremely appreciated !!. See below my comments in bold. I will see if I can fix the custom request unicode issue

Sorry it took a bit longer, but testing on my bigger test account was a bit slow (200k mails)

general:
oauth2 clean usage !worked!
oauth2 switch !worked! (old auth was still available, but not used)

OK good

sync:
--resume !worked!
updating (full run on already synced db) !worked!
syncing fresh database with 200k mails !worked!
unicode in mail body !worked!
unicode in mail subject !worked!
unicode in mail headers !not tested! =>This will also work as I have tested it
unicode in label !not working!
what I am seeing:
Perform custom synchronisation with gmail request: label:unicde.
what I should be seeing
Perform custom synchronisation with gmail request: label:unicöde.

It is working when emails are synced and stored on disk. What is not working for you is when you want pass some label with unicode character through the command line. I will check that one but it might come from the shell settings. Let's see

Export is a wrecked ship. It wasn't developed by me but by a contributor and was full of bugs.
I will need to have a look at it but not in that version as I want to quickly release and update (next week) because of the oauth2 issue. Hope you understand
export:

export !worked!
export growing issue #146 !still unfixed!
unicode failure (unicode in body, db and export dir, subject)
Start emails export for default labels.
Error: 'ascii' codec can't decode byte 0xc3 in position 21: ordinal not in range(128).

=== Exception traceback ===
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmv_cmd.py", line 753, in run
self._export(args)
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmv_cmd.py", line 602, in _export
exporter.export()
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmvault_export.py", line 69, in export
default_folder = self.GM_ALL, use_labels = True)
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmvault_export.py", line 104, in export_ids
self.mailbox.add(msg, folder, meta[gmvault_db.GmailStorer.FLAGS_K])
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmvault_export.py", line 288, in add
self.subdir(folder).add(mmsg)
File "/usr/local/lib/python2.7/dist-packages/gmvault-1.9_beta-py2.7.egg/gmv/gmvault_export.py", line 275, in subdir
mbox_path = os.path.join(cur_path, s)
File "/usr/lib/python2.7/posixpath.py", line 78, in join
path += b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 21: ordinal not in range(128)

=== End of Exception traceback ===

gaubert · 2015-04-21T19:36:06Z

@stp-ip Please check new release v1.9.
Left open is unicode in label and export.
I will work on it for v1.9.1 but I wanted to release quickly 1.9

gaubert · 2015-05-05T08:32:17Z

@stp-ip I worked on the support for unicode in labels and I think I understand why it doesn't work in your case but would need more info from your side. Could you please run the sync --type custom -g "label:unicöde" with --debug and send me the console outputs up to the start of the synchronisation.
I should then see what is the language setting from your terminal. Thanks in advance

stp-ip · 2015-05-05T09:33:49Z

Odd. Use default python2.7.x or 2.6.x distribution.

Use gmvault-db located in /con/data/unicode.

Disable deletion of emails that are in Gmvault db and not anymore in Gmail.
Activate debugging information.
Debugging logs are going to be saved in file /home/gmvault/gmvault.log.

no ENV variable $GMVAULT_DIR defined. Set by default $GMVAULT_DIR to $HOME/.gmvault (/home/gmvault/.gmvault)
Load defaults from /home/gmvault/.gmvault/gmvault_defaults.conf
Authentication performed with Gmail OAuth2 access token.

Get OAuth2 credential from /home/gmvault/.gmvault/michael@stp-ip.net.oauth2.

oauth2 creds = -
Access Token is expired. Renew it
json_resp = {u'access_token': u'-', u'token_type': u'Bearer', u'expires_in': 3600}
auth_str generated: user=user@example.net auth=Bearer -
Successfully read oauth2 credentials with get_oauth2_credential_from_refresh_token

Connect to Gmail server.

credential['value'] = user=user@example.net auth=Bearer -
Do not enable imap compression.
Connected
No Storage DB in /con/data/unicode. Create it.

raw unicode     = label:unicde.
chosen encoding = ANSI_X3.4-1968.
unicode_escape val = label:unicde.
hexval l=hex[0x6c],a=hex[0x61],b=hex[0x62],e=hex[0x65],l=hex[0x6c],:=hex[0x3a],u=hex[0x75],n=hex[0x6e],i=hex[0x69],c=hex[0x63],d=hex[0x64],e=hex[0x65],
Perform custom synchronisation with gmail request: label:unicde.

stp-ip · 2015-05-05T09:35:02Z

Furthermore I'm testing the 1.9 release with my docker image and will release it once testing is successful at https://registry.hub.docker.com/u/seetheprogress/gmvault/

stp-ip · 2015-05-05T09:52:55Z

Will try the unicode from cli again with the latest changes from the 1.9.1 branch, after the container release.

stp-ip · 2015-05-05T10:39:07Z

So far the container seems fine, except the gm-label issue: #198 and other known issues, such as #157, unicode via cli etc. WIll upload and move onto testing the 1.9.1 branch.

gaubert · 2015-05-05T12:26:04Z

@stp-ip Ok I checked and your terminal encoding is not properly set because you are maybe login via a Windows (or another) Terminal to an remote machine via ssh.
Via the system variables the Term encoding provided is ANSI_X3.4-1968 but it seems to be something else as the decoding fails.
So here is what I will do in my pseudo code:

if not preset in configuration then
getTerminalEncoding and try to decode with error
if onError then
try to use chardet to detect it and try to decode with errors
if onError then
print Message indicating that the Terminal encoding is not correctly set
then decode with error=replace

You are always able to set the Terminal encoding via the conf file in order to force it.
If nor the term info nor chardet gives you are right answer then the characters will be eaten.

Note also that chardet could give you a wrong encoding which will mess up the characters so this is why I prefer to have a way to force the encoding via the configuration file.

Could you find the right encoding for your terminal just to be sure.

Thanks

gaubert · 2015-05-05T12:27:36Z

I will try to prepare a new version for testing purposes tonight

gaubert · 2015-05-05T20:05:13Z

@stp-ip Ok you can pull branch 1.9.1 to check the command line encoding.
https://github.com/gaubert/gmvault/tree/gmv-1.9.1
Try then to perform a custom synchronisation with gmail request: label:unicöde.
I am not sure Gmvault will be able to detect your encoding.
To check your local encoding do:
$>locale
You should see something like below but not UTF-8

LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

Please try the request without changing first the Terminal encoding, set the --debug option and send me the console output.
If Gmvault doesn't detect the right encoding. You can also set it using the [Localisation]term_encoding property. Set it to the Term encoding which is set for your env and re-run the sync command with --debug and send me the console output.

Then you should set your Terminal encoding to UTF-8 and re-run the command and then it is going to work.
just type:

export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8 export LANGUAGE=en_US.UTF-8

Let me know how it goes. Many thanks

gaubert · 2015-05-07T07:57:13Z

@stp-ip any news regarding the testing ? Many thanks.

stp-ip · 2015-05-10T11:53:33Z

@stp-ip will do another test run on monday. Will get back to you then.

stp-ip · 2015-05-11T20:04:03Z

Odd. Use default python2.7.x or 2.6.x distribution.   
sys.argv=[['/tmp/gmvault.bootstrap.1', 'sync', '-c', 'no', '--type', 'custom', '-d', '/con/data/unicode', '--gmail-req', 'label:unic\xc3\xb6de', '--debug', 'user@example.com']]

in get unicode ANSI_X3.4-1968
Detection = {'confidence': 0.99, 'encoding': 'TIS-620'}
Use gmvault-db located in /con/data/unicode.

Disable deletion of emails that are in Gmvault db and not anymore in Gmail.
parsed args = [{'oauth2': 'empty', 'encrypt': False, 'compression': True, 'passwd': 'not_seen', 'chats_only': False, 'parser': CmdLineParser(prog='gmvault', usage=None, description=None, version=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True), 'request': {'req': u'label:unic\xf6de', 'type': 'gmail'}, 'db-cleaning': False, 'port': 993, 'ownership_control': True, 'host': 'imap.gmail.com', 'command': 'sync', 'debug': True, 'db-dir': '/con/data/unicode', 'type': 'custom', 'email': 'user@example.com', 'restart': False, 'emails_only': False}]
Activate debugging information.
Debugging logs are going to be saved in file /home/gmvault/gmvault.log.

no ENV variable $GMVAULT_DIR defined. Set by default $GMVAULT_DIR to $HOME/.gmvault (/home/gmvault/.gmvault)
Load defaults from /home/gmvault/.gmvault/gmvault_defaults.conf
Authentication performed with Gmail OAuth2 access token.

Get OAuth2 credential from /home/gmvault/.gmvault/user@example.com.oauth2.

oauth2 creds = x
Access Token is still valid
auth_str generated: user=user@example.comauth=Bearer x
Successfully read oauth2 credentials with get_oauth2_credential_from_refresh_token

Connect to Gmail server.

credential['value'] = user=user@example.comauth=Bearer x
Do not enable imap compression.
Connected
Perform custom synchronisation with gmail request: label:unic?de.

Start synchronization (2015-05-11T19h59m27s).

Start emails synchronization.
cat0 emails to be fetched.

Emails synchronisation operation performed in 3s.

Start chats synchronization.
Before selection
Selection is finished
0 chats to be fetched.

chats synchronisation operation performed in 1s.

db_cleaning is off so ignore removing deleted emails from disk.
Sync operation performed in 3s.

error_report complete structure = {'emails_in_quarantine': [], 'operation': 'Sync', 'reconnections': 0, 'cannot_be_fetched': [], 'empty': [], 'operation_time': '3s'}

================================================================
Sync operation performed in 3s.
Number of reconnections: 0.
Number of emails quarantined: 0.
Number of emails that could not be fetched: 0.
Number of emails that were returned empty by gmail: 0
================================================================

locale (from docker image):

LANG=                                                                                                                                                              
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

stp-ip · 2015-05-11T20:07:15Z

Added setting locale to the gmvault docker image in the next revision:

# Set the locale
RUN locale-gen en_US.UTF-8  
ENV LANG en_US.UTF-8  
ENV LANGUAGE en_US:en  
ENV LC_ALL en_US.UTF-8

stp-ip · 2015-05-11T20:27:07Z

Even with locale:

LANG=en_US.UTF-8                                                                                                                                          
LANGUAGE=en_US:en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

I get a wrong encoding choice:

Detection = {'confidence': 0.99, 'encoding': 'TIS-620'}

gaubert · 2015-05-12T11:25:22Z

@stp-ip ok so you have to force it with the option [Localisation]terminal_encoding='utf-8' in ~/gmvault_defaults.conf
Could you please have a try and let me know. You of course used the branch 1.9.1 ?
Many thanks

stp-ip · 2015-05-12T17:44:07Z

Small update. I used the right branch, but not all your encoding commits it seems.
When utf-8 is set on the terminal it works as expected.
When nothing is set I get:

encoding found with locale.getpreferredencoding()
terminal encoding = ANSI_X3.4-1968.
'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128)
Convertion of label:unicöde from ANSI_X3.4-1968 to a unicode failed. Will now convert to unicode using utf-8 encoding and ignoring errors (non utf-8 characters will be eaten).
Please set properly the Terminal encoding or use the [Localisation]:terminal_encoding property to set it.
Perform custom synchronisation with gmail request: label:unic?de.

Which sounds right, but seems to fail and use the label "unic?de" even so it should use utf-8.

When using the forced encoding, it fails the same way:

Use terminal encoding forced from the configuration file.
terminal encoding = 'utf-8'.
unicode_escape val = label:unic\xf6de.
raw unicode     = label:unic?de.
Perform custom synchronisation with gmail request: label:unic?de.

Furthermore the config file had the option term_encoding='utf-8' set uncommented. Using terminal_encoding='utf-8' seems to be the right choice as you suggested.

gaubert · 2015-05-13T16:13:24Z

If your terminal is not properly configured, forcing the utf8 encoding might not work because the characters need first to be decoded with the right encoding and then encoded in UTF8. If the decoding doesn't work then the character is eaten. So the solution is to properly set your terminal encoding. So I would say it is working

gaubert · 2015-05-13T19:54:58Z

@stp-ip there is a problem with the configuration file as it should be been overwritten. I will check it but not this week-end as I am not at home.

gaubert · 2015-08-09T12:48:52Z

@stp-ip I have fixed that problem on 1.9.1. If I didn't here from you in the following days. I will close that issue. Million times thank you for your help.

gaubert added bugs v1.8.2 labels Apr 17, 2015

gaubert added v1.9.1 and removed v1.8.2 labels May 4, 2015

gaubert self-assigned this May 7, 2015

gaubert added the fixed label Aug 26, 2015

gaubert closed this as completed Aug 26, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Testing 1.9-beta #195

Testing 1.9-beta #195

stp-ip commented Apr 16, 2015

gaubert commented Apr 16, 2015

stp-ip commented Apr 16, 2015

gaubert commented Apr 16, 2015

gaubert commented Apr 16, 2015

stp-ip commented Apr 16, 2015

gaubert commented Apr 17, 2015

gaubert commented Apr 21, 2015

gaubert commented May 5, 2015

stp-ip commented May 5, 2015

stp-ip commented May 5, 2015

stp-ip commented May 5, 2015

stp-ip commented May 5, 2015

gaubert commented May 5, 2015

gaubert commented May 5, 2015

gaubert commented May 5, 2015

gaubert commented May 7, 2015

stp-ip commented May 10, 2015

stp-ip commented May 11, 2015

stp-ip commented May 11, 2015

stp-ip commented May 11, 2015

gaubert commented May 12, 2015

stp-ip commented May 12, 2015

gaubert commented May 13, 2015

gaubert commented May 13, 2015

gaubert commented Aug 9, 2015

Testing 1.9-beta #195

Testing 1.9-beta #195

Comments

stp-ip commented Apr 16, 2015

gaubert commented Apr 16, 2015

stp-ip commented Apr 16, 2015

gaubert commented Apr 16, 2015

gaubert commented Apr 16, 2015

stp-ip commented Apr 16, 2015

gaubert commented Apr 17, 2015

gaubert commented Apr 21, 2015

gaubert commented May 5, 2015

stp-ip commented May 5, 2015

stp-ip commented May 5, 2015

stp-ip commented May 5, 2015

stp-ip commented May 5, 2015

gaubert commented May 5, 2015

gaubert commented May 5, 2015

gaubert commented May 5, 2015

gaubert commented May 7, 2015

stp-ip commented May 10, 2015

stp-ip commented May 11, 2015

stp-ip commented May 11, 2015

stp-ip commented May 11, 2015

gaubert commented May 12, 2015

stp-ip commented May 12, 2015

gaubert commented May 13, 2015

gaubert commented May 13, 2015

gaubert commented Aug 9, 2015