inbox-auth warn about invalid character strings #207

vitalyster · 2015-10-06T21:21:20Z

Initial run of inbox-auth after google authentication spam tons of warnings like this:

/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:450: Warning: Invalid utf8mb4 character string: '9A570A'

The text was updated successfully, but these errors were encountered:

khamidou · 2015-10-07T05:28:56Z

Hi @vitalyster. Can you give us more details about your gmail account? Which locale are you using? Do you have email in specific languages, etc?

regards,

Karim

vitalyster · 2015-10-07T06:34:26Z

@khamidou most of emails are in Russian

askielboe · 2015-10-07T08:02:02Z

Running inbox-auth on a brand new Gmail account gives me three of these errors:

Enter authorization code:
[INFO] Starting new HTTPS connection (1): accounts.google.com
[DEBUG] "POST /o/oauth2/token HTTP/1.1" 200 None
[INFO] Starting new HTTPS connection (1): www.googleapis.com
[DEBUG] "GET /oauth2/v1/tokeninfo?access_token=  HTTP/1.1"   200 None
[INFO] Starting new HTTPS connection (1): www.googleapis.com
[DEBUG] "GET /oauth2/v1/userinfo?access_token= HTTP/1.1"   200   None
[INFO] Starting new HTTPS connection (1): accounts.google.com
[DEBUG] "POST /o/oauth2/token HTTP/1.1" 200 None
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:450: Warning: Invalid utf8mb4 character string: 'A84900'
  cursor.execute(statement, parameters)
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:450: Warning: Invalid utf8mb4 character string: 'F1A0B6'
  cursor.execute(statement, parameters)
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:450: Warning: Invalid utf8mb4 character string: '9F404B'
  cursor.execute(statement, parameters)
OK. Authenticated account for

I'm in Denmark, but I selected my Gmail locale to be USA and I still get these. On my private Gmail account I also see tons of these messages.

bmathews · 2015-10-07T16:47:24Z

I'm getting a ton of these during the initial sync process. It looks like I get a handful for almost every synced message? I'm in the US.

[INFO] {"state": "initial", "level": "info", "account_id": 1, "provider": "gmail", "timestamp": "2015-10-07T16:44:18.008341Z", "folder": "[Gmail]/All Mail", "greenlet_id": 78165328, "new_committed_message_count": 1, "module": "inbox.mailsync.backends.gmail:332", "event": "Committed new UIDs"}
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:450: Warning: Invalid utf8mb4 character string: 'BA5BEE'
  cursor.execute(statement, parameters)
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:450: Warning: Invalid utf8mb4 character string: 'D51387'
  cursor.execute(statement, parameters)
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:450: Warning: Invalid utf8mb4 character string: 'B45C11'
  cursor.execute(statement, parameters)
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:450: Warning: Invalid utf8mb4 character string: '859981'
  cursor.execute(statement, parameters)
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:450: Warning: Invalid utf8mb4 character string: 'F3934C'
  cursor.execute(statement, parameters)
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:450: Warning: Invalid utf8mb4 character string: 'DD575C'
  cursor.execute(statement, parameters)
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:450: Warning: Invalid utf8mb4 character string: 'EFA8CF'
  cursor.execute(statement, parameters)

jaredly · 2015-10-08T15:58:23Z

I have the same as @bmathews . US, english, gmail

jangondol · 2015-10-08T17:54:20Z

Same here. Gmail. US locale, emails mostly with Central European characters (CP-1250 and UTF-8).

khamidou · 2015-10-08T18:23:38Z

@vitalyster, @jaredly or @bmathews --- could you do two things to help me diagnose the issue:

paste the contents of /etc/mysql/my.cnf
log into your mysql server (mysql -uroot -proot inbox if you're using our vagrant image) and run SHOW CREATE TABLE message;).

Thanks!

bmathews · 2015-10-08T19:04:13Z

vagrant@precise64:/vagrant$ cat /etc/mysql/my.cnf
# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA

#
# The MySQL Community Server configuration file.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

[client]
port        = 3306
socket      = /var/run/mysqld/mysqld.sock

[mysqld_safe]
pid-file    = /var/run/mysqld/mysqld.pid
socket      = /var/run/mysqld/mysqld.sock
nice        = 0

[mysqld]
user        = mysql
pid-file    = /var/run/mysqld/mysqld.pid
socket      = /var/run/mysqld/mysqld.sock
port        = 3306
basedir     = /usr
datadir     = /var/lib/mysql
tmpdir      = /tmp
lc-messages-dir = /usr/share/mysql
explicit_defaults_for_timestamp

# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address    = 127.0.0.1

log-error   = /var/log/mysql/error.log

# Recommended in standard MySQL setup
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

mysql> SHOW CREATE TABLE message;
+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table   | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| message | CREATE TABLE `message` (
  `public_id` binary(16) NOT NULL,
  `created_at` datetime NOT NULL,
  `updated_at` datetime NOT NULL,
  `deleted_at` datetime DEFAULT NULL,
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `namespace_id` int(11) NOT NULL,
  `thread_id` int(11) NOT NULL,
  `from_addr` text NOT NULL,
  `sender_addr` text,
  `reply_to` text,
  `to_addr` text NOT NULL,
  `cc_addr` text NOT NULL,
  `bcc_addr` text NOT NULL,
  `in_reply_to` text,
  `message_id_header` varchar(998) DEFAULT NULL,
  `subject` varchar(255) DEFAULT NULL,
  `received_date` datetime NOT NULL,
  `size` int(11) NOT NULL,
  `data_sha256` varchar(255) DEFAULT NULL,
  `is_read` tinyint(1) NOT NULL DEFAULT '0',
  `is_starred` tinyint(1) NOT NULL DEFAULT '0',
  `is_draft` tinyint(1) NOT NULL DEFAULT '0',
  `is_sent` tinyint(1) NOT NULL DEFAULT '0',
  `state` enum('draft','sending','sending failed','sent','actions_pending','actions_committed') DEFAULT NULL,
  `_compacted_body` longblob,
  `snippet` varchar(191) NOT NULL,
  `full_body_id` int(11) DEFAULT NULL,
  `decode_error` tinyint(1) NOT NULL DEFAULT '0',
  `references` text,
  `version` int(11) NOT NULL DEFAULT '0',
  `g_msgid` bigint(20) DEFAULT NULL,
  `g_thrid` bigint(20) DEFAULT NULL,
  `inbox_uid` varchar(64) DEFAULT NULL,
  `is_created` tinyint(1) NOT NULL DEFAULT '0',
  `is_reply` tinyint(1) DEFAULT NULL,
  `reply_to_message_id` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `thread_id` (`thread_id`),
  KEY `full_body_id_fk` (`full_body_id`),
  KEY `reply_to_message_id` (`reply_to_message_id`),
  KEY `ix_message_inbox_uid` (`inbox_uid`),
  KEY `ix_message_namespace_id_deleted_at` (`namespace_id`,`deleted_at`),
  KEY `ix_message_decode_error` (`decode_error`),
  KEY `ix_message_updated_at` (`updated_at`),
  KEY `ix_message_ns_id_is_draft_received_date` (`namespace_id`,`is_draft`,`received_date`),
  KEY `ix_message_deleted_at` (`deleted_at`),
  KEY `ix_message_namespace_id` (`namespace_id`),
  KEY `ix_message_public_id` (`public_id`),
  KEY `ix_message_g_msgid` (`g_msgid`),
  KEY `ix_message_received_date` (`received_date`),
  KEY `ix_message_namespace_id_is_created` (`namespace_id`,`is_created`),
  KEY `ix_message_g_thrid` (`g_thrid`),
  KEY `ix_message_created_at` (`created_at`),
  KEY `ix_message_data_sha256` (`data_sha256`(191)),
  KEY `ix_message_subject` (`subject`(191)),
  CONSTRAINT `full_body_id_fk` FOREIGN KEY (`full_body_id`) REFERENCES `block` (`id`),
  CONSTRAINT `message_ibfk_1` FOREIGN KEY (`namespace_id`) REFERENCES `namespace` (`id`) ON DELETE CASCADE,
  CONSTRAINT `message_ibfk_2` FOREIGN KEY (`thread_id`) REFERENCES `thread` (`id`) ON DELETE CASCADE,
  CONSTRAINT `message_ibfk_3` FOREIGN KEY (`reply_to_message_id`) REFERENCES `message` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=11 DEFAULT CHARSET=utf8mb4 |
+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

bmathews · 2015-10-08T19:06:48Z

@khamidou Let me know if there's anything else you want to see! Happy to help!

jangondol · 2015-10-08T19:23:38Z

@khamidou FWIW, my two outputs are identical to @bmathews except for: AUTO_INCREMENT=1105 (not 11). Currently running: 4f90903.

michaelrommel · 2015-10-14T19:09:58Z

Same problem here in Germany. my.cnf is identical, the create table is a bit different, sometimes bigint instead of int:

message | CREATE TABLE `message` (
  `public_id` binary(16) NOT NULL,
  `created_at` datetime NOT NULL,
  `updated_at` datetime NOT NULL,
  `deleted_at` datetime DEFAULT NULL,
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `namespace_id` bigint(20) NOT NULL,
  `thread_id` bigint(20) NOT NULL,
  `from_addr` text NOT NULL,
  `sender_addr` text,
  `reply_to` text,
  `to_addr` text NOT NULL,
  `cc_addr` text NOT NULL,
  `bcc_addr` text NOT NULL,
  `in_reply_to` text,
  `message_id_header` varchar(998) DEFAULT NULL,
  `subject` varchar(255) DEFAULT NULL,
  `received_date` datetime NOT NULL,
  `size` int(11) NOT NULL,
  `data_sha256` varchar(255) DEFAULT NULL,
  `is_read` tinyint(1) NOT NULL DEFAULT '0',
  `is_starred` tinyint(1) NOT NULL DEFAULT '0',
  `is_draft` tinyint(1) NOT NULL DEFAULT '0',
  `is_sent` tinyint(1) NOT NULL DEFAULT '0',
  `state` enum('draft','sending','sending failed','sent','actions_pending','actions_committed') DEFAULT NULL,
  `_compacted_body` longblob,
  `snippet` varchar(191) NOT NULL,
  `full_body_id` bigint(20) DEFAULT NULL,
  `decode_error` tinyint(1) NOT NULL DEFAULT '0',
  `references` text,
  `version` int(11) NOT NULL DEFAULT '0',
  `g_msgid` bigint(20) DEFAULT NULL,
  `g_thrid` bigint(20) DEFAULT NULL,
  `inbox_uid` varchar(64) DEFAULT NULL,
  `is_created` tinyint(1) NOT NULL DEFAULT '0',
  `is_reply` tinyint(1) DEFAULT NULL,
  `reply_to_message_id` bigint(20) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `thread_id` (`thread_id`),
  KEY `full_body_id_fk` (`full_body_id`),
  KEY `reply_to_message_id` (`reply_to_message_id`),
  KEY `ix_message_namespace_id_is_created` (`namespace_id`,`is_created`),
  KEY `ix_message_ns_id_is_draft_received_date` (`namespace_id`,`is_draft`,`received_date`),
  KEY `ix_message_decode_error` (`decode_error`),
  KEY `ix_message_received_date` (`received_date`),
  KEY `ix_message_data_sha256` (`data_sha256`(191)),
  KEY `ix_message_g_thrid` (`g_thrid`),
  KEY `ix_message_created_at` (`created_at`),
  KEY `ix_message_g_msgid` (`g_msgid`),
  KEY `ix_message_inbox_uid` (`inbox_uid`),
  KEY `ix_message_updated_at` (`updated_at`),
  KEY `ix_message_deleted_at` (`deleted_at`),
  KEY `ix_message_namespace_id_deleted_at` (`namespace_id`,`deleted_at`),
  KEY `ix_message_public_id` (`public_id`),
  KEY `ix_message_namespace_id` (`namespace_id`),
  KEY `ix_message_subject` (`subject`(191)),
  CONSTRAINT `full_body_id_fk` FOREIGN KEY (`full_body_id`) REFERENCES `block` (`id`),
  CONSTRAINT `message_ibfk_1` FOREIGN KEY (`namespace_id`) REFERENCES `namespace` (`id`) ON DELETE CASCADE,
  CONSTRAINT `message_ibfk_2` FOREIGN KEY (`thread_id`) REFERENCES `thread` (`id`) ON DELETE CASCADE,
  CONSTRAINT `message_ibfk_3` FOREIGN KEY (`reply_to_message_id`) REFERENCES `message` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=724 DEFAULT CHARSET=utf8mb4

spang · 2015-10-27T00:34:57Z

This turned out to be due to MySQL 5.6.27 spuriously emitting warnings for binary data as if it were supposed to be UTF8 character data. We've worked around this issue in ea4a9ec by squelching the warnings.

spang closed this as completed Oct 27, 2015

hozn mentioned this issue Jan 4, 2018

SQLAlchemy/Geoalchemy errors for binary geometry col data freezingsaddles/freezing-web#36

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

inbox-auth warn about invalid character strings #207

inbox-auth warn about invalid character strings #207

vitalyster commented Oct 6, 2015

khamidou commented Oct 7, 2015

vitalyster commented Oct 7, 2015

askielboe commented Oct 7, 2015

bmathews commented Oct 7, 2015

jaredly commented Oct 8, 2015

jangondol commented Oct 8, 2015

khamidou commented Oct 8, 2015

bmathews commented Oct 8, 2015

bmathews commented Oct 8, 2015

jangondol commented Oct 8, 2015

michaelrommel commented Oct 14, 2015

spang commented Oct 27, 2015

inbox-auth warn about invalid character strings #207

inbox-auth warn about invalid character strings #207

Comments

vitalyster commented Oct 6, 2015

khamidou commented Oct 7, 2015

vitalyster commented Oct 7, 2015

askielboe commented Oct 7, 2015

bmathews commented Oct 7, 2015

jaredly commented Oct 8, 2015

jangondol commented Oct 8, 2015

khamidou commented Oct 8, 2015

bmathews commented Oct 8, 2015

bmathews commented Oct 8, 2015

jangondol commented Oct 8, 2015

michaelrommel commented Oct 14, 2015

spang commented Oct 27, 2015