Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix mysql return bytes as field name type #787

Merged
merged 4 commits into from May 12, 2018
Merged

Conversation

sdvcrx
Copy link
Contributor

@sdvcrx sdvcrx commented May 9, 2018

This fix different behavior of mysql-connector-python C extension and pure python version.

Fix #758 #766

How to repeat:

Install mysql, pyspider, then install mysql-connector-python via pip :

pip install mysql-connector-python  # current version 8.0.11

Run python script:

from pyspider.database.mysql.taskdb import TaskDB

db = TaskDB(user='root', database='db', passwd='password')
print(db.get_task('projectname', 'taskid'))

and output:

{'taskid': 'taskid',
 'project': 'project',
 'url': 'url',
 'status': 1,
 b'schedule': '{"itag": "v2", "age": 86400}',    // <- b'schedule' but expect 'schedule'
 b'fetch': None,
 b'process': None,
 b'track': None,
 'lastcrawltime': None,
 'updatetime': None}

Related: mysql/mysql-connector-python#37

sdvcrx added 4 commits May 9, 2018 11:30
This error raise on mysql-connector-python with C extension
Pure version raise InterfaceError,
  but C extension version raise DatabaseError
@binux
Copy link
Owner

binux commented May 12, 2018

Thank you very much!

@binux binux merged commit 124ffef into binux:master May 12, 2018
@sdvcrx sdvcrx deleted the fix-mysql branch May 15, 2018 13:00
brian6peng pushed a commit to brian6peng/pyspider that referenced this pull request Jun 20, 2018
* use pip version of mysql-connector-python for testing

* fix mysql return bytes as field names type

* fix raise Unread result found error

This error raise on mysql-connector-python with C extension

* fix test

Pure version raise InterfaceError,
  but C extension version raise DatabaseError
StarUI added a commit to StarUI/pyspider that referenced this pull request Mar 16, 2021
* fix test

1.1.1.1 took by cloudflare

* Fixed db inconsistency (binux#779)

* Fixed db creation inconsistency in taskdb, projectdb and resultdb

* Fixed typo

* using reserved ip address for testing

rolling out version 0.3.10

* Fix mysql return bytes as field name type (binux#787)

* use pip version of mysql-connector-python for testing

* fix mysql return bytes as field names type

* fix raise Unread result found error

This error raise on mysql-connector-python with C extension

* fix test

Pure version raise InterfaceError,
  but C extension version raise DatabaseError

* fix binux#799

* optimise scheluler dynamic select limit and improve task queue (binux#796)

* optimise scheduler select-limit and task queue

* fix test case in python2.6

* fix: time priority queue only compare exetime

* update:add test case for time priority queue

* optimise: add globally auto increasing value for task to keep priority queue in order

* change async to async_ (binux#803)

* change async to async_

* change async to async_ in tests

* change async_ to async_mode

* modify async to async_mode to support python3.7

* add python3.7 CI test

* add python3.7 CI test

* add python3.7 CI test

* add python3.7 CI test

* remove python3.7 CI test

* add py3.7-dev CI test

* add support py3.7-dev CI test

* removed 2.6 due to lack of support, changed pip install for 3.5 due to pip versioning

* feature puppeteer js engine

* features: add opened pages maximum limit, default 5

* fix: python3.5 install lxml error

* add puppeteer fetcher

* update

* fix bugs
1. some args "async" haven't been replaced completely yet
2. delete Python 3.3 in .travis.yml because the current version of lxml is not supported by Python3.3

* use suggested python3.7 build

* fix build for 3.3

* 1. python2.7 image is different when using metrix
2. pip install just works now days

* sudo not required any more?

* try not to specify a version for apt-get

* fix setup.py test for py3.3

* try manually install

* try again

* fix for 3.7

* try install librt

* try again

* allow fail

* updated requirements.txt to fixed package versions

* port to python 3.6

* upgrade python-six

* updated travis.yml

* fixed "connect to scheduler rpc error: error(111, Connection refused)" error

* fixed phantomjs libssl_conf.so error

* travis test

* another Travis test

* trying to trace "cannot find module express" error in Travis

* using NODE_PATH env var

* moved NODE_PATH assignment after install

* making symlink to node_modules

* travis test

* node modules are currently missing from travis

* added npm install to travis.yml

* fixed travis node dependancy issues

* using run_in_thread for scheduler and fetcher dispatch again

* accommodate changes made in run.py to tests

* changed test_90_docker_scheduler

* added extra asserts to tests

* test

* upgraded sqlAlchemy

* sqlalchemy upgrade

* sqlalchemy upgrade

* sqlalchemy upgrade

* sqlalchemy upgrade

* sqlalchemy upgrade

* sqlalchemy upgrade

* sqlalchemy upgrade fix

* sqlalchemy upgrade

* added extra assertions

* sqlalchemy upgrade

* sqlalchemy upgrade

* sqlalchemy upgrade

* undo previous

* tracing errors

* fix sqlalchemy data encoding

* sqlalchemy changed dict encoding to pure json string

* test_10_save mongodb fix

* undo previous

* tracing test_10_save mongodb bug

* tracing test_10_save mongodb bug

* upgraded pymongo

* mongo tests now passing

* fixed test_a110_one failing by "fetcher() got an unexpected keyword argument xmlrpc"

* upgraded pika

* tracing RabbitMQ ConnectionRefusedError: [Errno 111] Connection refused

* fixed typo

* tracing RabbitMQ ConnectionRefusedError: [Errno 111] Connection refused

* tracing RabbitMQ ConnectionRefusedError: [Errno 111] Connection refused

* tracing RabbitMQ ConnectionRefusedError: [Errno 111] Connection refused

* switching to Pika for Rabbitmq

* skip TestAmqpRabbitMQ

* travis test

* travis build failing with 0 errors and 0 failures, 40 "unexpected successes"

* added updated docker-compose.yaml

* cleanup

* initial couchdb projectdb implementation

* test url parser

* fix couchdb connect url

* fix couchdb connect url

* fix couchdb json encoding

* fix couchdb json encoding

* fix couchdb url encoding

* fix couchdb urls

* fixed couchdb request headers

* travis upgrade couchdb

* travis upgrade couchdb

* travis upgrade couchdb

* travis upgrade couchdb

* travis upgrade couchdb

* fixed "Fields must be an array of strings, not: null" eroor

* fixed responses

* fixed drop database

* tracing insertion issue

* fixed default values

* tracing update bug

* fixed update bug

* fixed drop bug

* changed default fields

* fixed drop bug

* fixed _default_fields usage

* fixed update bug

* fixed update bug

* fixed drop bug

* tracing update bug

* fixed drop bug

* tracing drop bug

* fixed drop bug

* fixed db naming issue

* fixed drop bug

* initial resultdb implementation

* added resultdb tests

* fix resultdb tests

* fix resultdb init

* fix resultdb init

* fix missing class var

* fixed get_docs

* fixed db naming

* fixed db naming

* fixed db naming

* fixed get_docs

* minor fixes

* fixed update_doc

* fixed update_doc

* fixed get_doc

* fixed get_docs

* fixed get_docs

* fixed parse

* fixed get_all_docs

* fixed get_doc

* fixed update_doc

* minor fixes

* fixed select

* initial taskdb implementation

* added debug prints

* added collection_prefix

* minor fixes

* minor fixes

* fixed update

* fixed test_25_get_task

* fixed status_count selector

* fixed update

* tracing test_create_project bug

* fixed collection naming

* Revert "fixed collection naming"

This reverts commit 0d89a0d.

* fixed collection naming

* minor fixes

* minor fixes

* fixed test_z10_drop

* fixed test_50_load_tasks

* fixed get_docs

* fixed get methods

* cleanup

* removed python 3.3 and added 3.7 and 3.8

* added index

* tracing index create bug

* fixed index create bug

* fixed index create bug

* fixed index create bug

* minor test fixes

* added couchdb test run

* added couchdb test run

* full working example

* fixed test setup

* fixed test setup

* updated travis file for couchdb auth

* updated travis file for couchdb auth

* added credentials exception

* fixed credentials

* fixed test auth

* fixed test auth

* tracing auth issue

* tracing auth issue

* fixed test auth issue

* fixed test test_60a_docker_couchdb

* fixed test test_60a_docker_couchdb

* cleanup

* attempting to remove "unexpected successes"

* tracing "unexpected successes"

* tracing "unexpected successes"

* tracing "unexpected successes"

* tracing "unexpected successes"

* tracing "unexpected successes"

* tracing "unexpected successes"

* Revert "tracing "unexpected successes""

This reverts commit 829da8c.

* tracing "unexpected successes"

* tracing "unexpected successes" in crawl

* tracing "unexpected successes" in crawl

* tracing "unexpected successes"

* tracing "unexpected successes"

* tracing "unexpected successes"

* tracing "unexpected successes"

* tracing "unexpected successes"

* tracing "unexpected successes"

* tracing "unexpected successes"

* fixed "unexpected successes"

* fixed TestFetcherProcessor

* fixed TestFetcherProcessor

* fixed TestFetcherProcessor

* fix BaseHandler

* fix BaseHandler

* fix BaseHandler

* fix BaseHandler

* fix BaseHandler

* fix BaseHandler

* fix BaseHandler

* removed beanstalkc

* cleanup

* removed 3.8 from travis

* removed python 3.8 from setup.py

* fixed test_60_relist_projects change

* fixed .travis

* added https to couchdb + cleanup + added couchdb to docs

* added extra comment on top of docker-compose example

* fixed docker-compose issue

* improve docker-compose sample

* remove demo link

* fix test break because couchdb failing to start

* try to use non-auth for CouchDB test

* more couchdb_password

* improve couchdb allow empty username password

* drop support for couchdb

Co-authored-by: Roy Binux <root@binux.me>
Co-authored-by: jxltom <jxltom@users.noreply.github.com>
Co-authored-by: binux <roy@binux.me>
Co-authored-by: sdvcrx <memory.silentvoyage@gmail.com>
Co-authored-by: Lucas <comeson@126.com>
Co-authored-by: vibiu <540650312@qq.com>
Co-authored-by: farmercode <wangchangchun120@gmail.com>
Co-authored-by: Phillip <phillip1.peterson@umontana.edu>
Co-authored-by: feiyang <feiyang@ibantang.com>
Co-authored-by: clchen <ccl0326@163.com>
Co-authored-by: v1nc3nt <vinsechsz@gmail.com>
Co-authored-by: Keith Tunstead <tunstek@tcd.ie>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug TypeError: startswith first arg must be bytes or a tuple of bytes, not str
2 participants