Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
d725c3e
Make project/spider/job keys consistent
vshlapakov Feb 7, 2017
22d0428
collection.set() should return None
vshlapakov Feb 7, 2017
23a7d52
Better support for filter parameter
vshlapakov Feb 7, 2017
459c928
Add base validation for adding activities
vshlapakov Feb 7, 2017
79dd889
Add list & drop iter* for collections
vshlapakov Feb 7, 2017
ed6482e
Add AuthorizationError and handle HTTP 401
vshlapakov Feb 7, 2017
d40a2f8
Unify authentication errors
vshlapakov Feb 7, 2017
b5d4942
Refactor logic for wrapping iter* params
vshlapakov Feb 8, 2017
5c5a9ff
Accept strings in iter filter param
vshlapakov Feb 9, 2017
793a4e1
Some minor fixes after CR
vshlapakov Feb 9, 2017
68f0428
Unify test data (projects,spiders) in docstrings
vshlapakov Feb 9, 2017
3b18654
Add list method for all entities (with a warning)
vshlapakov Feb 9, 2017
1d1d5ee
Use string keys everywhere
vshlapakov Feb 9, 2017
707a6ad
Spider key should also be a string
vshlapakov Feb 9, 2017
1ee768d
Add validation for collection.delete input
vshlapakov Feb 9, 2017
100476c
Check input for update_tags logic
vshlapakov Feb 9, 2017
f8f846d
Add spider.key for consistency, minor fixes
vshlapakov Feb 9, 2017
f9330fb
Fix minor bug when using project.jobs
vshlapakov Feb 9, 2017
c457d40
Move check for update_tags input on server-side
vshlapakov Feb 9, 2017
360a12f
Add spider.update_tags logic
vshlapakov Feb 16, 2017
ffcbcbb
Multiple minor changes after running tests
vshlapakov Feb 17, 2017
fdab51f
Add some tests for iter-filter logic
vshlapakov Feb 17, 2017
5aa7593
Spider id should be a private attribute
vshlapakov Feb 17, 2017
0dad052
Rename col.delete() _keys argument for simplicity
vshlapakov Feb 17, 2017
b48bfc0
Add missing iter methods for more consistent API
vshlapakov Feb 17, 2017
d5ef2c3
Allow authorisation via jwt tokens
vshlapakov Feb 21, 2017
cafc15a
Extend auth logic to cover all possible cases
vshlapakov Feb 22, 2017
ac7272a
Check for bad apikey: should be hex-encoded
vshlapakov Feb 22, 2017
6d05df0
Rework auth logic after code-review
vshlapakov Feb 22, 2017
f611720
Add pytest.ini to enable doctests
vshlapakov Feb 22, 2017
1f98bb9
Provide extended API for Frontier
vshlapakov Feb 22, 2017
57a688e
Drop outdated test for modified auth logic
vshlapakov Feb 22, 2017
50222c4
Add spider_args to schedule method
vshlapakov Feb 22, 2017
e75fd4d
Add spider_args param to the docs
vshlapakov Feb 22, 2017
50a10c0
Use spider_args in tests
vshlapakov Feb 22, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ And select a particular project to work with::
>>> project = client.get_project(123)
>>> project
<scrapinghub.client.Project at 0x106cdd6a0>
>>> project.id
123
>>> project.key
'123'

The above is a shortcut for ``client.projects.get(123)``.

Expand All @@ -88,7 +88,7 @@ Jobs instance is described well in ``Jobs`` section below.

For example, to schedule a spider run (it returns a job object)::

>>> project.jobs.schedule('spider1', arg1='val1')
>>> project.jobs.schedule('spider1', spider_args={'arg1':'val1'})
<scrapinghub.client.Job at 0x106ee12e8>>

Project instance also has the following fields:
Expand Down Expand Up @@ -116,8 +116,8 @@ To select a particular spider to work with::
>>> spider = project.spiders.get('spider2')
>>> spider
<scrapinghub.client.Spider at 0x106ee3748>
>>> spider.id
2
>>> spider.key
'123/2'
>>> spider.name
spider2

Expand All @@ -128,7 +128,7 @@ Like project instance, spider instance has ``jobs`` field to work with the spide

To schedule a spider run::

>>> spider.jobs.schedule(arg1='val1')
>>> spider.jobs.schedule(spider_args={'arg1:'val1'})
<scrapinghub.client.Job at 0x106ee12e8>>

Note that you don't need to specify spider name explicitly.
Expand All @@ -144,7 +144,7 @@ get
To select a specific job for a project::

>>> job = project.jobs.get('123/1/2')
>>> job.id
>>> job.key
'123/1/2'

Also there's a shortcut to get same job with client instance::
Expand All @@ -160,6 +160,7 @@ Use ``schedule`` method to schedule a new job for project/spider::

Scheduling logic supports different options, like

- spider_args to provide spider arguments for the job
- units to specify amount of units to schedule the job
- job_settings to pass additional settings for the job
- priority to set higher/lower priority of the job
Expand Down
2 changes: 2 additions & 0 deletions pytest.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[pytest]
addopts = --doctest-glob='scrapinghub/*.py'
Loading