Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DB fixes and Scheduler HWM #389

Merged
merged 3 commits into from Apr 21, 2011
Merged

DB fixes and Scheduler HWM #389

merged 3 commits into from Apr 21, 2011

Conversation

minrk
Copy link
Member

@minrk minrk commented Apr 17, 2011

This branch is actually based on my 'windows' branch, so the real diff is here:
minrk/ipython@windows...db

until PR #374 is merged

Basic features:

add HWM to Python scheduler, allowing users to tune level of job queuing. (HWM=1 behaves like old TaskScheduler, HWM=0 is default, and behaves like pure 0MQ).

DB backend changes:

  • add db.get_history() method (like client/view.history, but returns all known msg_ids)
  • db.find_records returns list, allows filtering of keys
  • add db_backend tests

Two new Client methods exposing db backends:

  • rc.hub_history() # proxy for db.get_history
  • rc.db_query() # proxy for db.find_records

Up for discussion: These are Client methods, not View methods. To me, they seem like low-level methods, and thus appropriately bound to the low-level object, but They can be easily propagated up to the Views.

Extra fixes along the way:

  • Harder for db issues to crash the Hub
  • Better JSON handling of date objects
  • Don't raise NoEnginesRegistered when load-balancing on all engines
  • rc.get_result works with whole history if database is persistent

@minrk
Copy link
Member Author

minrk commented Apr 17, 2011

Among the DB bugs fixed by this is a failure in MongoDB.add_record, if buffers were unspecified (and a method name typo), as reported in #391.

@kaazoo
Copy link

kaazoo commented Apr 18, 2011

Requesting all known tasks and creating tasks in advance (when no engines are running) works now. Thanks.

@ellisonbg
Copy link
Member

I agree that these should be Client level methods, not View level.

@ellisonbg
Copy link
Member

As long as this is tested, I think it can be merged. I gave it quick look through and things look fine.

@minrk
Copy link
Member Author

minrk commented Apr 20, 2011

I've got db_backend tests, but I'll add them to the client methods before merging.

@ellisonbg
Copy link
Member

OK great.

DB backend changes:
* add db.get_history method
* db.find_records returns list, allows filtering of keys
* add db_backend tests

New Client methods exposing db backends:
* rc.hub_history() # proxy for db.get_history
* rc.db_query() # proxy for db.find_records

Extra fixes along the way:
* Harder for db issues to crash the Hub
* Better JSON handling of date objects
* Don't raise NoEnginesRegistered when load-balancing on all engines
* rc.get_result works with whole history if database is persistent

closes ipythongh-391
This option allows users to limit the level of greedy task assignment.
@minrk minrk merged commit deccce4 into ipython:master Apr 21, 2011
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants