Provide support for background tasks #2977
Comments
|
@rossjones , how can I work on it, should I just start and then push a PR? What if some else will start working on it at the same time, will only one of us get the bounty? |
|
I think you need to go to BountySource ( https://www.bountysource.com/issues/33227756-provide-support-for-background-tasks ) and mark yourself as working on it ... will update the original message to warn people. |
|
@GildedHonour Are you still working on this? BountySource says so but it also says you expected to be done by end of May. |
|
Sorry, no. Sent from my ASUS -------- Mensaje original --------
|
|
@GildedHonour I tried to remove you from the bounty, but it wouldn't let me, and it'd be cool if it didn't put anyone off attempting the issue. If you get 5 minutes, would you consider popping along to bountysource and removing your name? |
Adds a basic support of background jobs using python-rq. Jobs can be managed via the `ckan.lib.jobs` module, the `job_*` API functions and the `jobs` paster command.
Adds a basic support of background jobs using python-rq. Jobs can be managed via the `ckan.lib.jobs` module, the `job_*` API functions and the `jobs` paster command.
|
I've started working on this, see #3165. Note that I'm not interested in the bounty. |
Check Redis availability on server start but initialize queue only once it is actually required.
Check Redis availability on server start but initialize queue only once it is actually required.
|
@torfsen It'd be cool if you did take the bounty, you could always give it to charity, or attach it as a bounty elsewhere? |
|
@rossjones I'm already being paid for the work that I'm doing here through other means, and accepting the bounty would probably be a lot of work from a legal perspective. Feel free to give it to charity or re-assign it to another issue! |
The background job system now supports multiple queues. By default, a single queue is used.
The background job system now supports multiple queues. By default, a single queue is used. In addition, RQ queue names are now prefixed with the CKAN site ID so that multiple CKAN instances can share the same Redis database.
The background job system now supports multiple queues. By default, a single queue is used. In addition, RQ queue names are now prefixed with the CKAN site ID so that multiple CKAN instances can share the same Redis database.
The background job system now supports multiple queues. By default, a single queue is used. In addition, RQ queue names are now prefixed with the CKAN site ID so that multiple CKAN instances can share the same Redis database.
Adds a basic support of background jobs using python-rq. Jobs can be managed via the `ckan.lib.jobs` module, the `job_*` API functions and the `jobs` paster command.
Check Redis availability on server start but initialize queue only once it is actually required.
Mention that it's usually best to simply use the `ckan.tests.recorded_logs` context manager.
Adds a basic support of background jobs using python-rq. Jobs can be managed via the `ckan.lib.jobs` module, the `job_*` API functions and the `jobs` paster command.
Check Redis availability on server start but initialize queue only once it is actually required.
The background job system now supports multiple queues. By default, a single queue is used. In addition, RQ queue names are now prefixed with the CKAN site ID so that multiple CKAN instances can share the same Redis database.
Documents that Redis keys should be prefixed with the site ID and the extension name.
This output doesn't go into the CKAN logs but into a separate worker log (`/var/log/ckan-worker.log` if the default Supervisor configuration is used).
In most cases one can simply use `tempfile.NamedTemporaryFile` instead.
Mention that it's usually best to simply use the `ckan.tests.recorded_logs` context manager.
The documentation previously talked about sharing databases which could be misunderstood as the possibility of sharing PostgreSQL databases between CKAN instances (instead of sharing Redis databases, which was the intended meaning).
A previous commit for ckan#2977 removed `ckan/config/celery-supervisor.conf` as part of deprecating the old Celery background task system. However, the old documentation told people to copy *or link* that file, so removing it could break existing installations. Hence this commit restores the file, it should be kept around until support for the Celery system is removed completely.
The docs now suggest to use `my_extension.my_setting` instead of the previously suggested `ckan.my_extension.my_setting`.
[#2977] Basic implementation of background jobs via python-rq
|
@rossjones @torfsen Hi, in my job im working on a scrapper inside CKAN that reads datasets from other webpages. If you want and my boss let do it, i can try to give it opensource, or at least the part of the background task Also, I can't take the bounty and Im happy with give it to charity or for another task |
|
@deinok This sounds like it might be a better fit for https://github.com/ckan/ckanext-harvest than for CKAN core. I've created an idea ticket at ckan/ideas#184 so feel free to claim it there. As an aside, because of the CKAN licence (AGPL) if you writes scrapers inside CKAN, you are required to release it under the same open source licence. If nobody wants the bounty, I'll donate it to CKAN Assoc to help someone with travel costs to the next CKANCON :) |
If you are interested in the bounty, please mark yourself as 'working on it' at https://www.bountysource.com/issues/33227756-provide-support-for-background-tasks
Currently running background tasks (sending email and so forth) is a bit too complicated.
There is an incomplete PR at #2706 which introduces support for this using python-rq. There is also more discussion at ckan/ideas#66
CKAN Version if known (or site URL)
Based off
masterPlease describe the expected behaviour
Support as per Ideas issue and PR
Please describe the actual behaviour
PR to be completed, with basic documentation and tests.
The text was updated successfully, but these errors were encountered: