Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to RQ #72

Merged
merged 1 commit into from May 14, 2018
Merged

Switch to RQ #72

merged 1 commit into from May 14, 2018

Conversation

bmbouter
Copy link
Member

@bmbouter bmbouter commented Apr 20, 2018

  • Updates Travis to use RQ worker runners
  • Remove usage of Celery
  • port apply_async_with_reservation to enqueue_with_reservation

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp#3454

@pep8speaks
Copy link

pep8speaks commented Apr 20, 2018

Hello @bmbouter! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on May 08, 2018 at 16:46 Hours UTC

bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 20, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/devel that referenced this pull request Apr 20, 2018
This commit:

* Removes rabbitMQ and Qpid installation
* Installs, enables, and starts Redis
* Updates the systemd assets to use the RQ worker syntax

This PR should be tested against these core and pulp_file PRs:

pulp/pulp#3454
pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/devel that referenced this pull request Apr 20, 2018
This commit:

* Removes rabbitMQ and Qpid installation
* Installs, enables, and starts Redis
* Updates the systemd assets to use the RQ worker syntax

This PR modifies the environment for these changes:

pulp/pulp#3454
pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp-smash that referenced this pull request Apr 20, 2018
The attribute name in the status API changed when RQ with Redis replaced
AMQP brokers like RabbitMQ.

This PR is meant to be used with these core and pulp_file PRs:

pulp/pulp#3454
pulp/pulp_file#72
@bmbouter bmbouter added the 3.0 label Apr 20, 2018
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 20, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp-smash that referenced this pull request Apr 23, 2018
The attribute name in the status API changed when RQ with Redis replaced
AMQP brokers like RabbitMQ.

This PR is meant to be used with these core and pulp_file PRs:

pulp/pulp#3454
pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 23, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 23, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 23, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 23, 2018
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 23, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 23, 2018
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 23, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
dkliban added a commit to dkliban/pulp that referenced this pull request Apr 23, 2018
dkliban added a commit to dkliban/pulp that referenced this pull request Apr 23, 2018
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 23, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/devel that referenced this pull request Apr 26, 2018
This commit:

* Removes rabbitMQ and Qpid installation
* Installs, enables, and starts Redis
* Updates the systemd assets to use the RQ worker syntax

This PR modifies the environment for these changes:

pulp/pulp#3454
pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 26, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 27, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 27, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request Apr 27, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp-smash that referenced this pull request Apr 27, 2018
The attribute name in the status API changed when RQ with Redis replaced
AMQP brokers like RabbitMQ.

This PR is meant to be used with these core and pulp_file PRs:

pulp/pulp#3454
pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp-smash that referenced this pull request May 7, 2018
The attribute name in the status API changed when RQ with Redis replaced
AMQP brokers like RabbitMQ.

This PR is meant to be used with these core and pulp_file PRs:

pulp/pulp#3454
pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/devel that referenced this pull request May 7, 2018
This commit:

* Removes rabbitMQ and Qpid installation
* Installs, enables, and starts Redis
* Updates the systemd assets to use the RQ worker syntax
* Installs a specific commit of RQ to workaround a temporary RQ release
  issue

This PR modifies the environment for these changes:

pulp/pulp#3454
pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request May 7, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request May 7, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request May 7, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request May 7, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/devel that referenced this pull request May 8, 2018
This commit:

* Removes rabbitMQ and Qpid installation
* Installs, enables, and starts Redis
* Updates the systemd assets to use the RQ worker syntax
* Installs a specific commit of RQ to workaround a temporary RQ release
  issue

This PR modifies the environment for these changes:

pulp/pulp#3454
pulp/pulp_file#72
* Updates Travis to use RQ worker runners
* Remove usage of Celery
* port apply_async_with_reservation to enqueue_with_reservation

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp#3454
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request May 8, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request May 8, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
@bmbouter
Copy link
Member Author

bmbouter commented May 8, 2018

@daviddavis this repo doesn't require a LGTM, but would you want to give one if you feel that is right?

All the other PRs now have LGTMs including the pulp/pulp changes: pulp/pulp#3454 (review)

bmbouter pushed a commit to bmbouter/pulp that referenced this pull request May 8, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
Copy link
Contributor

@daviddavis daviddavis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

bmbouter pushed a commit to bmbouter/pulp that referenced this pull request May 14, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to bmbouter/pulp that referenced this pull request May 14, 2018
This PR does the following:

* port travis to RQ
* remove all celery references
* install, systemd docs updates
* create a custom RQ worker method for Pulp's needs
* ports the status API to report on the Redis connection
* port orphan cleanup tasks to work with RQ
* port all pulp tasks to RQ
* handle fatal error exception recording on tasks
* replace apply_async_with_reservation to enqueue_with_reservation
* task cancellation kills a running task correctly
* work discovery, normal shutdown, and crash shutdown tested
* adds settings for Redis connection to settings files
* removes celery as a dependency
* adds rq as a dependency

There is a devel repo PR here to be used by `vagrant up` when testing
this PR: pulp/devel#146

Required PR: pulp/pulp-smash#960
Required PR: pulp/pulp_file#72
bmbouter pushed a commit to pulp/pulp-smash that referenced this pull request May 14, 2018
The attribute name in the status API changed when RQ with Redis replaced
AMQP brokers like RabbitMQ.

This PR is meant to be used with these core and pulp_file PRs:

pulp/pulp#3454
pulp/pulp_file#72
@bmbouter bmbouter merged commit 3bf1361 into pulp:master May 14, 2018
@bmbouter bmbouter deleted the port-to-rq branch May 14, 2018 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants