Pulp with multiple celeryBeat instances #1940

shubham90 · 2015-06-19T19:19:38Z

bmbouter · 2015-06-19T19:24:56Z

docs/user-guide/installation.rst


   On some Pulp system, configure, start and enable the Celerybeat process. This process performs a
   job similar to a cron daemon for Pulp. Edit ``/etc/default/pulp_celerybeat`` to your liking, and
-   then enable and start it. Again, do not enable this on more than one host. For Upstart::
+   then enable and start it. Multiple instances of this may be enabled, which will make the system


s/this/pulp_celerybeat/
s/enabled/run concurrently/
s/the system/the Pulp installation/

bmbouter · 2015-06-19T19:29:00Z

Everywhere that you refer in the docs to pulp_celerybeat, ensure to wrap it in two backticks to rst escape the term.

``pulp_celerybeat``

pulpbot · 2015-06-19T19:30:27Z

Refer to this link for build results (access rights to CI server needed):
https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com//job/unittest-pulp-pr/953/
Test PASSed.

bmbouter · 2015-06-19T19:31:40Z

docs/user-guide/server.rst

@@ -30,7 +30,7 @@ For a recap of Pulp components and the work they are responsible for, read :ref:
  available workers. If a worker with the same name is started again after being missing, it is
  added into the pool of workers as any worker starting up normally would.

-* If ``pulp_celerybeat`` dies, and in case then new workers start, they won't be given work. If
+* If all instances of ``pulp_celerybeat`` dies, and in case then new workers start, they won't be given work. If


shorten this line.

s/dies/die/
s/ in case then//

bmbouter · 2015-06-19T19:42:55Z

A lot of these files have had their permissions changed and show 100644 → 100755 at the top. This needs to not change their file permissions.

bmbouter · 2015-06-19T19:50:35Z

server/pulp/server/async/scheduler.py

+            collection.remove({'lock_timestamp': {'$lte': old_timestamp}})
+            try:
+                # Insert new lock entry
+                collection.insert({


This should use the CeleryBeatLock model definition and use save() instead.

bmbouter · 2015-06-19T20:03:38Z

@shubham90 I'm done for now I'll re-review after the changes are made.

bmbouter · 2015-06-22T20:00:58Z

docs/user-guide/release-notes/master.rst

@@ -8,6 +8,8 @@ Pulp master
 New Features
 ------------

+* Multiple instances of ``pulp_celerybeat`` can now run simultaneously.
+  If one of them goes down, another instance will re-dispatch tasks as usual.


s/re-dispatch tasks/dispatch scheduled tasks/

bmbouter · 2015-06-26T20:40:29Z

server/pulp/server/db/model/dispatch.py

+    Single document collection which gives information about the current celerybeat lock.
+
+        :param celerybeat_name: string representing the celerybeat instance name
+        :type  celerybeat_name: basestring


type should have a single space after it.

bmbouter · 2015-06-29T16:25:49Z

I was hand testing this, and I got the following traceback when I run pulp-manage-db:

pulp.server.db.manage:CRITICAL: 'module' object has no attribute 'CeleryBeatLock'
pulp.server.db.manage:CRITICAL: (18666-81760) Traceback (most recent call last):
pulp.server.db.manage:CRITICAL: (18666-81760)   File "/home/bmbouter/Documents/pulp/server/pulp/server/db/manage.py", line 160, in main
pulp.server.db.manage:CRITICAL: (18666-81760)     return _auto_manage_db(options)
pulp.server.db.manage:CRITICAL: (18666-81760)   File "/home/bmbouter/Documents/pulp/server/pulp/server/db/manage.py", line 221, in _auto_manage_db
pulp.server.db.manage:CRITICAL: (18666-81760)     migrate_database(options)
pulp.server.db.manage:CRITICAL: (18666-81760)   File "/home/bmbouter/Documents/pulp/server/pulp/server/db/manage.py", line 104, in migrate_database
pulp.server.db.manage:CRITICAL: (18666-81760)     ensure_database_indexes()
pulp.server.db.manage:CRITICAL: (18666-81760)   File "/home/bmbouter/Documents/pulp/server/pulp/server/db/manage.py", line 124, in ensure_database_indexes
pulp.server.db.manage:CRITICAL: (18666-81760)     _ensure_indexes(model.CeleryBeatLock) 
pulp.server.db.manage:CRITICAL: (18666-81760) AttributeError: 'module' object has no attribute 'CeleryBeatLock'
pulp.server.db.manage:CRITICAL: (18666-81760)

The issue is that the CeleryBeatLock model needs to be moved out of dispatch and into pulp.server.db with the other subclasses of Document.

pulpbot · 2015-06-29T22:49:26Z

Refer to this link for build results (access rights to CI server needed):
https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com//job/unittest-pulp-pr/1017/

Build result: FAILURE

[...truncated 6 lines...] > git config remote.origin.url https://github.com/pulp/pulp.git # timeout=10Fetching upstream changes from https://github.com/pulp/pulp.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/pulp/pulp.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse 536f1f7^{commit} # timeout=10Checking out Revision 536f1f7 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 536f1f7 > git rev-list cdd7ba5 # timeout=10Triggering rhel5-npTriggering f20-npTriggering rhel7-npTriggering rhel6-npTriggering f21-nprhel5-np completed with result SUCCESSf20-np completed with result FAILURErhel7-np completed with result FAILURErhel6-np completed with result FAILUREf21-np completed with result FAILUREStarted calculate disk usage of buildFinished Calculation of disk usage of build in 0 secondsStarted calculate disk usage of workspaceFinished Calculation of disk usage of workspace in 2 secondSetting status of 536f1f7 to FAILURE with url https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/unittest-pulp-pr/1017/ and message: Build finished.
Test FAILed.

shubham90 · 2015-06-30T15:28:59Z

ok test

pulpbot · 2015-06-30T15:36:28Z

Refer to this link for build results (access rights to CI server needed):
https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com//job/unittest-pulp-pr/1027/
Test PASSed.

asmacdo · 2015-07-02T14:45:20Z

server/pulp/server/db/model/__init__.py

+    """
+    Single document collection which gives information about the current celerybeat lock.
+
+    :param celerybeat_name: string representing the celerybeat instance name


Very minor, but these are all instance variables, not params.

:ivar celerybeat_name: string representing the celerybeat instance name

asmacdo · 2015-07-02T14:54:54Z

@shubham90, you don't need to, but feel free to delete the old pulpbot test comments and triggers to clean up.

shubham90 · 2015-07-02T18:30:30Z

@asmacdo Thanks for reviewing. I have made the changes you suggested. Those were really helpful.

shubham90 · 2015-07-02T18:46:09Z

ok test

mhrivnak · 2015-07-02T18:49:16Z

server/pulp/server/async/scheduler.py

@@ -6,18 +6,20 @@
 import platform
 import threading
 import time
+import uuid
+import mongoengine


for pep8, this should be in the second group of imports. The first group is reserved for standard library imports.

Also within each section imports should be alphabetized so mongoengine would come before uuid.

pulpbot · 2015-07-02T18:54:25Z

Refer to this link for build results (access rights to CI server needed):
https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com//job/unittest-pulp-pr/1056/
Test PASSed.

mhrivnak · 2015-07-02T18:57:40Z

This looks good. Well done!

bmbouter · 2015-07-06T16:17:37Z

server/pulp/server/async/scheduler.py

@@ -27,6 +29,7 @@


 _logger = logging.getLogger(__name__)
+celery_name = uuid.uuid4()


This should be removed.

https://pulp/plan.io/issues/1060 closes #1060

pulpbot · 2015-07-06T17:39:06Z

Refer to this link for build results (access rights to CI server needed):
https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com//job/unittest-pulp-pr/1060/
Test PASSed.

bmbouter · 2015-07-06T19:18:10Z

I did several rounds of hand testing. I tested the following scenarios:

when two pulp_celerybeat processes are running only one scheduled task gets sent out
when the pulp_celerybeat who aquired the lock was killed the other failed over in the expected amount of time
starting a the first pulp_celerybeat behaves as expected
starting a pulp_celerybeat when old lock records are in place works as expected

@shubham90 great job, I'm putting the LGTM on it!

Story 1060: Pulp with multiple celeryBeat instances

bmbouter self-assigned this Jun 19, 2015

bmbouter reviewed Jun 19, 2015
View reviewed changes

bmbouter assigned shubham90 and unassigned bmbouter Jun 19, 2015

bmbouter reviewed Jun 22, 2015
View reviewed changes

bmbouter reviewed Jun 26, 2015
View reviewed changes

asmacdo reviewed Jul 2, 2015
View reviewed changes

mhrivnak reviewed Jul 2, 2015
View reviewed changes

bmbouter reviewed Jul 6, 2015
View reviewed changes

Story 1060: Pulp with multiple celerybeat instances

df6ad28

https://pulp/plan.io/issues/1060 closes #1060

bmbouter added Feature LGTM labels Jul 6, 2015

shubham90 pushed a commit that referenced this pull request Jul 6, 2015

Merge pull request #1940 from shubham90/celeryHA

04a91c9

Story 1060: Pulp with multiple celeryBeat instances

shubham90 merged commit 04a91c9 into pulp:master Jul 6, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pulp with multiple celeryBeat instances #1940

Pulp with multiple celeryBeat instances #1940

shubham90 commented Jun 19, 2015

bmbouter Jun 19, 2015

bmbouter commented Jun 19, 2015

pulpbot commented Jun 19, 2015

bmbouter Jun 19, 2015

bmbouter Jun 19, 2015

bmbouter commented Jun 19, 2015

bmbouter Jun 19, 2015

bmbouter commented Jun 19, 2015

bmbouter Jun 22, 2015

bmbouter Jun 26, 2015

bmbouter commented Jun 29, 2015

pulpbot commented Jun 29, 2015

shubham90 commented Jun 30, 2015

pulpbot commented Jun 30, 2015

asmacdo Jul 2, 2015

asmacdo commented Jul 2, 2015

shubham90 commented Jul 2, 2015

shubham90 commented Jul 2, 2015

mhrivnak Jul 2, 2015

bmbouter Jul 6, 2015

pulpbot commented Jul 2, 2015

mhrivnak commented Jul 2, 2015

bmbouter Jul 6, 2015

pulpbot commented Jul 6, 2015

bmbouter commented Jul 6, 2015

		@@ -27,6 +29,7 @@


		_logger = logging.getLogger(__name__)
		celery_name = uuid.uuid4()

Pulp with multiple celeryBeat instances #1940

Pulp with multiple celeryBeat instances #1940

Conversation

shubham90 commented Jun 19, 2015

Choose a reason for hiding this comment

bmbouter commented Jun 19, 2015

pulpbot commented Jun 19, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmbouter commented Jun 19, 2015

Choose a reason for hiding this comment

bmbouter commented Jun 19, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmbouter commented Jun 29, 2015

pulpbot commented Jun 29, 2015

Build result: FAILURE

shubham90 commented Jun 30, 2015

pulpbot commented Jun 30, 2015

Choose a reason for hiding this comment

asmacdo commented Jul 2, 2015

shubham90 commented Jul 2, 2015

shubham90 commented Jul 2, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pulpbot commented Jul 2, 2015

mhrivnak commented Jul 2, 2015

Choose a reason for hiding this comment

pulpbot commented Jul 6, 2015

bmbouter commented Jul 6, 2015