Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Global workqueue throwing an exception when the database is compacting #11187

Closed
amaltaro opened this issue Jun 17, 2022 · 0 comments · Fixed by #11193
Closed

Global workqueue throwing an exception when the database is compacting #11187

amaltaro opened this issue Jun 17, 2022 · 0 comments · Fixed by #11193

Comments

@amaltaro
Copy link
Contributor

Impact of the bug
Global workqueue

Describe the bug
When central workqueue/workqueue_inbox database is compacting, some tasks might be skipped until the compaction is over.
With that, it can cause a TypeError exception with unexpected data type returned by a given function, as can be seen in this log [1]

How to reproduce it
Run global workqueue while couchdb is compacting the workqueue database.

Expected behavior
Properly deal with this scenario, i.e., workClosed should be 0 instead of None.

Additional context and error message
[1]

2022-06-17 20:26:48,997:INFO:WorkQueueReqMgrInterface:Closing open requests
2022-06-17 20:26:49,023:INFO:WorkQueueBackend:CouchDB compacting - try again later.
2022-06-17 20:26:49,023:WARNING:WorkQueue:Backend busy or down: Can not close work at this time
2022-06-17 20:26:49,023:ERROR:WorkQueueReqMgrInterface:Generic error while closing open requests. Details: object of type 'NoneType' has no len()
Traceback (most recent call last):
  File "/data/srv/HG2206e/sw/slc7_amd64_gcc630/cms/workqueue/2.0.4.pre4/lib/python3.8/site-packages/WMCore/WorkQueue/WorkQueueReqMgrInterface.py", line 63, in __call__
    msg += "Work closed: %d, " % len(workClosed)
TypeError: object of type 'NoneType' has no len()
2022-06-17 20:26:49,024:INFO:WorkQueueReqMgrInterface:Adding new elements to open requests
@amaltaro amaltaro added this to To do in Planned for Q2 - 2022 via automation Jun 17, 2022
@amaltaro amaltaro self-assigned this Jun 23, 2022
@amaltaro amaltaro moved this from To do to In progress in Planned for Q2 - 2022 Jun 23, 2022
Planned for Q2 - 2022 automation moved this from In progress to Work Done Jun 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

1 participant