Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

badfish.py needs to utilize equivalent of JID_CLEARALL_FORCE #261

Open
sadsfae opened this issue Jun 8, 2019 · 3 comments

Comments

Projects
None yet
2 participants
@sadsfae
Copy link
Member

commented Jun 8, 2019

It seems that occasionally Dell systems (thus far r620 but probably others) don't always get their job queues cleared with badfish. These cannot be cleared either via the iDRAC UI.

This causes JID's to pile up and never complete.

2019-06-08 12:49:38,875 Clearing job queue for job IDs: ['RID_843181825116', 'RID_843175376755', 'RID_843175362071', 'RID_843201688905', 'RID_843208253120', 'RID_843158310720', 'RID_843158246559', 'RID_843208272388', 'RID_843053267400', 'RID_843188446456', 'RID_843181842184', 'RID_843195047909', 'RID_843188431875', 'RID_843201674287', 'RID_843195032975'].
2019-06-08 12:51:31,438 Job queue not cleared, current job queue contains jobs: ['RID_843181825116', 'RID_843175376755', 'RID_843175362071', 'RID_843201688905', 'RID_843208253120', 'RID_843158310720', 'RID_843158246559', 'RID_843208272388', 'RID_843053267400', 'RID_843188446456', 'RID_843181842184', 'RID_843195047909', 'RID_843188431875', 'RID_843201674287', 'RID_843195032975'].
2019-06-08 12:51:31,438 Could not set boot order via Badfish.
Traceback (most recent call last):
  File "/opt/quads/quads/tools/move_and_rebuild_hosts.py", line 151, in move_and_rebuild
    "../../conf/idrac_interfaces.yml"
  File "/opt/quads/quads/tools/badfish.py", line 306, in change_boot
    self.clear_job_queue()
  File "/opt/quads/quads/tools/badfish.py", line 412, in clear_job_queue
    sys.exit(1)
SystemExit: 1

Screenshot_2019-06-08_14-15-30

See full details here on upstream badfish, it should be backported to the badfish.py library.

redhat-performance/badfish#30

@sadsfae sadsfae added this to the 1.1.0 milestone Jun 8, 2019

@grafuls

This comment has been minimized.

Copy link
Contributor

commented Jun 10, 2019

I tried before to use JID_CLEARALL_FORCE on badfish but it did't clear the queue neither.
I'll do some research on this one.

@grafuls

This comment has been minimized.

sadsfae pushed a commit that referenced this issue Jun 10, 2019

Multi-fix: foreman creds, badfish, validate
Fixes issues with move and rebuild on clearing job queue.
Added force method for badfish clear_job_queue.
Fix for foreman validate_credentials.
Fix for validate_env on timedelta and older schedules.

Fixes: #261
Change-Id: I10e170785a99c7f2d96bd5e45a55b7a5427a4c26

@sadsfae sadsfae closed this Jun 10, 2019

@sadsfae

This comment has been minimized.

Copy link
Member Author

commented Jun 24, 2019

Re-opening this, it seems that the redfish API method still doesn't always clear the same (or as effective as) jobqueue delete -i JID_CLEARALL_FORCE

@sadsfae sadsfae reopened this Jun 24, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.