Rq not seen as queue to increase capacity

[freecurve]

Hello. Thank you very much for a great tool!

1. rescheduled jobs in Rq state don't seem to trigger load alarms. 'Rq' should probably be treated just like 'qw'.

[dougalb]

The qstat command used to count pending is here: https://github.com/awslabs/cfncluster-cookbook/blob/develop/templates/default/publish_pending.sge.erb#L20

Is this not catching jobs in Rq state?

[freecurve]

Hi Dougal -

I just put a job into Rq and you're correct, it did register.

This situation does not come up for us frequently but within a week it will
so I will check in in a live scaling group and will report, and we can
resolve.

[ec2-user@ip-172-31-33-228 ~]$ qstat -g d -s p -u '' | tail -n+3
1774 0.50500 Sleeper ec2-user Rq 01/11/2016 21:15:19
1
[ec2-user@ip-172-31-33-228 ~]$ qstat -g d -s p -u '' | tail -n+3 | awk
'{total = total+ $8}END{print total}'
1

On Mon, Jan 11, 2016 at 10:04 AM, Dougal Ballantyne <
notifications@github.com> wrote:

The qstat command used to count pending is here:
https://github.com/awslabs/cfncluster-cookbook/blob/develop/templates/default/publish_pending.sge.erb#L20

Is this not catching jobs in Rq state?

—
Reply to this email directly or view it on GitHub
#77 (comment).

b.

[freecurve]

And yet somehow after a tumultuous run with all the scale-up-downs across
availability zones with spot instances, the remaining Rq's do not trigger
an alarm. Adding some fresh dummy jobs does trigger the alarm. That's about
as far as I can help.

On Mon, Jan 11, 2016 at 1:17 PM, Boris Fain boris@freecurve.com wrote:

Hi Dougal -

I just put a job into Rq and you're correct, it did register.

This situation does not come up for us frequently but within a week it
will so I will check in in a live scaling group and will report, and we can
resolve.

[ec2-user@ip-172-31-33-228 ~]$ qstat -g d -s p -u '' | tail -n+3
1774 0.50500 Sleeper ec2-user Rq 01/11/2016 21:15:19
1
[ec2-user@ip-172-31-33-228 ~]$ qstat -g d -s p -u '' | tail -n+3 | awk
'{total = total+ $8}END{print total}'
1

On Mon, Jan 11, 2016 at 10:04 AM, Dougal Ballantyne <
notifications@github.com> wrote:

The qstat command used to count pending is here:
https://github.com/awslabs/cfncluster-cookbook/blob/develop/templates/default/publish_pending.sge.erb#L20

Is this not catching jobs in Rq state?

—
Reply to this email directly or view it on GitHub
#77 (comment).

b.

b.

[dougalb]

across availability zones

Have you modified CfnCluster to operate in multiple AZs?

[dougalb]

Can you send me your AWS account id, region and cluster name to dougalb at amazon dot com

[freecurve]

Thank you.

On Mon, Feb 15, 2016 at 2:03 PM, Dougal Ballantyne <notifications@github.com

wrote:

Closed #77 #77.

—
Reply to this email directly or view it on GitHub
#77 (comment).

b.

[freecurve]

hi - new version of cfncluster again does not count Rq state as pending. Fix is
pending=$(qstat -g d -s p -u '*' | tail -n+3 | awk '$5 == "qw" || $5 == "Rq" {total = total+ $8} END {print total}')

in publish_pending_jobs (at least for SGE).

[sean-smith]

Fixed in aws/aws-parallelcluster-node#43