Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ui, deployments] Promote Canary and Unhealthy Allocations in the deployment status panel #17547

Merged
merged 4 commits into from
Jun 19, 2023
Merged

[ui, deployments] Promote Canary and Unhealthy Allocations in the deployment status panel #17547

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3bfbcfd
A wild health status appears
philrenaud Jun 14, 2023
bb97069
autoPromote notification conditions
philrenaud Jun 15, 2023
cab4fdd
Legend fixes etc
philrenaud Jun 15, 2023
ef8995f
Acceptance tests for new canary alerts
philrenaud Jun 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions ui/app/components/job-status/allocation-status-block.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@
<span class="alloc-health-indicator">
{{#if (eq @health "healthy")}}
<FlightIcon @name="check" @color="#25ba81" />
{{else if (eq @health "unhealthy")}}
<FlightIcon @name="x" @color="#c84034" />
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the little top-right-corner status indicator that shows up when enough of an alloc exist to summarize it in a "+N" block

{{else}}
<FlightIcon @name="running" @color="black" />
{{/if}}
Expand Down
2 changes: 2 additions & 0 deletions ui/app/components/job-status/individual-allocation.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
<span class="alloc-health-indicator">
{{#if (eq @health "healthy")}}
<FlightIcon @name="check" @color="white" />
{{else if (eq @health "unhealthy")}}
<FlightIcon @name="x" @color="white" />
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the indicator for when an allocation is un-aggregated/summarized, in say a small enough cluster that each allocation gets its own square.

{{else}}
<FlightIcon @name="running" @color="white" />
{{/if}}
Expand Down
44 changes: 33 additions & 11 deletions ui/app/components/job-status/panel/deploying.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,18 +27,38 @@
@disabled={{this.fail.isRunning}}
@onConfirm={{perform this.fail}} />
{{/if}}
{{#if @job.latestDeployment.requiresPromotion}}
<button
data-test-promote-canary
type="button"
class="button is-warning is-small {{if this.promote.isRunning "is-loading"}}"
disabled={{this.promote.isRunning}}
onclick={{perform this.promote}}>Promote Canary</button>
{{/if}}
</div>
</div>
</div>
<div class="boxed-section-body">
<div class="boxed-section-body {{if @job.latestDeployment.requiresPromotion "requires-promotion"}}">
{{#if @job.latestDeployment.requiresPromotion}}
<div class="canary-promotion-alert">
{{#if this.canariesHealthy}}
<Hds::Alert @type="inline" @color="warning" as |A|>
<A.Title>Deployment requires promotion</A.Title>
<A.Description>Your deployment requires manual promotion — all canary allocations have passed their health checks.</A.Description>
<A.Button data-test-promote-canary @text="Promote Canary" @color="primary" {{on "click" (perform this.promote)}} />
</Hds::Alert>
{{else}}
{{#if this.someCanariesHaveFailed}}
<Hds::Alert @type="inline" @color="critical" as |A|>
<A.Title>Some Canaries have failed</A.Title>
<A.Description>Your canary allocations have failed their health checks. Please have a look at the error logs and task events for the allocations in question.</A.Description>
</Hds::Alert>
{{else}}
<Hds::Alert @type="inline" @color="highlight" as |A|>
<A.Title>Checking Canary health</A.Title>
{{#if this.deploymentIsAutoPromoted}}
<A.Description>Your canary allocations are being placed and health-checked. If they pass, they will be automatically promoted and your deployment will continue.</A.Description>
{{else}}
<A.Description>Your job requires manual promotion, and your canary allocations are being placed and health-checked.</A.Description>
{{/if}}
</Hds::Alert>
{{/if}}
{{/if}}
</div>
{{/if}}

<div class="deployment-allocations">
{{#if this.oldVersionAllocBlockIDs.length}}
<h4 class="title is-5" data-test-old-allocation-tally>Previous allocations: {{#if this.oldVersionAllocBlocks.running}}{{this.oldRunningHealthyAllocBlocks.length}} running{{/if}}</h4>
Expand Down Expand Up @@ -86,16 +106,18 @@

{{#each-in this.newAllocsByHealth as |health count|}}
<span class="legend-item {{if (eq count 0) "faded"}}">
<span class="represented-allocation legend-example">
<span class="represented-allocation legend-example {{health}}">
<span class="alloc-health-indicator">
{{#if (eq health "healthy")}}
<FlightIcon @name="check" @color="#25ba81" />
{{else if (eq health "unhealthy")}}
<FlightIcon @name="x" @color="#c84034" />
{{else}}
<FlightIcon @name="running" @color="black" class="not-animated" />
{{/if}}
</span>
</span>
{{count}} {{capitalize health}}
{{count}} {{humanize health}}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

humanize to drop the _ in health_unknown.

(in case you were wondering: we don't use unknown here because that has a dedicated orange colour associated on elements from "Node status unknown" / max client disconnect stuff.)

</span>
{{/each-in}}

Expand Down
53 changes: 49 additions & 4 deletions ui/app/components/job-status/panel/deploying.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,32 @@ export default class JobStatusPanelDeployingComponent extends Component {
);
}

/**
* Promotion of a deployment will error if the canary allocations are not of status "Healthy";
* this function will check for that and disable the promote button if necessary.
* @returns {boolean}
*/
get canariesHealthy() {
const relevantAllocs = this.job.allocations.filter(
(a) => !a.isOld && a.isCanary && !a.hasBeenRescheduled
);
return relevantAllocs.every(
(a) => a.clientStatus === 'running' && a.isHealthy
);
}

get someCanariesHaveFailed() {
const relevantAllocs = this.job.allocations.filter(
(a) => !a.isOld && a.isCanary && !a.hasBeenRescheduled
);
return relevantAllocs.some(
(a) =>
a.clientStatus === 'failed' ||
a.clientStatus === 'lost' ||
a.isUnhealthy
);
}

@task(function* () {
try {
yield this.job.latestDeployment.content.promote();
Expand Down Expand Up @@ -70,6 +96,7 @@ export default class JobStatusPanelDeployingComponent extends Component {
alloGroups[status] = {
healthy: { nonCanary: [] },
unhealthy: { nonCanary: [] },
health_unknown: { nonCanary: [] },
};
}
alloGroups[status].healthy.nonCanary.push(currentAlloc);
Expand All @@ -88,6 +115,7 @@ export default class JobStatusPanelDeployingComponent extends Component {
categories[type.label] = {
healthy: { canary: [], nonCanary: [] },
unhealthy: { canary: [], nonCanary: [] },
health_unknown: { canary: [], nonCanary: [] },
};
return categories;
}, {});
Expand All @@ -107,8 +135,10 @@ export default class JobStatusPanelDeployingComponent extends Component {
status === 'running'
? alloc.isHealthy
? 'healthy'
: 'unhealthy'
: 'unhealthy';
: alloc.isUnhealthy
? 'unhealthy'
: 'health_unknown'
: 'health_unknown';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not thrilled that I'm nesting a ternary here but it is only causing me mild discomfort


if (allocationCategories[status]) {
// If status is failed or lost, we only want to show it IF it's used up its restarts/rescheds.
Expand All @@ -129,6 +159,7 @@ export default class JobStatusPanelDeployingComponent extends Component {
allocationCategories['unplaced'] = {
healthy: { canary: [], nonCanary: [] },
unhealthy: { canary: [], nonCanary: [] },
health_unknown: { canary: [], nonCanary: [] },
};
allocationCategories['unplaced']['healthy']['nonCanary'] = Array(
availableSlotsToFill
Expand All @@ -149,6 +180,13 @@ export default class JobStatusPanelDeployingComponent extends Component {
];
}

get newRunningUnhealthyAllocBlocks() {
return [
...this.newVersionAllocBlocks['running']['unhealthy']['canary'],
...this.newVersionAllocBlocks['running']['unhealthy']['nonCanary'],
];
}

get rescheduledAllocs() {
return this.job.allocations.filter((a) => !a.isOld && a.hasBeenRescheduled);
}
Expand Down Expand Up @@ -183,8 +221,11 @@ export default class JobStatusPanelDeployingComponent extends Component {
get newAllocsByHealth() {
return {
healthy: this.newRunningHealthyAllocBlocks.length,
'health unknown':
this.totalAllocs - this.newRunningHealthyAllocBlocks.length,
unhealthy: this.newRunningUnhealthyAllocBlocks.length,
health_unknown:
this.totalAllocs -
this.newRunningHealthyAllocBlocks.length -
this.newRunningUnhealthyAllocBlocks.length,
};
}
// #endregion legend
Expand All @@ -205,4 +246,8 @@ export default class JobStatusPanelDeployingComponent extends Component {
// v----- Realistic method: Tally a job's task groups' "count" property
return this.args.job.taskGroups.reduce((sum, tg) => sum + tg.count, 0);
}

get deploymentIsAutoPromoted() {
return this.job.latestDeployment?.get('isAutoPromoted');
}
}
1 change: 1 addition & 0 deletions ui/app/components/job-status/panel/steady.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ export default class JobStatusPanelSteadyComponent extends Component {
* @typedef {Object} AllocationStatus
* @property {HealthStatus} healthy
* @property {HealthStatus} unhealthy
* @property {HealthStatus} health unknown
*/

/**
Expand Down
5 changes: 5 additions & 0 deletions ui/app/models/allocation.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,15 @@ export default class Allocation extends Model {
return this.deploymentStatus?.Canary;
}

// deploymentStatus.Healthy can be true, false, or null. Null implies pending
get isHealthy() {
return this.deploymentStatus?.Healthy;
}

get isUnhealthy() {
return this.deploymentStatus?.Healthy === false;
}

get willNotRestart() {
return this.clientStatus === 'failed' || this.clientStatus === 'lost';
}
Expand Down
9 changes: 8 additions & 1 deletion ui/app/models/deployment.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ export default class Deployment extends Model {

// If any task group is not promoted yet requires promotion and the deployment
// is still running, the deployment needs promotion.
@computed('status', 'taskGroupSummaries.@each.promoted')
@computed('status', 'taskGroupSummaries.@each.{promoted,requiresPromotion}')
get requiresPromotion() {
return (
this.status === 'running' &&
Expand All @@ -36,6 +36,13 @@ export default class Deployment extends Model {
);
}

@computed('taskGroupSummaries.@each.autoPromote')
get isAutoPromoted() {
return this.taskGroupSummaries
.toArray()
.every((summary) => summary.get('autoPromote'));
}

@attr('string') status;
@attr('string') statusDescription;

Expand Down
1 change: 1 addition & 0 deletions ui/app/models/task-group-deployment-summary.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ export default class TaskGroupDeploymentSummary extends Fragment {
@attr('string') name;

@attr('boolean') autoRevert;
@attr('boolean') autoPromote;
@attr('boolean') promoted;
@gt('desiredCanaries', 0) requiresPromotion;

Expand Down
23 changes: 23 additions & 0 deletions ui/app/styles/components/job-status-panel.scss
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,28 @@
gap: 1rem;
grid-auto-columns: 100%;

&.requires-promotion {
grid-template-areas:
'promotion-alert'
'deployment-allocations'
'legend-and-summary'
'history-and-params';

& > .canary-promotion-alert {
button {
background-color: $orange;
border-color: darken($orange, 5%);
&:hover {
background-color: darken($orange, 5%);
}
}
}
}

& > .promotion-alert {
grid-area: promotion-alert;
}

& > .deployment-allocations {
grid-area: deployment-allocations;
display: grid;
Expand Down Expand Up @@ -68,6 +90,7 @@
legend {
display: grid;
grid-template-columns: repeat(4, 1fr);
grid-auto-rows: max-content;
gap: 0.5rem;
}
.versions {
Expand Down
Loading