Leader election using locks #649

Juanadelacuesta · 2023-06-28T11:57:16Z

Add mechanism to run on high availability by using a lock to maintain the synchrony among instances. Currently is a free standing package that depends on the future implementation of the lock CLI on nomad.

It is still pending on exposing some metrics and error handling

For more information, refer to the RFC: [NMD-179] Leader election using locking mechanism for the Autoscaler

ha/lock.go

lgfa29

Looking great! Excited to see where this is going 😄

I think the tests not running in CI is the biggest blocker, most of the comments are me trying to understand stuff 😬

ha/lock.go

ha/lock_test.go

ha/lock.go

ha/lock_test.go

ha/lock.go

ha/lock_test.go

jrasell

Looking good and most of my comments are around UX and style, so we can catch in Slack if that's easier than GitHub :D

It would also be nice to add some tests to cover the new config parsing addition.

agent/agent.go

agent/config/config.go

command/agent.go

… data race condition on testing

Co-authored-by: Luiz Aoqui <luiz@hashicorp.com>

Co-authored-by: James Rasell <jrasell@users.noreply.github.com>

agent/config/config.go

command/agent.go

jrasell

Approving, with a final few questions.

…nd 30 s respectively

Juanadelacuesta force-pushed the f-gh-17449 branch from 34ee0e6 to 897efae Compare June 28, 2023 12:00

Juanadelacuesta requested a review from pkazmierczak July 3, 2023 11:35

Juanadelacuesta changed the title ~~F gh 17449~~ Leader election using locks Jul 3, 2023

Juanadelacuesta marked this pull request as ready for review July 3, 2023 11:42

Juanadelacuesta requested review from jrasell and lgfa29 as code owners July 3, 2023 11:42

Juanadelacuesta force-pushed the f-gh-17449 branch from 9a7b7d8 to fb81a21 Compare July 3, 2023 14:12

Juanadelacuesta requested a review from tgross July 5, 2023 13:19

tgross reviewed Jul 5, 2023

View reviewed changes

ha/lock.go Outdated Show resolved Hide resolved

ha/lock.go Outdated Show resolved Hide resolved

ha/lock.go Outdated Show resolved Hide resolved

Juanadelacuesta requested a review from tgross July 6, 2023 14:58

tgross reviewed Jul 6, 2023

View reviewed changes

ha/lock.go Outdated Show resolved Hide resolved

Juanadelacuesta requested a review from tgross July 6, 2023 20:39

lgfa29 suggested changes Jul 6, 2023

View reviewed changes

Juanadelacuesta force-pushed the f-gh-17449 branch 2 times, most recently from 3663302 to 443a376 Compare July 7, 2023 12:44

Juanadelacuesta requested a review from lgfa29 July 7, 2023 16:08

Juanadelacuesta requested a review from gulducat July 19, 2023 15:36

Juanadelacuesta force-pushed the f-gh-17449 branch from 9f95138 to f370a91 Compare August 23, 2023 19:17

jrasell requested changes Sep 13, 2023

View reviewed changes

Juanadelacuesta and others added 11 commits September 13, 2023 13:54

func: add initial draft of main cycle for the ha

147d396

func: add tests for the package

d18ea3f

fix: update dependencies

85eb788

fix: update dependecies on target plugins

5b1cb1c

style: add mutex to avoid warning on race condition

c9f8d94

fix: exclude the lock test from the race condition test

bccc321

style: rename ha controller

cc21345

func: add mutex and getter methods to mock lock and service, to avoid…

a7fcbf6

… data race condition on testing

Update ha/lock_test.go

78e027b

Co-authored-by: Luiz Aoqui <luiz@hashicorp.com>

Update ha/lock.go

35fad3d

Co-authored-by: Luiz Aoqui <luiz@hashicorp.com>

func: add lock leaser on nomad agent run

de57bf5

Juanadelacuesta and others added 8 commits September 13, 2023 13:54

func: update the useage of the lockleaser

a0d3732

func: update leaser implementation

51c2221

func: add definitions for ha mode

a7128b9

Update agent/agent.go

10e17bd

Co-authored-by: James Rasell <jrasell@users.noreply.github.com>

Update agent/config/config.go

1b3201d

Co-authored-by: James Rasell <jrasell@users.noreply.github.com>

Update command/agent.go

7e3477d

Co-authored-by: James Rasell <jrasell@users.noreply.github.com>

temp: use branch for feature of lock in nomad

25be047

fix: update deps and add extra config values

369b7af

Juanadelacuesta force-pushed the f-gh-17449 branch from fa03aa9 to 369b7af Compare September 13, 2023 18:54

Juanadelacuesta added 2 commits September 13, 2023 15:59

style: fix the enable

fbae869

func: add ha flags to tests and ttl and delay string parsing

1b27e02

Juanadelacuesta requested a review from jrasell September 14, 2023 17:10

func: add tests for ha flags

c001580

jrasell reviewed Sep 15, 2023

View reviewed changes

agent/config/config.go Outdated Show resolved Hide resolved

jrasell reviewed Sep 15, 2023

View reviewed changes

command/agent.go Show resolved Hide resolved

jrasell approved these changes Sep 15, 2023

View reviewed changes

func: add ttl and delay to flags docs and change the defaults to 60 a…

53ce81c

…nd 30 s respectively

Juanadelacuesta merged commit edb3175 into main Sep 15, 2023
23 checks passed

Juanadelacuesta deleted the f-gh-17449 branch September 15, 2023 17:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Leader election using locks #649

Leader election using locks #649

Juanadelacuesta commented Jun 28, 2023 •

edited

lgfa29 left a comment

jrasell left a comment

jrasell left a comment

Leader election using locks #649

Leader election using locks #649

Conversation

Juanadelacuesta commented Jun 28, 2023 • edited

lgfa29 left a comment

Choose a reason for hiding this comment

jrasell left a comment

Choose a reason for hiding this comment

jrasell left a comment

Choose a reason for hiding this comment

Juanadelacuesta commented Jun 28, 2023 •

edited