Initial implementation of StatusStream #917

mwildehahn · 2016-07-08T00:53:27Z

Split out setting up events from: #916

Calling these "Scheduler Events" was too confusing given that we also have "Empire Events" that publish to various sources. I went with calling this a "StatusStream" that can be created and passed to an action (like Deploy) and can be published and read from throughout the lifecycle of the action.

I ended up creating a separate package status because otherwise we would have had to import the scheduler within deployments.go which felt off to me.

ejholmes · 2016-07-08T00:57:50Z

deployments.go

@@ -78,5 +80,22 @@ func (s *deployerService) Deploy(ctx context.Context, db *gorm.DB, opts DeployOp
 		return r, err
 	}

+	if s, ok := opts.Updates.(status.SubscribableStream); ok {
+		for update := range s.Subscribe() {
+			msg := fmt.Sprintf("Status: %s", update.String())


I think this will already get prefixed with Status: when the jsonmessage is written.

Scratch that, maybe not.

ejholmes · 2016-07-08T01:31:34Z

I ended up creating a separate package status because otherwise we would have had to import the scheduler within deployments.go which felt off to me.

remind101/empire already depends on remind101/empire/scheduler, so it would be fine to put StatusStream within remind101/empire/scheduler. I actually think that would be better until there's a reason to extract it.

ejholmes · 2016-07-08T01:32:59Z

This is looking great so far.

mwildehahn · 2016-07-08T01:34:41Z

i'm fine moving within remind101/empire/scheduler. just to confirm, that is just moving remind101/empire/status/status.go to remind101/empire/scheduler/status.go or should we just include everything within remind101/empire/scheduler/scheduler.go?

ejholmes · 2016-07-08T01:35:53Z

I think everything in remind101/empire/scheduler/scheduler.go would be best.

mwildehahn · 2016-07-08T01:42:55Z

ok cool

ejholmes · 2016-07-08T02:05:01Z

status/status.go

+	Done(error)
+}
+
+type SubscribableStream interface {


Just food for thought. Another possibility here would be to forgo channels entirely and make the interface:

// StatusStream is an interface for publishing status updates while executing // an Empire action. type StatusStream interface { // Publish publishes an update to the status stream Publish(Status) error // Done finalizes the status stream Done(error) // Wait returns a channel that that receives once Done() is called. // Consumers should call the Err() method to determine if an error // ocurred. Wait() <-chan struct{} // Returns the error from calling Done(). Err() error }

Then we just pass in an implementation that writes to the logstream:

type jsonmessageStatusStream struct { sync.Mutex done chan struct{} err error w io.Writer } func (s *jsonmessageStream) Publish(message scheduler.Status) { select { case <-done: panic("Publish called on finalized stream") default: } json.NewEncoder(s.w).Encode(jsonmessage.Message{Status: status.Message}) } func (s *jsonmessageStream) Done(err error) { close(s.done) s.err = err } func (s *jsonmessageStream) Err() error { return s.err } func (s *jsonmessageStream) Wait() <-chan struct{} { return s.done }

Main advantage would be that we don't need to worry about buffering the channel, calls to Publish will immediately write to the stream and it's easy to wrap with middleware (e.g. writing to the apps kinesis stream, etc).

i like that.

another thing i'm coming across is having to do: https://github.com/remind101/empire/pull/917/files#diff-48d76aef8c283046a79db32824933290R523.

not the end of the world, but definitely annoying.

i was thinking of adding an active flag within the stream, if it never gets activated, then the subscriber never blocks. there could either be a specific activate method or as soon as you publish the stream would get activated. but that defeats the purpose of being able to write to the stream at any point within a go routine and having the client subscribe.

also, i'm not a fan of throwing: panic("Publish called on finalized stream"). given these are just status updates, it feels aggressive to throw a panic. what would be the proper way to warn instead? i couldn't find any examples in the code.

also, i'm not a fan of throwing: panic("Publish called on finalized stream"). given these are just status updates, it feels aggressive to throw a panic. what would be the proper way to warn instead? i couldn't find any examples in the code.

Agreed. If we change the interface to Publish(context.Context, Status), then you can do a logger.Warn(ctx, "message").

This allows us to subscribe to the StatusStream without blocking concurrent deploys.

mwildehahn · 2016-07-08T05:22:46Z

ok, i implemented the jsonmessageStatusStream as well as the other pieces of feedback.

the only part i'm not happy with is having to do: https://github.com/remind101/empire/pull/917/files#diff-48d76aef8c283046a79db32824933290R523

ejholmes · 2016-07-08T07:01:39Z

deployments.go

 	var msg jsonmessage.JSONMessage

-	r, err := s.deploy(ctx, db, opts)
+	tx := s.db.Begin()


I think we'd also wanna rollback here. Might be easier to wrap the transaction logic in a separate method:

func (s *deployerService) deployInTransaction(ctx context.Context, stream scheduler.StatusStream, opts De tx := s.db.Begin() r, err := s.deploy(ctx, tx, stream, opts) if err != nil { tx.Rollback() return r, err } if err := tx.Commit().Error; err != nil { return r, err } return r, nil }

i was rolling back here: https://github.com/remind101/empire/pull/917/files#diff-ff47559c0383dd1959cdbe50cd2c036dR76 which is roughly equivalent, but +1 on the separation, i was thinking the same

ejholmes · 2016-07-08T07:43:04Z

I just hacked together a couple changes in the CloudFormation backend to use this. So awesome seeing more context in the deployment stream:

https://asciinema.org/a/b6vnccpg2u2uykf9wyyb1ijej

This looks gtm 👍. We can make changes to Scheduler implementations to use this in a separate PR (or if you wanna just open PR's against this branch, that's cool too).

phobologic · 2016-07-08T17:26:08Z

ejholmes · 2016-07-08T22:24:33Z

scheduler/scheduler.go

+
+// StatusStream is an interface for publishing status updates while a scheduler
+// is executing.
+type StatusStream interface {


I was thinking some more about this last night and I think there's one more small change we might wanna do that will make this much simpler. We can give semantic meaning to when the StatusStream is nil, and change the signature of scheduler.Submit to:

// When StatusStream is nil, Submit should return as quickly as possible, // usually when the new version has been received, and validated. If // StatusStream is not nil, it's recommended that the method not return until // the deployment has fully completed. scheduler.Submit(context.Context, *App, StatusStream) error

This would mean, that we can remove the Done, Wait, and Err methods on the StatusStream interface so it's just Publish. For the CloudFormation backend, we'd just change this line to be if ss != nil.

The primary advantage will be in cases like the migration scheduler where we'd call Submit on two different schedulers, that might both call Done.

i like this. i think it helps differentiate EventStream as async empire notifications and StatusStream as synchronous notifications during the execution of an action.

…d wait

ejholmes · 2016-07-11T23:23:24Z

Awesome 👍 (sorry for the churn :)).

mwildehahn · 2016-07-12T18:11:52Z

@ejholmes c6441b7 is actually the commit I was talking about, I forgot I had done it in a different branch.

Without this the db transaction surrounds both the creation and the submission of the release, which breaks concurrency.

This will enable the status stream for the command.

…ation stack

mwildehahn · 2016-07-12T22:36:40Z

@ejholmes ok i think this is good to go. i added an -s flag so we'll only setup the status stream if someone enables it. This means people can opt in to the new UX.

mwildehahn · 2016-07-12T22:49:32Z

Deploy now looks like:

$ ./build/emp deploy remind101/acme-inc:master -s
master: Pulling from remind101/acme-inc

420890c9e918: Already exists
a3ed95caeb02: Already exists
1e14aab0083d: Already exists
4393485f2bb3: Already exists
a2a051cf14f7: Already exists
1f9177248efa: Already exists
Digest: sha256:d05682347b56191a950f55ab06f11a6c6e34041921af32281b01f4a6a9a0f75f
Status: Image is up to date for remind101/acme-inc:master
Status: Created cloudformation template: https://empire-0b96756d-templatebucket-fzr5jqii49z4.s3.amazonaws.com/acme-inc/e904d241-a249-4b13-aa77-9dc0a3639617/57c5f02918c0fd1714d4689075924b42523c4fb6
Status: Stack update submitted
Status: Waiting for stack update to complete
Status: Stack update complete
Status: Finished processing events for release v9 of acme-inc

When a deploy has been superseded:

$ ./build/emp deploy remind101/acme-inc:master -s
master: Pulling from remind101/acme-inc

420890c9e918: Already exists
a3ed95caeb02: Already exists
1e14aab0083d: Already exists
4393485f2bb3: Already exists
a2a051cf14f7: Already exists
1f9177248efa: Already exists
Digest: sha256:d05682347b56191a950f55ab06f11a6c6e34041921af32281b01f4a6a9a0f75f
Status: Image is up to date for remind101/acme-inc:master
Status: Created cloudformation template: https://empire-0b96756d-templatebucket-fzr5jqii49z4.s3.amazonaws.com/acme-inc/86874600-74f0-4ee4-9517-51c13a3e7240/3c156706ec606aa39375c6dfd7182f2f8a1d0ee4
Status: Waiting for existing stack operation to complete
Status: Operation superseded by newer release
Status: Finished processing events for release v4 of acme-inc

While waiting for another deploy to finish:

$ ./build/emp deploy remind101/acme-inc:master -s
master: Pulling from remind101/acme-inc

420890c9e918: Already exists
a3ed95caeb02: Already exists
1e14aab0083d: Already exists
4393485f2bb3: Already exists
a2a051cf14f7: Already exists
1f9177248efa: Already exists
Digest: sha256:d05682347b56191a950f55ab06f11a6c6e34041921af32281b01f4a6a9a0f75f
Status: Image is up to date for remind101/acme-inc:master
Status: Created cloudformation template: https://empire-0b96756d-templatebucket-fzr5jqii49z4.s3.amazonaws.com/acme-inc/86874600-74f0-4ee4-9517-51c13a3e7240/d5e7525e4fb98ee49bd3aceceed7e726a9029e64
Status: Waiting for existing stack operation to complete
Status: Stack update submitted
Status: Waiting for stack update to complete
Status: Stack update complete
Status: Finished processing events for release v5 of acme-inc

Create looks like:

$ ./build/emp deploy remind101/acme-inc:master -s
master: Pulling from remind101/acme-inc

420890c9e918: Already exists
a3ed95caeb02: Already exists
1e14aab0083d: Already exists
4393485f2bb3: Already exists
a2a051cf14f7: Already exists
1f9177248efa: Already exists
Digest: sha256:d05682347b56191a950f55ab06f11a6c6e34041921af32281b01f4a6a9a0f75f
Status: Image is up to date for remind101/acme-inc:master
Status: Created cloudformation template: https://empire-0b96756d-templatebucket-fzr5jqii49z4.s3.amazonaws.com/acme-inc/04a75f65-c7c4-4e36-943b-0a09614db1b4/67f96646de01688a45cc9d8a5f25ff9cb1ce479c
Status: Creating stack
Status: Stack created
Status: Finished processing events for release v2 of acme-inc

I wanted to get the actual cloudformation url, but i didn't see that we had easy access to the stack arn.

ejholmes · 2016-07-12T23:08:27Z

scheduler/cloudformation/cloudformation.go

@@ -208,6 +208,10 @@ func (s *Scheduler) submit(ctx context.Context, tx *sql.Tx, app *scheduler.App,
 		return err
 	}

+	if err := scheduler.Publish(ss, fmt.Sprintf("Created cloudformation template: %v", *t.URL)); err != nil {


Part of me feels like we should just ignore errors from Publish. If for example, the emp client disconnected from the network, this would result in err == io.EOF and would cause the deployment to fail.

yea i was thinking the same thing initially but also didn't want to just swallow errors. maybe we just log these?

sounds good

ejholmes · 2016-07-12T23:15:13Z

We should also add Stream: true to

empire/server/github/deployer.go

Lines 59 to 63 in 0f2cc1a

    
           _, err = d.empire.Deploy(ctx, empire.DeployOpts{ 
        
           	Image:  img, 
        
           	Output: p, 
        
           	User:   &empire.User{Name: event.Deployment.Creator.Login}, 
        
           })

.

ejholmes · 2016-07-12T23:17:31Z

releases.go

-
-	// Schedule the new release onto the cluster.
-	return r, s.Release(ctx, r)
+	return r, err


Minor, you can just return releasesCreate(db, r).

ejholmes · 2016-07-12T23:37:53Z

Couple of minor things, which we can follow up with later. This is awesome 👍.

ejholmes · 2016-07-13T00:24:18Z

Oh, can you update the changelog too?

We'll log these as warnings but we don't want it to disrupt a deploy

Initial implementation of StatusStream

f3afbd7

ejholmes reviewed Jul 8, 2016
View reviewed changes

Finalize the stream after submit

489b3dc

mwildehahn added 2 commits July 7, 2016 18:51

Remove StatusStream from DeployOpts

7d001a7

Move StatusStream within scheduler

7d1bb50

ejholmes reviewed Jul 8, 2016
View reviewed changes

mwildehahn added 5 commits July 7, 2016 19:09

Finalize the stream if passed

5c054ff

Handle errors before subscribing to the stream

f0013cf

Move transaction logic within the deployerService

e6afeea

This allows us to subscribe to the StatusStream without blocking concurrent deploys.

Warn instead of panicing

1b13ea2

Implement jsonmessageStatusStream

039659a

mwildehahn force-pushed the status-stream branch from 3b4b2ce to 039659a Compare July 8, 2016 05:20

ejholmes reviewed Jul 8, 2016
View reviewed changes

mwildehahn added 2 commits July 9, 2016 13:57

Simplify the interface for StatusStream

16de76a

Use the presence of a StatusStream as an indicator the function shoul…

c35af63

…d wait

mwildehahn added 2 commits July 12, 2016 11:09

Merge branch 'master' into status-stream

5ef2290

Only wrap the release creation in a transaction

c6441b7

mwildehahn added 2 commits July 12, 2016 15:00

Add -s flag to deploy command

1bfd746

This will enable the status stream for the command.

Publish status updates while the scheduler is deploying the cloudform…

6b00ba3

…ation stack

Publish statuses around creating a stack

5d983d9

ejholmes reviewed Jul 12, 2016
View reviewed changes

Ignore errors if we fail to publish

93c8d31

We'll log these as warnings but we don't want it to disrupt a deploy

mwildehahn mentioned this pull request Jul 13, 2016

Support streaming CF events #922

Open

mwildehahn added 4 commits July 12, 2016 17:43

Enable the stream for github deployments

c31b7e1

Simply return releasesCreate

2a99374

Always log a message when we create a release

2101c9b

Update CHANGELOG

dca56bb

mwildehahn mentioned this pull request Jul 13, 2016

Notification when deploy finishes #888

Closed

mwildehahn added 3 commits July 12, 2016 18:31

Update tests

6ca0536

Don't swallow error when writing it to Output

3e348ea

Update tests

b7a854b

mwildehahn merged commit d240a32 into remind101:master Jul 13, 2016

mwildehahn deleted the status-stream branch July 13, 2016 01:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial implementation of StatusStream #917

Initial implementation of StatusStream #917

mwildehahn commented Jul 8, 2016

ejholmes Jul 8, 2016

ejholmes Jul 8, 2016

ejholmes commented Jul 8, 2016

ejholmes commented Jul 8, 2016

mwildehahn commented Jul 8, 2016

ejholmes commented Jul 8, 2016

mwildehahn commented Jul 8, 2016

ejholmes Jul 8, 2016

mwildehahn Jul 8, 2016

ejholmes Jul 8, 2016 •

edited

Loading

mwildehahn Jul 8, 2016

mwildehahn commented Jul 8, 2016

ejholmes Jul 8, 2016

mwildehahn Jul 9, 2016

ejholmes commented Jul 8, 2016 •

edited

Loading

phobologic commented Jul 8, 2016

ejholmes Jul 8, 2016 •

edited

Loading

mwildehahn Jul 9, 2016

ejholmes commented Jul 11, 2016

mwildehahn commented Jul 12, 2016

mwildehahn commented Jul 12, 2016

mwildehahn commented Jul 12, 2016

ejholmes Jul 12, 2016

mwildehahn Jul 12, 2016

ejholmes Jul 12, 2016

ejholmes commented Jul 12, 2016

ejholmes Jul 12, 2016

ejholmes commented Jul 12, 2016

ejholmes commented Jul 13, 2016

Initial implementation of StatusStream #917

Initial implementation of StatusStream #917

Conversation

mwildehahn commented Jul 8, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ejholmes commented Jul 8, 2016

ejholmes commented Jul 8, 2016

mwildehahn commented Jul 8, 2016

ejholmes commented Jul 8, 2016

mwildehahn commented Jul 8, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ejholmes Jul 8, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwildehahn commented Jul 8, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ejholmes commented Jul 8, 2016 • edited Loading

phobologic commented Jul 8, 2016

ejholmes Jul 8, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ejholmes commented Jul 11, 2016

mwildehahn commented Jul 12, 2016

mwildehahn commented Jul 12, 2016

mwildehahn commented Jul 12, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ejholmes commented Jul 12, 2016

Choose a reason for hiding this comment

ejholmes commented Jul 12, 2016

ejholmes commented Jul 13, 2016

ejholmes Jul 8, 2016 •

edited

Loading

ejholmes commented Jul 8, 2016 •

edited

Loading

ejholmes Jul 8, 2016 •

edited

Loading