Configure heartbeat requests to go straight to API #473

pwnbus · 2018-11-09T16:55:03Z

This bypasses rabbitmq entirely for heartbeats. The agents will now post heartbeats to the API directly, which writes to postgres.

… from agents

…gents

…handler to enforce decoupling

…ions to use

…oad results

…ns of authorizing agents to upload heartbeats

…are authorized to upload heartbeats before saving them

… explicity

database/actions.go

database/schema.sql

mig-scheduler/agents_queues.go

mig-api/api.go

…dn't be a cause for failure

… reference

…agents in place when we deal with a heartbeat

…the rest of what was in the heartbeat

arcrose

I like how this looks at this point. A couple notes for reviewers who follow:

Heartbeat type is redefined for both the agent and API to decouple them from one another.
Opted not to re-implement authentication for heartbeats due to the low efficacy of the current approach.

Configure heartbeat requests to go straight to API

Zack Mullaly and others added 30 commits October 3, 2018 18:24

Documented a new API endpoint that agents can use to send heartbeats

bdc725f

Format fixup

19b904e

Really working on that formatting

88453e0

Create a markdown version of the API doc so we can format it better

b949da0

First stab at implementing a new API endpoint for handling heartbeats…

8516a37

… from agents

Started working on unit tests for new endpoint

dfecb6c

Removed unused variable

6fa81f8

Forgot to write a response in the success case

ee02d43

Do some validation on heartbeat data

7c6cccf

Fixed typo in JSON identifier

6070a2a

Documenting a new API endpoint that will handle result uploads from a…

1cb54ac

…gents

Writing a first implementation of the 'results upload' endpoint handler

8de90e1

Provide an abstraction over the modules.Result type for our endpoint …

3c87e7a

…handler to enforce decoupling

Fixed up some typos

e63d0fb

Wrote up some unit tests for new endpoint

8357bdb

Fixed a typo

2c009eb

Built out a more expressive error type for PersistResults implementat…

2cc13e1

…ions to use

Document that the API may inform us when we are not authorized to upl…

114cd67

…oad results

Fixed up tests so they fail in the right place

bbc5263

Started scaffolding for postgres wrapper

c948ed9

Adding an Authenticator interface we can implement with different mea…

fd29a91

…ns of authorizing agents to upload heartbeats

Format fix

df6d276

Typo fixed in method name

a51cf40

Verify that heartbeat messages contain sensible data and that agents …

fbabdda

…are authorized to upload heartbeats before saving them

Format fix

5e1a095

Document a new endpoint that we can use to retrieve actions

4e6637e

Agents will need to fetch an array of actions to run

f0324e9

We don't already have a way of authenticating requests to get actions…

e4485fd

… explicity

A first implementation of a request handler to list actions

da76299

Wrote unit tests for and fixed up actions list endpoint

605d1cb

Zack Mullaly and others added 4 commits November 6, 2018 15:20

removed tests for code we wont be using

4a53c96

Update docker exec command to use postgres name

41b976e

Add agent config for integration tests

dd57cce

Modify agent to post heartbeat to api

99fd4ee

jvehent reviewed Nov 9, 2018

View reviewed changes

database/actions.go Outdated Show resolved Hide resolved

pwnbus commented Nov 9, 2018

View reviewed changes

database/schema.sql Outdated Show resolved Hide resolved

pwnbus commented Nov 12, 2018

View reviewed changes

mig-scheduler/agents_queues.go Outdated Show resolved Hide resolved

pwnbus commented Nov 12, 2018

View reviewed changes

mig-api/api.go Show resolved Hide resolved

Zack Mullaly and others added 17 commits November 13, 2018 14:57

Merge branch 'master' into abstract-rabbit

91470ba

No longer have need for the relation between agents and actions

76a9624

test should look for >= 1 row because running it multiple times shoul…

61d886e

…dn't be a cause for failure

Added some fields that were missing from the heartbeat translation

5b8ca35

Merge branch 'master' into abstract-rabbit

41f46a1

Merge branch 'master' into abstract-rabbit

1e58cac

Adding the code from the scheduler that processed heartbeats back for…

262f2c4

… reference

Update agent information in place when a heartbeat is received

12392b0

Don't try to insert into table that no longer exists

d70bb40

Only insert a new agent when we find out about one. Otherwise update …

98d4eac

…agents in place when we deal with a heartbeat

Fixed imports

4dc4cb3

Add back in the information we accidentally threw away by neglecting …

c98623e

…the rest of what was in the heartbeat

Readd agent refresh ts for heartbeats

1132b4b

Add update role to agents table for migapi

9b32058

removing function that isn't being used yet

49476d6

Removing documentation that doesn't belong in this branch

ad7aee2

Removing unnecessary blank lines

5b386c1

arcrose approved these changes Nov 22, 2018

View reviewed changes

Revert scheduler back to normal operations so we can do a full deploy

deb4aba

arcrose merged commit e068d93 into master Nov 26, 2018

arcrose deleted the abstract-rabbit branch November 26, 2018 20:52

arcrose added a commit that referenced this pull request Jul 9, 2019

Merge pull request #473 from mozilla/abstract-rabbit

e51e983

Configure heartbeat requests to go straight to API

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure heartbeat requests to go straight to API #473

Configure heartbeat requests to go straight to API #473

pwnbus commented Nov 9, 2018

arcrose left a comment

Configure heartbeat requests to go straight to API #473

Configure heartbeat requests to go straight to API #473

Conversation

pwnbus commented Nov 9, 2018

arcrose left a comment

Choose a reason for hiding this comment