New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create an automated test suite for use against production deployments #1199

Closed
toolness opened this Issue Jan 4, 2017 · 15 comments

Comments

Projects
None yet
3 participants
@toolness
Contributor

toolness commented Jan 4, 2017

We've had a bunch of annoying regressions when deploying to calc.gsa.gov (instead of e.g. calc-dev.app.cloud.gov, or even calc-prod.app.cloud.gov).

It might be useful for us to create a simple automated test suite that we can run from our local dev machines against the prod deployment to make sure that regressions don't occur. For instance:

  • Add a field to the /healthcheck endpoint that includes the results of request.build_absolute_uri(), which the suite can use to verify that the server is at the hostname it thinks it's at (to guard against #1187).

  • Issue a csrf-protected POST to the server (one that isn't hidden behind auth, such as https://calc.gsa.gov/styleguide/radio-checkbox) and make sure it works (to guard against #1198).

  • Ensure the X-Requested-With header is being passed through to our origin server. Otherwise ajaxforms in particular will explode.

  • Ensure our API is delivering the proper CORS headers (#1307)

It might also be useful to surface this suite, or at least the kinds of things it's testing for, back to 18F as a whole, since it's likely that other cloud.gov-based projects will experience the same difficulties as us!

@toolness

This comment has been minimized.

Contributor

toolness commented Jan 4, 2017

@jseppi were there any other weird hiccups that happened when moving to calc.gsa.gov that you think we could guard against future regressions of?

@jseppi

This comment has been minimized.

Contributor

jseppi commented Jan 4, 2017

Haven't found any others yet...

@toolness

This comment has been minimized.

Contributor

toolness commented Jan 4, 2017

Okydoky.

Is it possible for a service like NewRelic to regularly issue a test like this against production CALC? It doesn't seem like the sort of thing we should do with a Travis cron job, but it'd be nice to have a test like this constantly running, I guess.

@jseppi

This comment has been minimized.

Contributor

jseppi commented Jan 5, 2017

I don't have any experience working with this feature, but New Relic apparently has some kind of "scripted monitoring" functionality: https://docs.newrelic.com/docs/synthetics/new-relic-synthetics/scripting-monitors

@toolness

This comment has been minimized.

Contributor

toolness commented Jan 5, 2017

Cool! I'm a bit confused by some of their documentation, but at a basic level it seems like their API tests are nodeJS scripts that use the request library (which their docs keep calling http-request for some reason).

So maybe I should just write up these tests as simple node scripts that use request, so we can execute them either from our own machines or via newrelic?

@jseppi

This comment has been minimized.

Contributor

jseppi commented Jan 5, 2017

Sounds good!

@toolness

This comment has been minimized.

Contributor

toolness commented Jan 6, 2017

Okay. For the sake of compatibility, since I have no idea what version of node they're using, maybe I should even use es5? Or I guess we could run it through a transpiler when we give it to newrelic...

@toolness

This comment has been minimized.

Contributor

toolness commented Jan 6, 2017

I guess it might be nice to also add a test to make sure the api umbrella host thingy works, i guess. blerg.

@toolness

This comment has been minimized.

Contributor

toolness commented Jan 9, 2017

Well, it seems like we only have New Relic Synthetics Lite at the moment, which only gives us the ability to do Ping checks--basically restricting us to pinging endpoints and making sure they return a 200-ish response, optionally validating that a string of text exists in the response. Upgrading to Synthetics Pro, which would let us run Selenium/WebDriver tests on the server, as well as those nodeJS scripts that use request, costs an extra $70 per month. 😞

@toolness

This comment has been minimized.

Contributor

toolness commented Jan 9, 2017

The request.build_absolute_uri() one could definitely be done as a Ping check. Not sure if there will be a way to easily guard against #1198 through a Ping check. There might be a way to make sure the API umbrella thing works, by issuing a simple Ping against it with a valid API key?

@jseppi

This comment has been minimized.

Contributor

jseppi commented Jan 9, 2017

We have a ping monitor against the api proxy already: https://synthetics.newrelic.com/accounts/987071/monitors/1e423fc5-3039-49e7-9898-ba59f89c2e14

For #1198 - yeah, doesn't look like there is any way to POST :(

$70 doesn't seem like it would break the bank...

@toolness

This comment has been minimized.

Contributor

toolness commented Jan 19, 2017

Another thing I just realized would be good to test: we should ensure that the X-Requested-With header is being passed through to our origin server. Otherwise ajaxforms in particular will explode.

@jseppi

This comment has been minimized.

Contributor

jseppi commented Jan 19, 2017

Yeah, would be good to test for it. I'm sure it is being passed through because the R10 upload definitely worked on prod the other day.

@toolness

This comment has been minimized.

Contributor

toolness commented Jan 19, 2017

Noice!!

Also, um, I just remembered that AWS lambda exists. If it's possible to schedule lambda jobs using our existing rq_scheduler process, it might be worth it to just do the automated testing that way, since it decouples us from the weird restrictions of the New Relic synthetics and might end up being cheaper. I dunno.

@jseppi

This comment has been minimized.

Contributor

jseppi commented Jan 19, 2017

I'm pretty sure we don't have access to Lambda through cloud.gov unfortunately

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment