ci: loadtest with different JWTs

[steve-chavez]

Problem

We need to run a loadtest similar to https://github.com/PostgREST/postgrest-benchmark/blob/master/k6/GETSingleDifferentJWT.js in our CI, to test the performance of JWT decoding coupled with the JWT cache.

Currently I run the above manually on every PR (example), and this takes a while and it gets repetitive.

Solution

• Generate a jwt-targets.http file, contents should be like:

  GET http://localhost:3000/authors_only
  Authorization: Bearer <JWT>
  GET http://localhost:3000/authors_only
  Authorization: Bearer <anotherJWT>

  • This can be done with Haskell or another language. A bash script might be too slow.

• Run vegeta with this jwt-targets.http file

• Put this on a nix script like the following and use it on CI

  postgrest/nix/tools/loadtest.nix

  Lines 36 to 48 in d89e7a2

  	loadtest =
  	checkedShellScript
  	{
  	name = "postgrest-loadtest";
  	docs = "Run the vegeta loadtests with PostgREST.";
  	args = [
  	"ARG_OPTIONAL_SINGLE([output], [o], [Filename to dump json output to], [./loadtest/result.bin])"
  	"ARG_OPTIONAL_SINGLE([testdir], [t], [Directory to load tests and fixtures from], [./test/load])"
  	"ARG_LEFTOVERS([additional vegeta arguments])"
  	];
  	workingDir = "/";
  	}
  	''

• This result show show on the "Summary" part of CI actions.

[taimoorzaeem]

This can be done with Haskell or another language. A bash script might be too slow.

Python seems good for this task as we already use it in our CI. It would also be easier to port the javascript code to python.

[steve-chavez]

Agree, python is a good choice.

[wolfgangwalther]

This is a very specific test / scenario. While I agree it makes sense to add a tool which makes it easy to run it locally in the postgrest repo, I don't think we necessarily need to add this to CI, though.

[taimoorzaeem]

Maybe we can run this locally with k6? It might give different results locally depending on the system. I think CI would make the results consistent.

$ k6 run GETSingleDifferentJWT.js --vus 10 --duration 10s

[wolfgangwalther]

CI is not more or less consistent than local (although it tends to be less consistent). What makes the results any useful is a comparison. You always need to compare a local run with the change and without it.

This is what postgrest-loadtest-against does. So no matter how you do it, you must do a comparison of sorts.

[steve-chavez]

I don't see a problem in adding one more loadtest to CI, it's only beneficial for everyone to not have to do manual tests.

JWT decoding is a crucial part of PostgREST so the fact that we measure its speed and ensure it doesn't drop down (if some change in the JWT lib happens for example) is a big benefit, regardless of the JWT cache feature.