Apparently when using the flask test app, exceptions thrown in the client do not result in 500 status codes as they do when running flask properly. Instead the exceptions propagate back up to check50 which results in the "check50 ran into an error running checks" message.