New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maintaining Your Test Regression Suite As Your Application Grows #7

Closed
johnathanbeal opened this Issue Oct 2, 2018 · 1 comment

Comments

2 participants
@johnathanbeal
Copy link

johnathanbeal commented Oct 2, 2018

As your application grows in the number of features it supports, how do you manage & maintain the large number of regression tests needed to verify that those features are working?

@SeanKilleen

This comment has been minimized.

Copy link
Owner

SeanKilleen commented Nov 21, 2018

Hi @johnathanbeal! Thanks for reaching out, and I'm sorry it took me so long to get back to you.

I think there are a few ways to maintain your test structures & number of regression tests as your application grows. I'll try to summarize below; feel free to follow up with additional questions or go deeper on those subjects.

  • Organization: Organization is key. In my experience it's best to have a test structure that mimics your production code structure as much as possible. This way, if I see a piece of production code, chances are I can find the unit tests for that code.
  • The testing pyramid: The regression tests are a combination of all automated tests you have -- they, collectively, are what prevent regression. Here is where the testing pyramid can help guide us. Your goal should generally be to have a large number of unit tests -- these can run very quickly and can go a long way to verifying that the individual units of code work as the developers intend them to. On top of those unit tests, it's helpful to have integration tests -- tests that use one or more real pieces of the code together. Beyond those, it's helpful to have functional tests, written in the language of the business. This way you also identify that the system is working as your users intend it to (you're using real-world examples from the perspective of the user here).
  • A good test runner: Having good development environment tooling that makes running your tests as easy as humanly possible is a good thing. For .NET, I love NCrunch because it is a super-fast continuous testing tool. It automatically runs all the tests affected by my code change, in real time, as I type. It makes doing TDD & coding feel like an amazing flow. That's a really big help for me psychologically when maintaining a test suite.
  • A Categorization of your test "buckets": While your unit tests should all take 1 second or less to run, generally, integration tests, functional tests, and UI tests can take much longer. Luckily, most test tools support adding categories to tests. This way, you can tag your tests with attributes like [Category("Long-Running")], [Category("UsesDatabase")], etc. I tend to suggest marking long-running tests (except for whole projects like UI tests where I assume every one is long running). I also try to mark recently failed tests, so that our CI system can run them more often. (more on this in a second).
  • A Continuous Integration system: You need something to help you out running these tests. It's good to run as many tests as possible as often as possible, so I suggest running your unit tests with every pull request or check-in. For longer-running tests, run them every so often (3 hours, 6 hours, every night at 3am -- whatever you can get away with). Say we run tests every 3 hours. If it's expensive to do that, I can then set those tests to run every 6 hours, but when one fails, I can move it up so that it's run much more often, to ensure that it gets back to stability.
  • Do the hard things more often: Find the pain points in writing your tests; the things that make you want to not write or run tests. Focus on doing those things often, both to get a muscle memory and to look at the problem many different times. You'll find good patterns; I can recommend xUnit Test Patterns as a book that can help here.

Let me know if you have any follow up questions, because I know this is kind of a broad answer.

Thanks for writing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment