New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use factories and not fixtures #11

Open
andreareginato opened this Issue Oct 3, 2012 · 37 comments

Comments

Projects
None yet
@andreareginato
Collaborator

andreareginato commented Oct 3, 2012

Write your thoughts about the "use factories and not fixtures" best practice.

@adimichele

This comment has been minimized.

Show comment
Hide comment
@adimichele

adimichele Oct 3, 2012

I think you should check out fixture_builder and reconsider this one. The basic idea is to auto-generate fixtures (using factories or whatever) at the beginning of the test suite and only when necessary. I've gotten huge test performance improvements using this method with a transaction-based db clean strategy.

adimichele commented Oct 3, 2012

I think you should check out fixture_builder and reconsider this one. The basic idea is to auto-generate fixtures (using factories or whatever) at the beginning of the test suite and only when necessary. I've gotten huge test performance improvements using this method with a transaction-based db clean strategy.

@myronmarston

This comment has been minimized.

Show comment
Hide comment
@myronmarston

myronmarston Oct 3, 2012

I'm not a fan of the binary good/bad statement here. There are good reasons not to use factories.

Really, the best practice is to use neither as much as possible: instead put as much of your domain logic in PORO that can be tested without needing complex, time consuming setup with either factories or fixtures. You'll need some tests that use DB records of course, and either factories or fixtures are valid tools to use there.

myronmarston commented Oct 3, 2012

I'm not a fan of the binary good/bad statement here. There are good reasons not to use factories.

Really, the best practice is to use neither as much as possible: instead put as much of your domain logic in PORO that can be tested without needing complex, time consuming setup with either factories or fixtures. You'll need some tests that use DB records of course, and either factories or fixtures are valid tools to use there.

@Spaceghost

This comment has been minimized.

Show comment
Hide comment
@Spaceghost

Spaceghost Oct 3, 2012

I don't use either in unit tests. I tend to just build objects using the actual classes. I've used fixtures, and factories. My preferential factory gem is Fabrication. I prefer the speed, clarity, and simple communication of just using the objects I need directly in unit tests.

If I'm doing functional testing, I'll definitely mix in some fabrication. The ease of using a factory in that case is apparent.

Spaceghost commented Oct 3, 2012

I don't use either in unit tests. I tend to just build objects using the actual classes. I've used fixtures, and factories. My preferential factory gem is Fabrication. I prefer the speed, clarity, and simple communication of just using the objects I need directly in unit tests.

If I'm doing functional testing, I'll definitely mix in some fabrication. The ease of using a factory in that case is apparent.

@robsaunders

This comment has been minimized.

Show comment
Hide comment
@robsaunders

robsaunders Oct 4, 2012

One caveat is to watch out you don't build up complex associations in your factory files - as this gives them the same downsides as fixtures. Structure should always be built up within the test itself.

Having complex structure in factory objects to DRY your tests obscures the object structure, adds unnecessary associations for tests that don't use them, and creates problems when you need a test which changes one small association or value, and all your other tests fail. It's an example of bad DRY.

robsaunders commented Oct 4, 2012

One caveat is to watch out you don't build up complex associations in your factory files - as this gives them the same downsides as fixtures. Structure should always be built up within the test itself.

Having complex structure in factory objects to DRY your tests obscures the object structure, adds unnecessary associations for tests that don't use them, and creates problems when you need a test which changes one small association or value, and all your other tests fail. It's an example of bad DRY.

@UncleGene

This comment has been minimized.

Show comment
Hide comment
@UncleGene

UncleGene Oct 4, 2012

While talking about fixtures, bad example shows plain object creation, not fixture. As comparison of fixtures and factories may be controversial, plain creation vs. factories has no contenders. Perhaps it makes sense to rename?

UncleGene commented Oct 4, 2012

While talking about fixtures, bad example shows plain object creation, not fixture. As comparison of fixtures and factories may be controversial, plain creation vs. factories has no contenders. Perhaps it makes sense to rename?

@dgm

This comment has been minimized.

Show comment
Hide comment
@dgm

dgm Oct 7, 2012

Factories fall apart when your application logic requires complex and numerous relationships set up in order to test - it makes the tests SLOW. I think there is much to be gained by setting up fixtures for a base system that has most of the relationships in place, and then modifying that as needed for specific tests, whether with factories or basic object creation.

I don't know why everyone says fixtures are hard to maintain. When your data gets complex, factories are equally hard to maintain. When dealing with over 50 models (government reporting here, it cannot be less complex), you have to put them in different files, and using factories means yet another DSL syntax to learn. YAML isn't that hard.

What is really missing is better tools to help manage those fixture files. There should be an easy way to load up the test db fixtures, edit them, save, and re-run tests to make sure you didn't break anything. (You say that's brittle? The same thing is possible by editing factory definitions. You deal with it by running tests.)

I like FactoryGirl, but only as an addition to the standard fixtures, to create border cases, test invalid data, etc.

dgm commented Oct 7, 2012

Factories fall apart when your application logic requires complex and numerous relationships set up in order to test - it makes the tests SLOW. I think there is much to be gained by setting up fixtures for a base system that has most of the relationships in place, and then modifying that as needed for specific tests, whether with factories or basic object creation.

I don't know why everyone says fixtures are hard to maintain. When your data gets complex, factories are equally hard to maintain. When dealing with over 50 models (government reporting here, it cannot be less complex), you have to put them in different files, and using factories means yet another DSL syntax to learn. YAML isn't that hard.

What is really missing is better tools to help manage those fixture files. There should be an easy way to load up the test db fixtures, edit them, save, and re-run tests to make sure you didn't break anything. (You say that's brittle? The same thing is possible by editing factory definitions. You deal with it by running tests.)

I like FactoryGirl, but only as an addition to the standard fixtures, to create border cases, test invalid data, etc.

@deanius

This comment has been minimized.

Show comment
Hide comment
@deanius

deanius Oct 18, 2012

Where are the upvote options ?? dgm - amen. YAML files are just data - a business person can write them, a QA - anyone. Factories are executable code. Executable code is a poor serialization of an object graph. For my money, fixtures solve that problem far more elegantly than whatever Factory-Gem-Of-The-Day.

deanius commented Oct 18, 2012

Where are the upvote options ?? dgm - amen. YAML files are just data - a business person can write them, a QA - anyone. Factories are executable code. Executable code is a poor serialization of an object graph. For my money, fixtures solve that problem far more elegantly than whatever Factory-Gem-Of-The-Day.

@Spaceghost

This comment has been minimized.

Show comment
Hide comment
@Spaceghost

Spaceghost Oct 18, 2012

I, for one, love the concept of VCR. Perhaps a nice factory2fixture gem could be made? Something like that?

Spaceghost commented Oct 18, 2012

I, for one, love the concept of VCR. Perhaps a nice factory2fixture gem could be made? Something like that?

@pepe

This comment has been minimized.

Show comment
Hide comment
@pepe

pepe Oct 18, 2012

+1 @robsaunders @myronmarston

The trick is not to abuse anything, and for me abusing the fixtures is just little bit easier.

And when cucumbering, good factories are pure gold.

pepe commented Oct 18, 2012

+1 @robsaunders @myronmarston

The trick is not to abuse anything, and for me abusing the fixtures is just little bit easier.

And when cucumbering, good factories are pure gold.

@dgm

This comment has been minimized.

Show comment
Hide comment
@dgm

dgm Oct 18, 2012

I will concede that factories are nice for cucumber, but that demands to ask if cucumber is worth anything. In all my experience, it is just extra steps, because only programmers write it, and it just adds extra steps and extra knowledge to the process. Cucumber only makes sense if you have QA, managers, or clients writing the stories... and even then I suspect programmers have to tweak it. I can take the same conversation with a client, write the story in pseudocode, and then program rspec to do the same thing with capybara and save the extra steps of regex parsing.

As I said before, fixtures are great for the base system, when there are a lot of dependencies due to the complexity of the domain model. Let the fixtures define the basic setup - almost a seed like structure, but a little more. Then use factories/object creation to define the corner cases.

Properly named fixtures help define the components that you want to test for, and proper tests know how to exclude the extra data (which is something that you should be testing for!). If the presence of additional data is throwing off your test, or if changing a fixture or factory breaks tests, you need to evaluate those tests and set them up to not be so brittle.

For example, if a test currently checks that there are 4 comments to a post after completing an addition, and you add a comment in the fixtures or factories, of course that breaks the test. The test should instead query the number before the action, and then verify that the action added 1 comment. This insulates your tests from changes in the fixtures OR factories.

@robsaunders says: "Structure should always be built up within the test itself." Well, that is nice until you have 500 tests (only 500?) that all need to build up 30 relationships before each test can run. Personally, I hate waiting four an hour for tests to finish. Mocking and stubbing don't help at all, because it is the very nature of these relationships that need the most testing. Fixtures and transactional tests take a lot of pain out the equation. I can depend on the base system and add 1-3 objects per test and achieve a huge speed boost.

And again, what is truly missing is proper tools to manage those fixtures - not just a dump of the test database, but to facilitate proper naming of the various models and organize it. But in the end, it is just editing text files, and yaml is no more difficult to understand than factory girl's DSL; in fact, I think yaml is far easier to understand!

Coming back around to cucumber: If you have decently named factories, I think my project managers would better understand the fixtures, because they could say: (I'm going to alter the syntax a little, this isn't necessarily cucumber)

With child_x_with_abc_registration
Sign up for y_class at x_site
x_site should show registration for child_x in y_class

My project managers have a great memory for the existence of predefined test data like child_x and abc_registration. This would make a lot of sense to them and doesn't require factories. It also doesn't require cucumber. That little pseudocode can be converted to rspec that is readable by managers, even if they cannot create it directly.

For me, tests that run in a decent amount of time, and not wasting time on unecessary programming steps, is pure gold.

dgm commented Oct 18, 2012

I will concede that factories are nice for cucumber, but that demands to ask if cucumber is worth anything. In all my experience, it is just extra steps, because only programmers write it, and it just adds extra steps and extra knowledge to the process. Cucumber only makes sense if you have QA, managers, or clients writing the stories... and even then I suspect programmers have to tweak it. I can take the same conversation with a client, write the story in pseudocode, and then program rspec to do the same thing with capybara and save the extra steps of regex parsing.

As I said before, fixtures are great for the base system, when there are a lot of dependencies due to the complexity of the domain model. Let the fixtures define the basic setup - almost a seed like structure, but a little more. Then use factories/object creation to define the corner cases.

Properly named fixtures help define the components that you want to test for, and proper tests know how to exclude the extra data (which is something that you should be testing for!). If the presence of additional data is throwing off your test, or if changing a fixture or factory breaks tests, you need to evaluate those tests and set them up to not be so brittle.

For example, if a test currently checks that there are 4 comments to a post after completing an addition, and you add a comment in the fixtures or factories, of course that breaks the test. The test should instead query the number before the action, and then verify that the action added 1 comment. This insulates your tests from changes in the fixtures OR factories.

@robsaunders says: "Structure should always be built up within the test itself." Well, that is nice until you have 500 tests (only 500?) that all need to build up 30 relationships before each test can run. Personally, I hate waiting four an hour for tests to finish. Mocking and stubbing don't help at all, because it is the very nature of these relationships that need the most testing. Fixtures and transactional tests take a lot of pain out the equation. I can depend on the base system and add 1-3 objects per test and achieve a huge speed boost.

And again, what is truly missing is proper tools to manage those fixtures - not just a dump of the test database, but to facilitate proper naming of the various models and organize it. But in the end, it is just editing text files, and yaml is no more difficult to understand than factory girl's DSL; in fact, I think yaml is far easier to understand!

Coming back around to cucumber: If you have decently named factories, I think my project managers would better understand the fixtures, because they could say: (I'm going to alter the syntax a little, this isn't necessarily cucumber)

With child_x_with_abc_registration
Sign up for y_class at x_site
x_site should show registration for child_x in y_class

My project managers have a great memory for the existence of predefined test data like child_x and abc_registration. This would make a lot of sense to them and doesn't require factories. It also doesn't require cucumber. That little pseudocode can be converted to rspec that is readable by managers, even if they cannot create it directly.

For me, tests that run in a decent amount of time, and not wasting time on unecessary programming steps, is pure gold.

@mpalmer

This comment has been minimized.

Show comment
Hide comment
@mpalmer

mpalmer Mar 3, 2013

I'd like to see a definition of "difficult to control". I have no idea what that means.

mpalmer commented Mar 3, 2013

I'd like to see a definition of "difficult to control". I have no idea what that means.

@andreareginato

This comment has been minimized.

Show comment
Hide comment
@andreareginato

andreareginato Mar 9, 2013

Collaborator

I've added a section where I describe the fact that if possible we should avoid both fixtures and factories.
Any correction and comment to the updated guideline is appreciated.

Collaborator

andreareginato commented Mar 9, 2013

I've added a section where I describe the fact that if possible we should avoid both fixtures and factories.
Any correction and comment to the updated guideline is appreciated.

@mpalmer

This comment has been minimized.

Show comment
Hide comment
@mpalmer

mpalmer Mar 10, 2013

I've read that "why I don't like factory girl" article before, and it has never felt "whole" to me, because it doesn't give any concrete way to do it better. So in some ways, this point has gotten worse for me now. I still don't understand why fixtures are "difficult to control" -- they've never seemed particularly painful to me (although I will concede that may be Stockholm Syndrome talking). In addition to that, I'm now being pointed to an article that says that factories are a bad idea too, but doesn't give me any way for me to tell if I'm suffering from the same problems the author of that article is worried about, and even if I am, they don't give any suggestions for exactly how to solve those problems.

A solid, real-world-applicable explanation of what we should be doing instead of both factories and fixtures is sorely needed, I think. I don't know of any relevant articles on the subject, though. Perhaps it still needs to be written...

mpalmer commented Mar 10, 2013

I've read that "why I don't like factory girl" article before, and it has never felt "whole" to me, because it doesn't give any concrete way to do it better. So in some ways, this point has gotten worse for me now. I still don't understand why fixtures are "difficult to control" -- they've never seemed particularly painful to me (although I will concede that may be Stockholm Syndrome talking). In addition to that, I'm now being pointed to an article that says that factories are a bad idea too, but doesn't give me any way for me to tell if I'm suffering from the same problems the author of that article is worried about, and even if I am, they don't give any suggestions for exactly how to solve those problems.

A solid, real-world-applicable explanation of what we should be doing instead of both factories and fixtures is sorely needed, I think. I don't know of any relevant articles on the subject, though. Perhaps it still needs to be written...

@robsaunders

This comment has been minimized.

Show comment
Hide comment
@robsaunders

robsaunders Mar 11, 2013

It is possible to mix traits together to create a similar behaviour to fixture-factories, but having the dependency lie on the test-side rather than the fixture side. It is more limited and less flexible, but the overall force here is to do the right thing rather than the easy thing. When it comes down to it, putting structure into your factory creation is the equivalent of writing fixtures and you WILL end up with a messy pile of unmaintainable tests.

As a slight panacea you could, for example, use traits and do something like this:

create(:user, :with_posts)

It is difficult to then go ahead and create sub-associations (:with_comments_on_posts for example), but this limitation is a good thing - it prevents you from having crazy levels of sub-associations within your tests. And if you need sub associations, you write them in your test setup. This is in no way slower, in fact it's slower creating a pile of unused associative objects EVERY time you call your 'monster factory create' function.

@dgm if you have 500+ tests that use crazy levels of associations, you're doing it wrong. The bulk of your tests should be testing models (unit) or controllers (functional) where you really don't want to be testing a complete set of associations at once. The tests that you do have this kind of setup (integration tests), you should be confident that the associations work and really just be testing behaviour. It takes time to discover this, but the more complex your test setup, the more you end up testing nothing, because you can't account for the trillions of different edge cases and permutations that occur. Setting up tests this way also reduces the time it takes your suite to run (as you have fewer slow and bulky integration tests).

The complexity of the above paragraph probably requires giving a whole talk on the subject and there are a few good ones out there on this topic.

@mpalmer hopefully this answers your question - it's not a matter of finding a 'better way to do the wrong thing', it's a matter of understanding why building associations in a global way is bad, and how you can write more efficient, more specific tests.

robsaunders commented Mar 11, 2013

It is possible to mix traits together to create a similar behaviour to fixture-factories, but having the dependency lie on the test-side rather than the fixture side. It is more limited and less flexible, but the overall force here is to do the right thing rather than the easy thing. When it comes down to it, putting structure into your factory creation is the equivalent of writing fixtures and you WILL end up with a messy pile of unmaintainable tests.

As a slight panacea you could, for example, use traits and do something like this:

create(:user, :with_posts)

It is difficult to then go ahead and create sub-associations (:with_comments_on_posts for example), but this limitation is a good thing - it prevents you from having crazy levels of sub-associations within your tests. And if you need sub associations, you write them in your test setup. This is in no way slower, in fact it's slower creating a pile of unused associative objects EVERY time you call your 'monster factory create' function.

@dgm if you have 500+ tests that use crazy levels of associations, you're doing it wrong. The bulk of your tests should be testing models (unit) or controllers (functional) where you really don't want to be testing a complete set of associations at once. The tests that you do have this kind of setup (integration tests), you should be confident that the associations work and really just be testing behaviour. It takes time to discover this, but the more complex your test setup, the more you end up testing nothing, because you can't account for the trillions of different edge cases and permutations that occur. Setting up tests this way also reduces the time it takes your suite to run (as you have fewer slow and bulky integration tests).

The complexity of the above paragraph probably requires giving a whole talk on the subject and there are a few good ones out there on this topic.

@mpalmer hopefully this answers your question - it's not a matter of finding a 'better way to do the wrong thing', it's a matter of understanding why building associations in a global way is bad, and how you can write more efficient, more specific tests.

@warmwaffles

This comment has been minimized.

Show comment
Hide comment
@warmwaffles

warmwaffles Mar 11, 2013

@robsaunders I totally agree. I see it as a sign of test and code smell when I start seeing crazy levels of associations needing to be done in order to test one thing. Which a lot of times the solution is to stub the crap out of that model and make sure the boundaries of that method are right.

warmwaffles commented Mar 11, 2013

@robsaunders I totally agree. I see it as a sign of test and code smell when I start seeing crazy levels of associations needing to be done in order to test one thing. Which a lot of times the solution is to stub the crap out of that model and make sure the boundaries of that method are right.

@zamith

This comment has been minimized.

Show comment
Hide comment
@zamith

zamith Apr 18, 2013

I prefer to use build instead of create as much as possible, as not hitting the db speeds up tests greatly. If you test behaviour instead of state, this should not be a problem as you don't want to test ActiveRecord or DataMapper or whatever.

zamith commented Apr 18, 2013

I prefer to use build instead of create as much as possible, as not hitting the db speeds up tests greatly. If you test behaviour instead of state, this should not be a problem as you don't want to test ActiveRecord or DataMapper or whatever.

@andreareginato

This comment has been minimized.

Show comment
Hide comment
@andreareginato

andreareginato May 16, 2013

Collaborator

@mpalmer, @robsaunders would you like to send a pull request better describing this point? It feels like this is the weakest point in all betterspecs and I want to fix it.

Collaborator

andreareginato commented May 16, 2013

@mpalmer, @robsaunders would you like to send a pull request better describing this point? It feels like this is the weakest point in all betterspecs and I want to fix it.

@Spaceghost

This comment has been minimized.

Show comment
Hide comment
@Spaceghost

Spaceghost May 16, 2013

I disagree that one should only use fixtures.

There are many reasons why you'd use both, and it's a bit presumptuous to
lay a blanket statement like that down.
I can't even decide if 'prefer factories' is a good way forward either.
Things like VCR which record data you're using to test against are almost
like a hybrid. Dynamically generating data and saving it as a fixture.

I don't believe this is an either/or situation, it seems to me that it's a
question of when and how much.

~Johnneylee

On Thu, May 16, 2013 at 6:47 AM, Andrea Reginato
notifications@github.comwrote:

@mpalmer https://github.com/mpalmer, @robsaundershttps://github.com/robsaunderswould you like to send a pull request better describing this point? It
feels like this is the weakest point in all betterspecs and I want to fix
it.


Reply to this email directly or view it on GitHubhttps://github.com/lelylan/betterspecs/issues/11#issuecomment-17993971
.

Spaceghost commented May 16, 2013

I disagree that one should only use fixtures.

There are many reasons why you'd use both, and it's a bit presumptuous to
lay a blanket statement like that down.
I can't even decide if 'prefer factories' is a good way forward either.
Things like VCR which record data you're using to test against are almost
like a hybrid. Dynamically generating data and saving it as a fixture.

I don't believe this is an either/or situation, it seems to me that it's a
question of when and how much.

~Johnneylee

On Thu, May 16, 2013 at 6:47 AM, Andrea Reginato
notifications@github.comwrote:

@mpalmer https://github.com/mpalmer, @robsaundershttps://github.com/robsaunderswould you like to send a pull request better describing this point? It
feels like this is the weakest point in all betterspecs and I want to fix
it.


Reply to this email directly or view it on GitHubhttps://github.com/lelylan/betterspecs/issues/11#issuecomment-17993971
.

@mpalmer

This comment has been minimized.

Show comment
Hide comment
@mpalmer

mpalmer May 16, 2013

@andreareginato I don't really have enough knowledge to be able to write a pull request. In my opinion, the point should probably just be removed entirely.

@robsaunders I'm afraid that doesn't answer my question at all -- I still lack "understanding why building associations in a global way is bad". I see some furious hand-waving, but not a lot of concrete example. A simple "here is what happens when you use fixtures, and here is how factories make it all better" would be nice.

mpalmer commented May 16, 2013

@andreareginato I don't really have enough knowledge to be able to write a pull request. In my opinion, the point should probably just be removed entirely.

@robsaunders I'm afraid that doesn't answer my question at all -- I still lack "understanding why building associations in a global way is bad". I see some furious hand-waving, but not a lot of concrete example. A simple "here is what happens when you use fixtures, and here is how factories make it all better" would be nice.

@dgm

This comment has been minimized.

Show comment
Hide comment
@dgm

dgm May 16, 2013

I don't think fixtures are necessarily bad, but their current implementation could be better.

Right now I'm working on a report that requires a lot of data in the database in order to fully test - it takes 5 minutes for the factories to whip up the test data. As a result, all the assertions are done in one massive test, as no one wants to wait for it to regenerate.

It would be much better if there was a way to use the factories to generate the data, and then snapshot the whole database into a cached sql file; then the test case could use that to load the database directly in a before section, and use transactions to roll back between tests that are augmented by further factory calls.

This would be a hybrid between fixtures and factories - but I would have to agree that using yaml fixtures would be painful... What I want is a VCR like gem for databases.

The reason there is no pull request for this, as far as I am concerned, is because there is still no truly good solution yet.

dgm commented May 16, 2013

I don't think fixtures are necessarily bad, but their current implementation could be better.

Right now I'm working on a report that requires a lot of data in the database in order to fully test - it takes 5 minutes for the factories to whip up the test data. As a result, all the assertions are done in one massive test, as no one wants to wait for it to regenerate.

It would be much better if there was a way to use the factories to generate the data, and then snapshot the whole database into a cached sql file; then the test case could use that to load the database directly in a before section, and use transactions to roll back between tests that are augmented by further factory calls.

This would be a hybrid between fixtures and factories - but I would have to agree that using yaml fixtures would be painful... What I want is a VCR like gem for databases.

The reason there is no pull request for this, as far as I am concerned, is because there is still no truly good solution yet.

@warmwaffles

This comment has been minimized.

Show comment
Hide comment
@warmwaffles

warmwaffles May 16, 2013

@dgm if you are using SQLite3 for your database, there is an in memory option. I have never gotten it to work but I've seen it done and it can speed tests up. I know it really doesn't fix the problem but it is definitely something to look in to in order to speed up your tests.

warmwaffles commented May 16, 2013

@dgm if you are using SQLite3 for your database, there is an in memory option. I have never gotten it to work but I've seen it done and it can speed tests up. I know it really doesn't fix the problem but it is definitely something to look in to in order to speed up your tests.

@mpalmer

This comment has been minimized.

Show comment
Hide comment
@mpalmer

mpalmer May 16, 2013

In-memory sqlite3 is awesome for testing. I use it in all my projects. I'm not sure it's an argument either for or against fixtures or factories, though -- it'll just make everything database-related significantly faster.

mpalmer commented May 16, 2013

In-memory sqlite3 is awesome for testing. I use it in all my projects. I'm not sure it's an argument either for or against fixtures or factories, though -- it'll just make everything database-related significantly faster.

@warmwaffles

This comment has been minimized.

Show comment
Hide comment
@warmwaffles

warmwaffles May 17, 2013

well he was arguing that the build up time for his tests with factories was
causing him to consider using fixtures and possible database switching.

just curious, do you have an example sqlite3 memory example I can look at?

warmwaffles commented May 17, 2013

well he was arguing that the build up time for his tests with factories was
causing him to consider using fixtures and possible database switching.

just curious, do you have an example sqlite3 memory example I can look at?

@maurogeorge

This comment has been minimized.

Show comment
Hide comment
@maurogeorge

maurogeorge Aug 16, 2013

You guys think is a good pattern use the FactoryGirl::Syntax::Methods to wirite only create(:user) over FactoryGirl.create(:user)? I use as default on all projects the short syntax.

maurogeorge commented Aug 16, 2013

You guys think is a good pattern use the FactoryGirl::Syntax::Methods to wirite only create(:user) over FactoryGirl.create(:user)? I use as default on all projects the short syntax.

@marnen

This comment has been minimized.

Show comment
Hide comment
@marnen

marnen Jan 2, 2014

@robsaunders:

One caveat is to watch out you don't build up complex associations in your factory files - as this gives them the same downsides as fixtures. Structure should always be built up within the test itself.

I completely disagree. A major purpose of factories is to be able to build complex associations easily—otherwise you'd just use User.new instead of FactoryGirl.create :user. This does not have the same downside as fixtures, because the object that you care about is specified and created right in your test (sure, associated objects are created too, but you can ignore them). Fixtures get less useful as your associations get more complex, while factories actually get more useful. Not using associations in your factories removes most of the benefit of using factories at all.

How do you test with complex associations, then? It seems to me you'd either have to build up the associations in your tests (making your tests ugly) or mock a lot (making your tests implementation-coupled). Is there a third way that I'm missing?

@mpalmer:

In-memory sqlite3 is awesome for testing. I use it in all my projects.

Careful! SQLite is single-user and therefore not suitable for production. If you're using it for testing, then that means you're testing on a different DB from production, and your test environment may behave significantly differently from your dev environment: although ActiveRecord does a good job of abstracting differences between DBs, it cannot do so completely, and SQLite is just plain missing a lot of useful features (such as foreign key constraints and true booleans), so that testing any part of your app involving those features will be automatically incorrect.

IMHO, for any but the most trivial uses of the database, this difference is too dangerous to bear, so I'm willing to live with the lower speed of my tests actually talking to Postgres—because then I know they're correct for my production environment.

marnen commented Jan 2, 2014

@robsaunders:

One caveat is to watch out you don't build up complex associations in your factory files - as this gives them the same downsides as fixtures. Structure should always be built up within the test itself.

I completely disagree. A major purpose of factories is to be able to build complex associations easily—otherwise you'd just use User.new instead of FactoryGirl.create :user. This does not have the same downside as fixtures, because the object that you care about is specified and created right in your test (sure, associated objects are created too, but you can ignore them). Fixtures get less useful as your associations get more complex, while factories actually get more useful. Not using associations in your factories removes most of the benefit of using factories at all.

How do you test with complex associations, then? It seems to me you'd either have to build up the associations in your tests (making your tests ugly) or mock a lot (making your tests implementation-coupled). Is there a third way that I'm missing?

@mpalmer:

In-memory sqlite3 is awesome for testing. I use it in all my projects.

Careful! SQLite is single-user and therefore not suitable for production. If you're using it for testing, then that means you're testing on a different DB from production, and your test environment may behave significantly differently from your dev environment: although ActiveRecord does a good job of abstracting differences between DBs, it cannot do so completely, and SQLite is just plain missing a lot of useful features (such as foreign key constraints and true booleans), so that testing any part of your app involving those features will be automatically incorrect.

IMHO, for any but the most trivial uses of the database, this difference is too dangerous to bear, so I'm willing to live with the lower speed of my tests actually talking to Postgres—because then I know they're correct for my production environment.

@robsaunders

This comment has been minimized.

Show comment
Hide comment
@robsaunders

robsaunders Jan 3, 2014

@marnen:

If that's how you feel about factories, you may be abusing them. What you are doing is pushing complex and brittle structure that actually BELONGS to the test, into another, communal area. I may have misunderstood you though, so if I did this is not relevant, but I am assuming you mean to put 'helpers' to build up associated data (complex structures of models with specific sets of children) inside your factory files, which are used by multiple tests at once, rather than in the model test file itself.

It's fine in a small project with a few people who all respect the code. But more than a handful of devs and tests and you run into chaos quickly. This isn't about 'pretty' code or convenient short term code. This is about robust code that works in the long run, and that is about modularising areas so you have fewer moving parts (areas of extreme churn) and less functions that start to look like 'The Blob', which are changed every time you need a little bit more functionality from new consumers of said function and potentially breaking other consumers in the process.

I think it does take being in projects where this is a problem to truly respect how bad it can get, I am always on the look out for this anti-pattern and the single biggest one I find is mis-using factories.

In my opinion (I do understand that other people differ) factories provide the following benefits:

  • They provide a nice syntactic sugar layer for quickly creating models in your tests
  • They allow for 'default' values for your objects so you don't have to worry so much about validations which are a cause of brittleness themselves.
  • They allow you to focus purely on what the test itself is testing (for example, you can create a 'default' object and each test unit can focus on values for its area knowing the other values are correct). This abstraction is probably the main value of factories.
  • They reduce bugs by separating off what is considered a valid model (eg. when you change something about the model so it is no longer valid for your tests, the factory is often the first thing that breaks, so you know your model is failing correctly, it's a good 'first place' to look when you're unsure if your model is valid).

This is why I don't just use Model.new instead of factories, and I think these are sensible benefits rather than shortcuts that satiate short-term developer laziness but create long term problems.

robsaunders commented Jan 3, 2014

@marnen:

If that's how you feel about factories, you may be abusing them. What you are doing is pushing complex and brittle structure that actually BELONGS to the test, into another, communal area. I may have misunderstood you though, so if I did this is not relevant, but I am assuming you mean to put 'helpers' to build up associated data (complex structures of models with specific sets of children) inside your factory files, which are used by multiple tests at once, rather than in the model test file itself.

It's fine in a small project with a few people who all respect the code. But more than a handful of devs and tests and you run into chaos quickly. This isn't about 'pretty' code or convenient short term code. This is about robust code that works in the long run, and that is about modularising areas so you have fewer moving parts (areas of extreme churn) and less functions that start to look like 'The Blob', which are changed every time you need a little bit more functionality from new consumers of said function and potentially breaking other consumers in the process.

I think it does take being in projects where this is a problem to truly respect how bad it can get, I am always on the look out for this anti-pattern and the single biggest one I find is mis-using factories.

In my opinion (I do understand that other people differ) factories provide the following benefits:

  • They provide a nice syntactic sugar layer for quickly creating models in your tests
  • They allow for 'default' values for your objects so you don't have to worry so much about validations which are a cause of brittleness themselves.
  • They allow you to focus purely on what the test itself is testing (for example, you can create a 'default' object and each test unit can focus on values for its area knowing the other values are correct). This abstraction is probably the main value of factories.
  • They reduce bugs by separating off what is considered a valid model (eg. when you change something about the model so it is no longer valid for your tests, the factory is often the first thing that breaks, so you know your model is failing correctly, it's a good 'first place' to look when you're unsure if your model is valid).

This is why I don't just use Model.new instead of factories, and I think these are sensible benefits rather than shortcuts that satiate short-term developer laziness but create long term problems.

@marnen

This comment has been minimized.

Show comment
Hide comment
@marnen

marnen Jan 3, 2014

@robsaunders Thanks very much for a thoughtful reply! I'm always trying to make my testing practices better.

What you are doing is pushing complex and brittle structure that actually BELONGS to the test, into another, communal area.

I'm not sure that the complex structure belongs explicitly in a unit test. Perhaps it does, but...

# user.rb
class User
  validates_presence_of :account_id
  validates_presence_of :preferred_language_id
  validates_presence_of 10 other things
  ...
end

#user_spec.rb
describe User do
  describe "#full_name" do
    it "should return the first and last names" do
      User.new(first_name: 'John', last_name: 'Smith').full_name.should == 'John Smith'
    end
  end
end

The unit spec will fail, because all the required attributes and associations have to be set up even though they're not relevant to this spec. So you either have to build them all explicitly (ugly), mock them out (ugly and risky), or put them in a factory. I'll take the factory every time.

I may have misunderstood you though, so if I did this is not relevant, but I am assuming you mean to put 'helpers' to build up associated data (complex structures of models with specific sets of children) inside your factory files, which are used by multiple tests at once, rather than in the model test file itself.

Certainly not. I've never heard of this helper technique. My factory file would be

Factory.define :user do
  association :account
  association :preferred_language
  # 10 more associations and default fields
end

So when I do FactoryGirl.create :user, I get all the associations needed to make a valid User, but I don't have to clutter up specs with them where they're not relevant.

marnen commented Jan 3, 2014

@robsaunders Thanks very much for a thoughtful reply! I'm always trying to make my testing practices better.

What you are doing is pushing complex and brittle structure that actually BELONGS to the test, into another, communal area.

I'm not sure that the complex structure belongs explicitly in a unit test. Perhaps it does, but...

# user.rb
class User
  validates_presence_of :account_id
  validates_presence_of :preferred_language_id
  validates_presence_of 10 other things
  ...
end

#user_spec.rb
describe User do
  describe "#full_name" do
    it "should return the first and last names" do
      User.new(first_name: 'John', last_name: 'Smith').full_name.should == 'John Smith'
    end
  end
end

The unit spec will fail, because all the required attributes and associations have to be set up even though they're not relevant to this spec. So you either have to build them all explicitly (ugly), mock them out (ugly and risky), or put them in a factory. I'll take the factory every time.

I may have misunderstood you though, so if I did this is not relevant, but I am assuming you mean to put 'helpers' to build up associated data (complex structures of models with specific sets of children) inside your factory files, which are used by multiple tests at once, rather than in the model test file itself.

Certainly not. I've never heard of this helper technique. My factory file would be

Factory.define :user do
  association :account
  association :preferred_language
  # 10 more associations and default fields
end

So when I do FactoryGirl.create :user, I get all the associations needed to make a valid User, but I don't have to clutter up specs with them where they're not relevant.

@robsaunders

This comment has been minimized.

Show comment
Hide comment
@robsaunders

robsaunders Jan 3, 2014

@marnen I may have misunderstood you. The kinds of functions I am talking about are things like functions called create_user with crazy levels of arguments to define how the user factory instance is made. Defining single-level associations within a factory is ok but not great (you're making much more database-heavy work than you need, and kind of starts to break the unit tests reason for being a unit test, but there are some situations where, without it you don't have a valid factory). I think as long as you're only doing the minimal work needed to create a valid factory, you won't run into what equates to a 'god function' (the equivalent of the god model antipattern).

robsaunders commented Jan 3, 2014

@marnen I may have misunderstood you. The kinds of functions I am talking about are things like functions called create_user with crazy levels of arguments to define how the user factory instance is made. Defining single-level associations within a factory is ok but not great (you're making much more database-heavy work than you need, and kind of starts to break the unit tests reason for being a unit test, but there are some situations where, without it you don't have a valid factory). I think as long as you're only doing the minimal work needed to create a valid factory, you won't run into what equates to a 'god function' (the equivalent of the god model antipattern).

@marnen

This comment has been minimized.

Show comment
Hide comment
@marnen

marnen Jan 3, 2014

@robsaunders:

The kinds of functions I am talking about are things like functions called create_user with crazy levels of arguments to define how the user factory instance is made.

People do that?!? That really defeats the point of factories. I could see doing that if you weren't using factories, but.

Defining single-level associations within a factory is ok but not great (you're making much more database-heavy work than you need, and kind of starts to break the unit tests reason for being a unit test, but there are some situations where, without it you don't have a valid factory).

Rails unit tests hit the DB. I understand the theoretical arguments for not having them do so, but so far, I haven't found a good way of making that convenient in practice. Even if you use something like NullDB, the presence or absence of the DB starts to leak into the tests, even though it should be a private implementation detail.

That being the case, then, I'm perfectly happy to have as many associations as necessary in my factories as necessary for a valid record of the class I care about. If that means that FactoryGirl.create :user makes 15 associated records, I really don't care very much. I'll use as many levels of association in my factories as I have to in order to make my test code easy to read.

Do you do otherwise? If so, how do you make it convenient?

I think as long as you're only doing the minimal work needed to create a valid factory,

That's my practice. In a complex application, that can still be a lot of work, but it's necessary.

you won't run into what equates to a 'god function' (the equivalent of the god model antipattern).

What do you consider to be a "god function" here?

marnen commented Jan 3, 2014

@robsaunders:

The kinds of functions I am talking about are things like functions called create_user with crazy levels of arguments to define how the user factory instance is made.

People do that?!? That really defeats the point of factories. I could see doing that if you weren't using factories, but.

Defining single-level associations within a factory is ok but not great (you're making much more database-heavy work than you need, and kind of starts to break the unit tests reason for being a unit test, but there are some situations where, without it you don't have a valid factory).

Rails unit tests hit the DB. I understand the theoretical arguments for not having them do so, but so far, I haven't found a good way of making that convenient in practice. Even if you use something like NullDB, the presence or absence of the DB starts to leak into the tests, even though it should be a private implementation detail.

That being the case, then, I'm perfectly happy to have as many associations as necessary in my factories as necessary for a valid record of the class I care about. If that means that FactoryGirl.create :user makes 15 associated records, I really don't care very much. I'll use as many levels of association in my factories as I have to in order to make my test code easy to read.

Do you do otherwise? If so, how do you make it convenient?

I think as long as you're only doing the minimal work needed to create a valid factory,

That's my practice. In a complex application, that can still be a lot of work, but it's necessary.

you won't run into what equates to a 'god function' (the equivalent of the god model antipattern).

What do you consider to be a "god function" here?

@nruth

This comment has been minimized.

Show comment
Hide comment
@nruth

nruth Jun 7, 2016

The kinds of functions I am talking about are things like functions called create_user with crazy levels of arguments to define how the user factory instance is made

Yes this does happen, e.g. wrapping the must-be-in-one-transaction-so-cant-be-belongs-to shortcoming of old tools. Anyway, it seems the document should be updated to say "Don't use Rails helpers spec/support helpers", instead of don't use fixtures.

In fact, for complex setup cases fixtures sound pretty good and I'm going to give them a spin for climbing out of a 4-7 year old hole of overlapping helper mess & slow factories.

Beyond that it'd mean showing sustainable ways to use both approaches, which is probably no good for a pithy guide like this.

nruth commented Jun 7, 2016

The kinds of functions I am talking about are things like functions called create_user with crazy levels of arguments to define how the user factory instance is made

Yes this does happen, e.g. wrapping the must-be-in-one-transaction-so-cant-be-belongs-to shortcoming of old tools. Anyway, it seems the document should be updated to say "Don't use Rails helpers spec/support helpers", instead of don't use fixtures.

In fact, for complex setup cases fixtures sound pretty good and I'm going to give them a spin for climbing out of a 4-7 year old hole of overlapping helper mess & slow factories.

Beyond that it'd mean showing sustainable ways to use both approaches, which is probably no good for a pithy guide like this.

@marnen

This comment has been minimized.

Show comment
Hide comment
@marnen

marnen Jun 7, 2016

@nruth That article you linked to drives me crazy, because it seems to go out of its way to show some of the ways that fixtures suck, but then unaccountably says "oh, they're not so bad"!

They are so bad; they encourage inflexible test data that's overly separated from your test functions. What you probably want instead is more factory templates to easily cover different cases.

marnen commented Jun 7, 2016

@nruth That article you linked to drives me crazy, because it seems to go out of its way to show some of the ways that fixtures suck, but then unaccountably says "oh, they're not so bad"!

They are so bad; they encourage inflexible test data that's overly separated from your test functions. What you probably want instead is more factory templates to easily cover different cases.

ecoologic pushed a commit to ecoologic/betterspecs that referenced this issue Mar 14, 2017

Erik Ecoologic
Solves issue #11
* Avoid `have` form, as it will be deprecated
* Describe class, not string
* Use `described_class`

See: http://rspec.info/blog/2013/11/rspec-2-99-and-3-0-betas-have-been-released

benoittgt added a commit that referenced this issue Mar 16, 2017

@ttilberg

This comment has been minimized.

Show comment
Hide comment
@ttilberg

ttilberg Sep 14, 2017

It's been five years of discussion, and the rule in the guide still makes a pretty dramatic claim about fixtures without stating why. The example given has nothing to do with fixtures. I've recently been trying to research the reasons people flatly say "don't use fixtures" but in every case I've come across, they are used in an arduous way that isn't necessary. If the example were actually using fixtures, it would look like this:

Use factories and not fixtures

This is an old topic, but it's still good to remember it. Do not use fixtures because they are difficult to control, use factories instead. Use them to reduce the verbosity on creating new data.

BAD
user = users(:guy)

GOOD
user = FactoryGirl.create :user

Well, that's not really compelling at all. In the published example, you aren't even using fixtures, so of course it's more verbose. A common argument is often that you have to maintain fixtures and give them default values and whatnot. But... don't you also do that with FactoryGirl?

Being that the fixtures can also include generated data if you really need it to (such as name: <%= Faker::Name.first_name %>, or create 1000 random objects in quick loop), I struggle to understand the argument here. What's difficult to control about sourcing an object with default values, and when necessary, changing them to fit your test using things like user = users(:tim); user.name = 'special name for this case'? Even complex associations can either be saved in the Fixture, or composed on the fly in your test, just like in FactoryGirl.

I fully admit that I don't have a lot of experience with rspec and FactoryGirl, as I've found MiniTest and fixtures very sufficient in my few years of study. It's very possible that my lack of experience in huge apps has me blind to something. That said, the motivation of my comment here is the search for that experience -- but everything I've found so far has been extremely unconvincing.

ttilberg commented Sep 14, 2017

It's been five years of discussion, and the rule in the guide still makes a pretty dramatic claim about fixtures without stating why. The example given has nothing to do with fixtures. I've recently been trying to research the reasons people flatly say "don't use fixtures" but in every case I've come across, they are used in an arduous way that isn't necessary. If the example were actually using fixtures, it would look like this:

Use factories and not fixtures

This is an old topic, but it's still good to remember it. Do not use fixtures because they are difficult to control, use factories instead. Use them to reduce the verbosity on creating new data.

BAD
user = users(:guy)

GOOD
user = FactoryGirl.create :user

Well, that's not really compelling at all. In the published example, you aren't even using fixtures, so of course it's more verbose. A common argument is often that you have to maintain fixtures and give them default values and whatnot. But... don't you also do that with FactoryGirl?

Being that the fixtures can also include generated data if you really need it to (such as name: <%= Faker::Name.first_name %>, or create 1000 random objects in quick loop), I struggle to understand the argument here. What's difficult to control about sourcing an object with default values, and when necessary, changing them to fit your test using things like user = users(:tim); user.name = 'special name for this case'? Even complex associations can either be saved in the Fixture, or composed on the fly in your test, just like in FactoryGirl.

I fully admit that I don't have a lot of experience with rspec and FactoryGirl, as I've found MiniTest and fixtures very sufficient in my few years of study. It's very possible that my lack of experience in huge apps has me blind to something. That said, the motivation of my comment here is the search for that experience -- but everything I've found so far has been extremely unconvincing.

@marnen

This comment has been minimized.

Show comment
Hide comment
@marnen

marnen Sep 14, 2017

@ttilberg:

A common argument is often that you have to maintain fixtures and give them default values and whatnot. But... don't you also do that with FactoryGirl?

Not in the same way. A factory is a template, with default values that you're expected to override as necessary when you call it. A fixture, OTOH, is a fixed record that you're expected to fetch as is. (At least, this was the case when I last used fixtures, admittedly a long time ago.)

So with fixtures, you define your test data far away from your test functions, and you need to predeclare every test record. With factories, you define your test data right in the test, and you do so on the fly without much predeclaration. The difference may look small at the beginning, but gets more apparent as the tests get more complex.

Predeclaring is time-consuming and takes you out of the creative flow when you're writing your test. If you need a different case of test record with fixtures, you'll be tempted to reuse an existing one because of the overhead of predeclaring, so your test data won't be sufficiently specific to actually test what you think you're testing. If you need a different case with factories, there is no overhead; you just say exactly what you want, so you can test exactly what you think you're testing.

What's difficult to control about sourcing an object with default values, and when necessary, changing them to fit your test

Nothing as stated. But if you really wanted a user named John, why did you have to first create Tim and then change his name? Something like Factory Girl would let you just create John right at the beginning without declaring a new template or fixture. In your example, there's not a huge difference, but in a system that had an audit log for each change to a record, there would be a big difference, and it could mess up your tests.

Moreover, if you only have a few fixtures and you're changing their data for every test, then you have no reason not to use factories: that is more or less the usage pattern that Factory Girl was designed for, and it provides a nicer interface for it.

Even complex associations can either be saved in the Fixture, or composed on the fly in your test, just like in FactoryGirl.

Saving a complex association in the fixture is usually bad, because it makes your test data too rigid. Composing on the fly without syntactic help is awkward as soon as there's more than one level of association. Factory Girl strikes a happy medium where the association pattern is assisted, but the association data is flexible on the fly.

I fully admit that I don't have a lot of experience with rspec and FactoryGirl, as I've found MiniTest and fixtures very sufficient in my few years of study.

If you don't have a lot of experience with factories, then why are you making sweeping statements about them? (Also, check out RSpec and/or MiniTest::Spec: the syntax is more readable and helps you focus your tests on interface, not implementation.)

Anyway, this sounds like classic Blub Paradox, but I know that's not an especially satisfactory answer. So: for me, a big reason that I like factories is that I can create exactly the test data I need, right where I need it, without predeclaring or confusing the test data.

I'll grant you that the differences may initially seem subtle and may not be immediately apparent to a novice. But they grow bigger as projects get more complex and as tests get more complete.

I'd go so far as to say that if you find fixtures sufficient, you probably fall into one of three cases:

  • Your project is trivial (so it doesn't matter which method you use right now, but your project will grow)
  • You're not testing well enough (because fixtures make it very hard to do so)
  • You're abusing fixtures by treating them like factories as suggested above (in which case, why not use factories?)

In closing, a thought: some people who switch from fixtures to factories try to use factories like fixtures by predeclaring every fixture as a factory template. If you're doing that, stop now: that's not how factories are meant to be used.


If you have sample test code available, I would be happy to have a look at how you use fixtures and see what I can tell you about the differences for your use case.

marnen commented Sep 14, 2017

@ttilberg:

A common argument is often that you have to maintain fixtures and give them default values and whatnot. But... don't you also do that with FactoryGirl?

Not in the same way. A factory is a template, with default values that you're expected to override as necessary when you call it. A fixture, OTOH, is a fixed record that you're expected to fetch as is. (At least, this was the case when I last used fixtures, admittedly a long time ago.)

So with fixtures, you define your test data far away from your test functions, and you need to predeclare every test record. With factories, you define your test data right in the test, and you do so on the fly without much predeclaration. The difference may look small at the beginning, but gets more apparent as the tests get more complex.

Predeclaring is time-consuming and takes you out of the creative flow when you're writing your test. If you need a different case of test record with fixtures, you'll be tempted to reuse an existing one because of the overhead of predeclaring, so your test data won't be sufficiently specific to actually test what you think you're testing. If you need a different case with factories, there is no overhead; you just say exactly what you want, so you can test exactly what you think you're testing.

What's difficult to control about sourcing an object with default values, and when necessary, changing them to fit your test

Nothing as stated. But if you really wanted a user named John, why did you have to first create Tim and then change his name? Something like Factory Girl would let you just create John right at the beginning without declaring a new template or fixture. In your example, there's not a huge difference, but in a system that had an audit log for each change to a record, there would be a big difference, and it could mess up your tests.

Moreover, if you only have a few fixtures and you're changing their data for every test, then you have no reason not to use factories: that is more or less the usage pattern that Factory Girl was designed for, and it provides a nicer interface for it.

Even complex associations can either be saved in the Fixture, or composed on the fly in your test, just like in FactoryGirl.

Saving a complex association in the fixture is usually bad, because it makes your test data too rigid. Composing on the fly without syntactic help is awkward as soon as there's more than one level of association. Factory Girl strikes a happy medium where the association pattern is assisted, but the association data is flexible on the fly.

I fully admit that I don't have a lot of experience with rspec and FactoryGirl, as I've found MiniTest and fixtures very sufficient in my few years of study.

If you don't have a lot of experience with factories, then why are you making sweeping statements about them? (Also, check out RSpec and/or MiniTest::Spec: the syntax is more readable and helps you focus your tests on interface, not implementation.)

Anyway, this sounds like classic Blub Paradox, but I know that's not an especially satisfactory answer. So: for me, a big reason that I like factories is that I can create exactly the test data I need, right where I need it, without predeclaring or confusing the test data.

I'll grant you that the differences may initially seem subtle and may not be immediately apparent to a novice. But they grow bigger as projects get more complex and as tests get more complete.

I'd go so far as to say that if you find fixtures sufficient, you probably fall into one of three cases:

  • Your project is trivial (so it doesn't matter which method you use right now, but your project will grow)
  • You're not testing well enough (because fixtures make it very hard to do so)
  • You're abusing fixtures by treating them like factories as suggested above (in which case, why not use factories?)

In closing, a thought: some people who switch from fixtures to factories try to use factories like fixtures by predeclaring every fixture as a factory template. If you're doing that, stop now: that's not how factories are meant to be used.


If you have sample test code available, I would be happy to have a look at how you use fixtures and see what I can tell you about the differences for your use case.

@ttilberg

This comment has been minimized.

Show comment
Hide comment
@ttilberg

ttilberg Sep 14, 2017

I appreciate the reply, and all points are taken with consideration. You're right that my apps are mostly small data management applications for ETL and client management -- fixtures work great. But I want to correct this part as it's directly related to this issue, and my original comment:

If you don't have a lot of experience with factories, then why are you making sweeping statements about them?

That's actually the exact opposite of what's going on here. I'm not making any assumptions at all about rspec/FactoryGirl, or criticizing them. In fact, I'm fully admitting I don't know them well. This is how I ended up on this project's page in the first place, and found this exact issue to discuss this point in the project.

When you are learning rails, minitest and fixtures are what's included -- so that's what you learn.

The project is making the sweeping statement, asserting that fixtures are bad and you should opt for factories without helping a user understand why. The given example has nothing to do with fixtures.

My goal with these comments isn't to criticize or argue about the merits of Fixtures v FactoryGirl, but to bring to light that for someone who uses fixtures regularly, the content of the subject at hand simply doesn't make sense. It makes a dramatic statement ("don't use the thing that comes bundled with Rails and is well documented in the official documentation") while intentionally misrepresenting the example.

So, like I mentioned -- I don't know enough about this subject to be able to write a PR for this to improve it. I just know that as it stands, this doesn't make much sense, and this is the forum for discussing that.

ttilberg commented Sep 14, 2017

I appreciate the reply, and all points are taken with consideration. You're right that my apps are mostly small data management applications for ETL and client management -- fixtures work great. But I want to correct this part as it's directly related to this issue, and my original comment:

If you don't have a lot of experience with factories, then why are you making sweeping statements about them?

That's actually the exact opposite of what's going on here. I'm not making any assumptions at all about rspec/FactoryGirl, or criticizing them. In fact, I'm fully admitting I don't know them well. This is how I ended up on this project's page in the first place, and found this exact issue to discuss this point in the project.

When you are learning rails, minitest and fixtures are what's included -- so that's what you learn.

The project is making the sweeping statement, asserting that fixtures are bad and you should opt for factories without helping a user understand why. The given example has nothing to do with fixtures.

My goal with these comments isn't to criticize or argue about the merits of Fixtures v FactoryGirl, but to bring to light that for someone who uses fixtures regularly, the content of the subject at hand simply doesn't make sense. It makes a dramatic statement ("don't use the thing that comes bundled with Rails and is well documented in the official documentation") while intentionally misrepresenting the example.

So, like I mentioned -- I don't know enough about this subject to be able to write a PR for this to improve it. I just know that as it stands, this doesn't make much sense, and this is the forum for discussing that.

@marnen

This comment has been minimized.

Show comment
Hide comment
@marnen

marnen Sep 14, 2017

Ah, good point; the example on the betterspecs website is terrible. I was responding more to the general principle than to the website content.

When you are learning rails, minitest and fixtures are what's included -- so that's what you learn.

Yes, and then you quickly discard them when you realize how much they suck. :) FWIW, I've worked as a Rails developer since about 2008, at a variety of companies. I think every one of them has used RSpec and factories. They are kind of community defaults at this point, even if the Rails core team doesn't want to distribute them with the Rails gem. :)

Now, I'm not recommending them because they're widely used, but rather because I think they're better than what comes standard with Rails. But I do think their near-universal acceptance is an interesting point.

marnen commented Sep 14, 2017

Ah, good point; the example on the betterspecs website is terrible. I was responding more to the general principle than to the website content.

When you are learning rails, minitest and fixtures are what's included -- so that's what you learn.

Yes, and then you quickly discard them when you realize how much they suck. :) FWIW, I've worked as a Rails developer since about 2008, at a variety of companies. I think every one of them has used RSpec and factories. They are kind of community defaults at this point, even if the Rails core team doesn't want to distribute them with the Rails gem. :)

Now, I'm not recommending them because they're widely used, but rather because I think they're better than what comes standard with Rails. But I do think their near-universal acceptance is an interesting point.

@redbar0n

This comment has been minimized.

Show comment
Hide comment
@redbar0n

redbar0n Oct 24, 2017

I think this illustrates the pain point with fixtures pretty well.

TL;DR: Changing a fixture, or even adding a new fixture will often break a vast number of previous tests*, and fixtures introduces mystery guests.

* - Sometimes you need to test methods that f.ex. outputs statistic based on the DB state. Even adding 1 new fixture later on, would break all those tests of methods relying on counting records in the DB. Maybe there's a workaround (?), but that is what I am experiencing.

For data sets that are a bit more involved, YAML fixtures are difficult to maintain and hard to change without affecting other tests. I mean, you can make them work, of course—after all, developers used them plenty in the past—but tons of developers will agree that the price to pay for managing fixtures is just a bit stingy.

One scenario we definitely want to avoid is changing little details on an existing fixture and causing tons of tests to fail. If these failing tests are unrelated, the situation is even worse—a good example of tests being too brittle. In order to “protect” existing tests from this scenario, this can also lead to growing your fixture set beyond any reasonable size—being DRY with fixtures is most likely not on the table anymore at that point.

To avoid breaking your test data when the inevitable changes occur, developers were happy to adopt newer strategies that offered more flexibility and dynamic behaviour. That’s where Factory Girl came in and kissed the YAML days goodbye.

Another issue is the heavy dependency between the test and the .yml fixture file. Since the fixtures are defined in a separate .yml file, mystery guests are also a major pain waiting to bite you due to being obscure.

Source: https://code.tutsplus.com/articles/antipatterns-basics-rails-tests--cms-26011

It also describes what Mystery Guests are, and the problem with them (their obscurity - especially to new developers coming into the project - are akin to a mode, and we all know why we generally want "no modes"). One counter-argument is that mystery guests is a result of stringently applying Convention over Configuration.

That said, I see the benefits of fixtures as well, and I'm torn between that and the admittedly more sexy FactoryGirl.

I tried finding an elaboration on DHH's stance on it, but the most detailed I could find was this:

Use the power of testing fixtures and the case-transaction loop to avoid having to do expensive and excessive setup work for every case, and merely tailor your prototypical fixtures to the case at hand by changing the single state needed. (More on this technique in that promised future post).

Source: http://david.heinemeierhansson.com/2014/test-induced-design-damage.html

I couldn't find that promised future post. Let me know if any of you find it.

The case-transaction loop he is referring to (a novel expression, I believe), is probably ActionModel::Base.transaction.
By "tailor your prototypical fixtures to the case at hand by changing the single state needed", I presume he means using Model.update_attributes (which will update the record in the DB which the YAML fixtures had ensured was already there.) in each test (or in the setup method).

I think this is doing exactly what @marnen advised against above:
"You're abusing fixtures by treating them like factories as suggested above (in which case, why not use factories?)" But to DHH that seems to be the golden standard, perhaps oddly enough. Maybe you can actually get the best of both worlds with this approach? Using a complete set of fully fleshed out fixtures (using real world data), in all their complexity and interconnectedness, while overriding in the few cases where it is necessary..

I think the whole issue with the betterspec page though, is that it uses "fixtures" in an ambiguous way: "fixtures" in the broad sense which means pre-created fixed data (like Model.create), and not "fixtures" in the narrow sense that is common in Rails, which refers to the YAML files approach. The page should probably be updated to remove this ambiguity.

I second the suggestion by @dgm "What I want is a VCR like gem for databases." to try to get the best from both factories and fixtures and the worst of neither.

(If I'm allowed to dream, outside of this discussion, I would also want at VCR-like gem that would allow a non-developer to click through a use case in the app web interface, and store that as a test case which then capybara or similar could run. Automatically generated/macroed integration tests. Sorry for digressing, I just wanted that idea to be out there.)

redbar0n commented Oct 24, 2017

I think this illustrates the pain point with fixtures pretty well.

TL;DR: Changing a fixture, or even adding a new fixture will often break a vast number of previous tests*, and fixtures introduces mystery guests.

* - Sometimes you need to test methods that f.ex. outputs statistic based on the DB state. Even adding 1 new fixture later on, would break all those tests of methods relying on counting records in the DB. Maybe there's a workaround (?), but that is what I am experiencing.

For data sets that are a bit more involved, YAML fixtures are difficult to maintain and hard to change without affecting other tests. I mean, you can make them work, of course—after all, developers used them plenty in the past—but tons of developers will agree that the price to pay for managing fixtures is just a bit stingy.

One scenario we definitely want to avoid is changing little details on an existing fixture and causing tons of tests to fail. If these failing tests are unrelated, the situation is even worse—a good example of tests being too brittle. In order to “protect” existing tests from this scenario, this can also lead to growing your fixture set beyond any reasonable size—being DRY with fixtures is most likely not on the table anymore at that point.

To avoid breaking your test data when the inevitable changes occur, developers were happy to adopt newer strategies that offered more flexibility and dynamic behaviour. That’s where Factory Girl came in and kissed the YAML days goodbye.

Another issue is the heavy dependency between the test and the .yml fixture file. Since the fixtures are defined in a separate .yml file, mystery guests are also a major pain waiting to bite you due to being obscure.

Source: https://code.tutsplus.com/articles/antipatterns-basics-rails-tests--cms-26011

It also describes what Mystery Guests are, and the problem with them (their obscurity - especially to new developers coming into the project - are akin to a mode, and we all know why we generally want "no modes"). One counter-argument is that mystery guests is a result of stringently applying Convention over Configuration.

That said, I see the benefits of fixtures as well, and I'm torn between that and the admittedly more sexy FactoryGirl.

I tried finding an elaboration on DHH's stance on it, but the most detailed I could find was this:

Use the power of testing fixtures and the case-transaction loop to avoid having to do expensive and excessive setup work for every case, and merely tailor your prototypical fixtures to the case at hand by changing the single state needed. (More on this technique in that promised future post).

Source: http://david.heinemeierhansson.com/2014/test-induced-design-damage.html

I couldn't find that promised future post. Let me know if any of you find it.

The case-transaction loop he is referring to (a novel expression, I believe), is probably ActionModel::Base.transaction.
By "tailor your prototypical fixtures to the case at hand by changing the single state needed", I presume he means using Model.update_attributes (which will update the record in the DB which the YAML fixtures had ensured was already there.) in each test (or in the setup method).

I think this is doing exactly what @marnen advised against above:
"You're abusing fixtures by treating them like factories as suggested above (in which case, why not use factories?)" But to DHH that seems to be the golden standard, perhaps oddly enough. Maybe you can actually get the best of both worlds with this approach? Using a complete set of fully fleshed out fixtures (using real world data), in all their complexity and interconnectedness, while overriding in the few cases where it is necessary..

I think the whole issue with the betterspec page though, is that it uses "fixtures" in an ambiguous way: "fixtures" in the broad sense which means pre-created fixed data (like Model.create), and not "fixtures" in the narrow sense that is common in Rails, which refers to the YAML files approach. The page should probably be updated to remove this ambiguity.

I second the suggestion by @dgm "What I want is a VCR like gem for databases." to try to get the best from both factories and fixtures and the worst of neither.

(If I'm allowed to dream, outside of this discussion, I would also want at VCR-like gem that would allow a non-developer to click through a use case in the app web interface, and store that as a test case which then capybara or similar could run. Automatically generated/macroed integration tests. Sorry for digressing, I just wanted that idea to be out there.)

@marnen

This comment has been minimized.

Show comment
Hide comment
@marnen

marnen Oct 24, 2017

@redbar0n FWIW, I no longer pay much attention to how DHH wants me to use Rails. Of course he did fantastic work in creating (the early versions of) the framework, but perhaps because of that, he's become so attached to his earlier vision that he can't seem to understand that there are other (IMHO better) ways of doing things than he envisioned. It's got to the point where by default, I take his recommendations as a slight negative these days.

That said, I see the benefits of fixtures as well

Then would you kindly enlighten me on what you think they are? IMHO there are none. :)

marnen commented Oct 24, 2017

@redbar0n FWIW, I no longer pay much attention to how DHH wants me to use Rails. Of course he did fantastic work in creating (the early versions of) the framework, but perhaps because of that, he's become so attached to his earlier vision that he can't seem to understand that there are other (IMHO better) ways of doing things than he envisioned. It's got to the point where by default, I take his recommendations as a slight negative these days.

That said, I see the benefits of fixtures as well

Then would you kindly enlighten me on what you think they are? IMHO there are none. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment