Data-driven tests #5098
Data-driven tests #5098
Conversation
Right now, we're not able to use data-driven tests at all. E.g.
given a set of tests which we'd like to repeat with different
test data, one has to hand-write any outer loop that iterates
over the test data and then call each of the test functions. Next
to being bothersome, this also has the downside that error
reporting is quite lacking, as one never knows which test data
actually produced failures.
So let's implement the ability to re-run complete test modules
with changing test data. To retain backwards compatibility and
enable trivial addition of new runs with changed test data, we
simply extend clar's generate.py. Instead of scanning for a
single `_initialize` function, each test module may now implement
multiple `_initialize_foo` functions. The generated test harness
will then run all test functions for each of the provided
initializer functions, making it possible to provide different
test data inside of each of the initializer functions. Example:
```
void test_git_example__initialize_with_nonbare_repo(void)
{
g_repo = cl_git_sandbox_init("testrepo");
}
void test_git_example__initialize_with_bare_repo(void)
{
g_repo = cl_git_sandbox_init("testrepo.git");
}
void test_git_example__cleanup(void)
{
cl_git_sandbox_cleanup();
}
void test_git_example__test1(void)
{
test1();
}
void test_git_example__test2(void)
{
test2();
}
```
Executing this test module will cause the following output:
```
$ ./libgit2_clar -sgit::example
git::example (with nonbare repo)..
git::example (with bare repo)..
```
The object::cache test module has two tests that do nearly the same thing: given a cache limit, load a certain set of objects and verify if those objects have been cached or not. Convert those tests to the new data-driven initializers to demonstrate how these are to be used. Furthermore, add some additional test data. This conversion is mainly done to show this new facility.
|
Well, isn't this clever. I'm just driving by at the moment, so I'll need to put a little more 👀 on the implementation but I love the idea. Hey @vmg, we've been making a lot of changes (okay, not a lot, but also not zero) to clar lately. Do you want us to try to upstream these back to you, or...? |
|
Nah, clearly the version here in libgit2 is the "definitive" version of Clar. I may just delete my repo and point people here. You're way ahead. 😅 |
I like that it's broken out into a separate repo though (I use it in other projects as well). 🤔 |
|
Could we place it in a separate folder here in libgit2 so it's easier to copy&paste into other projects? |
|
If you're anxious to get rid of https://github.com/vmg/clar then we can either fork that over to the @libgit2 account (or to one of our personal accounts) and make that the new canonical place. Don't know if anybody else has a strong preference. I lean away from @libgit2 a little bit, because I don't want anybody to think that the Team As A Whole supports it. |
|
Same here, I'd really like It might be possible to import it as a git submodule instead ? Not sure how we feel about that though, but it might be better located as I've started to use it as the test harness in libssh2, and I'm already carrying some custom stuff there (more vararg-y macros, so tests can build more complex messages). So I'm fine to be added to some @libgit2/clar team, if we feel one is needed. |
|
@tiennou: happy to convert more in case you know of some where it would make sense to do so! Merge analysis in #5017 is an obvious next one, but that's not merged yet.
I don't. I feel like I'm usually the one driving for new features in clar for the libgit2 project, so I might as well fork it into my own personal namespace. But in general, I always prefer the more "official" way of using an org for that. "clar" is taken already, but we may opt for something like "clar-tests". I'd also prefer the second approach in order to not become the sole maintainer, but instead to have a team of maintainers like in libgit2.
I agree that we should probably refactor this. Currently, we're quite inconsistent with regards to third-party dependencies, where we have some in "deps/" and some mixed into our own sources. If we agree to do that, then I might clean it up in the near-term future. |
I was thinking about the worktree/bare/non-bare. Arguably, I'm looking very hard at most of |
|
On Tue, Jun 11, 2019 at 12:52:00AM -0700, Etienne Samson wrote:
> happy to convert more in case you know of some where it would make sense to do so!
I was thinking about the worktree/bare/non-bare. Arguably, I'm
looking very hard at most of `repo`, because running that using
a bunch of normal, bare, shallow, in-memory, mostly-memory,
repositories would make the most sense, but it's harder (though
maybe taking a look at "how" would benefit the proposed
implementation).
I agree, but I think we should keep it out of this PR. As you
say, it's probably quite involved and will require some
refactoring/moving around, while the test case presented here is
a rather simple conversion that serves as a nice demonstration of
how to use the new functionality.
|
👍 This is neat, I like it a lot. Thanks @pks-t. |
|
Thanks for merging, @ethomson! We still didn't agree on how to fare with clar, though. Should we maintain it as a separate project "clar-{framework,tests,testing}/clar" (proposals welcome!) and then re-synchronize our own copy with such an upstream project? |
|
I've just filed clar-test/clar#81, which carries our current changes. |
|
Cool, thanks for your effort! If @vmg wants to keep maintaing clar then that's probably the best way ahead. To me it sounded like he didn't, though, but it might be that I interpreted too much into his words. :) |
|
Sounds like @vmg would like to create an org for this, so I've created https://github.com/clar-test/clar and invited all of you. @vmg do you want to do that magic to make that one the root of the network? I forget how that works. |
|
@ethomson I think I need admin access to the |
|
Fixed!
…On Mon, Jun 17, 2019 at 8:13 AM Vicent Martí ***@***.***> wrote:
@ethomson <https://github.com/ethomson> I think I need admin access to
the clar repo in the new org before I can do that!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#5098?email_source=notifications&email_token=AAIT4HSN4NLTKTQP54ACPWLP242QDA5CNFSM4HVWTD22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX2H7CI#issuecomment-502562697>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAIT4HRM6EK3QI2YQOGAJT3P242QDANCNFSM4HVWTD2Q>
.
|
|
@vmg I've also deleted https://github.com/clar-test/clar, so you're welcome to do that transfer if you'd like. |
tests: allow for simple data-driven tests
Right now, we're not able to use data-driven tests at all. E.g.
given a set of tests which we'd like to repeat with different
test data, one has to hand-write any outer loop that iterates
over the test data and then call each of the test functions. Next
to being bothersome, this also has the downside that error
reporting is quite lacking, as one never knows which test data
actually produced failures.
So let's implement the ability to re-run complete test modules
with changing test data. To retain backwards compatibility and
enable trivial addition of new runs with changed test data, we
simply extend clar's generate.py. Instead of scanning for a
single
_initializefunction, each test module may now implementmultiple
_initialize_foofunctions. The generated test harnesswill then run all test functions for each of the provided
initializer functions, making it possible to provide different
test data inside of each of the initializer functions. Example:
Executing this test module will cause the following output:
The goal was basically to have the least-intrusive change that gives us the most bang for the buck. No changes were required for any of our existing test modules, but we can now easily set up multiple test runs of a complete module by just specifying multiple initializers.