You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
At this point in time, many unit (and some integration) tests rely on checking for specific addresses to validate the output of different parts of the debugger. This is problematic because if the addresses in one of the example binaries used in testing change just a bit, the tests that expect them will fail. Those failures will usually be false negatives, since it's just the binary that contains a different address after compiling again.
An obvious solution that comes to mind is to compile all the examples once and check them into version control. This way they would stay the same no matter what machine the tests
run on. This approach has the disadvantage that the range of different compilers and platforms tests is much more narrow. Specifically the tests will exclusively make assertions about the platform they were once compiled on.
In the integration tests, regular expressions might be used to check that there are addresses at the right positions. The downside of this is that any address will be accepted. Since all addresses passed around internally have their own address type anyways, the type system already enforces the presence of addresses. Hence, this solution decreases the value of the assertions drastically, too.
Lastly, we could assert that any given address must be in a specific (narrow) range. I've seen multiple tests fail because of off-by-one assertions where the new address was only slightly different.
This is definitely still an open problem and I'd be happy to discuss different options.
The text was updated successfully, but these errors were encountered:
At this point in time, many unit (and some integration) tests rely on checking for specific addresses to validate the output of different parts of the debugger. This is problematic because if the addresses in one of the example binaries used in testing change just a bit, the tests that expect them will fail. Those failures will usually be false negatives, since it's just the binary that contains a different address after compiling again.
An obvious solution that comes to mind is to compile all the examples once and check them into version control. This way they would stay the same no matter what machine the tests
run on. This approach has the disadvantage that the range of different compilers and platforms tests is much more narrow. Specifically the tests will exclusively make assertions about the platform they were once compiled on.
In the integration tests, regular expressions might be used to check that there are addresses at the right positions. The downside of this is that any address will be accepted. Since all addresses passed around internally have their own address type anyways, the type system already enforces the presence of addresses. Hence, this solution decreases the value of the assertions drastically, too.
Lastly, we could assert that any given address must be in a specific (narrow) range. I've seen multiple tests fail because of off-by-one assertions where the new address was only slightly different.
This is definitely still an open problem and I'd be happy to discuss different options.
The text was updated successfully, but these errors were encountered: