Testing strategy

[Zalastax]

The discussion about testing in #54 needs its own place.

Highlights:

@Zalastax:

What's your suggestion regarding testing? As you said, some code will fail/pass type checking depending on this option's value. Two new subdirectories?

@zuiderkwast:

I don't know @Zalastax. I think unit tests using code in strings, as in the typechecker_tests suite, is better than the should_fail/pass ones.

@Zalastax:

I don't agree that strings are better - they lack syntax highlighting.
Type checkers are a prime case for golden testing and I was happy to see that it's already what's being used. Golden testing is easier to reason about since every test case is completely static (less logic in test = good). We should only resort to unit tests with custom logic when it's absolutely necessary.

@gomoripeti:

regarding testing I think the should_pass/fail are nice as smoke tests but a bit limited (for example you cannot test if there are 1 or more errors in a should_fail module). the typechecker_tests-style tests need less context, can reach edge cases easier, or eg can also test specific errors (when gradualizer will support returning error terms)

regarding module attributes, I think in real use the option should be per project (eg in rebar.config) and not per module - but for testing a module attribute can be a nice idea.

Thoughts

My two personal goals are that:

• Tests should be easy to add and read (actual erlang files as tests score high here)
• Tests should not require lots of boilerplate/duplication (testing multiple errors for a module / testing small variations might be difficult if using actual erlang files)

I've had quite good success with the dtslint approach to testing types but I am not sure that's the path forward.

[zuiderkwast]

Sounds good. Then I suggest using attributes specifically for this.

-gradualizer_expect_error({type_error, bla, bla, bla}).
f() -> integer_to_list(42) + [].

[Zalastax]

I still think this is a nice idea but I don't feel like implementing it. The current system + #129 does most of what we need and I don't see any pain points here at the moment.