Annotation @Test(expected=NullPointerException.class) ignored

[tautschnig]

Hello Infer Team,
I would consider the below a false-positive, but you might as well tell me that this is by design and @SuppressWarnings("infer") should be used instead. Here's an example code:

import org.junit.Test;

class NullTest
{
  public void convert()
  {
  }

  public void convert(NullTest nt)
  {
    nt.convert();
  }

  @Test(expected=NullPointerException.class)
  public void should_fail() throws Exception
  {
    convert(null);
  }
}

with the result

$ infer/bin/infer -- javac -cp /usr/share/java/junit4.jar NullTest.java
[...]
NullTest.java:17: error: NULL_DEREFERENCE
  object null could be null and is dereferenced by call to convert(...) at line 17
[...]

and infer-out/report.json confirms that the trace originates at should_fail:

"bug_trace":[{"level":0,"filename":"NullTest.java","line_number":15,"description":"start of procedure should_fail()", ...

My claim is that @Test with expected=NullPointerException.class should possibly only yield a report if there is no null dereference.

Thanks,
Michael

[sblackshear]

Hi Michael,
We don't currently have any special support for JUnit annotations like @Test, but we could add support if there is a need.

In general, I'm skeptical of the utility of statically analyzing small pieces of test code like your example (since one can always just run the code and let JUnit check if the right thing happens). But I'm intrigued by the possibility of using JUnit assertions to specify statically checkable contracts in non-test code. Let me me know if you are interested in that use case/have examples and we can try to throw some support together.

[tautschnig]

Thanks a lot for the feedback!

For us, this has come up as part of various larger code bases. (I agree that static analysis of small snippets is of limited use.) My example was just intended so as to have a reproducible test scenario. Arguably we could work on ways to exclude tests from the analysis altogether.

For specifications, I am wondering whether JUnit is most suitable; wouldn't JML possibly be a better approach?

[jberdine]

Another issue that is independent of annotations is that infer does consider whether there is an enclosing handler for RuntimeExceptions. In the larger code bases, do you rely on catching RuntimeExceptions and want infer to not report raising them in cases where they will be caught? Such a change might be quite involved.

[jvillard]

A tad unrelated direction is to use infer's blacklisting options to avoid analysing or reporting on these files, unless tests and regular source codes are mixed together.

For instance: --infer-blacklist-files-containing "@Test(" or --infer-blacklist-files-containing "import org.junit.Test;". These options (like all options) can also go into a .inferconfig file.

[sblackshear]

@tautschnig

For us, this has come up as part of various larger code bases. (I agree that static analysis of small snippets is of limited use.) My example was just intended so as to have a reproducible test scenario. Arguably we could work on ways to exclude tests from the analysis altogether.

I get it now, thanks!

For specifications, I am wondering whether JUnit is most suitable; wouldn't JML possibly be a better approach?

I don't have any strong opinion about this. If you already have a large codebase with lots of JML annotations or lots of programmers who would be willing to write JML annotations in exchange for having them statically checked, then I think JML sounds like a great choice.

[jvillard]

It looks like this issue is about not reporting on tests, basically. Does excluding these files from the analysis work for you @tautschnig?