Tests defined in nested classes result in an error

[sanderploegsma]

At least two exercises in the Java track use nested classes in their unit tests using JUnit 4's @Enclosed annotation:

• REST API
• Simple Cipher

Running these exercises in the test runner results in the following errors:

REST API

$ bin/run-in-docker.sh rest-api ../exercism-track-java/exercises/practice/rest-api/ ./
Exception in thread "main" java.lang.NullPointerException: Null testCode
	at com.exercism.report.AutoValue_TestDetails$Builder.setTestCode(AutoValue_TestDetails.java:143)
	at com.exercism.report.ReportGenerator.buildTestDetails(ReportGenerator.java:25)
	at com.exercism.report.ReportGenerator.lambda$generate$0(ReportGenerator.java:13)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
	at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
	at com.exercism.report.ReportGenerator.generate(ReportGenerator.java:13)
	at com.exercism.TestRunner.run(TestRunner.java:57)
	at com.exercism.TestRunner.main(TestRunner.java:40)

Simple Cipher

$ bin/run-in-docker.sh simple-cipher ../exercism-track-java/exercises/practice/simple-cipher/ ./
Exception in thread "main" java.lang.IllegalArgumentException: Multiple entries with same key: TestSource[packageName=, className=SimpleCipherTest, methodName=cipherCanEncode]=@Test
public void cipherCanEncode() {
    String plainText = "aaaaaaaaaa";
    String cipherText = "abcdefghij";
    assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);
} and TestSource[packageName=, className=SimpleCipherTest, methodName=cipherCanEncode]=/**
 * Here we take advantage of the fact that plaintext of "aaa..." doesn't output the key. This is a critical
 * problem with shift ciphers, some characters will always output the key verbatim.
 */
@Test
public void cipherCanEncode() {
    String plainText = "aaaaaaaaaa";
    String cipherText = cipherWithDefaultKey.getKey().substring(0, 10);
    assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);
}
	at com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:210)
	at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:204)
	at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:146)
	at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:109)
	at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:389)
	at com.exercism.junit.JUnitTestParser.buildTestCodeMap(JUnitTestParser.java:51)
	at com.exercism.TestRunner.run(TestRunner.java:57)
	at com.exercism.TestRunner.main(TestRunner.java:40)

The issue here is that the code that extracts the source from the test method isn't determining the class name correctly, causing the lookup to fail.

[sanderploegsma]

This probably also affects JUnit 5 test classes that use @Nested.

[sanderploegsma]

I tried fixing the logic that determines the class name for each test method and while that does resolve the error, it also yields interesting results:

Simple Cipher test results

{
  "status" : "pass",
  "tests" : [ {
    "name" : "keyIsLowercaseLetters",
    "test_code" : "@Test\npublic void keyIsLowercaseLetters() {\n    assertThat(cipherWithDefaultKey.getKey()).matches(\"^[a-z]+$\");\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanDecode",
    "test_code" : "@Test\npublic void cipherCanDecode() {\n    String cipherText = \"aaaaaaaaaa\";\n    assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.getKey().substring(0, 10))).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanEncode",
    "test_code" : "/**\n * Here we take advantage of the fact that plaintext of \"aaa...\" doesn't output the key. This is a critical\n * problem with shift ciphers, some characters will always output the key verbatim.\n */\n@Test\npublic void cipherCanEncode() {\n    String plainText = \"aaaaaaaaaa\";\n    String cipherText = cipherWithDefaultKey.getKey().substring(0, 10);\n    assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherIsReversible",
    "test_code" : "@Test\npublic void cipherIsReversible() {\n    String plainText = \"abcdefghij\";\n    assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.encode(plainText))).isEqualTo(plainText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanDoubleShiftEncode",
    "test_code" : "@Test\npublic void cipherCanDoubleShiftEncode() {\n    String plainText = \"iamapandabear\";\n    String cipherText = \"qayaeaagaciai\";\n    assertThat(new Cipher(plainText).encode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanDecode",
    "test_code" : "@Test\npublic void cipherCanDecode() {\n    String plainText = \"abcdefghij\";\n    String cipherText = \"aaaaaaaaaa\";\n    assertThat(cipherWithDefaultKey.decode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanEncode",
    "test_code" : "@Test\npublic void cipherCanEncode() {\n    String plainText = \"aaaaaaaaaa\";\n    String cipherText = \"abcdefghij\";\n    assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherMessageLongerThanKey",
    "test_code" : "@Test\npublic void cipherMessageLongerThanKey() {\n    String plainText = \"iamapandabear\";\n    String key = \"abc\";\n    String cipherText = \"iboaqcnecbfcr\";\n    assertThat(new Cipher(key).encode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherIsReversibleGivenKey",
    "test_code" : "@Test\npublic void cipherIsReversibleGivenKey() {\n    String plainText = \"abcdefghij\";\n    assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.encode(plainText))).isEqualTo(plainText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanWrapDecode",
    "test_code" : "@Test\npublic void cipherCanWrapDecode() {\n    String plainText = \"zabcdefghi\";\n    String cipherText = \"zzzzzzzzzz\";\n    assertThat(cipherWithDefaultKey.decode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanWrapEncode",
    "test_code" : "@Test\npublic void cipherCanWrapEncode() {\n    String plainText = \"zzzzzzzzzz\";\n    String cipherText = \"zabcdefghi\";\n    assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanDoubleShiftEncode",
    "test_code" : "@Test\npublic void cipherCanDoubleShiftEncode() {\n    String plainText = \"iamapandabear\";\n    String cipherText = \"qayaeaagaciai\";\n    assertThat(new Cipher(plainText).encode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanDecode",
    "test_code" : "@Test\npublic void cipherCanDecode() {\n    String plainText = \"abcdefghij\";\n    String cipherText = \"aaaaaaaaaa\";\n    assertThat(cipherWithDefaultKey.decode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanEncode",
    "test_code" : "@Test\npublic void cipherCanEncode() {\n    String plainText = \"aaaaaaaaaa\";\n    String cipherText = \"abcdefghij\";\n    assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherMessageLongerThanKey",
    "test_code" : "@Test\npublic void cipherMessageLongerThanKey() {\n    String plainText = \"iamapandabear\";\n    String key = \"abc\";\n    String cipherText = \"iboaqcnecbfcr\";\n    assertThat(new Cipher(key).encode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherIsReversibleGivenKey",
    "test_code" : "@Test\npublic void cipherIsReversibleGivenKey() {\n    String plainText = \"abcdefghij\";\n    assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.encode(plainText))).isEqualTo(plainText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanWrapDecode",
    "test_code" : "@Test\npublic void cipherCanWrapDecode() {\n    String plainText = \"zabcdefghi\";\n    String cipherText = \"zzzzzzzzzz\";\n    assertThat(cipherWithDefaultKey.decode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanWrapEncode",
    "test_code" : "@Test\npublic void cipherCanWrapEncode() {\n    String plainText = \"zzzzzzzzzz\";\n    String cipherText = \"zabcdefghi\";\n    assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "keyIsLowercaseLetters",
    "test_code" : "@Test\npublic void keyIsLowercaseLetters() {\n    assertThat(cipherWithDefaultKey.getKey()).matches(\"^[a-z]+$\");\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanDecode",
    "test_code" : "@Test\npublic void cipherCanDecode() {\n    String cipherText = \"aaaaaaaaaa\";\n    assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.getKey().substring(0, 10))).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherCanEncode",
    "test_code" : "/**\n * Here we take advantage of the fact that plaintext of \"aaa...\" doesn't output the key. This is a critical\n * problem with shift ciphers, some characters will always output the key verbatim.\n */\n@Test\npublic void cipherCanEncode() {\n    String plainText = \"aaaaaaaaaa\";\n    String cipherText = cipherWithDefaultKey.getKey().substring(0, 10);\n    assertThat(cipherWithDefaultKey.encode(plainText)).isEqualTo(cipherText);\n}",
    "status" : "pass"
  }, {
    "name" : "cipherIsReversible",
    "test_code" : "@Test\npublic void cipherIsReversible() {\n    String plainText = \"abcdefghij\";\n    assertThat(cipherWithDefaultKey.decode(cipherWithDefaultKey.encode(plainText))).isEqualTo(plainText);\n}",
    "status" : "pass"
  } ],
  "version" : 3
}

The problem here is that for some reason all tests are reported twice. This seems to be caused by JUnit discovering the tests multiple times, but I'm not sure.

[danilopiazza]

The problem here is that for some reason all tests are reported twice. This seems to be caused by JUnit discovering the tests multiple times, but I'm not sure.

I think you're right. I was able to reproduce this using the JUnit Platform Launcher API: junit-team/junit-framework#3537.

This only happens on JUnit Vintage. JUnit 5 @Nested tests are not affected.

[sanderploegsma]

Thanks for sorting this out! Good to hear that JUnit 5 is not affected by this.

[sanderploegsma]

This issue should now be fixed!