Bug 62785. -Incomplete search path applied to the filenames used in the upload functionality of the HTTP sampler

[artem-fedorov]

https://bz.apache.org/bugzilla/show_bug.cgi?id=62785

[vlsi]

Why, why do you people use bin/jmeter.properties for unit testing?
Eh :(

[FSchumacher]

We have introduced a resources directory under test that should be used for test files -- and those should be test specific, but it was either really old code, or someone was lazy.

[vlsi]

@artem-fedorov , @FSchumacher , please refrain from computations inside asserttrue/false.
It produces a failure like the following

java.lang.AssertionError
        at org.junit.Assert.fail(Assert.java:86)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertFalse(Assert.java:64)
        at org.junit.Assert.assertFalse(Assert.java:74)
        at org.apache.jmeter.protocol.http.sampler.TestHTTPSamplers.testRawBodyFromFile(TestHTTPSamplers.java:385)

and it provides no clue on what really went wrong.

It would be so much better if you used

String msg = sample.getResponseDataAsString();
if (msg.contains("java.io.FileNotFoundException:")) {
  fail("sample.getResponseDataAsString()"+
      " should not contain java.io.FileNotFoundException,"+
      " but the actual value is " + msg);
}

Do you see? Thank you for the understanding.

[artem-fedorov]

@vlsi I'm agree with your remark. The second code style is better for understanding in fail cases.

[FSchumacher]

Thats is definitely better, but I think it would be even better to use hamcrests StringContains.constainsString and junits Assert.assertThat for the test.

Using static imports it would result in

assertThat(sample.getResponseDataAsString(), not(containsString("java.io.FileNotFoundException:")));

[vlsi]

Mr @pmouawad might argue that Groovy+Spock might even be better (e.g. https://twitter.com/VladimirSitnikv/status/1100666414548045825 ).

However please estimate how assert failures would look like. Would it look like a good bug report? Is the failure message+stacktrace enough to file a understandable bug report?

What we typically see is that "tests pass". We don't accept code changes if tests fail, do we?

However the important bit of the test code is the way it fails should something wrong happen. It is important that developers understand that, and committers acting as gatekeepers should watch for that as well.

I don't mean "assert messages" are the top priority, however having sane messages is vital for further maintenance of the codebase.

[pmouawad]

Mr @pmouawad might argue that Groovy+Spock might even be better (e.g. https://twitter.com/VladimirSitnikv/status/1100666414548045825 ).

I don't know if it's sarcastic or not :-)
Still, I find the messages very clear and text comparison a great improvement.
Integration with Eclipse is not that great (at least from my experience) but still interesting framework (Thanks Graham Russell (to Caesar what's belong to Caesar) for making me discover it). No firm conviction on JUnit 5 or Spock still, both do the job

[FSchumacher]

I like the spock error messages, too. They might be a bit over the top and frightening to the beginner, but very descriptive and therefore helpful.

... Would it look like a good bug report? Is the failure message+stacktrace enough to file a understandable bug report?

The error message would look like:

Expected: not a string containing "java.io.FileNotFoundException"
  but was: "the real content of the sampler ..."

In my opinion this is a good (enough) message, with the plus side, that is is code only and thus better suited for refactoring. But either way, we should get rid of simple assertFalse or assertTrue statements, that have no meaningful description message.

What we typically see is that "tests pass". We don't accept code changes if tests fail, do we?

What do you mean by this? If the code is wrong, or the error messages are difficult to read, we should -- and have done -- change the tests.

All in all, this seems to grow into a discussion, that would be good to have on the mailing list.