Some flaky tests

[Agorguy]

Hello,

We tried running your project and discovered that it contains some flaky tests (i.e., tests that nondeterministically pass and fail). We found these tests to fail more frequently when running them on certain machines of ours.

To prevent others from running this project and its tests in machines that may result in flaky tests, we suggest adding information to the README.md file indicating the minimum resource configuration for running the tests of this project as to prevent observation of test flakiness.

If we run this project in a machine with 1cpu and 1gb ram, we observe flaky tests. We found that the tests in this project did not have any flaky tests when we ran it on machines with 2cpu and 4gb ram.

Here is a list of the tests we have identified and their likelihood of failure on a system with less than the recommended 2 CPUs and 4 GB RAM.

1. delight.nashornsandbox.TestSwitch#test_graal#javadelight ( 4 out of 10)
2. delight.nashornsandbox.TestSwitch#test#javadelight ( 3 out of 10)
3. delight.nashornsandbox.TestEvalWithScriptContext#testWithCPUAndMemory#javadelight ( 6 out of 10)
4. delight.nashornsandbox.TestEvalWithScriptContextWithVariables#testWithCPUAndMemory#javadelight ( 2 cout of 10)
5. delight.nashornsandbox.TestLimitCPU#testCpuLmitInInvocable#javadelight ( 5 out of 10)
6. delight.nashornsandbox.TestLimitCPU#test_do_while#javadelight ( 6 out of 10)
7. delight.nashornsandbox.TestLimitCPU#test_evil_script#javadelight ( 4 out of 10)
8. delight.nashornsandbox.TestLimitCPU#test#javadelight ( 6 out of 10)
9. delight.nashornsandbox.TestMemoryLimit#test_issue_86_continued_use#javadelight ( 4 out of 10)
10. delight.nashornsandbox.TestMemoryLimit#test ( 3 out of 10)

Reproducing

FROM maven:3.5.4-jdk-11

WORKDIR /home/

RUN git clone https://github.com/javadelight/delight-nashorn-sandbox && \
  cd delight-nashorn-sandbox && \
  git checkout ac868e1cbae8c503ce5ddd8624e97139115d9d75 



WORKDIR /home/delight-nashorn-sandbox

RUN mvn install -DskipTests

ENTRYPOINT ["mvn", "test", "-fn"]

Build the image:

$> mkdir tmp

$> cp Dockerfile tmp

$> cd tmp

$> docker build -t delight-nashorn-sandbox . # estimated time of build 3m

Running:
this configuration likely prevents flakiness (no flakiness in 10 runs)

$> docker run --rm --memory=4g --cpus=2 --memory-swap=-1 delight-nashorn-sandbox | tee output.txt
$> grep "Failures:"  output.txt # checking results

checking results

this other configuration –similar to the previous– can’t prevent flaky tests (observation in 10 runs)

$> docker run --rm --memory=1g --cpus=1 --memory-swap=-1 delight-nashorn-sandbox | tee output2.txt
$> grep "Failures:"  output2.txt # checking results

Thank you for your attention to this matter. We hope that our recommendations will be helpful in improving the quality and performance of your project.

[mxro]

Thank you for raising this! Please see the PR above - I've added a test to assert there are sufficient resources in the test environment. Do the values I chose there look all right?

[Agorguy]

@mxro this looks great, just two questions:

1. In the command run: mvn -B package --file pom.xml -Drun.jvmArguments="-Xmx4048m -Xms1024" in the build.yml file, the value of the -Xms argument is missing an "m" (of megabytes) in the end, isn't it?
2. What do you think add in the assert message informing to try to run the tests again with these options (-Drun.jvmArguments...)

[mxro]

@Agorguy Great questions, I've amended the code as per above PR. Let me know if there is anything else we can improve.