Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extension doesn't always compile code before running tests. #1629

Open
TimKingtonFC opened this issue Oct 29, 2023 · 19 comments
Open

Extension doesn't always compile code before running tests. #1629

TimKingtonFC opened this issue Oct 29, 2023 · 19 comments
Labels
help wanted

Comments

@TimKingtonFC
Copy link

If I edit code and don't save the file, running the tests sometimes executes the old code.
When I click run, the files are saved, but there seems to be a race between the compiler and JUnit. Sometimes the compiler wins and things are okay, sometimes I get false test failures because JUnit is running the old code.

This is on MacOS 13.6, v0.40.0 of the runner, and vscode 1.83.1
@jdneo
Copy link
Member

jdneo commented Oct 30, 2023

What kind of project are you working on? Maven/Gradle?

@TimKingtonFC
Copy link
Author

The one with no build tools.

@jdneo
Copy link
Member

jdneo commented Oct 31, 2023

Hmm, I haven't heard this kind of feedback before. The extension should complete the compilation before running the tests.

Could you share a snippet of your test cases? And the settings you are using?

@TimKingtonFC
Copy link
Author

Here it is in action:
https://github.com/microsoft/vscode-java-test/assets/13016916/43c64055-a55e-4c1a-a498-88f3e8e37008

It doesn't look like I've changed any settings in the extension. My java related settings:

  "java.format.settings.url": ".vscode/java-formatter.xml",
"java.jdt.ls.java.home": "/Library/Java/JavaVirtualMachines/jdk-18.0.2.1.jdk/Contents/Home",
  "[java]": {
    "editor.defaultFormatter": "redhat.java",
    "editor.pasteAs.enabled": false
  },

The tests aren't anything special. They're only doing things in-memory with POJOs. I can send you the whole folder if you like, but I've seen this with multiple projects so I don't think the actual code is important. The project only depends on junit-platform-console-standalone-1.10.0.jar.

Here's an example. It's from a Java course I teach. They're simulating a VM, but the test doesn't create any real threads or do anything like that.:


    /** Test execution. */
    @Test
    public void testPriSimpleExec() {
        hintContext = "(prio)";
        TestUtil.compileMethod("t1",
            "push 5\n" +
                "pop_local x\n" +
                "push 3\n" +
                "push_local x\n" +
                "add\n" +
                "pop_local y\n");
        VMThread t1 = new VMThread("t1");
        Map<String, Integer> t1Symbols = t1.getEntryPointLocals();

        List<VMThread> threadList = new ArrayList<VMThread>();
        threadList.add(t1);

        PriorityScheduler scheduler = new PriorityScheduler(threadList);

        assertSame(hint("getCurrentThread returns wrong object"), t1, scheduler.getCurrentThread());
        assertEquals(hint("symbol table should be empty initially"), 0, t1Symbols.size());

        scheduler.run(1);
        assertEquals(hint("symbol table should still be empty"), 0, t1Symbols.size());

        scheduler.run(1);
        assertEquals(hint("symbol table contains wrong number of entries"), 1, t1Symbols.size());
        assertEquals(hint("t1 symbol didn't have expected value"), 5, (int) t1Symbols.get("x"));

        scheduler.run(3);
        assertEquals(hint("symbol table contains wrong number of entries"), 1, t1Symbols.size());
        assertSame(hint("getCurrentThread returns wrong object"), t1, scheduler.getCurrentThread());

        scheduler.run(1);
        assertEquals(hint("symbol table contains wrong number of entries"), 2, t1Symbols.size());
        assertEquals(hint("t1 symbol didn't have expected value"), 5, (int) t1Symbols.get("x"));
        assertEquals(hint("symbol didn't have expected value"), 8, (int) t1Symbols.get("y"));
        assertNull(hint("getCurrentThread should return null"), scheduler.getCurrentThread());
    }

@jdneo
Copy link
Member

jdneo commented Nov 1, 2023

Have you set java.debug.settings.forceBuildBeforeLaunch? Please check in both USER and WORKSPACE scope.

@TimKingtonFC
Copy link
Author

No, it wasn't set. I set it just to be sure, no change.

@jdneo
Copy link
Member

jdneo commented Nov 1, 2023

Hmm..

The video about looks very like java.debug.settings.forceBuildBeforeLaunch is disabled.

First time failed because it's running with the old class file. But the auto build will update the class file then, so the second time, the new class file has been generated.

Without this, so far, I cannot imagine other causes that makes this.

@jdneo
Copy link
Member

jdneo commented Nov 1, 2023

Maybe share the whole folder to let me check if I can repro from my side

@TimKingtonFC
Copy link
Author

TimKingtonFC commented Nov 1, 2023 via email

@TimKingtonFC
Copy link
Author

VM.zip

@jdneo
Copy link
Member

jdneo commented Nov 1, 2023

It works on my side. What is the probability of this happening?
https://github.com/microsoft/vscode-java-test/assets/6193897/8dbb8fb2-fc00-4231-a164-0175e0cdea18

@TimKingtonFC
Copy link
Author

About 50%. I just tried it 20 times, 10 changing the test to fail and 10 changing it to pass. It worked correctly 5/10 changing it to fail, and 4/10 changing it to pass.

@jdneo
Copy link
Member

jdneo commented Nov 2, 2023

I still suspect it an environment specific problem, because 50% is a very high ratio. If that applies to other users, we should receive quite a lot complains about this.

What java related extensions are you installed? Is this behavior also observable from your colleague/students?

@TimKingtonFC
Copy link
Author

TimKingtonFC commented Nov 2, 2023 via email

@jdneo
Copy link
Member

jdneo commented Nov 7, 2023

Sorry @TimKingtonFC, one more question:

Besides the test, is this behavior also observable if you run a main method?

@TimKingtonFC
Copy link
Author

TimKingtonFC commented Nov 7, 2023 via email

@jdneo
Copy link
Member

jdneo commented Nov 8, 2023

I tried it again on a Windows machine today, but still cannot find a way to reproduce it.

Is this behavior also observable from your colleague/students?

Sorry that I could not give more help so far.

@TimKingtonFC
Copy link
Author

TimKingtonFC commented Nov 17, 2023 via email

@TimKingtonFC
Copy link
Author

I've noticed a couple of other things that might be related.

Fairly often - about once a day - I try to run my tests, and it gives me the popup that says the build failed, but the code is fine. It appears to be trying to build/run code from a few seconds earlier that didn't compile - I can tell from the error messages. Sometimes I can fix it by editing a file and saving, and sometimes I have to clean the Java workspace.

A little less often - maybe every other day - something weird happens when I'm editing. I'll be typing code, and then some of the code I've just typed disappears. It's usually about half a line of code. IIRC, hitting undo readds the code that disappears, but I'm not 100% sure on that point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jdneo @TimKingtonFC