Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

More test stability fixes #120

Merged
merged 32 commits into from
Nov 7, 2022
Merged

More test stability fixes #120

merged 32 commits into from
Nov 7, 2022

Commits on Nov 7, 2022

  1. Typos and remove unused SuppressWarnings 

    Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    f707199 View commit details
    Browse the repository at this point in the history

  2. Use correct version of GDB on Jenkins 

    In 56ee2c3 I got github actions
working by using default GDB on GHA, but on Jenkins we should
continue to use CDT's pre-built version of GDB.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    1251f6b View commit details
    Browse the repository at this point in the history

  3. Convert pdom tests to JUnit5 format 

    This will allow, if needed, to mark tests as flaky

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    41ab286 View commit details
    Browse the repository at this point in the history

  4. Reduce API warning in test 

    Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    6cfff65 View commit details
    Browse the repository at this point in the history

  5. Fail test if there are any files left over in the workspace directory 

    Some tests sort of clean up after themselves, but still leave files
around.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    f3d1267 View commit details
    Browse the repository at this point in the history

  6. Remove incorrect assumptions of lifecycle of test 

    Maybe once upon a time this lifecycle did something,
but now in setUp fProject is always null and therefore
the project was never getting deleted as the fProject
that deleteProject saw was different than
the tests.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    c9c9b17 View commit details
    Browse the repository at this point in the history

  7. Convert test to JUnit5 and cleanup projects made during test 

    Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    6d1abe1 View commit details
    Browse the repository at this point in the history

  8. Clean up projects created during test 

    Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    e04e023 View commit details
    Browse the repository at this point in the history

  9. Delete contents on disk when cleaning up after test 

    The resource helper is widely used, but when it deletes
projects, it leaves their contents on disk. Fix this
so that the contents on disk is deleted.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    aba8b23 View commit details
    Browse the repository at this point in the history

  10. Better test error messages when workspace is not empty after test 

    On GHA the error does not really give a good indication of what
went wrong with minimal to no stack traces

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    95524a0 View commit details
    Browse the repository at this point in the history

  11. Remove test for code that is no longer supported 

    Ideally the code itself should also be deleted from CDT, but
this test is super flaky and I cannot seem to convert it to
JUnit5 so I can properly mark it as flaky. Therefore
the test is now simply gone.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    8879798 View commit details
    Browse the repository at this point in the history

  12. Convert more of the tests to JUnit5 

    Some of these tests left behind projects, by chaning them
to extend BaseTestCase5 the resource cleanup happens
and the tests are cleaned up properly

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    95b10be View commit details
    Browse the repository at this point in the history

  13. Delete projects fully at the end of tests 

    This code seems to be trying to optimize across tests.
This change isolated each individual test better.
Also removed is a bunch of effectively unused test code.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    a28fbaf View commit details
    Browse the repository at this point in the history

  14. Try to catch uncleaned up files earlier 

    Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    8974260 View commit details
    Browse the repository at this point in the history

  15. Delete test that was never run 

    This is an ancient (2004) test that does not apply, was never
referenced in the Suites and whose name doesn't match standard
pattern.

It tried to import ancient versions of projects as well, which
would kick off the project converter UI that can't be disabled
from the core plug-in.

All in all, this test adds nothing of value.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    3b93d3f View commit details
    Browse the repository at this point in the history

  16. Make ASTWriterTester abstract so it doesn't look like a test 

    JDT thinks this is a test and will run it in the IDE and display
an error. But it is only used to compose other tests, by making
it abstract the IDE won't see it anymore.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    c8a6953 View commit details
    Browse the repository at this point in the history

  17. There is a new way to mark tests as failing 

    And by "new" I mean 15 years old.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    72c55b9 View commit details
    Browse the repository at this point in the history

  18. Add JUnit5 dependencies 

    While it may be that the tests don't directly rely
on JUnit5, the IDE requires JUnit5 in the classpath
or else the launch config doesn't work with this error:

Cannot find class 'org.junit.platform.commons.annotation.Testable'
on project build path.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    695a63b View commit details
    Browse the repository at this point in the history

  19. Pass exceptions up the stack 

    Make sure not to lose stack trace when a test fails.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    ad2177a View commit details
    Browse the repository at this point in the history

  20. Add missing abstract keyword to abstract tests 

    Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    5b21f4f View commit details
    Browse the repository at this point in the history

  21. Clean up created projects in tearDown 

    Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    658ac45 View commit details
    Browse the repository at this point in the history

  22. Create temporary files with ResourceHelper so they get cleanedup 

    Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    d5dbb36 View commit details
    Browse the repository at this point in the history

  23. Delete settings.xml file at end of test 

    Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    a6ba37d View commit details
    Browse the repository at this point in the history

  24. Convert to JUnit5 and cleanup resources 

    Also stop catching exceptions and discarding stacktraces

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    2fcbee3 View commit details
    Browse the repository at this point in the history

  25. Convert test to JUnit5 

    The test "abused" the JUnit3 ability to control test order and
required all tests to run in the given order to work.

Refactored the test to remove all redundant try/catch and
provide a more traditional flow.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    5d8de2c View commit details
    Browse the repository at this point in the history

  26. Make sure test does not leave projects around 

    Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    71a2fa1 View commit details
    Browse the repository at this point in the history

  27. Cosmetic changes to unit test 

    Instead of using assertTrue, but assertEquals we get nice
error messages that show the contents of the collections.

Using assertEquals(List.of() instead of assertArrayEquals(...
because that gives nicer error messages.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    6518d81 View commit details
    Browse the repository at this point in the history

  28. Delete test for CDT 2.x functionality 

    Rather than update this test, simply delete this test of
ancient functionality

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    9bf8421 View commit details
    Browse the repository at this point in the history

  29. Make tests inherit from BaseTestCase or BaseTestCase5 

    This provides the shared pre/post condition assertions to
help make sure tests are running in a clean environment.

If I had time I would convert all the tests to BaseTestCase5
which is JUnit5, but I only did the tests that had something
wrong with them.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    9c6bc92 View commit details
    Browse the repository at this point in the history

  30. Place each test run in its own project 

    This helps add some isolation between tests in case background
threads are accessing a project. However I am not sure
this solves any of the actual outstanding flaky tests.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    e4b8ae5 View commit details
    Browse the repository at this point in the history

  31. Wait until all threads are stopped before continuing test 

    These tests all had the same mistake, they had 5 x waitForEvent
with the comment "at this point all 5 threads should be stopped"
but there are actually 6 threads (5 spawned ones + main thread).
Most of the time all 6 threads would be stopped in time, but
sometimes one of them wouldn't be stopped, leading to a
"CoreException: Context is running" error in the test

This fixes the ThreadStackFrameSyncTest item

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    059e16d View commit details
    Browse the repository at this point in the history

  32. Include the full stack trace in exception message 

    The format of this error message used to look like:

```java
Expected number (0) of Non-OK status objects in log differs from actual (1).
	Error while parsing /projC_testTripleDownwardV/h3.h. java.lang.reflect.InvocationTargetException

Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.AssertionError: Need to hold a write lock to clear result caches
```

and it was hard to impossible to identify what the cause is.

The hope is that capturing this fuller stack trace into the log
will make it easier to identify the problem.

Part of eclipse-cdt#117
    @jonahgraham
    jonahgraham committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    3e63144 View commit details
    Browse the repository at this point in the history