Random failures: test success but should not

[uhafner]

I have a test that checks that some specific static calls are not used in my code: PluginArchitectureTest.java#L33. I introduced a violation of this rule in my PR, see IssuesChartPortlet.java#L54.

When I use this rule it works perfectly in a VM and in GitHub Actions, but the rule fails on my local machine (MacBook) or sometimes in our CI (see PR jenkinsci/warnings-ng-plugin#546 and CI results (Windows fails, Linux succeeds).

I'm not sure if I use some concepts in the wrong way or if this is a bug that occurs only sometimes.

E.g. if I use the annotation @AnalyzeClasses(packages = "io.jenkins.plugins.analysis") then the test succeeds on my machine. If I restrict the scanned packages to @AnalyzeClasses(packages = "io.jenkins.plugins.analysis.core.portlets") then the test correctly fails. Is this a concurrency problem? Or is the usage of the static fields for the tests a problem? I tried to debug this issue but actually I am a bit of lost when the rules are dynamically invoked. Maybe you can give me some hints on how to find the root cause for this problem.

[codecholeric]

My first instinct would be that this is a classpath problem 🤔 Something in the direction of #319.
I might help to print the classpath within the test to easily compare how it looks when it fails and when it succeeds. That might also explain, why a subpackage might get found, but if you use the parent package it doesn't. Although I thought I fixed this with version 0.14.0 (and AFAIS you're using the current version, right?)
You could also compare it to the DEBUG log output of com.tngtech.archunit.core.importer.UrlSource. And last but not least, you could add a simple method

@ArchTest
static void printClasses(JavaClasses classes) {
    assertThat(classes.contain("io.jenkins.plugins.analysis.core.portlets.IssuesChartPortlet")).isTrue();
}

to make sure that the class that you expect to violate the rule is actually imported.

[codecholeric]

Did you manage to find the reason? Is there anything we need to fix in ArchUnit?

[uhafner]

Sorry, I forgot to dig into it any further.

Now I added the logging you mentioned but the class seems to be loaded. I added another test in the PR so that you see the results of both tests (one with the sub package and one with the top level package). So I think there is something wrong (or I am using the API in the wrong way?).

[uhafner]

I added the logging output of the class paths:
classpath.1.txt
classpath.2.txt
And the whole log:
everything.log
Maybe it helps?

[uhafner]

I put my code in the PR jenkinsci/warnings-ng-plugin#546.

[codecholeric]

Oh man, I completely lost track of this. I guess you didn't find out the reason? I actually don't see a classpath problem and I have trouble to run this locally. One thing I have noticed from the linked PR is though, that there was a change where before it was

@AnalyzeClasses(packages = "io.jenkins.plugins.analysis..")

I don't think you can use the package matcher syntax here, i.e. ... It also would not make sense, because ArchUnit automatically imports sub-packages with the ClassFileImporter, you can't just import a package without sub-packages with this API.
But even then, for the current state I would expect both PluginArchitectureTest and PluginArchitectureTest2 to fail, because both define the same rule and import both classes that form the violation 🤔
Too bad the project doesn't build on my machine at all 😞

[uhafner]

Hmm, I should have written the comment of jenkinsci/warnings-ng-plugin#546 here:

After merging with master now both tests are failing. This means, ArchUnit is now correctly running!

I really have no idea what caused the problem. Maybe something related to Jenkins complex class loader structure or dependencies. So I think I we can close #395. If it happens again I'll try to provide a simpler way to reproduce.

[uhafner]

I also remove the 2 dots in the @AnalyzeClasses(packages = "io.jenkins.plugins.analysis..") annotations now in all my tests. Seems to work in all my plugins in HEAD.

[codecholeric]

Okay, let me know if there are further problems! 😃