New Action Runner Ubuntu image fails to start Junit test

[mariokoehler]

Description

with the new action runner ubuntu 22.04 image announced here (#7660) we see problems running our Arquillian Junit tests. The maven surefire plugin indicates that it has run the tests, but the actual test is not really executed (debug logs normally written by the test show up nowhere, and files that should have been created by the tests are missing, thats why we are certain the test hasn't actually run). we had random individual builds that still pulled the old image and they were fine, but whenever the new image gets pulled this behavior appears.

we had to switch to Ubuntu 20.04 to avoid the problem for now.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

Ubuntu 22.04:

20230604.1
20230611.1

Only Ubuntu 22.04 is affected! Ubuntu 20.04 works fine.

Is it regression?

20230517.1

Expected behavior

after the unit test has run, we expect to find a directory and files in that directory

Actual behavior

the directory and the files that the unit test creates are missing. debug messages included into the unit test for analysis are not found in the log. conclusion: the test is never run, even though surefire plugin indicates it has

[INFO] Running de.otto.payments.voucher.gum.queue.outbound.vouchers.GenerateVouchersBluewhaleCDCExampleMessagesTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.139 s -- in de.otto.payments.voucher.gum.queue.outbound.vouchers.GenerateVouchersBluewhaleCDCExampleMessagesTest

A later step in the build can't find the files/directory:

ls: cannot access 'target/messages/bluewhale': No such file or directory

Repro steps

in our case its enough to switch to the new Ubuntu 22.04 image to consistently reproduce the effect. switching back to Ubuntu 20.04 or being "lucky" and getting a runner that still uses the older Ubuntu 22.04 image resolves the problem.

[Alexey-Ayupov]

Hi @mariokoehler, could you please provide actual repro steps (a code snippet will be the best)? It will help us to better understand where the problem is.

[mariokoehler]

to be able to do that, i would have to pick apart our build and try to isolate the "minimum viable project" that still triggers the problem, which would be a huge task. and i'm not willing to invest that kind of time at this point. frankly, it was already a huge time drain to identify the image version as the source of the problem...it wasn't the first thing we looked at ;-) All i can tell you at this time is that we use maven to run our Arquillian unit tests, and we have done it this way for literal years in half a dozen projects without any problems. and with the new image version it no longer works. as a fellow software engineer, i get that its gonna be hard for you to analyze this, and honestly, i didn't expect you to pull a solution out of the hat right away. the intention was more like "raising a flag" and see if others rally around it with the same problem, so maybe we get a better idea whats behind it.

...and maybe don't delete the old image just yet ;-)

[ilia-shipitsin]

@mariokoehler , sorry we do not provide image pinning. maybe containerized build can pin images for you, for example

jobs:
  build:
    runs-on: ubuntu-latest
    container:
      image: php:${{ matrix.php_version }}
    strategy:
      matrix:
        php_version: ["8.2", "8.1"]

as for "minimum viable project", I'm afraid we have no option to move forward without repro steps

[vpolikarpov-akvelon]

Hey @mariokoehler.
You said that you use Arquillian in many projects. Do you experience this strange behavior you described in all of them? If not, could you try to analyze what common features affected projects have.

Also I would like you to examine update notes for Ubuntu 22 runner image: #7674, #7701. Do you see updates for any software that you use in your project? Arquillian project doesn't seem well-maintained so any little update may break it.

Anyway, the simplest assumption is that files you are looking for are either in another directory or being deleted after successful test run. We can't check it as we don't know what software you use and how your workflows are organized so I ask you to verify that there were no updates for software you are using that could introduce such changes.

[mariokoehler]

we're currently only experiencing this in one of our projects. that project is the first that we migrated from javax to jakarta. so it uses many "newer" versions of the dependencies that we usually use, like Payara 6 instead of Payara 5 and so forth. We believe it might have something to do with the javax/jakarta switch. AND starting today the newest Version of the Ubuntu 20.04 has the same problem as well. so i guess now we have to invest the additional time to investigate this, because we have no workaround anymore.

[mariokoehler]

the "weirdness" continues...today, with Ubuntu 22.04 and image version 20230619.1.0 the build started working again (we haven't really changed anything in our build, because we don't know where to start, yet). the newest version of Ubuntu 20.04 is still broken, though. no additional insights other than that from our side at this point.

[mikhailkoliada]

@mariokoehler Please provide a minimal repro steps workflows, otherwise we have no data to work with to reproduce the problem

[erik-bershel]

Hi @mariokoehler!
Please update information on this issue. Whats going on with the latest runners? How about any of repro steps?

[mariokoehler]

it comes and goes...Ubuntu 22.04 20230619.1.0 was fine again, but 20230625.1.0 reintroduced the behaviour. i think we're at an impasse. we're not gonna be able to give you minimal repro steps. so i think i'm gonna close the issue for now.