Test archive is empty due sub test passing

[AltamashShaikh]

Issue

Faced this issue when trying to sync failed test cases using the below command.
./console development:sync-system-test-processed --repository="innocraft/plugin-CustomReports" --http-user={USER} --http-password="{PASSWORD}" 674

On investigating further it was found out that the artifact archive for failed system test case is empty even though we can see failed test cases. On further digging it was found out that the build has 5 sub builds and out of which 1,2,3 and 5 failed and 4 passed.
The build which passed have made the archive empty.

Expected Behavior

The archive should not be empty and should contain all the processed files of individual sub builds.

[sgiehl]

Yes. It's currently kind of expected, that the last build that finished overwrites the artifacts. Not sure if we should try to fix that for travis or keep that in mind when building new scripts for github actions.

[tsteur]

@sgiehl for github actions: we could move the issue over to matomo-org then? Or not sure where issues for it will be managed and if we have again a submodule or not?

[AltamashShaikh]

Okay, so for now the workaround will be restart the single job after all the jobs are completed and then run a sync command to fix the testcases

[sgiehl]

@AltamashShaikh If some jobs fails and other succeed, that's most likely an issue of using different PHP versions or using another Matomo version. In that case simply syncing the files from the broken job, might let the previously working one fail. You then should consider ignoring the differences in the expected files by removing certain fields, not running the test for older Matomo versions at all or using different expected files depending on PHP or Matomo version.

[sgiehl]

With the new github action artifacts should by default only be uploaded by the job running the lowest php version. That way it is at least consistent. Nevertheless it can happen that a job using a newer PHP version fails. It won't be possible to fetch those artifacts.