GitLab generated pipelines fail when a test depends on two or more tests from previous stage

[sabo]

The following test workflow fails when run as individual GitLab tests generated using the --ci-generate option.

class BaseTest(rfm.RunOnlyRegressionTest):
    def __init__(self):
        self.valid_systems = ['*']
        self.valid_prog_environs = ['*']
        self.executable = 'echo'
        self.sanity_patterns = sn.assert_true(True)

@rfm.simple_test
class T0(BaseTest):
    pass

@rfm.simple_test
class T1(BaseTest):
    pass

@rfm.simple_test
class T2(BaseTest):
    def __init__(self):
        super().__init__()
        self.depends_on('T0')
        self.depends_on('T1')

Here's the pipeline.yaml generated:

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
  - rfm-stage/${CI_COMMIT_SHORT_SHA}
stages:
- rfm-stage-0
- rfm-stage-1
T0:
  stage: rfm-stage-0
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA}  -c /home/gitlab-runner/builds/jG5qeF1K/0/sablosky/reframe_ci_bug/rfm-tests/deps_complex.py  --report-file=rfm-report-0.json  -n T0 -r
  artifacts:
    paths:
    - rfm-report-0.json
  needs: []
T1:
  stage: rfm-stage-0
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA}  -c /home/gitlab-runner/builds/jG5qeF1K/0/sablosky/reframe_ci_bug/rfm-tests/deps_complex.py  --report-file=rfm-report-0.json  -n T1 -r
  artifacts:
    paths:
    - rfm-report-0.json
  needs: []
T2:
  stage: rfm-stage-1
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA}  -c /home/gitlab-runner/builds/jG5qeF1K/0/sablosky/reframe_ci_bug/rfm-tests/deps_complex.py  --report-file=rfm-report-1.json --restore-session=rfm-report-0.json -n T2 -r
  artifacts:
    paths:
    - rfm-report-1.json
  needs:
  - T0
  - T1

Jobs T0 and T1, as you would expect, run fine. T2 fails with this error:

run session stopped: reframe error: could not restore testcase ('T0', 'generic:default', 'builtin'): not found in the report file

Fairly sure what's happening is, when T2 runs, it grabs the artifacts emitted by the T0 and T1 jobs. But both are named rfm-report-0.json, and the second one from T1 clobbers the first from T0. So when the T2 job reads in the session file to restore from, it only sees the result of T1.

[sabo]

The only way I can think to work around this is for each job to have a unique report file, and then have some way of merging together the results of the test cases. I've hacked together a basic workaround in a branch in my repo. Don't think you'd want to munge the reports together the way I do, but the generated pipeline works:

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
  - rfm-stage/${CI_COMMIT_SHORT_SHA}
stages:
- rfm-stage-0
- rfm-stage-1
T0:
  stage: rfm-stage-0
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA}  -c /home/gitlab-runner/builds/jG5qeF1K/0/sablosky/reframe_ci_bug/rfm-tests/deps_complex.py  --report-file=T0-report.json  -n T0 -r
  artifacts:
    paths:
    - T0-report.json
  needs: []
T1:
  stage: rfm-stage-0
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA}  -c /home/gitlab-runner/builds/jG5qeF1K/0/sablosky/reframe_ci_bug/rfm-tests/deps_complex.py  --report-file=T1-report.json  -n T1 -r
  artifacts:
    paths:
    - T1-report.json
  needs: []
T2:
  stage: rfm-stage-1
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA}  -c /home/gitlab-runner/builds/jG5qeF1K/0/sablosky/reframe_ci_bug/rfm-tests/deps_complex.py  --report-file=T2-report.json --restore-session=T0-report.json,T1-report.json -n T2 -r
  artifacts:
    paths:
    - T2-report.json
  needs:
  - T0
  - T1

[vkarak]

Hi @sabo, I was able to reproduce it here. It is indeed what you say. The original test case that we were using got around it because the way the dependencies were topologically sorted.