Unexpected failure when running test

[vkarak]

We get the following stack trace when running a test:

FAILURE INFO for AffinityOpenMPTest_omp_bind_threads 
  * System partition: daint:gpu
  * Environment: PrgEnv-gnu
  * Stage directory: /scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/stage/daint/gpu/PrgEnv-gnu/AffinityOpenMPTest_omp_bind_threads
  * Job type: batch job (id=None)
  * Maintainers: ['RS', 'VK']
  * Failing phase: run
  * Reason: unexpected error: sequence item 2: expected str instance, _io.TextIOWrapper found
Traceback (most recent call last):
  File "/scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/reframe/core/schedulers/slurm.py", line 143, in _run_command
    return os_ext.run_command(cmd, check=True, timeout=timeout)
  File "/scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/reframe/utility/os_ext.py", line 29, in run_command
    proc.stderr, timeout) from None
reframe.core.exceptions.SpawnedProcessTimeout: <unprintable SpawnedProcessTimeout object>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/reframe/frontend/executors/__init__.py", line 58, in _safe_call
    return fn(*args, **kwargs)
  File "/scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/reframe/core/pipeline.py", line 1051, in run
    self._job.submit()
  File "/scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/reframe/core/schedulers/slurm.py", line 149, in submit
    completed = self._run_command(cmd, settings().job_submit_timeout)
  File "/scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/reframe/core/schedulers/slurm.py", line 145, in _run_command
    raise JobError(jobid=self._jobid) from e
reframe.core.exceptions.JobError: <unprintable JobError object>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/reframe/frontend/executors/__init__.py", line 58, in _safe_call
    return fn(*args, **kwargs)
  File "/scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/reframe/core/logging.py", line 446, in __exit__
    getlogger().log(self._level, msg.format(exc_fullname, exc_value))
  File "/scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/reframe/core/exceptions.py", line 195, in __str__
    msg = super().__str__()
  File "/scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/reframe/core/exceptions.py", line 22, in __str__
    ret += ': ' + str(self.__cause__)
  File "/scratch/snx3000/jenscscs/reframe-ci-daint-c955313-1434/reframe/core/exceptions.py", line 175, in __str__
    return '\n'.join(lines)
TypeError: sequence item 2: expected str instance, _io.TextIOWrapper found

It seems that an exception is thrown during run and when trying to form the exception message, we are getting this error.

More info here: https://jenkins.cscs.ch/blue/organizations/jenkins/ReframeCI/detail/ReframeCI/1434/pipeline

[teojgo]

This happens because the communicate method returns a tuple of _io.TextIOWrapper objects if the subprocess is not finished. This throws an error when trying to use the __str__ method of the SpawnedProcessTimeout exception.