[feat] Add a new infrastructure for build systems #340
Conversation
The compilation phase is now separated from the programming environment.
It is part of the new concept of build systems. A build system is
responsible for generating the required commands for compiling a
code. The framework then uses these commands to generate a build job
script which is then submitted. Currently, only local compilation is
supported.
The current behavior of the `sourcepath` and `sourcesdir` attributes is
maintained for convenience. Internally, they translate to concrete build
systems that are set up accordingly.
All build systems share some basic attributes and behavior:
- The compilers and compilation flags. By default, if not specified the
corresponding values from the current programming environment will be
used.
- The ability to ignore completely the current programming
environment. A build system may be configured independently of the
current programming environment by explicitly setting the compilers
and compilation flags.
The above design allows the programming environment to become immutable
holding the global default values for each system.
Currently, for backward compatibility, it is not yet immutable, but
setting its attributes is now deprecated.
Two build systems are provided by this commit:
1. `SingleSource`: This build system is responsible for compiling a
single source file in any of the recognized programming languages,
i.e., C, C++, Fortran and CUDA.
2. `Make`: This build system is responsible for compiling a project
using the `make` command.
Create a different type that accepts Sequences.
| self._fflags = fflags | ||
| self._ldflags = ldflags | ||
| self._include_search_path = [] | ||
| self._propagate = True |
There was a problem hiding this comment.
I am not treating this variable in the new build systems. I should fix that for compatibility reasons.
| class BuildSystemError(ReframeError): | ||
| """Raised when a build system is not configured properly.""" | ||
|
|
||
|
|
There was a problem hiding this comment.
I should also remove the CompilationError. Not valid any more.
There was a problem hiding this comment.
Actually, I need to keep the CompilationError and change it. Before we used to print the output of the compilation output. We still need to treat such errors specially. Perhaps, we could print the path to the compilation output.
unittests/test_environments.py
Outdated
|
|
||
|
|
||
| class TestProgEnvironment(unittest.TestCase): | ||
| class _TestProgEnvironment: |
There was a problem hiding this comment.
Haha, a workaround I forgot to remove. I should fix this file.
reframe/core/pipeline.py
Outdated
|
|
||
| # FIXME: this check is not reliable for certain scheduler backends | ||
| if self._build_job.exitcode != 0: | ||
| raise PipelineError('compilation failed') |
There was a problem hiding this comment.
I should perhaps raise a more specific error here, e.g., BuildError. What do you think?
Codecov Report
@@ Coverage Diff @@
## master #340 +/- ##
=========================================
+ Coverage 91.13% 91.2% +0.06%
=========================================
Files 68 70 +2
Lines 8268 8581 +313
=========================================
+ Hits 7535 7826 +291
- Misses 733 755 +22
Continue to review full report at Codecov.
|
Shell script generation is revised significantly. Here are the key
points of this revision:
- ReFrame generates only Bash. This has always been the case, but the
previous design was such as to "enable" generation of other types of
shell scripts. This pseudo-generic design was completely dropped and
replaced by a Bash-only script generator. The rationale behind this is
that in order to support fully portable script generation, it would
require an intermediate language that would interface with the shell
script generators and would require users to program in that language
when setting the `pre_run`, `post_run` etc. attributes. This is way to
much effort with zero or negative value.
Another solution considered was to globally control the shell type
used by ReFrame from the settings. The downside with this solution is
that user tests are becoming inherently less portable, if a user wants
to write his `pre_run`, `post_run` etc. commands in, say, fish.
Finally, supporting multiple shell script generation backends would
make the internal design more complicated and less consistent, since
the different parts of the framework that need to emit shell commands
would be required to do that through another API and not directly.
For all these reasons, and given that Bash is the "standard" POSIX
shell, I decided to drop completely the old pseudo-generic design in a
favor of a simpler and more consistent behavior across the framework.
- The generated shell scripts are more sophisticated now. They can trap
different events (signals, errors and exit) during their execution and
act accordingly. These additions are crucial for two parts of the
framework:
1. For the generated build script. We want to trap errors in order to
exit immediately if any of the commands fail without requiring the
user to take extra precaution for that when setting the
`prebuild_cmd` and `postbuild_cmd`.
2. For getting reliable exit and signal code information for job
scheduler backends that do not support it. By trapping
signals (with a "terminate" or "core dump" default action) and the
shell exit, we can always record the both the signal number and the
exit code in the output. Then the scheduler backend can retrieve
this information.
NOTE: This feature is not yet implemented.
- A shell script inside ReFrame consists of the three parts:
1. The shebang, i.e., the very first line of the script.
2. The prolog
3. The body
4. The epilog
Anyone wanting to generate a script may decide in which part (except
the shebang) to emit the commands he likes. When the `finalize()`
method is called the whole script will be generated. ReFrame emits its
traps between the prolog and the body.
- The `Job` abstract class does not hold any more information not
relating directly to the job creation and status, i.e., `pre_run`,
`post_run`, `executable` etc. Instead, it provides a new, richer
version of the `prepare` method for generating the job script. This
new `prepare()` has the following signature:
```python
def prepare(self, commands, environs, **gen_opts):
```
The `commands` argument is a list of the actual shell commands to be
emitted in the job script. The caller is responsible for filling up
this list.
The `environs` argument is a list of the environments to be set loaded
before emitting the commands.
The `gen_opts` arguments are passed through the bash script generator.
- This PR establishes also conventions for the functions emitting shell
code. These should start with the `emit_` prefix and must return a
list of shell commands. They must not accept a shell script generator
object as argument. The standard way of generating a shell script is
the following:
```python
import reframe.core.shell as shell
with shell.generate_script(filename) as gen:
gen.write_prolog(emit_foo1())
gen.write_body(emit_bar())
```
The `generate_script` context manager takes care of the file creation
and the finalization of the shell script.
Finally, this PR brings some fixes in the unit tests regarding the
resources directory.
vkarak
changed the title
|
@jenkins-cscs retry daint kesch monch |
|
@teojgo @victorusu When reviewing this PR, I suggest playing a bit with your tests and try to adapt them to the build system syntax. This way you can have a better feeling of how it works and whether there are things that you don't like and need to be done differently. |
- The old `CompilationError` is removed. - This new exception just prints the location of the standard output and error of the build job.
|
The regression tests affected by this PR are a lot. I don't think it makes sense to include them in this one, cos it will become huge. I see two options: either we merge this to the master and cope with the deprecation warnings for some time in production or I will do a separate PR on this one (to be reviewed separately) that will update the affected tests. What do you think? |
|
I think that everyone can take some regression tests that is mostly familiar with and adapt them to the new syntax/new build system. That way we can also practice both of these new features and completed the transformation as quickly as possible. |
vkarak
changed the title
|
Removed WIP. |
vkarak
changed the title
victorusu
left a comment
victorusu
left a comment
There was a problem hiding this comment.
There are a lot of changes... I reviewed twice and I could not find anything that is broken...
I have also tested with the new build system with the DGEMM test and it worked fine for me...
teojgo
left a comment
teojgo
left a comment
There was a problem hiding this comment.
Other than the duplicate job name line it looks fine.
| def emit_preamble(self): | ||
| preamble = [ | ||
| self._format_option(self.name, '--job-name="{0}"'), | ||
| self._format_option(self.name, '--job-name="{0}"'), |
There was a problem hiding this comment.
The --job-name is set twice here.
…eframe into feature/buildsys-infrastructure
4 participants
This PR provides two major contributions:
New build system infrastructure
The compilation phase is now separated from the programming environment.
It is part of the new concept of build systems. A build system is
responsible for generating the required commands for compiling a
code. The framework then uses these commands to generate a build job
script which is then submitted. Currently, only local compilation is
supported.
The current behavior of the
sourcepathandsourcesdirattributes ismaintained for convenience. Internally, they translate to concrete build
systems that are set up accordingly.
All build systems share some basic attributes and behavior:
corresponding values from the current programming environment will be
used.
environment. A build system may be configured independently of the
current programming environment by explicitly setting the compilers
and compilation flags.
The above design allows the programming environment to become immutable
holding the global default values for each system.
Currently, for backward compatibility, it is not yet immutable, but
setting its attributes is now deprecated.
Two build systems are provided by this commit:
SingleSource: This build system is responsible for compiling asingle source file in any of the recognized programming languages,
i.e., C, C++, Fortran and CUDA.
Make: This build system is responsible for compiling a projectusing the
makecommand.Redesign of the shell script generation infrastructure
Shell script generation is revised significantly. Here are the key
points of this revision:
ReFrame generates only Bash. This has always been the case, but the
previous design was such as to "enable" generation of other types of
shell scripts. This pseudo-generic design was completely dropped and
replaced by a Bash-only script generator. The rationale behind this is
that in order to support fully portable script generation, it would
require an intermediate language that would interface with the shell
script generators and would require users to program in that language
when setting the
pre_run,post_runetc. attributes. This is way tomuch effort with zero or negative value.
Another solution considered was to globally control the shell type
used by ReFrame from the settings. The downside with this solution is
that user tests are becoming inherently less portable, if a user wants
to write his
pre_run,post_runetc. commands in, say, fish.Finally, supporting multiple shell script generation backends would
make the internal design more complicated and less consistent, since
the different parts of the framework that need to emit shell commands
would be required to do that through another API and not directly.
For all these reasons, and given that Bash is the "standard" POSIX
shell, I decided to drop completely the old pseudo-generic design in a
favor of a simpler and more consistent behavior across the framework.
The generated shell scripts are more sophisticated now. They can trap
different events (signals, errors and exit) during their execution and
act accordingly. These additions are crucial for two parts of the
framework:
For the generated build script. We want to trap errors in order to
exit immediately if any of the commands fail without requiring the
user to take extra precaution for that when setting the
prebuild_cmdandpostbuild_cmd.For getting reliable exit and signal code information for job
scheduler backends that do not support it. By trapping
signals (with a "terminate" or "core dump" default action) and the
shell exit, we can always record the both the signal number and the
exit code in the output. Then the scheduler backend can retrieve
this information.
NOTE: This feature is not yet implemented.
A shell script inside ReFrame consists of the three parts:
Anyone wanting to generate a script may decide in which part (except
the shebang) to emit the commands he likes. When the
finalize()method is called the whole script will be generated. ReFrame emits its
traps between the prolog and the body.
The
Jobabstract class does not hold any more information notrelating directly to the job creation and status, i.e.,
pre_run,post_run,executableetc. Instead, it provides a new, richerversion of the
preparemethod for generating the job script. Thisnew
prepare()has the following signature:The
commandsargument is a list of the actual shell commands to beemitted in the job script. The caller is responsible for filling up
this list.
The
environsargument is a list of the environments to be set loadedbefore emitting the commands.
The
gen_optsarguments are passed through the bash script generator.This PR establishes also conventions for the functions emitting shell
code. These should start with the
emit_prefix and must return alist of shell commands. They must not accept a shell script generator
object as argument. The standard way of generating a shell script is
the following:
The
generate_scriptcontext manager takes care of the file creationand the finalization of the shell script.
Fixes #295, #297, #296, #339, #176.
Still todo: