Adapt backend API for CLI

[DropD]

This is the first step towards the implementation of the CLI (GDP-1). This issue serves to track the implementation of the described changes. Each change will be treated in a separate issue.

Current situation

The GDP-1 proof-of-concept had to use internals of concrete backends which are not part of the public API to achieve a number of things. This would obviously not be maintainable.

Specifically the problems arise from the intention of making the CLI output easy to integrate into an external build system, while the current backend API is geared towards JIT generation/compilation for use from python only. Code generation is not cleanly separated from compilation and stencil-ids are baked into file / code object names and paths.

Changes to the API

Functionality:

• Generate the primary language code without (or with optional) unique stencil-id, returning a hierarchy of source files ready to be independently compiled in a client-defined location (or in-memory in a way that allows programmatically writing out).
• Generate language bindings (for python or other secondary language), again without (or optionally) baking in stencil-id and return in a format ready to be written / copied programatically to client-defined location (without compiling anything).
• Generate a secondary language module / source file (for bindings) without actually compiling the bindings, in a way that if the language bindings are compiled correctly the module / source file can be used from the target language.
• After generating bindings and secondary language source, compile bindings to make the target language source immediately usable. (not a backend refactoring anymore)

Data:

• set of secondary languages for which bindings can be generated

Design:

• separate caching from generating in backend API

Explanation:

Generate primary language code

For the GridTools backends this would be C++ or Cuda code. Both the JIT process as well as the CLI require the source, however for the CLI to embed transparently into external build systems the unique stencil-id must be optional. Also the user of the CLI should have control over where the source files end up to be written and the source file hierarchy (if there are more than one) must make sense for an external build tool. Obviously compiling for JIT usage must hapen separately from this step.

Generate language bindings

For GridTools, this is the pybind11 .cpp file for python as a secondary language. If no stencil-id is given, it should be generated under the assumption that the source files it refers to are generated without stencil-id and the output must make it clear where this file expects to be located relative to the primary language source files.

If a backend does not support secondary languages (if python is the primary language), or if the secondary language can call the primary at runtime this functionality may be absent.

Generate secondary language source

For GridTools backends this is the extension module that imports the compiled bindings from an .so object file and provides the python wrapper on top of that. In general it is the entry point for the secondary language to call the optimized stencils in an idiomatic way.

This may be absent if no secondary languages are supported (if python is the primary) or if the primary and secondary languages are so compatible that they do not require a wrapper.

Compile bindings

This is for when the CLI is also to be used as the build system, when the client code calling the stencils is written in a secondary language supported by the backend. This should be the same process as JIT compilation, except without / optional stencil-id in the file names of course, with the source files in the CLI-user specified location. The Idea is that afterwards the secondary language wrapper can be used immediately.

[DropD]

Since this affects backends which are already quite advanced in implementation I propose to implement these changes as optional features of the backend API (i.e. not abstract methods, but methods that by default raise NotImplemented, None default values for data members).

The CLI will then treat absence of any of these features as declaration of incompatibility with the CLI and gtpyc --backend "backend_with_old_interface" ... will simply error out with "Error: backend is not (yet) CLI enabled."

[DropD]

However the goal should be to enable all backends distributed with GT4Py sooner rather than later!

[egparedes]

The first two tasks (generation of the source code and generation of the bindings) are clear. I would only add that the output should be a single root folder (which may have sub-folders if needed) and we should also add a build tool config to let the user to build the generated code with a single command (with or without python bindings). I suggest to use CMake as the default build tool (you should check with @havogt to set a CMakeLists.txt using best practices) and later we could potentially add other tools, allowing the user to select which one to use with a CLI option.

The third task ("Generate secondary language source") is potentially unnecessary, since we might want to get rid of this wrapper in the future and implement the full interface of a StencilObject directly in the pybind11/bindings files. Of course, this is a different discussion (the API of the generated Python objects) which also depends on doing performance analysis of the code and running benchmarks. In my opinion, once the first 2 tasks are done it could be useful to evaluate again this task according to the state of gt4py and the other discussion going on, before starting the actual implementation.