Best practices for compiled dependencies

[mrtnkb]

Hi everyone,

I'm working in a scientific codebase which is mostly Python but has a reasonable amount of C++ and Fortran. Right now the Python is not managed by any build tool, the C++ is Bazel and the Fortran is compiled through scikit-build and build_ext.

I'm wondering what best practices would be for moving the Python to a build tool like Pants. Some things I've tried to not a lot of success:

• Using Bazel for everything. That's ok and pybind11 for calling C++ code is really nicely integrated but Python support is not ideal and breaks a lot of what many Python developers are familiar with. I also couldn't find a solution to integrating python setup.py build_ext; python setup.py install.
• Using Pants for Python and somehow dealing with C++ and Fortran deps. Pants is a real pleasure for pure Python but I couldn't work out a clean way to handle building and integrating C++/pybind11 or the esoteric Fortran package.

So my question: Is there a standard, recommended way to do this? Ideally I would have Python and C++ all in one repo (with protobufs compiled for both languages). Without C++ support is there a way to cleanly depend on files compiled by another tool in Pants?

Thanks a lot!

[benjyw]

Hi! It sounds like at least the Fortran, and maybe the C++ code, is built into native extensions?

And it sounds like you have enough C++ code that simply building it via build_ext is not performant, so you actually need something like Bazel there?

You could potentially invoke your existing C++/Fortran builds via Pants's adhoc_tool functionality, which lets you run arbitrary processes and use their outputs downstream. cc @chrisjrn for more on that.

@sureshjoshi has been looking at some C++ support so I'll ask him to weigh in as well.

[chrisjrn]

I've had success running CPython's makefile-based builds under adhoc_tool, so I'd definitely consider looking into that as a possible approach.

[mrtnkb]

It's not so much performance that leads us to Bazel for C++, but it's enough code that being able to break up into many small libararies with independent unit tests is very valuable. We'd also like to share the C++ libraries between both Python and our primary C++ codebase (all Bazel). If we just wanted to use build_ext I'm assuming that would that mean maintaining a parallel set of build rules in a make file?

@chrisjrn That sounds super relevant, have you got any example code? I looked briefly at using adhoc_tool, but it seems that the runnable needs to be a Pants target. Is that right?

shell_command seemed to be more likely to work? But for both shell_command or adhoc_tool I had a question about the source file dependencies. Would the entire set of source files need to be listed under execution_dependencies so that they can be used in the sandbox the tool is run in?