Enable CMake Based Build System for Torchtext

[Nayef211]

🚀 Feature

Create native C++ binaries using a CMake based build system to allow third-parties to integrate torchtext in their applications

Motivation

Currently torchtext’s C++ extension and third party libraries are built using PyTorch’s CppExtension module. Though this approach has some pros, the main downsides are that it makes integrating third party libraries challenging and the resulting library is linked against libpython, via libtorch_python. Due to the dependency on libpython, OSS torchtext does not have a standalone C++ binary that downstream users can depend on. This is a blocker for some customers who want to use torchtext operators in a non-Python environment (using TorchScript or as a C++ API).

There are some issues created by users that further motivate this work:

• How to build pytorch text with system third_party libraries? #1615
• Error loading saved tokenizer  #1255

Pitch

We aim to resolve these problems by moving our build system to use CMake. Firstly, CMake should make it easier to integrate third party libraries simply by adding a reference to it’s own CMake file to build the library. Secondly, since CMake is language-agnostic, there is no need to link against libpython.

Before making any changes to torchtext’s build system, we need to ensure that there is adequate testing across all platforms in our OSS CI. This means enabling MacOS unit tests in CircleCI to catch platform specific build failures caused by the switch to CMake.

Milestones

• Enable MacOS Unit tests Adding MacOS unit tests on CircleCI #1672
• Update C++ includes to use imports relative to root directory (to resolve CMake build errors) [CMake 1/3] Update C++ includes to use imports relative to root directory #1666
• Switch the build mechanism from PyTorch’s CppExtension to CMake for torchtext and create a single _torchtext.so library with all source files from csrc/ folder [CMake 2/3] Add CMake Build to torchtext to create single _torchtext library #1673
• Separate out source files that have a Python dependency to _torchtext.so library which will link against a libtorchtext.so library containing source files with pure C++ dependencies [CMake 3/3] Split source files with Python dependency to separate library #1660
• Create a simple tutorial showcasing how to depend on libtorchtext.so within a simple C++ application Add libtorchtext cpp example #1817