[mlir][CAPI] Proposal: Always building a libMLIRPublicAPI.so.

We were discussing on discord regarding the need for extension-based systems like Python to dynamically link against MLIR (or else you can only have one extension that depends on it). Currently, when I set that up, I piggy-backed off of the flag that enables build libLLVM.so and libMLIR.so and depended on libMLIR.so from the python extension if shared library building was enabled. However, this is less than ideal.

In the current setup, libMLIR.so exports both all symbols from the C++ API and the C-API. The former is a kitchen sink and the latter is curated. We should be splitting them and for things that are properly factored to depend on the C-API, they should have the option to *only* depend on the C-API, and we should build that shared library no matter what. Its presence isn't just an optimization: it is a key part of the system.

To do this right, I needed to:

* Introduce visibility macros into mlir-c/Support.h. These should work on both *nix and windows as-is.
* Create a new libMLIRPublicAPI.so with just the mlir-c object files.
* Compile the C-API with -fvisibility=hidden.
* Conditionally depend on the libMLIR.so from libMLIRPublicAPI.so if building libMLIR.so (otherwise, also links against the static libs and will produce a mondo libMLIRPublicAPI.so).
* Disable re-exporting of static library symbols that come in as transitive deps.

This gives us a dynamic linked C-API layer that is minimal and should work as-is on all platforms. Since we don't support libMLIR.so building on Windows yet (and it is not very DLL friendly), this will fall back to a mondo build of libMLIRPublicAPI.so, which has its uses (it is also the most size conscious way to go if you happen to know exactly what you need).

Sizes (release/stripped, Ubuntu 20.04):

Shared library build:
	libMLIRPublicAPI.so: 121Kb
	_mlir.cpython-38-x86_64-linux-gnu.so: 1.4Mb
	mlir-capi-ir-test: 135Kb
	libMLIR.so: 21Mb

Static build:
	libMLIRPublicAPI.so: 5.5Mb (since this is a "static" build, this includes the MLIR implementation as non-exported code).
	_mlir.cpython-38-x86_64-linux-gnu.so: 1.4Mb
	mlir-capi-ir-test: 44Kb

Things like npcomp and circt which bring their own dialects/transforms/etc would still need the shared library build and code that links against libMLIR.so (since it is all C++ interop stuff), but hopefully things that only depend on the public C-API can just have the one narrow dep.

I spot checked everything with nm, and it looks good in terms of what is exporting/importing from each layer.

I'm not in a hurry to land this, but if it is controversial, I'll probably split off the Support.h and API visibility macro changes, since we should set that pattern regardless.

Reviewed By: mehdi_amini, benvanik

Differential Revision: https://reviews.llvm.org/D90824

mlir/cmake/modules/AddMLIR.cmake

@@ -165,6 +165,33 @@ function(add_mlir_library_install name)
   set_property(GLOBAL APPEND PROPERTY MLIR_EXPORTS ${name})
 endfunction()
 
+# Declare an mlir library which is part of the public C-API and will be
+# compiled and exported into libMLIRPublicAPI.so/MLIRPublicAPI.dll.
+# This shared library is built regardless of the overall setting of building
+# libMLIR.so (which exports the C++ implementation).
+function(add_mlir_public_c_api_library name)
+  add_mlir_library(${name}
+    ${ARGN}
+    # NOTE: Generates obj.${name} which is used for shared library building.
+    OBJECT
+    EXCLUDE_FROM_LIBMLIR
+    ADDITIONAL_HEADER_DIRS
+    ${MLIR_MAIN_INCLUDE_DIR}/mlir-c
+  )
+  # API libraries compile with hidden visibility and macros that enable
+  # exporting from the DLL. Only apply to the obj lib, which only affects
+  # the exports via a shared library.
+  set_target_properties(obj.${name}
+    PROPERTIES
+    CXX_VISIBILITY_PRESET hidden
+  )
+  target_compile_definitions(obj.${name}
+    PRIVATE
+    -DMLIR_CAPI_BUILDING_LIBRARY=1
+  )
+  set_property(GLOBAL APPEND PROPERTY MLIR_PUBLIC_C_API_LIBS ${name})
+endfunction()
+
 # Declare the library associated with a dialect.
 function(add_mlir_dialect_library name)
   set_property(GLOBAL APPEND PROPERTY MLIR_DIALECT_LIBS ${name})

mlir/cmake/modules/AddMLIRPythonExtension.cmake

@@ -13,9 +13,6 @@ function(add_mlir_python_extension libname extname)
   if ("${ARG_SOURCES}" STREQUAL "")
     message(FATAL_ERROR " Missing SOURCES argument to add_mlir_python_extension(${libname}, ...")
   endif()
-  if(NOT LLVM_BUILD_LLVM_DYLIB)
-    message(FATAL_ERROR "Building MLIR Python extension require -DLLVM_BUILD_LLVM_DYLIB=ON")
-  endif()
 
   # Normally on unix-like platforms, extensions are built as "MODULE" libraries
   # and do not explicitly link to the python shared object. This allows for
@@ -85,13 +82,23 @@ function(add_mlir_python_extension libname extname)
   # to take place.
   set_target_properties(${libname} PROPERTIES CXX_VISIBILITY_PRESET "hidden")
 
+  # Python extensions depends *only* on the public API and LLVMSupport unless
+  # if further dependencies are added explicitly.
   target_link_libraries(${libname}
     PRIVATE
-    MLIR # Always link to libMLIR.so
+    MLIRPublicAPI
+    LLVMSupport
     ${ARG_LINK_LIBS}
     ${PYEXT_LIBADD}
   )
 
+  target_link_options(${libname}
+    PRIVATE
+      # On Linux, disable re-export of any static linked libraries that
+      # came through.
+      $<$<PLATFORM_ID:Linux>:LINKER:--exclude-libs,ALL>
+  )
+
   llvm_setup_rpath(${libname})
 
   ################################################################################

mlir/include/mlir-c/AffineExpr.h

@@ -42,128 +42,146 @@ DEFINE_C_API_STRUCT(MlirAffineExpr, const void);
 #undef DEFINE_C_API_STRUCT
 
 /// Gets the context that owns the affine expression.
-MlirContext mlirAffineExprGetContext(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED MlirContext
+mlirAffineExprGetContext(MlirAffineExpr affineExpr);
 
 /** Prints an affine expression by sending chunks of the string representation
  * and forwarding `userData to `callback`. Note that the callback may be called
  * several times with consecutive chunks of the string. */
-void mlirAffineExprPrint(MlirAffineExpr affineExpr, MlirStringCallback callback,
-                         void *userData);
+MLIR_CAPI_EXPORTED void mlirAffineExprPrint(MlirAffineExpr affineExpr,
+                                            MlirStringCallback callback,
+                                            void *userData);
 
 /// Prints the affine expression to the standard error stream.
-void mlirAffineExprDump(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED void mlirAffineExprDump(MlirAffineExpr affineExpr);
 
 /** Checks whether the given affine expression is made out of only symbols and
  * constants. */
-int mlirAffineExprIsSymbolicOrConstant(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED int
+mlirAffineExprIsSymbolicOrConstant(MlirAffineExpr affineExpr);
 
 /** Checks whether the given affine expression is a pure affine expression, i.e.
  * mul, floordiv, ceildic, and mod is only allowed w.r.t constants. */
-int mlirAffineExprIsPureAffine(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED int mlirAffineExprIsPureAffine(MlirAffineExpr affineExpr);
 
 /** Returns the greatest known integral divisor of this affine expression. The
  * result is always positive. */
-int64_t mlirAffineExprGetLargestKnownDivisor(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED int64_t
+mlirAffineExprGetLargestKnownDivisor(MlirAffineExpr affineExpr);
 
 /// Checks whether the given affine expression is a multiple of 'factor'.
-int mlirAffineExprIsMultipleOf(MlirAffineExpr affineExpr, int64_t factor);
+MLIR_CAPI_EXPORTED int mlirAffineExprIsMultipleOf(MlirAffineExpr affineExpr,
+                                                  int64_t factor);
 
 /** Checks whether the given affine expression involves AffineDimExpr
  * 'position'. */
-int mlirAffineExprIsFunctionOfDim(MlirAffineExpr affineExpr, intptr_t position);
+MLIR_CAPI_EXPORTED int mlirAffineExprIsFunctionOfDim(MlirAffineExpr affineExpr,
+                                                     intptr_t position);
 
 //===----------------------------------------------------------------------===//
 // Affine Dimension Expression.
 //===----------------------------------------------------------------------===//
 
 /// Creates an affine dimension expression with 'position' in the context.
-MlirAffineExpr mlirAffineDimExprGet(MlirContext ctx, intptr_t position);
+MLIR_CAPI_EXPORTED MlirAffineExpr mlirAffineDimExprGet(MlirContext ctx,
+                                                       intptr_t position);
 
 /// Returns the position of the given affine dimension expression.
-intptr_t mlirAffineDimExprGetPosition(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED intptr_t
+mlirAffineDimExprGetPosition(MlirAffineExpr affineExpr);
 
 //===----------------------------------------------------------------------===//
 // Affine Symbol Expression.
 //===----------------------------------------------------------------------===//
 
 /// Creates an affine symbol expression with 'position' in the context.
-MlirAffineExpr mlirAffineSymbolExprGet(MlirContext ctx, intptr_t position);
+MLIR_CAPI_EXPORTED MlirAffineExpr mlirAffineSymbolExprGet(MlirContext ctx,
+                                                          intptr_t position);
 
 /// Returns the position of the given affine symbol expression.
-intptr_t mlirAffineSymbolExprGetPosition(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED intptr_t
+mlirAffineSymbolExprGetPosition(MlirAffineExpr affineExpr);
 
 //===----------------------------------------------------------------------===//
 // Affine Constant Expression.
 //===----------------------------------------------------------------------===//
 
 /// Creates an affine constant expression with 'constant' in the context.
-MlirAffineExpr mlirAffineConstantExprGet(MlirContext ctx, int64_t constant);
+MLIR_CAPI_EXPORTED MlirAffineExpr mlirAffineConstantExprGet(MlirContext ctx,
+                                                            int64_t constant);
 
 /// Returns the value of the given affine constant expression.
-int64_t mlirAffineConstantExprGetValue(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED int64_t
+mlirAffineConstantExprGetValue(MlirAffineExpr affineExpr);
 
 //===----------------------------------------------------------------------===//
 // Affine Add Expression.
 //===----------------------------------------------------------------------===//
 
 /// Checks whether the given affine expression is an add expression.
-int mlirAffineExprIsAAdd(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED int mlirAffineExprIsAAdd(MlirAffineExpr affineExpr);
 
 /// Creates an affine add expression with 'lhs' and 'rhs'.
-MlirAffineExpr mlirAffineAddExprGet(MlirAffineExpr lhs, MlirAffineExpr rhs);
+MLIR_CAPI_EXPORTED MlirAffineExpr mlirAffineAddExprGet(MlirAffineExpr lhs,
+                                                       MlirAffineExpr rhs);
 
 //===----------------------------------------------------------------------===//
 // Affine Mul Expression.
 //===----------------------------------------------------------------------===//
 
 /// Checks whether the given affine expression is an mul expression.
-int mlirAffineExprIsAMul(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED int mlirAffineExprIsAMul(MlirAffineExpr affineExpr);
 
 /// Creates an affine mul expression with 'lhs' and 'rhs'.
-MlirAffineExpr mlirAffineMulExprGet(MlirAffineExpr lhs, MlirAffineExpr rhs);
+MLIR_CAPI_EXPORTED MlirAffineExpr mlirAffineMulExprGet(MlirAffineExpr lhs,
+                                                       MlirAffineExpr rhs);
 
 //===----------------------------------------------------------------------===//
 // Affine Mod Expression.
 //===----------------------------------------------------------------------===//
 
 /// Checks whether the given affine expression is an mod expression.
-int mlirAffineExprIsAMod(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED int mlirAffineExprIsAMod(MlirAffineExpr affineExpr);
 
 /// Creates an affine mod expression with 'lhs' and 'rhs'.
-MlirAffineExpr mlirAffineModExprGet(MlirAffineExpr lhs, MlirAffineExpr rhs);
+MLIR_CAPI_EXPORTED MlirAffineExpr mlirAffineModExprGet(MlirAffineExpr lhs,
+                                                       MlirAffineExpr rhs);
 
 //===----------------------------------------------------------------------===//
 // Affine FloorDiv Expression.
 //===----------------------------------------------------------------------===//
 
 /// Checks whether the given affine expression is an floordiv expression.
-int mlirAffineExprIsAFloorDiv(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED int mlirAffineExprIsAFloorDiv(MlirAffineExpr affineExpr);
 
 /// Creates an affine floordiv expression with 'lhs' and 'rhs'.
-MlirAffineExpr mlirAffineFloorDivExprGet(MlirAffineExpr lhs,
-                                         MlirAffineExpr rhs);
+MLIR_CAPI_EXPORTED MlirAffineExpr mlirAffineFloorDivExprGet(MlirAffineExpr lhs,
+                                                            MlirAffineExpr rhs);
 
 //===----------------------------------------------------------------------===//
 // Affine CeilDiv Expression.
 //===----------------------------------------------------------------------===//
 
 /// Checks whether the given affine expression is an ceildiv expression.
-int mlirAffineExprIsACeilDiv(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED int mlirAffineExprIsACeilDiv(MlirAffineExpr affineExpr);
 
 /// Creates an affine ceildiv expression with 'lhs' and 'rhs'.
-MlirAffineExpr mlirAffineCeilDivExprGet(MlirAffineExpr lhs, MlirAffineExpr rhs);
+MLIR_CAPI_EXPORTED MlirAffineExpr mlirAffineCeilDivExprGet(MlirAffineExpr lhs,
+                                                           MlirAffineExpr rhs);
 
 //===----------------------------------------------------------------------===//
 // Affine Binary Operation Expression.
 //===----------------------------------------------------------------------===//
 
 /** Returns the left hand side affine expression of the given affine binary
  * operation expression. */
-MlirAffineExpr mlirAffineBinaryOpExprGetLHS(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED MlirAffineExpr
+mlirAffineBinaryOpExprGetLHS(MlirAffineExpr affineExpr);
 
 /** Returns the right hand side affine expression of the given affine binary
  * operation expression. */
-MlirAffineExpr mlirAffineBinaryOpExprGetRHS(MlirAffineExpr affineExpr);
+MLIR_CAPI_EXPORTED MlirAffineExpr
+mlirAffineBinaryOpExprGetRHS(MlirAffineExpr affineExpr);
 
 #ifdef __cplusplus
 }

mlir/include/mlir-c/AffineMap.h

@@ -41,114 +41,120 @@ DEFINE_C_API_STRUCT(MlirAffineMap, const void);
 #undef DEFINE_C_API_STRUCT
 
 /// Gets the context that the given affine map was created with
-MlirContext mlirAffineMapGetContext(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED MlirContext mlirAffineMapGetContext(MlirAffineMap affineMap);
 
 /// Checks whether an affine map is null.
-inline int mlirAffineMapIsNull(MlirAffineMap affineMap) {
+static inline int mlirAffineMapIsNull(MlirAffineMap affineMap) {
   return !affineMap.ptr;
 }
 
 /// Checks if two affine maps are equal.
-int mlirAffineMapEqual(MlirAffineMap a1, MlirAffineMap a2);
+MLIR_CAPI_EXPORTED int mlirAffineMapEqual(MlirAffineMap a1, MlirAffineMap a2);
 
 /** Prints an affine map by sending chunks of the string representation and
  * forwarding `userData to `callback`. Note that the callback may be called
  * several times with consecutive chunks of the string. */
-void mlirAffineMapPrint(MlirAffineMap affineMap, MlirStringCallback callback,
-                        void *userData);
+MLIR_CAPI_EXPORTED void mlirAffineMapPrint(MlirAffineMap affineMap,
+                                           MlirStringCallback callback,
+                                           void *userData);
 
 /// Prints the affine map to the standard error stream.
-void mlirAffineMapDump(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED void mlirAffineMapDump(MlirAffineMap affineMap);
 
 /** Creates a zero result affine map with no dimensions or symbols in the
  * context. The affine map is owned by the context. */
-MlirAffineMap mlirAffineMapEmptyGet(MlirContext ctx);
+MLIR_CAPI_EXPORTED MlirAffineMap mlirAffineMapEmptyGet(MlirContext ctx);
 
 /** Creates a zero result affine map of the given dimensions and symbols in the
  * context. The affine map is owned by the context. */
-MlirAffineMap mlirAffineMapGet(MlirContext ctx, intptr_t dimCount,
-                               intptr_t symbolCount);
+MLIR_CAPI_EXPORTED MlirAffineMap mlirAffineMapGet(MlirContext ctx,
+                                                  intptr_t dimCount,
+                                                  intptr_t symbolCount);
 
 /** Creates a single constant result affine map in the context. The affine map
  * is owned by the context. */
-MlirAffineMap mlirAffineMapConstantGet(MlirContext ctx, int64_t val);
+MLIR_CAPI_EXPORTED MlirAffineMap mlirAffineMapConstantGet(MlirContext ctx,
+                                                          int64_t val);
 
 /** Creates an affine map with 'numDims' identity in the context. The affine map
  * is owned by the context. */
-MlirAffineMap mlirAffineMapMultiDimIdentityGet(MlirContext ctx,
-                                               intptr_t numDims);
+MLIR_CAPI_EXPORTED MlirAffineMap
+mlirAffineMapMultiDimIdentityGet(MlirContext ctx, intptr_t numDims);
 
 /** Creates an identity affine map on the most minor dimensions in the context.
  * The affine map is owned by the context. The function asserts that the number
  * of dimensions is greater or equal to the number of results. */
-MlirAffineMap mlirAffineMapMinorIdentityGet(MlirContext ctx, intptr_t dims,
-                                            intptr_t results);
+MLIR_CAPI_EXPORTED MlirAffineMap
+mlirAffineMapMinorIdentityGet(MlirContext ctx, intptr_t dims, intptr_t results);
 
 /** Creates an affine map with a permutation expression and its size in the
  * context. The permutation expression is a non-empty vector of integers.
  * The elements of the permutation vector must be continuous from 0 and cannot
  * be repeated (i.e. `[1,2,0]` is a valid permutation. `[2,0]` or `[1,1,2]` is
  * an invalid invalid permutation.) The affine map is owned by the context. */
-MlirAffineMap mlirAffineMapPermutationGet(MlirContext ctx, intptr_t size,
-                                          unsigned *permutation);
+MLIR_CAPI_EXPORTED MlirAffineMap mlirAffineMapPermutationGet(
+    MlirContext ctx, intptr_t size, unsigned *permutation);
 
 /** Checks whether the given affine map is an identity affine map. The function
  * asserts that the number of dimensions is greater or equal to the number of
  * results. */
-int mlirAffineMapIsIdentity(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED int mlirAffineMapIsIdentity(MlirAffineMap affineMap);
 
 /// Checks whether the given affine map is a minor identity affine map.
-int mlirAffineMapIsMinorIdentity(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED int mlirAffineMapIsMinorIdentity(MlirAffineMap affineMap);
 
 /// Checks whether the given affine map is an empty affine map.
-int mlirAffineMapIsEmpty(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED int mlirAffineMapIsEmpty(MlirAffineMap affineMap);
 
 /** Checks whether the given affine map is a single result constant affine
  * map. */
-int mlirAffineMapIsSingleConstant(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED int mlirAffineMapIsSingleConstant(MlirAffineMap affineMap);
 
 /** Returns the constant result of the given affine map. The function asserts
  * that the map has a single constant result. */
-int64_t mlirAffineMapGetSingleConstantResult(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED int64_t
+mlirAffineMapGetSingleConstantResult(MlirAffineMap affineMap);
 
 /// Returns the number of dimensions of the given affine map.
-intptr_t mlirAffineMapGetNumDims(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED intptr_t mlirAffineMapGetNumDims(MlirAffineMap affineMap);
 
 /// Returns the number of symbols of the given affine map.
-intptr_t mlirAffineMapGetNumSymbols(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED intptr_t mlirAffineMapGetNumSymbols(MlirAffineMap affineMap);
 
 /// Returns the number of results of the given affine map.
-intptr_t mlirAffineMapGetNumResults(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED intptr_t mlirAffineMapGetNumResults(MlirAffineMap affineMap);
 
 /** Returns the number of inputs (dimensions + symbols) of the given affine
  * map. */
-intptr_t mlirAffineMapGetNumInputs(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED intptr_t mlirAffineMapGetNumInputs(MlirAffineMap affineMap);
 
 /** Checks whether the given affine map represents a subset of a symbol-less
  * permutation map. */
-int mlirAffineMapIsProjectedPermutation(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED int
+mlirAffineMapIsProjectedPermutation(MlirAffineMap affineMap);
 
 /** Checks whether the given affine map represents a symbol-less permutation
  * map. */
-int mlirAffineMapIsPermutation(MlirAffineMap affineMap);
+MLIR_CAPI_EXPORTED int mlirAffineMapIsPermutation(MlirAffineMap affineMap);
 
 /// Returns the affine map consisting of the `resultPos` subset.
-MlirAffineMap mlirAffineMapGetSubMap(MlirAffineMap affineMap, intptr_t size,
-                                     intptr_t *resultPos);
+MLIR_CAPI_EXPORTED MlirAffineMap mlirAffineMapGetSubMap(MlirAffineMap affineMap,
+                                                        intptr_t size,
+                                                        intptr_t *resultPos);
 
 /** Returns the affine map consisting of the most major `numResults` results.
  * Returns the null AffineMap if the `numResults` is equal to zero.
  * Returns the `affineMap` if `numResults` is greater or equals to number of
  * results of the given affine map. */
-MlirAffineMap mlirAffineMapGetMajorSubMap(MlirAffineMap affineMap,
-                                          intptr_t numResults);
+MLIR_CAPI_EXPORTED MlirAffineMap
+mlirAffineMapGetMajorSubMap(MlirAffineMap affineMap, intptr_t numResults);
 
 /** Returns the affine map consisting of the most minor `numResults` results.
  * Returns the null AffineMap if the `numResults` is equal to zero.
  * Returns the `affineMap` if `numResults` is greater or equals to number of
  * results of the given affine map. */
-MlirAffineMap mlirAffineMapGetMinorSubMap(MlirAffineMap affineMap,
-                                          intptr_t numResults);
+MLIR_CAPI_EXPORTED MlirAffineMap
+mlirAffineMapGetMinorSubMap(MlirAffineMap affineMap, intptr_t numResults);
 
 #ifdef __cplusplus
 }