[clang-repl] JIT linkage issues with extern templates

[jeaye]

Overview

Hey folks! I've found that the clang interpreter on the latest 19 build cannot link to extern templates definitions. I have verified that this bug also existed in release/18.x, so it's not new to 19.

Pinging @vgvassilev for vis, since I originally reported this to him with a more complex repro case.

Test case

This requires linking to an extern template which is defined either within the current process or in a loaded shared library. In my test case, I follow the existing pattern of shared object loading to build a shared object containing our defined function. You can see the patch to add the test here:

diff --git a/clang/test/Interpreter/Inputs/extern-template.cpp b/clang/test/Interpreter/Inputs/extern-template.cpp
new file mode 100644
index 000000000000..e08263a96c8f
--- /dev/null
+++ b/clang/test/Interpreter/Inputs/extern-template.cpp
@@ -0,0 +1,3 @@
+#include <cstdio>
+
+char foo(char c) { printf("good %c", c); return c; }
diff --git a/clang/test/Interpreter/Inputs/extern-template.hpp b/clang/test/Interpreter/Inputs/extern-template.hpp
new file mode 100644
index 000000000000..ac067a202dc8
--- /dev/null
+++ b/clang/test/Interpreter/Inputs/extern-template.hpp
@@ -0,0 +1,5 @@
+#pragma once
+
+template <typename T> T foo(T t) { return t; }
+
+extern template char foo(char c);
diff --git a/clang/test/Interpreter/extern-template.cpp b/clang/test/Interpreter/extern-template.cpp
new file mode 100644
index 000000000000..edb586285bdf
--- /dev/null
+++ b/clang/test/Interpreter/extern-template.cpp
@@ -0,0 +1,16 @@
+// REQUIRES: host-supports-jit, x86_64-linux
+
+// To generate libextern-template.so :
+// clang -xc++ -o Inputs/libextern-template.so -fPIC -shared Inputs/extern-template.cpp
+
+// RUN: cat %s | env LD_LIBRARY_PATH=%S/Inputs:$LD_LIBRARY_PATH clang-repl -Xcc -I%S/Inputs | FileCheck %s
+// RUN: cat %s | env LD_LIBRARY_PATH=%S/Inputs:$LD_LIBRARY_PATH clang-repl -Xcc -I%S/Inputs -Xcc -O2 | FileCheck %s
+
+extern "C" int printf(const char* format, ...);
+
+#include "extern-template.hpp"
+
+%lib libextern-template.so
+
+foo('a');
+// CHECK: good

The output of running this test (after having compiled libextern-template.so):

❯ ./bin/llvm-lit -j1 -v ../clang/test/Interpreter/extern-template.cpp
llvm-lit: /home/jeaye/projects/third-party/llvm-project/llvm/utils/lit/lit/llvm/config.py:509: note: using clang: /home/jeaye/projects/third-party/llvm-project/build/bin/clang
-- Testing: 1 tests, 1 workers --
FAIL: Clang :: Interpreter/extern-template.cpp (1 of 1)
******************** TEST 'Clang :: Interpreter/extern-template.cpp' FAILED ********************
Exit Code: 1

...

JIT session error: Symbols not found: [ _Z3fooIcET_S0_ ]
error: Failed to materialize symbols: { (main, { __orc_init_func.incr_module_15, $.incr_module_15.__inits.0 }) }
error: Failed to materialize symbols: { (main, { __orc_init_func.incr_module_15 }) }
/home/jeaye/projects/third-party/llvm-project/clang/test/Interpreter/extern-template.cpp:16:11: error: CHECK: expected string not found in input
// CHECK: good

...

********************
********************
Failed Tests (1):
  Clang :: Interpreter/extern-template.cpp
Testing Time: 0.05s
Total Discovered Tests: 1
  Failed: 1 (100.00%)

Additional notes

In the Inputs/extern-template.hpp file, we have this line:

extern template char foo(char c);

If it is instead changed to this, the test will pass:

char foo(char c);

Real world example

My test case is reduced down from a real world use case I originally noticed in the fmt library: https://github.com/fmtlib/fmt/blob/b61c8c3d23b7e6fdf9d44593877dba1c8a291be1/include/fmt/format.h#L4283

[llvmbot]

@llvm/issue-subscribers-orcjit

Author: Jeaye Wilkerson (jeaye)

## Overview Hey folks! I've found that the clang interpreter on the latest 19 build cannot link to extern templates definitions. I have verified that this bug also existed in `release/18.x`, so it's not new to 19.

Pinging @vgvassilev for vis, since I originally reported this to him with a more complex repro case.

Test case

This requires linking to an extern template which is defined either within the current process or in a loaded shared library. In my test case, I follow the existing pattern of shared object loading to build a shared object containing our defined function. You can see the patch to add the test here:

diff --git a/clang/test/Interpreter/Inputs/extern-template.cpp b/clang/test/Interpreter/Inputs/extern-template.cpp
new file mode 100644
index 000000000000..e08263a96c8f
--- /dev/null
+++ b/clang/test/Interpreter/Inputs/extern-template.cpp
@@ -0,0 +1,3 @@
+#include <cstdio>
+
+char foo(char c) { printf("good %c", c); return c; }
diff --git a/clang/test/Interpreter/Inputs/extern-template.hpp b/clang/test/Interpreter/Inputs/extern-template.hpp
new file mode 100644
index 000000000000..ac067a202dc8
--- /dev/null
+++ b/clang/test/Interpreter/Inputs/extern-template.hpp
@@ -0,0 +1,5 @@
+#pragma once
+
+template <typename T> T foo(T t) { return t; }
+
+extern template char foo(char c);
diff --git a/clang/test/Interpreter/extern-template.cpp b/clang/test/Interpreter/extern-template.cpp
new file mode 100644
index 000000000000..edb586285bdf
--- /dev/null
+++ b/clang/test/Interpreter/extern-template.cpp
@@ -0,0 +1,16 @@
+// REQUIRES: host-supports-jit, x86_64-linux
+
+// To generate libextern-template.so :
+// clang -xc++ -o Inputs/libextern-template.so -fPIC -shared Inputs/extern-template.cpp
+
+// RUN: cat %s | env LD_LIBRARY_PATH=%S/Inputs:$LD_LIBRARY_PATH clang-repl -Xcc -I%S/Inputs | FileCheck %s
+// RUN: cat %s | env LD_LIBRARY_PATH=%S/Inputs:$LD_LIBRARY_PATH clang-repl -Xcc -I%S/Inputs -Xcc -O2 | FileCheck %s
+
+extern "C" int printf(const char* format, ...);
+
+#include "extern-template.hpp"
+
+%lib libextern-template.so
+
+foo('a');
+// CHECK: good

The output of running this test (after having compiled libextern-template.so):

❯ ./bin/llvm-lit -j1 -v ../clang/test/Interpreter/extern-template.cpp
llvm-lit: /home/jeaye/projects/third-party/llvm-project/llvm/utils/lit/lit/llvm/config.py:509: note: using clang: /home/jeaye/projects/third-party/llvm-project/build/bin/clang
-- Testing: 1 tests, 1 workers --
FAIL: Clang :: Interpreter/extern-template.cpp (1 of 1)
******************** TEST 'Clang :: Interpreter/extern-template.cpp' FAILED ********************
Exit Code: 1

...

JIT session error: Symbols not found: [ _Z3fooIcET_S0_ ]
error: Failed to materialize symbols: { (main, { __orc_init_func.incr_module_15, $.incr_module_15.__inits.0 }) }
error: Failed to materialize symbols: { (main, { __orc_init_func.incr_module_15 }) }
/home/jeaye/projects/third-party/llvm-project/clang/test/Interpreter/extern-template.cpp:16:11: error: CHECK: expected string not found in input
// CHECK: good

...

********************
********************
Failed Tests (1):
  Clang :: Interpreter/extern-template.cpp
Testing Time: 0.05s
Total Discovered Tests: 1
  Failed: 1 (100.00%)

Additional notes

In the Inputs/extern-template.hpp file, we have this line:

extern template char foo(char c);

If it is instead changed to this, the test will pass:

char foo(char c);

Real world example

My test case is reduced down from a real world use case I originally noticed in the fmt library: https://github.com/fmtlib/fmt/blob/b61c8c3d23b7e6fdf9d44593877dba1c8a291be1/include/fmt/format.h#L4283