Code that uses a libcall marks internal symbol as exported when making object file

[gbaraldi]

This is showing up as JuliaLang/julia#57658 (comment) which is holding up julia being able to compile with LLVM20.

The issue is that a defining a libcall in a module and then having it be used is making the libcall symbol become external

source_filename = "text"
target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-macosx10.14.0-macho"

declare i16 @julia_float_to_half(float)

define internal i16 @__truncsfhf2(float %0) {
  %2 = call i16 @julia_float_to_half(float %0)
  ret i16 %2
}

define hidden swiftcc half @julia_fp(float %0) {
  %2 = fptrunc float %0 to half
  ret half %2
}

@llvm.compiler.used = appending global [1 x ptr] [ptr @__truncsfhf2], section "llvm.metadata"

❯ /Users/gabrielbaraldi/julia9/usr/tools/llc lala.ll -filetype=obj -o llvm20.o
usr/lib/julia on  debug-llvm-20 [?] 
❯ nm llvm20.o
0000000000000000 T ___truncsfhf2
                 U _julia_float_to_half
0000000000000010 T _julia_fp
usr/lib/julia on  debug-llvm-20 [?] 
❯ llc lala.ll -filetype=obj -o llvm19.o
usr/lib/julia on  debug-llvm-20 [?] 
❯ nm llvm19.o
0000000000000000 t ___truncsfhf2
                 U _julia_float_to_half
0000000000000010 T _julia_fp

Not sure if this is an intentional change or not but it's between LLVM 19 and 20

[Zentrik]

bisected to 3d0846b with the help of manyclangs

[llvmbot]

@llvm/issue-subscribers-backend-x86

Author: Gabriel Baraldi (gbaraldi)

This is showing up as https://github.com/JuliaLang/julia/pull/57658#issuecomment-2727385403 which is holding up julia being able to compile with LLVM20.

The issue is that a defining a libcall in a module and then having it be used is making the libcall symbol become external

source_filename = "text"
target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-macosx10.14.0-macho"

declare i16 @<!-- -->julia_float_to_half(float)

define internal i16 @<!-- -->__truncsfhf2(float %0) {
  %2 = call i16 @<!-- -->julia_float_to_half(float %0)
  ret i16 %2
}

define hidden swiftcc half @<!-- -->julia_fp(float %0) {
  %2 = fptrunc float %0 to half
  ret half %2
}

@<!-- -->llvm.compiler.used = appending global [1 x ptr] [ptr @<!-- -->__truncsfhf2], section "llvm.metadata"

❯ /Users/gabrielbaraldi/julia9/usr/tools/llc lala.ll -filetype=obj -o llvm20.o
usr/lib/julia on  debug-llvm-20 [?] 
❯ nm llvm20.o
0000000000000000 T ___truncsfhf2
                 U _julia_float_to_half
0000000000000010 T _julia_fp
usr/lib/julia on  debug-llvm-20 [?] 
❯ llc lala.ll -filetype=obj -o llvm19.o
usr/lib/julia on  debug-llvm-20 [?] 
❯ nm llvm19.o
0000000000000000 t ___truncsfhf2
                 U _julia_float_to_half
0000000000000010 T _julia_fp

Not sure if this is an intentional change or not but it's between LLVM 19 and 20

[Zentrik]

Based on #3045, it seems at least in 2008 it was intentional that calls to libcalls have symbols of type MO_ExternalSymbol, if I'm understanding things correctly. @weiweichen was removing support for an internal libcall definition intentional?'

[weiweichen]

Based on #3045, it seems at least in 2008 it was intentional that calls to libcalls have symbols of type MO_ExternalSymbol, if I'm understanding things correctly. @weiweichen was removing support for an internal libcall definition intentional?'

Not sure I have the context here (or why am I tagged??)

[Zentrik]

Sorry, so if you look at the ir in the first comment, it's no longer producing an internal symbol as you can see in the nm output. I bisected the change to 3d0846b, which is a pr you authored, #108880.

I was trying to understand what the point of that pr was and if it was intentional that it had caused the symbol to no longer be internal.

Thanks.

[weiweichen]

Sorry, so if you look at the ir in the first comment, it's no longer producing an internal symbol as you can see in the nm output. I bisected the change to 3d0846b, which is a pr you authored, #108880.

I was trying to understand what the point of that pr was and if it was intentional that it had caused the symbol to no longer be internal.

Thanks.

Shouldn’t MO_ExternalSymbol be external (as the name says)? Why would it be internal?
As for the context, I think we were having codegen linking issues for this type of symbol not being marked as external. What is the fundamental issue here for your case with this symbol type difference (the commit has been in for a while (6months ago?), and am a bit surprised with it causing problem till now 🤔 ).

[Zentrik]

The Julia Language currently will define __truncsfhf2 in every module (as in IR below) to call our own function and then try to link all the modules together. This fails due to 3d0846b as the symbol is now marked external.

As far as I can tell, a libcall is always represented with a MO_ExternalSymbol even if it's defined in the same module with internal linkage. The only reason this IR isn't producing an internal __truncsfhf2 is because it's a libcall, if it was renamed it would be internal.

source_filename = "text"
target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-macosx10.14.0-macho"

declare i16 @julia_float_to_half(float)

define internal i16 @__truncsfhf2(float %0) {
  %2 = call i16 @julia_float_to_half(float %0)
  ret i16 %2
}

define hidden swiftcc half @julia_fp(float %0) {
  %2 = fptrunc float %0 to half
  ret half %2
}

@llvm.compiler.used = appending global [1 x ptr] [ptr @__truncsfhf2], section "llvm.metadata"

(the commit has been in for a while (6months ago?), and am a bit surprised with it causing problem till now 🤔

The Julia language typically only upgrades llvm when a new version comes out which is why it has gone unnoticed so far.

[weiweichen]

The Julia Language currently will define __truncsfhf2 in every module (as in IR below) to call our own function and then try to link all the modules together. This fails due to 3d0846b as the symbol is now marked external.

As far as I can tell, a libcall is always represented with a MO_ExternalSymbol even if it's defined in the same module with internal linkage. The only reason this IR isn't producing an internal __truncsfhf2 is because it's a libcall, if it was renamed it would be internal.

source_filename = "text"
target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-macosx10.14.0-macho"

declare i16 @julia_float_to_half(float)

define internal i16 @__truncsfhf2(float %0) {
%2 = call i16 @julia_float_to_half(float %0)
ret i16 %2
}

define hidden swiftcc half @julia_fp(float %0) {
%2 = fptrunc float %0 to half
ret half %2
}

@llvm.compiler.used = appending global [1 x ptr] [ptr @__truncsfhf2], section "llvm.metadata"

(the commit has been in for a while (6months ago?), and am a bit surprised with it causing problem till now 🤔

The Julia language typically only upgrades llvm when a new version comes out which is why it has gone unnoticed so far.

Thank you for the details on the failure symptoms! This describes why Julia is currently failing, though it doesn't seem to explain why a MO_ExternalSymbol should be internal which seems contradictory. Since #3045 was authored by Chris, wondering if @lattner has more context/advice on this issue?

(From another train of thought, perhaps less technical)I understand that Julia only pulls upstream every 6months hence the issue just popped, but llvm backend has many downstream users other than Julia, and there hasn't been much issue with this change for them. Could this be a more Julia specific thing that needs to be adjusted on Julia side instead of x86 backend? (CC @npanchen @dgurchenkov)