Skip to content
This repository

a test for when the clang compiler method signature mismatches with the julia ccall one #978

Closed
wants to merge 1 commit into from

4 participants

Jameson Nash Jeff Bezanson Viral B. Shah Stefan Karpinski
Jameson Nash
Collaborator

The clang / llvm-gcc compiler that ships with Apple does not cast 8 bit values to the appropriate size in function arguments. I worked around this bug in the fall to correct boxing of 8 bit values and get the code to compile, but I realized this could affect ccall also (if the target code was compiled with llvm-gcc / clang) so I started to put together this testsuite for ccall to demonstrate the error.

Jeff Bezanson
Owner

Needs a rebase; then I will merge it.

Viral B. Shah
Owner

Is this the likely reason for the instability we see with clang in #1013?

Stefan Karpinski
Owner

@vtjnash: can you fix this up? Should we merge it?

Viral B. Shah
Owner

@vtjnash can you check if this is fixed with Xcode 4.4?

Jameson Nash
Collaborator

It wasn't fixed with clang 3.1 (although I feel I need to find a way to trigger this test more reliably). Really this needs to be fixed somehow in Julia / LLVM.

Jeff Bezanson
Owner

Bump

Jeff Bezanson

Should this be filed as a bug in clang? What should we do next?

Jameson Nash
Collaborator

Sorry for the huge delay. I didn't know llvm well enough. The problem is with Julia not LLVM, I just didn't understand where or how to look previously.

@JeffBezanson I've come to realize that method signatures are slightly incorrect. It is very important to tell llvm the sign of the arguments (signext or zeroext). I can show you why later / on IRC / in person (in short: llvm will remove the previous bit casts because it assumes they weren't necessary). GCC is much more forgiving regarding mismatching type signatures than clang.
e.g. the method signature for jl_box_int8 in llvm optcodes should have been define void* @jl_box_int8(i8 zeroext %x) { } (before we changed this last fall to ignore the issue)

Jameson Nash
Collaborator

http://llvm.org/docs/LangRef.html#paramattrs
Lots of interesting attributes to read about. also, clang -S -emit-llvm main.c -o - can be really informative

Stefan Karpinski

I have to say it frequently seems to me like llvm should have just gone with having signed and unsigned integer types. Oh well.

Jeff Bezanson

I'm kind of glad it doesn't have them, since they don't really exist. All that's really there is a bit string after all.

@vtjnash do you think we should add the following:

--- a/src/intrinsics.cpp
+++ b/src/intrinsics.cpp
@@ -994,6 +994,8 @@ static Function *boxfunc_llvm(FunctionType *ft, const std::string &cname,
 {
     Function *f =
         Function::Create(ft, Function::ExternalLinkage, cname, jl_Module);
+    Function::arg_iterator AI = f->arg_begin();
+    (*AI).addAttr(Attribute::ZExt);
     jl_ExecutionEngine->addGlobalMapping(f, addr);
     return f;
 }
Jameson Nash
Collaborator

@JeffBezanson: yes, that seems right, but you need to select zext or sext based on the sign of the argument.

I agree with Stefan. Looking at how the code generator works, they should have just gone with explicitly signed and unsigned types.

I've observed that llvm seems to mostly ignore explicit bit modification instructions, so they don't actually happen until just before the function call, and then they happen based upon this hinting. (they probably are observed more closely in the presence of load / store instructions, but I haven't investigated). Additionally, this looks like it is necessary for every function created in LLVM, even if they are purely JIT functions, for any bit type. Pointer types I think are OK without it.

Also from that link above, most ccall functions would benefit from having the sspreq attribute set, to protect the stack. Also byval seems rather interesting, although I'm not sure if it can be any more efficient that the current implementation.

Jeff Bezanson

I don't fully understand why we would need this, since in every call I generate the types match exactly, as required by the llvm verifier. And the type of the actual argument should matter more, since for example you would zero extend a uint8 to pass it as a signed int32.

Jameson Nash
Collaborator

That's what this pull request was originally about. Correctly matching the types is not enough. The signature of the function must also be made to match, or (I've observed that) the assembly to guarantee conversion of the types will not be generated. And you can't pass an int8 to a function that requires an int32, their sizes don't match.

Here's my final code testcase (in c first, for readability):

int8_t ptr[300];
int8_t f2a(int8_t x) { return ptr[x]; }
int8_t f1a(int64_t x) { return f2a( (int8_t)(x&0xff) ); }

int8_t f2b(int8_t x) { return x+1; }
int8_t f1b(int64_t x) { return f2b( (int8_t)(x&0xff) ); }

int main() {
  s1 = f1a( 5000 );
  s2 = f2a( 5000 );
  return (int)s2;
}

(disclaimer: I just wrote this for readability, I didn't actually use and/or test it. It should generate something equivalent to the below LLVM, if the optimizations don't eliminate everything)

Compiling this by hand into raw llvm opt codes (this is what I really was using):

@ptr = global i8 300

define i8 @f2a(i8 zeroext %x) {
  %1 = getelementptr i8* @ptr, i8 %x
  %2 = load i8* %1, align 4
  ret i8 %2
}

define i8 @f1a(i64 %A) {
  %s1 = trunc i64 %A to i8
  %s2 = call i8 @f2a(i8 %s1)
  ret i8 %s2
}

define i8 @f2b(i8 zeroext %x) {
  %1 = add i8 %x, 1
  ret i8 %1
}

define i8 @f1b(i64 %A) {
  %s1 = trunc i64 %A to i8
  %s2 = call i8 @f2b(i8 %s1)
  ret i8 %s2
}

define i32 @main() {
  %s1 = call i8 @f1a(i64 5000)
  %s2 = call i8 @f1b(i64 5000)
  %s3 = sext i8 %s2 to i32
  ret i32 %s3
}

Experiment with removing and changing the various zeroext and truncate instructions and you will find that LLVM does not generate sufficient code to remove the extra bits if the type of the argument is not specified as zero/sign extended.

Generate the associated assembly with $ llc test.c -o -
(painful assembly output follows, comments are at the end)

    .section    __TEXT,__text,regular,pure_instructions
    .globl  _f2a
    .align  4, 0x90
_f2a:                                   ## @f2a
    .cfi_startproc
## BB#0:
                                        ## kill: EDI<def> EDI<kill> RDI<def>
    movsbq  %dil, %rax
    leaq    _ptr(%rip), %rcx
    movb    (%rax,%rcx), %al
    ret
    .cfi_endproc

    .globl  _f1a
    .align  4, 0x90
_f1a:                                   ## @f1a
    .cfi_startproc
## BB#0:
    pushq   %rax
Ltmp1:
    .cfi_def_cfa_offset 16
    movzbl  %dil, %edi
    callq   _f2a
    popq    %rdx
    ret
    .cfi_endproc

    .globl  _f2b
    .align  4, 0x90
_f2b:                                   ## @f2b
    .cfi_startproc
## BB#0:
    incb    %dil
    movb    %dil, %al
    ret
    .cfi_endproc

    .globl  _f1b
    .align  4, 0x90
_f1b:                                   ## @f1b
    .cfi_startproc
## BB#0:
    pushq   %rax
Ltmp3:
    .cfi_def_cfa_offset 16
    movzbl  %dil, %edi
    callq   _f2b
    popq    %rdx
    ret
    .cfi_endproc

    .globl  _main
    .align  4, 0x90
_main:                                  ## @main
    .cfi_startproc
## BB#0:
    pushq   %rax
Ltmp5:
    .cfi_def_cfa_offset 16
    movl    $5000, %edi             ## imm = 0x1388
    callq   _f1a
    movl    $5000, %edi             ## imm = 0x1388
    callq   _f1b
    movsbl  %al, %eax
    popq    %rdx
    ret
    .cfi_endproc

    .section    __DATA,__data
    .globl  _ptr                    ## @ptr
_ptr:
    .byte   44                      ## 0x2c


.subsections_via_symbols

Note that if zeroext is removed, the movzbl instructions will also disappear. Then both of these functions will return an incorrect answer, despite the attempt at type matching! The argument types in the function did not require it, so the callers did not bother to clear the high bits.
Further note that the return value isn't actually returned as a byte either, but as the lower 8 bits of a certain register, with the other bits unchanged. It is assumed that the caller will take care of these details, but only if this is communicated via the function argument types.

Jeff Bezanson

Ah, so an i8 argument with no attribute behaves as a word where the higher bits are just undefined? That strikes me as a bad way to avoid a single mov instruction. Alternatively, perhaps this can be seen as a flaw in getelementptr; it should require the offset to be extended to a pointer-size integer.

Jeff Bezanson

Actually, julia might be less susceptible to this problem, since internally all array indexes are word-size integers, and any other integer types are explicitly converted first. I don't think we generate any GEPs with non-word-size integer offsets.

Jameson Nash
Collaborator

I played with this some more and found (no big surprise here) that the important thing is to be consistent (I was analyzing the above assembly slightly incorrectly). Thus, if the function is declared with zeroext in the parameter type, then LLVM will generate code that assumes the high bits of that parameter are already zero (including optimizing away explicit casts) but it will also generate code at the call site to ensure this condition is met (otherwise it will optimize those casts away).

When clang is compiling functions, it declares the function parameter types to include the zeroext / signext modifiers (e.g. it assumes that the caller will take care of the conversion). By contrast, when GCC compiles functions, it appears to take the safe approach and has both the caller and callee perform the bit truncation. Therefore, only external functions (those using addGlobalMapping, esp. those in ccall) must be annotated (to match clang's expectations) to ensure proper behavior. Internally to Julia, this parameter is optional and only serves to occasionally allow slight additional optimization.

Jameson Nash
Collaborator

Following up on on conversation last night, clang only applies signext/zext to i8 and i16 arguments (not pointers, floats, or larger inteters). It's probably still worth asking them what their algorithm is.

Jeff Bezanson JeffBezanson closed this in c8041e7 January 06, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 2 authors.

Jan 06, 2013
Jameson Nash test for incomplete ccall method declaration behavior for 8-bit integ…
…er parameters in clang-compiled code
bdfedd3
This page is out of date. Refresh to see the latest.
2  .gitignore
@@ -13,6 +13,8 @@
13 13
 *.dll
14 14
 *.do
15 15
 *.o
  16
+*.dylib
  17
+*.dSYM
16 18
 
17 19
 *.out
18 20
 
1  test/.gitignore
... ...
@@ -0,0 +1 @@
  1
+/ccall
9  test/Makefile
@@ -10,7 +10,7 @@ random math functional bigint bigfloat combinatorics \
10 10
 statistics glpk linprog poly file Rmath remote zlib image \
11 11
 iostring gzip
12 12
 
13  
-$(TESTS) ::
  13
+$(TESTS) :: libccall.$(SHLIB_EXT) ccall
14 14
 	$(QUIET_JULIA) $(JULIA_EXECUTABLE) ./runtests.jl $@
15 15
 
16 16
 perf:
@@ -25,6 +25,13 @@ benchmark:
25 25
 
26 26
 clean:
27 27
 	@$(MAKE) -C perf $@
  28
+	-rm -f libccall.${SHLIB_EXT} ccall
28 29
 
29 30
 .PHONY: $(TESTS) perf benchmark clean
30 31
 
  32
+libccall.$(SHLIB_EXT): ccall.c
  33
+	    $(CC) $(CFLAGS) $(DEBUGFLAGS) -O3 $< -shared -o $@ $(LDFLAGS) -DCC=$(CC)
  34
+
  35
+ccall: ccall.c
  36
+	$(CC) $(CFLAGS) $(DEBUGFLAGS) -O3 $< -o $@ $(LDFLAGS) -DCC=$(CC)
  37
+
24  test/ccall.c
... ...
@@ -0,0 +1,24 @@
  1
+#include <stdio.h>
  2
+
  3
+int xs[300] = {0,0,0,1,0};
  4
+
  5
+int __attribute((noinline)) testUcharX(unsigned char x) {
  6
+	return xs[x];
  7
+}
  8
+
  9
+#define xstr(s) str(s)
  10
+#define str(s) #s
  11
+volatile int (*fptr)(unsigned char x);
  12
+volatile int a;
  13
+volatile int b;
  14
+
  15
+int main() {
  16
+    printf("all of the following should be 1 except xs[259] = 0");
  17
+	a = 3;
  18
+    b = 259;
  19
+    fptr = (volatile int (*)(unsigned char x))&testUcharX;
  20
+    if ((((long)fptr)&((long)1)<<32) == 1) fptr = NULL;
  21
+	printf("compiled with: '%s'\nxs[3] = %d\nxs[259] = %d\ntestUcharX(3) = %d\ntestUcharX(%d) = %d\nfptr(3) = %d\nfptr(259) = %d\n",
  22
+		   xstr(CC), xs[a], xs[b], testUcharX(a), b, testUcharX(b), fptr(a), fptr(b));
  23
+}
  24
+
3  test/ccall.jl
... ...
@@ -0,0 +1,3 @@
  1
+ccall_test_func(x) = ccall((:testUcharX, :libccall), Int32, (Int8,), x)
  2
+@assert ccall_test_func(3) == 1
  3
+@assert ccall_test_func(259) == 1
1  test/extra.jl
@@ -13,3 +13,4 @@ runtests("image")
13 13
 runtests("gzip")
14 14
 
15 15
 runtests("perf")
  16
+runtests("ccall")
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.