Use objc_msgSend_fpret for backingStoreFactor.

[pcwalton]

This mistake led to a bizarre chain reaction in Servo that ultimately
led to a hang only when optimization was enabled.

r? @bjz

[SSheldon]

@pcwalton, did this hang happen on 32-bit OSX? I was under the impression that, for x86_64, msg_send is supposed to be used for double, and msg_send_fpret is only necessary for long double (which ObjC doesn't support). http://www.sealiesoftware.com/blog/archive/2008/11/16/objc_explain_objc_msgSend_fpret.html

[pcwalton]

@SSheldon It was on 64-bit OS X.

[SSheldon]

@pcwalton, interesting! I'd like to test this in my own ObjC crate, any info about the case where you were seeing weird behavior? msg_send was just always returning the wrong value? Or maybe was this when messaging nil?

[pcwalton]

@SSheldon What I saw was mov xmm0,rax after a call to call objc_msgSend that returned the double value in xmm0. This clobbered the xmm0 return value with the garbage value in rax and caused incorrect results. Switching to objc_msgSend_fpret caused xmm0 to be preserved properly.

[SSheldon]

@pcwalton and @bjz, I don't think objc_msgSend_fpret is the culprit here. I was able to come up with a simple repro case:

extern crate cocoa;

use std::mem;
use cocoa::base::{class, id, msg_send, objc_msgSend, SEL, selector};

fn main() {
    unsafe {
        let number = msg_send()(class("NSNumber"), selector("numberWithDouble:"), 3.14f64);

        let f: extern fn(id, SEL, ...) -> f64 = mem::transmute(objc_msgSend);
        let value = f(number, selector("doubleValue"));

        assert!(value == 3.14);
    }
}

This fails when compiled with optimizations. However, if you change the type of f from extern fn(id, SEL, ...) -> f64 to extern fn(id, SEL) -> f64? It stops failing.

I think the true problem here is that this crate treats objc_msgSend as a function that accepts variadic arguments, when that isn't true; objc_msgSend needs to be cast to the true type of the method.

[SSheldon]

Here's how the assembly changes when you switch from extern fn(id, SEL, ...) -> f64 to extern fn(id, SEL) -> f64:

        movq    %rax, -32(%rbp)
        leaq    -32(%rbp), %rdi
        callq   __ZN4base8selector20hdba17a2bd7f3a148eHeE
-       movq    %rax, %rcx
-       xorl    %eax, %eax
        movq    %rbx, %rdi
-       movq    %rcx, %rsi
+       movq    %rax, %rsi
        callq   _objc_msgSend
-       movd    %rax, %xmm0
        ucomisd LCPI0_0(%rip), %xmm0
        jne     LBB0_10
        jnp     LBB0_11

(I'm not much of an assembly programmer, so I'm not sure why this fixes the bug, just thought it might be informative.)