max=! fails with inff not known to nvrtc

[prigoyal]

FAILS
def softmax(float(N, D) I) -> (O, expsum, maxVal) {
        maxVal(n) max=! I(n, d)
        expsum(n) +=! exp(I(n, d) - maxVal(n))
        O(n, d) = exp(I(n, d) - maxVal(n)) / expsum(n)
}
Error: https://gist.github.com/prigoyal/0318a964368f86b3a6a09228c7bfbf71

PASSES (bang removed from max=)
def softmax(float(N, D) I) -> (O, expsum, maxVal) {
        maxVal(n) max= I(n, d)
        expsum(n) +=! exp(I(n, d) - maxVal(n))
        O(n, d) = exp(I(n, d) - maxVal(n)) / expsum(n)
}

[prigoyal]

https://gist.github.com/prigoyal/0318a964368f86b3a6a09228c7bfbf71 - it doesn't strictly feel like frontend issue to me. the TC passes semantic analysis etc and then fails at NVRTC compilation but maybe the frontend didn't do the parsing correctly.

@zdevito do you know what's going on here? repro in #61

[abadams]

Oh, it's due to how we codegen inf. The identity of max=! is -inf, which we don't print in a way cuda source would understand.

[abadams]

Anyone know the correct way to write inf in cuda?

[prigoyal]

haha, yeah, inff looked very suspicious in first place. https://devtalk.nvidia.com/default/topic/486689/representing-infinity/?offset=3 this seems to denote we could use

const unsigned long long ieee754inf =  0x7ff0000000000000;

so we might need to set this custom type

[prigoyal]

cc @boncheol.gu