Different size/alignment rules for ffi/LLVM cause general FUBAR #132

Closed
paniq opened this Issue Nov 19, 2015 · 4 comments

Comments

Projects
None yet
2 participants
@paniq

paniq commented Nov 19, 2015

I've been running in some weird bugs when working with composite structs, which I only started doing lately. Fields would not contain the values they were supposed to, writes would happen out of bounds despite there being no reason for them to be, etc.

Often I got the error fixed by changing the order of declarations in structs, but the problem has gotten so bad that these simple fixes are no longer applicable. I basically need to add my own padding on a per-declaration basis, which interferes with templated struct generation, and, well, the situation needs fixing.

I hope you can figure out the source of this problem soon.

Here is a simple reproduce for the problem:

local errors = 0

-- compare ffi's sizeof with the element offset generated by LLVM
local function assert_size(basetype)
    local ffi_size = terralib.sizeof(basetype)
    local intp = &int
    local terra get_size () : int
        var k : &basetype = nil
        var o = k + 1
        return @(intp(&o))
    end
    local llvm_size = get_size()
    if (ffi_size ~= llvm_size) then
        errors = errors + 1
        print(("%s: size mismatch: %s != %s"):format(tostring(basetype), ffi_size, llvm_size))
    end
end

-- compare ffi's offsetof with the member offset generated by LLVM
local function assert_offset(basetype, name)
    local ffi_offset = terralib.offsetof(basetype, name)
    local intp = &int
    local terra get_offset () : int
        var k : &basetype = nil
        var o = &k.[name]
        return @(intp(&o))
    end
    local llvm_offset = get_offset()
    if (ffi_offset ~= llvm_offset)  then
        errors = errors + 1
        print(("%s.%s: offset mismatch: %s != %s"):format(tostring(basetype), name, ffi_offset, llvm_offset))
    end
end

local struct Reference {
    v1 : uint32;
    v2 : uint32;
    v3 : float;
    v4 : uint32;
}

-- all fine
assert_size(Reference)
assert_offset(Reference, "v1")
assert_offset(Reference, "v2")
assert_offset(Reference, "v3")
assert_offset(Reference, "v4")

local struct Entry {
    key : uint8;
    value : uint64;
}

-- fails: "Entry: size mismatch: 16 != 12"
assert_size(Entry)
assert_offset(Entry, "key")
-- fails: "Entry.value: offset mismatch: 8 != 4"
assert_offset(Entry, "value")

local struct Map {
    count : uint;
    map : Entry[1024];
    mask : uint;
}

-- fails: "Map: size mismatch: 16400 != 12296"
assert_size(Map)
assert_offset(Map, "count")
-- fails: "Map.map: offset mismatch: 8 != 4"
assert_offset(Map, "map")
-- fails: Map.mask: offset mismatch: 16392 != 12292
assert_offset(Map, "mask")

local struct MetaMap {
    count : uint;
    map : Map;
    actors : uint64[10000];
}

-- fails: "MetaMap: size mismatch: 96408 != 92300"
assert_size(MetaMap)
assert_offset(MetaMap, "count")
-- fails: "MetaMap.map: offset mismatch: 8 != 4"
assert_offset(MetaMap, "map")
-- fails: "MetaMap.actors: offset mismatch: 16408 != 12300"
assert_offset(MetaMap, "actors")

assert(errors == 0, ("%i errors occurred"):format(errors))

My platform is Ubuntu 15.10, LLVM 3.6

@paniq paniq changed the title from Different size/alignment rules for ffi/LLVM cause general fubar to Different size/alignment rules for ffi/LLVM cause general FUBAR Nov 19, 2015

@zdevito

This comment has been minimized.

Show comment
Hide comment
@zdevito

zdevito Nov 19, 2015

Owner

This code runs without errors on a computer I have running OSX and and
another running Ubuntu. Can you give me more details of your system?

On Thu, Nov 19, 2015 at 12:26 AM, Leonard Ritter notifications@github.com
wrote:

I've been running in some weird bugs when working with composite structs,
which I only started doing lately. Fields would not contain the values they
were supposed to, writes would happen out of bounds despite there being no
reason for them not to, etc.

Often I got the error fixed by changing the order of declarations in
struct, but the problem has gotten so bad that these simple fixes are no
longer applicable. I basically need to add my own padding on a
per-declaration basis, which interferes with templated struct generation,
and, well, the situation needs fixing.

I hope you can figure out the source of this problem soon.

Here is a simple reproduce for the problem:

local errors = 0

-- compare ffi's sizeof with the element offset generated by LLVM
local function assert_size(basetype)
local ffi_size = terralib.sizeof(basetype)
local intp = &int
local terra get_size () : int
var k : &basetype = nil
var o = k + 1
return @(intp(&o))
end
local llvm_size = get_size()
if (ffi_size ~= llvm_size) then
errors = errors + 1
print(("%s: size mismatch: %s != %s"):format(tostring(basetype), ffi_size, llvm_size))
end
end

-- compare ffi's offsetof with the member offset generated by LLVM
local function assert_offset(basetype, name)
local ffi_offset = terralib.offsetof(basetype, name)
local intp = &int
local terra get_offset () : int
var k : &basetype = nil
var o = &k.[name]
return @(intp(&o))
end
local llvm_offset = get_offset()
if (ffi_offset ~= llvm_offset) then
errors = errors + 1
print(("%s.%s: offset mismatch: %s != %s"):format(tostring(basetype), name, ffi_offset, llvm_offset))
end
end

local struct Reference {
v1 : uint32;
v2 : uint32;
v3 : float;
v4 : uint32;
}

-- all fine
assert_size(Reference)
assert_offset(Reference, "v1")
assert_offset(Reference, "v2")
assert_offset(Reference, "v3")
assert_offset(Reference, "v4")

local struct Entry {
key : uint8;
value : uint64;
}

-- fails: "Entry: size mismatch: 16 != 12"
assert_size(Entry)
assert_offset(Entry, "key")
-- fails: "Entry.value: offset mismatch: 8 != 4"
assert_offset(Entry, "value")

local struct Map {
count : uint;
map : Entry[1024];
mask : uint;
}

-- fails: "Map: size mismatch: 16400 != 12296"
assert_size(Map)
assert_offset(Map, "count")
-- fails: "Map.map: offset mismatch: 8 != 4"
assert_offset(Map, "map")
-- fails: Map.mask: offset mismatch: 16392 != 12292
assert_offset(Map, "mask")

local struct MetaMap {
count : uint;
map : Map;
actors : uint64[10000];
}

-- fails: "MetaMap: size mismatch: 96408 != 92300"
assert_size(MetaMap)
assert_offset(MetaMap, "count")
-- fails: "MetaMap.map: offset mismatch: 8 != 4"
assert_offset(MetaMap, "map")
-- fails: "MetaMap.actors: offset mismatch: 16408 != 12300"
assert_offset(MetaMap, "actors")

assert(errors == 0, ("%i errors occurred"):format(errors))


Reply to this email directly or view it on GitHub
#132.

Owner

zdevito commented Nov 19, 2015

This code runs without errors on a computer I have running OSX and and
another running Ubuntu. Can you give me more details of your system?

On Thu, Nov 19, 2015 at 12:26 AM, Leonard Ritter notifications@github.com
wrote:

I've been running in some weird bugs when working with composite structs,
which I only started doing lately. Fields would not contain the values they
were supposed to, writes would happen out of bounds despite there being no
reason for them not to, etc.

Often I got the error fixed by changing the order of declarations in
struct, but the problem has gotten so bad that these simple fixes are no
longer applicable. I basically need to add my own padding on a
per-declaration basis, which interferes with templated struct generation,
and, well, the situation needs fixing.

I hope you can figure out the source of this problem soon.

Here is a simple reproduce for the problem:

local errors = 0

-- compare ffi's sizeof with the element offset generated by LLVM
local function assert_size(basetype)
local ffi_size = terralib.sizeof(basetype)
local intp = &int
local terra get_size () : int
var k : &basetype = nil
var o = k + 1
return @(intp(&o))
end
local llvm_size = get_size()
if (ffi_size ~= llvm_size) then
errors = errors + 1
print(("%s: size mismatch: %s != %s"):format(tostring(basetype), ffi_size, llvm_size))
end
end

-- compare ffi's offsetof with the member offset generated by LLVM
local function assert_offset(basetype, name)
local ffi_offset = terralib.offsetof(basetype, name)
local intp = &int
local terra get_offset () : int
var k : &basetype = nil
var o = &k.[name]
return @(intp(&o))
end
local llvm_offset = get_offset()
if (ffi_offset ~= llvm_offset) then
errors = errors + 1
print(("%s.%s: offset mismatch: %s != %s"):format(tostring(basetype), name, ffi_offset, llvm_offset))
end
end

local struct Reference {
v1 : uint32;
v2 : uint32;
v3 : float;
v4 : uint32;
}

-- all fine
assert_size(Reference)
assert_offset(Reference, "v1")
assert_offset(Reference, "v2")
assert_offset(Reference, "v3")
assert_offset(Reference, "v4")

local struct Entry {
key : uint8;
value : uint64;
}

-- fails: "Entry: size mismatch: 16 != 12"
assert_size(Entry)
assert_offset(Entry, "key")
-- fails: "Entry.value: offset mismatch: 8 != 4"
assert_offset(Entry, "value")

local struct Map {
count : uint;
map : Entry[1024];
mask : uint;
}

-- fails: "Map: size mismatch: 16400 != 12296"
assert_size(Map)
assert_offset(Map, "count")
-- fails: "Map.map: offset mismatch: 8 != 4"
assert_offset(Map, "map")
-- fails: Map.mask: offset mismatch: 16392 != 12292
assert_offset(Map, "mask")

local struct MetaMap {
count : uint;
map : Map;
actors : uint64[10000];
}

-- fails: "MetaMap: size mismatch: 96408 != 92300"
assert_size(MetaMap)
assert_offset(MetaMap, "count")
-- fails: "MetaMap.map: offset mismatch: 8 != 4"
assert_offset(MetaMap, "map")
-- fails: "MetaMap.actors: offset mismatch: 16408 != 12300"
assert_offset(MetaMap, "actors")

assert(errors == 0, ("%i errors occurred"):format(errors))


Reply to this email directly or view it on GitHub
#132.

@paniq

This comment has been minimized.

Show comment
Hide comment
@paniq

paniq Nov 20, 2015

Just did a fresh checkout and build, and I get the errors on these systems:

Ubuntu 15.10 (64-bit) + LLVM 3.6.2 (gcc 5.2.1)
Ubuntu 15.04 (64-bit) + LLVM 3.6.2 (gcc 4.9.2)

I haven't tested any other yet.

paniq commented Nov 20, 2015

Just did a fresh checkout and build, and I get the errors on these systems:

Ubuntu 15.10 (64-bit) + LLVM 3.6.2 (gcc 5.2.1)
Ubuntu 15.04 (64-bit) + LLVM 3.6.2 (gcc 4.9.2)

I haven't tested any other yet.

@zdevito

This comment has been minimized.

Show comment
Hide comment
@zdevito

zdevito Nov 20, 2015

Owner

This was due to a not-very-well-documented LLVM API that I was apparently not using correctly for many years.

Owner

zdevito commented Nov 20, 2015

This was due to a not-very-well-documented LLVM API that I was apparently not using correctly for many years.

@paniq

This comment has been minimized.

Show comment
Hide comment
@paniq

paniq Nov 21, 2015

retested with new version, all good now, thanks a lot!

paniq commented Nov 21, 2015

retested with new version, all good now, thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment