8244938: Crash in foreign ABI CallArranger class when a test native function returns a nested struct #162
This is a nasty issue with the SysV struct classification algorithm (which is rather complex).
The algortigm as specified, is designed to work on 8-byte chunks at a time (the SysV spec calls them
Here we have an int field (
The currently implemented logic, sees the first field, and classifies it as
To address this issue I first considered flattening structs, but then I quickly gave up, since it was pretty messy to make sure that flattening worked correctly with respect to unions (e.g. structs inside unions).
I then settled on a simpler scheme: since the classification logic is meant to work on one eightbyte at a time, I just wrote a routine that parses the incoming
We recursively scan the layout, trying to find all the fields, and keeping track of their offsets. Eventually, when we come to leaves layouts (values) we add their corresponding ArgumentClassImpl to the array slot that corresponds to the eightbyte associated with the offset being considered.
Once this processing is done, classifying the struct is a breeze, as what's left to do is simply to merge all the classes in a single eightbyte slot (which can be done with a simple reduce step).
Note: for this logic to work, we have to assume that all value layouts in the group are not bigger than 8 bytes. In practice this is not a big issue, since bigger value layouts were not supported anyway. I also believe it won't be an issue moving forward, since we can simply make sure that e.g. the SysV type
And so forth for vector types. In other words, rather than making the classification logic more complex, we can simply define the ABI layout constants accordingly, so that they are already broken up into 8-byte chunks.
JornVernee left a comment
A nice chunk of code getting vaporized
@mcimadamore This change now passes all automated pre-integration checks, type
Since the source branch of this PR was last updated there have been 2 commits pushed to the
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid automatic rebasing, please merge
@mcimadamore The following commits have been pushed to foreign-abi since your change was applied:
Your commit was automatically rebased without conflicts.
Pushed as commit 32d1f6c.