irregexp corrupts the stack

[classilla]

Steps:

1. Go to https://github.com/classilla/tenfourfox/blob/master/README.md
2. Try to edit it

The editor has lots of problems. Debug build asserts in RootingAPI.h, even with Ion and Baseline off. Occasionally release builds will crash. Doesn't occur in 31.

Patch for RootingAPI.h in 38.3.0 fixes the interpreter case but does not fix the JIT case(s).

[classilla]

(But the patch does seem to wallpaper the crashes, so it's still better than before.)

[classilla]

Disabling Ion doesn't fix it.
Changing VMWrappers to always use a "temp temp" register for any rooted object, not just RootValue, doesn't fix it.

[classilla]

New URL: http://vinacc.blogspot.co.at/2015/09/article-challenge-to-reach-identity-of.html
This bombs in the interpreter, even, at least in a debug build.

[classilla]

It does not bomb if native regex is turned off (and it does bomb when it is turned back on).

[classilla]

Turning off regexes gets the github example URL to work, even with Ion and Baseline on.
Turning off regexes gets the vinacc example URL to not crash, even with Ion and Baseline on.
This seems to be a problem with irregexp.

[classilla]

Disabling auto-growth of the irregexp stack allocation doesn't fix it.

[classilla]

Comparison of irregexp opcodes from a couple minutes of random browsing with the vinacc site:

% diff irr_working.txt irr_nworking.txt 
15a16,17
> [Irregexp] !!! CheckNotBackReference
> [Irregexp] !!! CheckNotBackReferenceIgnoreCase
22a25
> [Irregexp] !!! IfRegisterEqPos
36d38
< [Irregexp] !!! SetCurrentPositionFromEnd

In CheckNotBackReferenceIgnoreCase, there is a codepath that makes an ABI-compliant call. Putting a trap here traps on the affected sites, but not (so far) unaffected ones.

[classilla]

[Irregexp] Created RegExp (raw 0x3332b5b4 length 3120)
Program received signal SIGTRAP, Trace/breakpoint trap.
0x33e18130 in ?? ()
(gdb) disas $pc-0x24 $pc+0x80
Dump of assembler code from 0x33e1810c to 0x33e181b0:
0x33e1810c:     addi    r1,r1,-24
0x33e18110:     stw     r10,20(r1)
0x33e18114:     stw     r8,16(r1)
0x33e18118:     stw     r6,12(r1)
0x33e1811c:     stw     r5,8(r1)
0x33e18120:     stw     r4,4(r1)
0x33e18124:     stw     r3,0(r1)
0x33e18128:     add     r4,r4,r3
0x33e1812c:     add     r5,r5,r3
0x33e18130:     trap
0x33e18134:     lis     r12,2795
0x33e18138:     ori     r12,r12,58288
0x33e1813c:     mtctr   r12
0x33e18140:     mr      r3,r4
0x33e18144:     mr      r4,r5
0x33e18148:     mr      r5,r8
0x33e1814c:     andi.   r0,r1,4
0x33e18150:     mr      r16,r1
0x33e18154:     subf    r1,r0,r1
0x33e18158:     andi.   r0,r1,8
0x33e1815c:     subf    r1,r0,r1
0x33e18160:     addi    r1,r1,-512
0x33e18164:     stw     r18,0(r1)
0x33e18168:     mflr    r18
0x33e1816c:     bctrl
0x33e18170:     mtlr    r18
0x33e18174:     lwz     r18,0(r1)
0x33e18178:     mr      r1,r16
0x33e1817c:     mr      r7,r3
0x33e18180:     lwz     r10,20(r1)
0x33e18184:     lwz     r8,16(r1)
0x33e18188:     lwz     r6,12(r1)
0x33e1818c:     lwz     r5,8(r1)
0x33e18190:     lwz     r4,4(r1)
0x33e18194:     lwz     r3,0(r1)
0x33e18198:     addi    r1,r1,24
0x33e1819c:     and.    r0,r7,r7
0x33e181a0:     bne-    0x33e181b4
0x33e181a4:     b       0x33e20e50
0x33e181a8:     nop
0x33e181ac:     nop
End of assembler dump.
(gdb) i reg r3 r4 r5
r3             0x3702ec8c       922938508
r4             0x13702eb92      5217905554
r5             0x13702eb96      5217905558

[classilla]

r10??
[Irregexp] Starting RegExp (input_end_pointer r3) (current_character r4) (current_position r5) (backtrack_stack_pointer r6) (temp0 r7) temp1 (r8) temp2 (r9)

[classilla]

Fixed by hand-writing the ABI call like we did for the backstack grow routine. I'm still not sure what's wrong, but both sites work now.

[classilla]

Shipp'd
If fix sticks, remove the old RootingAPI shim in 38.5 (leaving open for that).

[classilla]

Shim to rooted GC is removed.