Vm Crash caused by unaligned float64 read on ARM

[mkustermann]

It looks like the DartVM makes unaligned memory access for doubles in certain cases.

We were recently seeing a VM crash with exit code SIGEMT ("Emulator trap; this results from certain unimplemented instructions which might be emulated in software, or the operating system's failure to properly emulate them."). Looking at it with gdb:

(gdb) bt            
#0  0x00649f84 in dart::BootstrapNatives::DN_TypedData_GetFloat64(_Dart_NativeArguments*) ()
#1  0xb5dff17c in ?? ()
#2  0xb5dff17c in ?? ()
(gdb) disas
Dump of assembler code for function _ZN4dart16BootstrapNatives23DN_TypedData_GetFloat64EP21_Dart_NativeArguments:
...
   0x00649f7c <+728>:   add     r3, r3, r2, asr #1
   0x00649f80 <+732>:   vldr    d0, [r3]
=> 0x00649f84 <+736>:   bl      0x2b557c <_ZN4dart6Double3NewEdNS_4Heap5SpaceE>
   0x00649f88 <+740>:   mov     r2, r0
...
End of assembler dump.
(gdb) info registers
r0             0x0      0
r1             0x8      8
r2             0xa      10
r3             0xb5611a5d       3043039837
r4             0xb5dabd4c       3051011404
r5             0xb5dab928       3051010344
r6             0xb5dabf90       3051011984
r7             0xb5dabd54       3051011412
r8             0xb5dab920       3051010336
r9             0x914bd0 9522128
r10            0x81ab60 8498016
r11            0xb5dabf8c       3051011980
r12            0x1      1
sp             0xb5dab918       0xb5dab918
lr             0x649f6c 6594412
pc             0x649f84 0x649f84 <dart::BootstrapNatives::DN_TypedData_GetFloat64(_Dart_NativeArguments*)+736>
cpsr           0x60000010       1610612752
(gdb) p *(double*)0xb5611a5d
$1 = 42.420000000000002

The instruction which caused the SIGEMT signal is vldr which tries to load from 0xb5611a5d which is not aligned.

The causing dart code called most likely this function:

static double readDoubleFromBuffer(Uint8List buffer, int offset) {
  return buffer.buffer.asByteData(buffer.offsetInBytes)
      .getFloat64(offset, commandEndianness);
}

There seems to be a linux kernel option named CONFIG_ALIGNMENT_TRAP which, when enabled, will cause a software emulation of the unaligned access, thereby not causing any issues. The option is apparently enabled by default in newer kernels. See here.
=> Therefore I'm not sure if our buildbots would catch the issue.

Might be related to #23810 and #12868 .

$ dart --version
Dart VM version: 1.12.0-edge.b2ccdc92f8f87e8ccdac6a7468e2e69ca726fbc9 (Mon Jun 29 13:24:14 2015) on "linux_arm"

cc possibly interested people: @zanderso, @johnmccutchan, @iposva

[zanderso]

If the typed data API is meant to support arbitrarily aligned accesses to ByteData buffers, then we'll have to emulate this on ARM, probably both in the functions generated by the TYPED_GETTER_SETTER macro in the ExternalTypedData class defined in //runtime/vm/object.h, and in {Load,Store}IndexedInstr. Since 32 and 64 bit integer accesses will be fixed up by either hardware or the kernel, we can probably just cast to/from an int type before the access. John, WDYT?

[johnmccutchan]

@zanderso I don't think we can guarantee alignment of the loads / stores. We will need some fallback for unaligned I/O.

[rmacnak-google]

Another duplicate: #22151

[peter-ahe-google]

I was able to work around the crash using this: dart-archive/sdk@cad1d1f

(Note to people attempting to copy my work around: be aware that it assumes that offset is a multiple of 8).

[mkustermann]

Since 32 and 64 bit integer accesses will be fixed up by either hardware or the kernel

As a side question: Does anyone know what big of a performance drop it is if we rely on the kernel to fix it? [I would naively assume it's quite big -- in which case a programmer might be very surprised if he adds e.g. one more byte in front of a messages and suddenly everything gets much slower.]

[zanderso]

@mkustermann I'd guess that doing something to prevent kernel handling of unaligned accesses would entail a big penalty to aligned accesses. I'll see if it's easy to convenient to get some numbers as I'm working on a fix.

[zanderso]

@fsc8000 would fb75e1e help with this?

[fsc8000]

My CL fixed the discrepancy in class member offsets between the host- and the target-toolchain. So I don't think it will help here. The ALIGN8 macro just aligns offset of the member - so there would still be unaligned access if the object start is misaligned.

[mkustermann]

@sortie Not sure if the slow down you experience is related to this, but sometimes the kernel traps unaligned access and emulates them which is significantly slower.

[eseidelGoogle]

We saw a Flutter customer hitting this in the wild in flutter/flutter#10701.

[rmacnak-google]

Closing in favor of #22151.