vm/dart/use_dwarf_stack_traces_flag_test fails on AOT bots

[sstrickl]

There are new test failures on Enable overridden_fields in analyzer/....Resolve RecordPattern(s)..

The tests

vm/dart_2/use_dwarf_stack_traces_flag_test RuntimeError (expected Pass)

are failing on configurations

dartkp-linux-debug-x64c

Strangely enough, the bisection points to a5ad599, which is completely unrelated. Will continue looking into this, but approving for now, especially since the failure doesn't cause any issues in using the native_stack_traces package to decode dwarf stack traces since all the other tests using this functionality passes. This is just failing while testing something I thought would be an invariant that perhaps isn't as much of one as I thought.

[mkustermann]

Today it seems this test started failing again. Though running it locally seems to pass.

Maybe there is something not quite deterministic in this test?

The specific place it fails is when comparing virtual addresses to be matching across runs. Are those virtual addresses relative to dso base or actual runtime virtual addresses (which may not match across runs due to ASLR)?

[sstrickl]

Those should be relocated addresses, IIRC, e.g., relative to the DSO base. That's what's confusing to me, as that calculation shouldn't be non-deterministic, since the comparisons that are failing are only run when we're compiling directly to ELF, in which case we control the relocated addresses and thus they should be the same in both separate debugging info and the snapshot. I'll take a look at this again in a bit.

However, I might just remove the checks entirely, since we never actually end up using the virt-marked addresses when actually symbolicizing stack traces, and, as the lack of other failing tests shows, that symbolicization is currently always succeeding. It's just odd that those values aren't consistently equal to the other calculations of relocated addresses, since they should be, but maybe it's the calculation of the virt-marked addresses in the runtime that's to blame (or something that calculation depends on).

[sstrickl]

Seeing a failure on vm-kernel-precomp-win-debug-x64c today as well. The fact that it only seems to be happening on compressed pointer architectures is interesting, and might be the clue to figuring this out. It's my gardening day, so I'll allocate cycles to pinning this down.

[sstrickl]

Note that this does seem to be somewhat flaky, as dartkp-linux-debug-x64c is currently passing (not just approved failing).

[alexmarkov]

vm/dart/use_dwarf_stack_traces_flag_test started failing consistently on vm-aot-msan-linux-release-x64 bot.

It is also flaky on several other AOT bots: https://dart-current-results.web.app/#/filter=vm/dart/use_dwarf_stack_traces_flag_test&showAll

[alexmarkov]

Historically, it has been flipping between Pass, flaky and RuntimeError:

https://dart-ci.firebaseapp.com/#showLatestFailures=false&test=vm/dart/use_dwarf_stack_traces_flag_test

[sstrickl]

I think I've figured out why this is happening while working on CL 362380. It looks like sometimes the call info for the address returned by VMInternalsForTesting.randomInstructionsOffsetInsideAllocateObjectStub() isn't attributed to the stub, but rather just a random address in the instructions section. (That is, I got a failure at this check during one of my runs of the new test, which is derived from the test that's flaky and so would be similarly flaky.)

I'll take a look and see which part is at fault: the VM runtime for returning an address that shouldn't be within the stub or the generated DWARF for not appropriately capturing the proper address range for the stub.