Ensure the same options are used for jitdump compilations

[fjeremic]

When generating a jitdump we create the TR::Options object and modify it slightly so it enables various tracing options [1], however because generating the system core dump, javacore, and heapdump take precedence over a jitdump in the dump agent order, this means we may have to wait several seconds since the crash to start generating the jitdump.

This delay can cause problems because some options, most notoriously the use of NextGenHCR [2], depend on the phase the JIT is in (startup vs. rampup vs. idle vs. etc.). These options can drastically affect the IL which is generated, which means we may not be able to reproduce the original issue in the jitdump, thus making the jitdump useless.

To remedy this problem we need to ensure that the options used in the original compilation can be reproduced in the jitdump compilation. There are two cases to consider:

1. A crash during compilation
2. A crash in a JIT compiled method body

For 1. it should be easy to get a hold of the options since the TR::Compilation* is still valid at the time of jitdump generation, so we can just copy over the options from the original compilation. The issue is how to handle 2.?

[1] https://github.com/eclipse/openj9/blob/8fdc4f47505253f24ac82e4fd48fe0ca10657c1c/runtime/compiler/control/CompilationThread.cpp#L7317-L7341
[2] https://github.com/eclipse/openj9/blob/8fdc4f47505253f24ac82e4fd48fe0ca10657c1c/runtime/compiler/control/CompilationThread.cpp#L7572-L7576

[fjeremic]

@dsouzai @mpirvu looking for thoughts about 2. There are about ~1200 options at the moment. It seems unfeasible to persist those options for each compilation. The next best thing we can do is try to identify which options are enabled/disabled based on "the state of the world" or environment at the time, such as the one in [2], and isolate such options in a single place somewhere and only persist those options with the metadata.

Do you have any other thoughts on how to handle 2.? @0xdaryl FYI as well.

[mpirvu]

Since the crash can theoretically happen in any jitted method (moreover we generate a compilation log for any jitted method found on the stack of the faulting thread) we need to preserve the options for all compilations. Now, how do we minimize the amount of data stored for each compilation?

identify which options are enabled/disabled based on "the state of the world" or environment at the time, such as the one in [2], and isolate such options in a single place somewhere

We could that if the set of such options is small.

I wonder if we can use some delta encoding. The options for a compilation are derived from the master cmdLineOptions and then particularized for the compilation. If we serialize the options and compute the diff vs the master options, we could store just the diff.
Metadata is the obvious choice for storing this additional information.

[fjeremic]

@mpirvu do you have any sense of how many distinct option sets there would be during the lifetime of a typical JVM? The option set in currently ~150 bytes worth of information. Obviously we don't want to store that for every method. Could we instead store it in a hash map and only store the hash for every method?

If we expect such a hash map to have few distinct values (~150 bytes per value) then this would be feasible as well. Thinking about it I would expect there only to be a handful of entries, but given your experience you may be able to better gauge whether going down this path would be worth while.

Delta compression would also work, however I'm not sure what the compression ratio would be on 150 bytes.

[fjeremic]

This issue is partly addressed by #11959. I'll continue to monitor if we are unable to reproduce original crashes to see if we need to come back to this issue. I think I've covered the majority of the cases. As a future work item we may get someone new or as a project for a beginner to take a look at whether it is feasible to cache the option bits for past compilations with the previously mentioned method.