Migrate WithWarnings to use EnsoHashMap to speed them up significantly

[JaroslavTulach]

This is a long time open task that keeps popping up from time to time. For example again in

• Support on_problems=Problem_Behavior.Report_Warning and Map_Error wrapping in Vector.map #8595 (comment)

The task has been hidden as a future checkbox in #5233, but apparently it is so hidden there it deserves its own issue.

WithWarnings uses EconomicMap implementation which (despite its good name) doesn't deliver on it promise as it cannot be partially evaluated and has to be hidden behind @TruffleBoundary. That slows things down by magnitudes as Greg's benchmark also indicate.

Since

• 400x faster with linear hashing of the hash map entries #8425

there is a better way. Replace EconomicMap with direct use of EnsoHashMap (which is carefully designed to be PE friendly) and things will get lightening fast!

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-07-16):

Progress: - Collect profiling from benchmarks to see the real hotspots of WithWarnings.

• Addressing review in System.exit does proper context hard exit. #10363 It should be finished by 2024-07-24.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-07-17):

Progress: - Migrated WithWarnings to use EnsoHashMap, the performance is much worse so far, will investigate closely tomorrow. It should be finished by 2024-07-24.

[JaroslavTulach]

the performance is much worse so far

I cannot say why that's the case. I believe your approach is correct, Pavel.

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2024-07-18):

Progress: - Continue investigation of performance issues on WithWarnings. It should be finished by 2024-07-24.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-07-19):

Progress: - Let's not remove WarningsLibrary now, it would be too much changes at once.

• Created dedicated AppendWarningNode that handles warning attachment functionality.
  • Is a lot of manual code rewritting, since warning handling is all over the code base.
• The goal is to rewrite all warning handling into dedicated Truffle nodes. It should be finished by 2024-07-24.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-07-22):

Progress: - Still working on the AppendWarningNode - debugging an issue with more warnings appended than expected. It should be finished by 2024-07-24.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-07-23):

Progress: - Fixed all the tests for the newly introduced AppendWarningNode.

• I did not test the performance yet, just fixed the tests, let's first make sure that everything works as expected. It should be finished by 2024-07-24.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-07-24):

Progress: - Starting to analyze the performance. The warnings benchmark is so far twice as slow as it used to be.

• Started a simple fix Atom fields are visible in debugger #10661 that is simple but there are some things to discuss about Atom interop protocol. It should be finished by 2024-07-24.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-07-25):

Progress: - reassignments of warnings are the most costly and most frequent operation and they server no purpose, let's remove them.

• Removing them sped up the benchmark - WithWarnings uses EnsoHashMap to speed things up #10555 (comment)
• There is still an unecessary array allocation. Let's remove it by refactoring WarningsLibrary.getWarnings to return EnsoHashMap directly.
• Finished the small fix in Atom fields are visible in debugger #10661 and created a new follow-up issue Debugger doesn't show atom fields when there are multiple constructors #10675 It should be finished by 2024-07-31.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-07-26):

Progress: - Debugging performance - looking into IGV graphs, experimenting with profiles.

• Managed to get the performance to a better shape, but still not better than on develop. It should be finished by 2024-07-31.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-07-29):

Progress: - Discovered std lib benchmark regression (#10702)

• The investigation is partially blocked by Cannot run stdlib benchmarks locally via JMH #10706
• I have strong suspicion that it is caused by Implement Runtime.Context.Dataflow_Stack_Trace for dataflow errors thrown from Enso #9625
• Experimenting with some parameters for the WarningBenchmarks.diffWarningRandomElementsVecSum.
  • Truffle graphs suggest that the handling of warning on a vector with max warnings should be no-op. However, the score of the benchmark is still 3x worse than on develop.
  • Does the benchmark really measure only the warning handling? Or does it measure also vector creation and warnings attaching? It should be finished by 2024-07-31.

[enso-bot]

Pavel Marek reports a new STANDUP for yesterday (2024-07-29):

Progress: - JPMS help on #10714

• Turning std libs into JPMS modules.
• Consulting Jaroslav about next steps for the warning performance.
• Started to analyse requirements for Store FramePointer in the IR and cache it #10129 It should be finished by 2024-07-31.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-07-31):

Progress: - More JPMS help

• Some experimenting on Warnings performance.
• Created skeletons of new compiler pass in Store FramePointer in IR #10729 It should be finished by 2024-07-31.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-08-05):

Progress: - Finally reveal the worst bottleneck. The benchmark results are now 10x better than on develop.

• Integrating rest of PR reviews and test failures, try to merge ASAP. It should be finished by 2024-08-07.

[enso-bot]

Pavel Marek reports a new STANDUP for today (2024-08-06):

Progress: - Tests seem OK, scheduling benchmarks on this PR and tomorrow, will compare the results. It should be finished by 2024-08-07.