Add support for using Probe DFG OSR Exit behind a runtime flag.

https://bugs.webkit.org/show_bug.cgi?id=177844
<rdar://problem/34801425>

Reviewed by Saam Barati.

Source/JavaScriptCore:

This is based on the code originally posted in https://bugs.webkit.org/show_bug.cgi?id=175144
(in r221774 and r221832) with some optimizations and bug fixes added.  The probe
based DFG OSR Exit is only enabled if Options::useProbeOSRExit() is true.  We're
landing this behind an option switch to make it easier to tune performance using
the probe based OSR exit.

* JavaScriptCore.xcodeproj/project.pbxproj:
* assembler/MacroAssembler.cpp:
(JSC::stdFunctionCallback):
* assembler/MacroAssemblerPrinter.cpp:
(JSC::Printer::printCallback):
* assembler/ProbeContext.cpp:
(JSC::Probe::executeProbe):
(JSC::Probe::flushDirtyStackPages):
* assembler/ProbeContext.h:
(JSC::Probe::Context::Context):
(JSC::Probe::Context::arg):
* assembler/ProbeFrame.h: Added.
(JSC::Probe::Frame::Frame):
(JSC::Probe::Frame::argument):
(JSC::Probe::Frame::operand):
(JSC::Probe::Frame::setArgument):
(JSC::Probe::Frame::setOperand):
(JSC::Probe::Frame::get):
(JSC::Probe::Frame::set):
* assembler/ProbeStack.cpp:
(JSC::Probe::Page::lowWatermarkFromVisitingDirtyChunks):
(JSC::Probe::Stack::Stack):
(JSC::Probe::Stack::lowWatermarkFromVisitingDirtyPages):
* assembler/ProbeStack.h:
(JSC::Probe::Stack::Stack):
(JSC::Probe::Stack::lowWatermark):
(JSC::Probe::Stack::set):
(JSC::Probe::Stack::savedStackPointer const):
(JSC::Probe::Stack::setSavedStackPointer):
(JSC::Probe::Stack::newStackPointer const): Deleted.
(JSC::Probe::Stack::setNewStackPointer): Deleted.
* bytecode/ArrayProfile.h:
(JSC::ArrayProfile::observeArrayMode):
* bytecode/CodeBlock.cpp:
(JSC::CodeBlock::updateOSRExitCounterAndCheckIfNeedToReoptimize):
* bytecode/CodeBlock.h:
(JSC::CodeBlock::addressOfOSRExitCounter): Deleted.
* bytecode/ExecutionCounter.h:
(JSC::ExecutionCounter::hasCrossedThreshold const):
(JSC::ExecutionCounter::setNewThresholdForOSRExit):
* bytecode/MethodOfGettingAValueProfile.cpp:
(JSC::MethodOfGettingAValueProfile::reportValue):
* bytecode/MethodOfGettingAValueProfile.h:
* dfg/DFGDriver.cpp:
(JSC::DFG::compileImpl):
* dfg/DFGJITCompiler.cpp:
(JSC::DFG::JITCompiler::linkOSRExits):
(JSC::DFG::JITCompiler::link):
* dfg/DFGOSRExit.cpp:
(JSC::DFG::jsValueFor):
(JSC::DFG::restoreCalleeSavesFor):
(JSC::DFG::saveCalleeSavesFor):
(JSC::DFG::restoreCalleeSavesFromVMEntryFrameCalleeSavesBuffer):
(JSC::DFG::copyCalleeSavesToVMEntryFrameCalleeSavesBuffer):
(JSC::DFG::saveOrCopyCalleeSavesFor):
(JSC::DFG::createDirectArgumentsDuringExit):
(JSC::DFG::createClonedArgumentsDuringExit):
(JSC::DFG::emitRestoreArguments):
(JSC::DFG::OSRExit::executeOSRExit):
(JSC::DFG::reifyInlinedCallFrames):
(JSC::DFG::adjustAndJumpToTarget):
(JSC::DFG::printOSRExit):
* dfg/DFGOSRExit.h:
(JSC::DFG::OSRExitState::OSRExitState):
* dfg/DFGThunks.cpp:
(JSC::DFG::osrExitThunkGenerator):
* dfg/DFGThunks.h:
* dfg/DFGVariableEventStream.cpp:
(JSC::DFG::tryToSetConstantRecovery):
(JSC::DFG::VariableEventStream::reconstruct const):
(JSC::DFG::VariableEventStream::tryToSetConstantRecovery const): Deleted.
* dfg/DFGVariableEventStream.h:
* profiler/ProfilerOSRExit.h:
(JSC::Profiler::OSRExit::incCount):
* runtime/JSCJSValue.h:
* runtime/JSCJSValueInlines.h:
* runtime/Options.h:

Tools:

Enable --useProbeOSrExit=true for dfg-eager and ftl-no-cjit-validate-sampling-profiler
test configurations.

* Scripts/run-jsc-stress-tests:



Canonical link: https://commits.webkit.org/194150@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@222871 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Source/JavaScriptCore/ChangeLog

@@ -1,3 +1,95 @@
+2017-10-04  Mark Lam  <mark.lam@apple.com>
+
+        Add support for using Probe DFG OSR Exit behind a runtime flag.
+        https://bugs.webkit.org/show_bug.cgi?id=177844
+        <rdar://problem/34801425>
+
+        Reviewed by Saam Barati.
+
+        This is based on the code originally posted in https://bugs.webkit.org/show_bug.cgi?id=175144
+        (in r221774 and r221832) with some optimizations and bug fixes added.  The probe
+        based DFG OSR Exit is only enabled if Options::useProbeOSRExit() is true.  We're
+        landing this behind an option switch to make it easier to tune performance using
+        the probe based OSR exit.
+
+        * JavaScriptCore.xcodeproj/project.pbxproj:
+        * assembler/MacroAssembler.cpp:
+        (JSC::stdFunctionCallback):
+        * assembler/MacroAssemblerPrinter.cpp:
+        (JSC::Printer::printCallback):
+        * assembler/ProbeContext.cpp:
+        (JSC::Probe::executeProbe):
+        (JSC::Probe::flushDirtyStackPages):
+        * assembler/ProbeContext.h:
+        (JSC::Probe::Context::Context):
+        (JSC::Probe::Context::arg):
+        * assembler/ProbeFrame.h: Added.
+        (JSC::Probe::Frame::Frame):
+        (JSC::Probe::Frame::argument):
+        (JSC::Probe::Frame::operand):
+        (JSC::Probe::Frame::setArgument):
+        (JSC::Probe::Frame::setOperand):
+        (JSC::Probe::Frame::get):
+        (JSC::Probe::Frame::set):
+        * assembler/ProbeStack.cpp:
+        (JSC::Probe::Page::lowWatermarkFromVisitingDirtyChunks):
+        (JSC::Probe::Stack::Stack):
+        (JSC::Probe::Stack::lowWatermarkFromVisitingDirtyPages):
+        * assembler/ProbeStack.h:
+        (JSC::Probe::Stack::Stack):
+        (JSC::Probe::Stack::lowWatermark):
+        (JSC::Probe::Stack::set):
+        (JSC::Probe::Stack::savedStackPointer const):
+        (JSC::Probe::Stack::setSavedStackPointer):
+        (JSC::Probe::Stack::newStackPointer const): Deleted.
+        (JSC::Probe::Stack::setNewStackPointer): Deleted.
+        * bytecode/ArrayProfile.h:
+        (JSC::ArrayProfile::observeArrayMode):
+        * bytecode/CodeBlock.cpp:
+        (JSC::CodeBlock::updateOSRExitCounterAndCheckIfNeedToReoptimize):
+        * bytecode/CodeBlock.h:
+        (JSC::CodeBlock::addressOfOSRExitCounter): Deleted.
+        * bytecode/ExecutionCounter.h:
+        (JSC::ExecutionCounter::hasCrossedThreshold const):
+        (JSC::ExecutionCounter::setNewThresholdForOSRExit):
+        * bytecode/MethodOfGettingAValueProfile.cpp:
+        (JSC::MethodOfGettingAValueProfile::reportValue):
+        * bytecode/MethodOfGettingAValueProfile.h:
+        * dfg/DFGDriver.cpp:
+        (JSC::DFG::compileImpl):
+        * dfg/DFGJITCompiler.cpp:
+        (JSC::DFG::JITCompiler::linkOSRExits):
+        (JSC::DFG::JITCompiler::link):
+        * dfg/DFGOSRExit.cpp:
+        (JSC::DFG::jsValueFor):
+        (JSC::DFG::restoreCalleeSavesFor):
+        (JSC::DFG::saveCalleeSavesFor):
+        (JSC::DFG::restoreCalleeSavesFromVMEntryFrameCalleeSavesBuffer):
+        (JSC::DFG::copyCalleeSavesToVMEntryFrameCalleeSavesBuffer):
+        (JSC::DFG::saveOrCopyCalleeSavesFor):
+        (JSC::DFG::createDirectArgumentsDuringExit):
+        (JSC::DFG::createClonedArgumentsDuringExit):
+        (JSC::DFG::emitRestoreArguments):
+        (JSC::DFG::OSRExit::executeOSRExit):
+        (JSC::DFG::reifyInlinedCallFrames):
+        (JSC::DFG::adjustAndJumpToTarget):
+        (JSC::DFG::printOSRExit):
+        * dfg/DFGOSRExit.h:
+        (JSC::DFG::OSRExitState::OSRExitState):
+        * dfg/DFGThunks.cpp:
+        (JSC::DFG::osrExitThunkGenerator):
+        * dfg/DFGThunks.h:
+        * dfg/DFGVariableEventStream.cpp:
+        (JSC::DFG::tryToSetConstantRecovery):
+        (JSC::DFG::VariableEventStream::reconstruct const):
+        (JSC::DFG::VariableEventStream::tryToSetConstantRecovery const): Deleted.
+        * dfg/DFGVariableEventStream.h:
+        * profiler/ProfilerOSRExit.h:
+        (JSC::Profiler::OSRExit::incCount):
+        * runtime/JSCJSValue.h:
+        * runtime/JSCJSValueInlines.h:
+        * runtime/Options.h:
+
 2017-10-04  Ryan Haddad  <ryanhaddad@apple.com>
 
         Unreviewed, rolling out r222840.

Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

@@ -4596,6 +4596,7 @@
 		FEA0C4001CDD7D0E00481991 /* FunctionWhitelist.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = FunctionWhitelist.cpp; sourceTree = "<group>"; };
 		FEA0C4011CDD7D0E00481991 /* FunctionWhitelist.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = FunctionWhitelist.h; sourceTree = "<group>"; };
 		FEB137561BB11EEE00CD5100 /* MacroAssemblerARM64.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = MacroAssemblerARM64.cpp; sourceTree = "<group>"; };
+		FEB41CCB1F73284200C5481E /* ProbeFrame.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ProbeFrame.h; sourceTree = "<group>"; };
 		FEB51F6A1A97B688001F921C /* Regress141809.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = Regress141809.h; path = API/tests/Regress141809.h; sourceTree = "<group>"; };
 		FEB51F6B1A97B688001F921C /* Regress141809.mm */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.objcpp; name = Regress141809.mm; path = API/tests/Regress141809.mm; sourceTree = "<group>"; };
 		FEB58C12187B8B160098EF0B /* ErrorHandlingScope.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ErrorHandlingScope.cpp; sourceTree = "<group>"; };
@@ -7295,6 +7296,7 @@
 				FE63DD531EA9B60E00103A69 /* Printer.h */,
 				FE10AAF31F46826D009DEDC5 /* ProbeContext.cpp */,
 				FE10AAED1F44D946009DEDC5 /* ProbeContext.h */,
+				FEB41CCB1F73284200C5481E /* ProbeFrame.h */,
 				FE10AAE91F44D510009DEDC5 /* ProbeStack.cpp */,
 				FE10AAEA1F44D512009DEDC5 /* ProbeStack.h */,
 				FE533CA01F217C310016A1FE /* testmasm.cpp */,

Source/JavaScriptCore/assembler/MacroAssembler.cpp

@@ -38,7 +38,7 @@ const double MacroAssembler::twoToThe32 = (double)0x100000000ull;
 #if ENABLE(MASM_PROBE)
 static void stdFunctionCallback(Probe::Context& context)
 {
-    auto func = static_cast<const std::function<void(Probe::Context&)>*>(context.arg);
+    auto func = context.arg<const std::function<void(Probe::Context&)>*>();
     (*func)(context);
 }
 

Source/JavaScriptCore/assembler/MacroAssemblerPrinter.cpp

@@ -175,7 +175,7 @@ void printMemory(PrintStream& out, Context& context)
 void printCallback(Probe::Context& probeContext)
 {
     auto& out = WTF::dataFile();
-    PrintRecordList& list = *reinterpret_cast<PrintRecordList*>(probeContext.arg);
+    PrintRecordList& list = *probeContext.arg<PrintRecordList*>();
     for (size_t i = 0; i < list.size(); i++) {
         auto& record = list[i];
         Context context(probeContext, record.data);

Source/JavaScriptCore/assembler/ProbeContext.cpp

@@ -52,8 +52,9 @@ void executeProbe(State* state)
 #endif
 
     if (context.hasWritesToFlush()) {
-        context.stack().setNewStackPointer(state->cpu.sp());
-        state->cpu.sp() = std::min(context.stack().lowWatermark(), state->cpu.sp());
+        context.stack().setSavedStackPointer(state->cpu.sp());
+        void* lowWatermark = context.stack().lowWatermark(state->cpu.sp());
+        state->cpu.sp() = std::min(lowWatermark, state->cpu.sp());
 
         state->initializeStackFunction = flushDirtyStackPages;
         state->initializeStackArg = context.releaseStack();
@@ -64,7 +65,7 @@ static void flushDirtyStackPages(State* state)
 {
     std::unique_ptr<Stack> stack(reinterpret_cast<Probe::Stack*>(state->initializeStackArg));
     stack->flushWrites();
-    state->cpu.sp() = stack->newStackPointer();
+    state->cpu.sp() = stack->savedStackPointer();
 }
 
 // Not for general use. This should only be for writing tests.

Source/JavaScriptCore/assembler/ProbeContext.h

@@ -191,11 +191,13 @@ class Context {
     using FPRegisterID = MacroAssembler::FPRegisterID;
 
     Context(State* state)
-        : m_state(state)
-        , arg(state->arg)
-        , cpu(state->cpu)
+        : cpu(state->cpu)
+        , m_state(state)
     { }
 
+    template<typename T>
+    T arg() { return reinterpret_cast<T>(m_state->arg); }
+
     uintptr_t& gpr(RegisterID id) { return cpu.gpr(id); }
     uintptr_t& spr(SPRegisterID id) { return cpu.spr(id); }
     double& fpr(FPRegisterID id) { return cpu.fpr(id); }
@@ -224,13 +226,10 @@ class Context {
     bool hasWritesToFlush() { return m_stack.hasWritesToFlush(); }
     Stack* releaseStack() { return new Stack(WTFMove(m_stack)); }
 
-private:
-    State* m_state;
-public:
-    void* arg;
     CPUState& cpu;
 
 private:
+    State* m_state;
     Stack m_stack;
 
     friend JS_EXPORT_PRIVATE void* probeStateForContext(Context&); // Not for general use. This should only be for writing tests.

Source/JavaScriptCore/assembler/ProbeFrame.h

@@ -0,0 +1,94 @@
+/*
+ * Copyright (C) 2017 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL APPLE INC. OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#pragma once
+
+#if ENABLE(MASM_PROBE)
+
+#include "CallFrame.h"
+#include "ProbeStack.h"
+
+namespace JSC {
+namespace Probe {
+
+class Frame {
+public:
+    Frame(void* frameBase, Stack& stack)
+        : m_frameBase { reinterpret_cast<uint8_t*>(frameBase) }
+        , m_stack { stack }
+    { }
+
+    template<typename T = JSValue>
+    T argument(int argument)
+    {
+        return get<T>(CallFrame::argumentOffset(argument) * sizeof(Register));
+    }
+    template<typename T = JSValue>
+    T operand(int operand)
+    {
+        return get<T>(static_cast<VirtualRegister>(operand).offset() * sizeof(Register));
+    }
+    template<typename T = JSValue>
+    T operand(int operand, ptrdiff_t offset)
+    {
+        return get<T>(static_cast<VirtualRegister>(operand).offset() * sizeof(Register) + offset);
+    }
+
+    template<typename T>
+    void setArgument(int argument, T value)
+    {
+        return set<T>(CallFrame::argumentOffset(argument) * sizeof(Register), value);
+    }
+    template<typename T>
+    void setOperand(int operand, T value)
+    {
+        set<T>(static_cast<VirtualRegister>(operand).offset() * sizeof(Register), value);
+    }
+    template<typename T>
+    void setOperand(int operand, ptrdiff_t offset, T value)
+    {
+        set<T>(static_cast<VirtualRegister>(operand).offset() * sizeof(Register) + offset, value);
+    }
+
+    template<typename T = JSValue>
+    T get(ptrdiff_t offset)
+    {
+        return m_stack.get<T>(m_frameBase + offset);
+    }
+    template<typename T>
+    void set(ptrdiff_t offset, T value)
+    {
+        m_stack.set<T>(m_frameBase + offset, value);
+    }
+
+private:
+    uint8_t* m_frameBase;
+    Stack& m_stack;
+};
+
+} // namespace Probe
+} // namespace JSC
+
+#endif // ENABLE(MASM_PROBE)

Source/JavaScriptCore/assembler/ProbeStack.cpp

@@ -34,6 +34,8 @@
 namespace JSC {
 namespace Probe {
 
+static void* const maxLowWatermark = reinterpret_cast<void*>(std::numeric_limits<uintptr_t>::max());
+
 #if ASAN_ENABLED
 // FIXME: we should consider using the copy function for both ASan and non-ASan builds.
 // https://bugs.webkit.org/show_bug.cgi?id=176961
@@ -49,7 +51,7 @@ static void copyStackPage(void* dst, void* src, size_t size)
         *dstPointer++ = *srcPointer++;
 }
 #else
-#define copyStackPage(dst, src, size) std::memcpy(dst, src, size);
+#define copyStackPage(dst, src, size) std::memcpy(dst, src, size)
 #endif
 
 Page::Page(void* baseAddress)
@@ -84,12 +86,24 @@ void Page::flushWrites()
     m_dirtyBits = 0;
 }
 
+void* Page::lowWatermarkFromVisitingDirtyChunks()
+{
+    uint64_t dirtyBits = m_dirtyBits;
+    size_t offset = 0;
+    while (dirtyBits) {
+        if (dirtyBits & 1)
+            return reinterpret_cast<uint8_t*>(m_baseLogicalAddress) + offset;
+        dirtyBits = dirtyBits >> 1;
+        offset += s_chunkSize;
+    }
+    return maxLowWatermark;
+}
+
 Stack::Stack(Stack&& other)
-    : m_newStackPointer(other.m_newStackPointer)
-    , m_lowWatermark(other.m_lowWatermark)
-    , m_stackBounds(WTFMove(other.m_stackBounds))
+    : m_stackBounds(WTFMove(other.m_stackBounds))
     , m_pages(WTFMove(other.m_pages))
 {
+    m_savedStackPointer = other.m_savedStackPointer;
 #if !ASSERT_DISABLED
     other.m_isValid = false;
 #endif
@@ -128,6 +142,18 @@ Page* Stack::ensurePageFor(void* address)
     return m_lastAccessedPage;
 }
 
+void* Stack::lowWatermarkFromVisitingDirtyPages()
+{
+    void* low = maxLowWatermark;
+    for (auto it = m_pages.begin(); it != m_pages.end(); ++it) {
+        Page& page = *it->value;
+        if (!page.hasWritesToFlush() || low < page.baseAddress())
+            continue;
+        low = std::min(low, page.lowWatermarkFromVisitingDirtyChunks());
+    }
+    return low;
+}
+
 } // namespace Probe
 } // namespace JSC