[JSC] Make DFG::OSRExit data unlinked

https://bugs.webkit.org/show_bug.cgi?id=239828

Reviewed by Saam Barati.

This patch makes DFG::OSRExit unlinked. While generated OSR exit code is linked version,
we no longer put linked data to DFG::OSRExit so that unlinked DFG can use DFG::OSRExit.
Key changes are two things.

1. Now, we always store compiled MacroAssemblerCodeRef in DFG::JITData regardless of whether
   we have linked / unlinked DFG. While linked DFG uses repatching to jump to this code,
   unlinked DFG looks into this vector in JITData and jump to that.
2. MethodOfGettingAValueProfile was including CodeBlock*, ValueProfile* in CodeBlock* etc.,
   so it was linked data structure which unlinked DFG cannot use. Instead, we encode how to
   retrieve these pointers when generating OSR exit code actually, and just storing CodeOrigin,
   type, and Operand to make MethodOfGettingAValueProfile unlinked data structure. While
   CodeOrigin can include InlineCallFrame, but our first version of unlinked DFG will not perform
   inlining thus we will not include it. It also makes sizeof(MethodOfGettingAValueProfile) smaller
   from 32 bytes to 16 bytes (50% reduction).

* Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.h:
(JSC::MacroAssemblerCodeRef::offsetOfCodePtr):
* Source/JavaScriptCore/bytecode/CodeBlock.cpp:
(JSC::CodeBlock::updateOSRExitCounterAndCheckIfNeedToReoptimize): Deleted.
* Source/JavaScriptCore/bytecode/CodeBlock.h:
* Source/JavaScriptCore/bytecode/MethodOfGettingAValueProfile.cpp:
(JSC::MethodOfGettingAValueProfile::emitReportValue const):
(JSC::MethodOfGettingAValueProfile::fromLazyOperand): Deleted.
(JSC::MethodOfGettingAValueProfile::reportValue): Deleted.
* Source/JavaScriptCore/bytecode/MethodOfGettingAValueProfile.h:
(JSC::MethodOfGettingAValueProfile::unaryArithProfile):
(JSC::MethodOfGettingAValueProfile::binaryArithProfile):
(JSC::MethodOfGettingAValueProfile::argumentValueProfile):
(JSC::MethodOfGettingAValueProfile::bytecodeValueProfile):
(JSC::MethodOfGettingAValueProfile::lazyOperandValueProfile):
(JSC::MethodOfGettingAValueProfile::operator bool const):
(JSC::MethodOfGettingAValueProfile::MethodOfGettingAValueProfile): Deleted.
* Source/JavaScriptCore/dfg/DFGGraph.cpp:
(JSC::DFG::Graph::methodOfGettingAValueProfileFor):
* Source/JavaScriptCore/dfg/DFGJITCode.cpp:
(JSC::DFG::JITCode::JITCode):
(JSC::DFG::JITCode::findPC):
* Source/JavaScriptCore/dfg/DFGJITCode.h:
* Source/JavaScriptCore/dfg/DFGJITCompiler.cpp:
(JSC::DFG::JITCompiler::JITCompiler):
(JSC::DFG::JITCompiler::linkOSRExits):
(JSC::DFG::JITCompiler::link):
* Source/JavaScriptCore/dfg/DFGJITFinalizer.cpp:
(JSC::DFG::JITFinalizer::finalize):
* Source/JavaScriptCore/dfg/DFGOSRExit.cpp:
(JSC::DFG::JSC_DEFINE_JIT_OPERATION):
(JSC::DFG::OSRExit::compileExit):
(JSC::DFG::OSRExit::codeLocationForRepatch const): Deleted.
* Source/JavaScriptCore/dfg/DFGOSRExit.h:
(JSC::DFG::OSRExit::codeLocationForRepatch const):
(JSC::DFG::OSRExitState::OSRExitState): Deleted.
* Source/JavaScriptCore/dfg/DFGPlan.cpp:
(JSC::DFG::Plan::finalizeJITData):
* Source/JavaScriptCore/dfg/DFGPlan.h:
* Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::lower):
* Source/JavaScriptCore/ftl/FTLOSRExitCompiler.cpp:
(JSC::FTL::compileStub):
* Source/WTF/wtf/FixedVector.h:

Canonical link: https://commits.webkit.org/250111@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@293605 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Source/JavaScriptCore/ChangeLog

@@ -1,3 +1,69 @@
+2022-04-27  Yusuke Suzuki  <ysuzuki@apple.com>
+
+        [JSC] Make DFG::OSRExit data unlinked
+        https://bugs.webkit.org/show_bug.cgi?id=239828
+
+        Reviewed by Saam Barati.
+
+        This patch makes DFG::OSRExit unlinked. While generated OSR exit code is linked version,
+        we no longer put linked data to DFG::OSRExit so that unlinked DFG can use DFG::OSRExit.
+        Key changes are two things.
+
+        1. Now, we always store compiled MacroAssemblerCodeRef in DFG::JITData regardless of whether
+           we have linked / unlinked DFG. While linked DFG uses repatching to jump to this code,
+           unlinked DFG looks into this vector in JITData and jump to that.
+        2. MethodOfGettingAValueProfile was including CodeBlock*, ValueProfile* in CodeBlock* etc.,
+           so it was linked data structure which unlinked DFG cannot use. Instead, we encode how to
+           retrieve these pointers when generating OSR exit code actually, and just storing CodeOrigin,
+           type, and Operand to make MethodOfGettingAValueProfile unlinked data structure. While
+           CodeOrigin can include InlineCallFrame, but our first version of unlinked DFG will not perform
+           inlining thus we will not include it. It also makes sizeof(MethodOfGettingAValueProfile) smaller
+           from 32 bytes to 16 bytes (50% reduction).
+
+        * assembler/MacroAssemblerCodeRef.h:
+        (JSC::MacroAssemblerCodeRef::offsetOfCodePtr):
+        * bytecode/CodeBlock.cpp:
+        (JSC::CodeBlock::updateOSRExitCounterAndCheckIfNeedToReoptimize): Deleted.
+        * bytecode/CodeBlock.h:
+        * bytecode/MethodOfGettingAValueProfile.cpp:
+        (JSC::MethodOfGettingAValueProfile::emitReportValue const):
+        (JSC::MethodOfGettingAValueProfile::fromLazyOperand): Deleted.
+        (JSC::MethodOfGettingAValueProfile::reportValue): Deleted.
+        * bytecode/MethodOfGettingAValueProfile.h:
+        (JSC::MethodOfGettingAValueProfile::unaryArithProfile):
+        (JSC::MethodOfGettingAValueProfile::binaryArithProfile):
+        (JSC::MethodOfGettingAValueProfile::argumentValueProfile):
+        (JSC::MethodOfGettingAValueProfile::bytecodeValueProfile):
+        (JSC::MethodOfGettingAValueProfile::lazyOperandValueProfile):
+        (JSC::MethodOfGettingAValueProfile::operator bool const):
+        (JSC::MethodOfGettingAValueProfile::MethodOfGettingAValueProfile): Deleted.
+        * dfg/DFGGraph.cpp:
+        (JSC::DFG::Graph::methodOfGettingAValueProfileFor):
+        * dfg/DFGJITCode.cpp:
+        (JSC::DFG::JITCode::JITCode):
+        (JSC::DFG::JITCode::findPC):
+        * dfg/DFGJITCode.h:
+        * dfg/DFGJITCompiler.cpp:
+        (JSC::DFG::JITCompiler::JITCompiler):
+        (JSC::DFG::JITCompiler::linkOSRExits):
+        (JSC::DFG::JITCompiler::link):
+        * dfg/DFGJITFinalizer.cpp:
+        (JSC::DFG::JITFinalizer::finalize):
+        * dfg/DFGOSRExit.cpp:
+        (JSC::DFG::JSC_DEFINE_JIT_OPERATION):
+        (JSC::DFG::OSRExit::compileExit):
+        (JSC::DFG::OSRExit::codeLocationForRepatch const): Deleted.
+        * dfg/DFGOSRExit.h:
+        (JSC::DFG::OSRExit::codeLocationForRepatch const):
+        (JSC::DFG::OSRExitState::OSRExitState): Deleted.
+        * dfg/DFGPlan.cpp:
+        (JSC::DFG::Plan::finalizeJITData):
+        * dfg/DFGPlan.h:
+        * ftl/FTLLowerDFGToB3.cpp:
+        (JSC::FTL::DFG::LowerDFGToB3::lower):
+        * ftl/FTLOSRExitCompiler.cpp:
+        (JSC::FTL::compileStub):
+
 2022-04-28  Yusuke Suzuki  <ysuzuki@apple.com>
 
         Support C files in Unified Builds

Source/JavaScriptCore/assembler/MacroAssemblerCodeRef.h

@@ -526,6 +526,8 @@ class MacroAssemblerCodeRef : private MacroAssemblerCodeRefBase {
         m_codePtr.dumpWithName("CodeRef", out);
     }
 
+    static ptrdiff_t offsetOfCodePtr() { return OBJECT_OFFSETOF(MacroAssemblerCodeRef, m_codePtr); }
+
 private:
     template<PtrTag otherTag>
     MacroAssemblerCodeRef(const MacroAssemblerCodeRef<otherTag>& otherCodeRef)

Source/JavaScriptCore/bytecode/CodeBlock.cpp

@@ -2610,55 +2610,6 @@ bool CodeBlock::checkIfOptimizationThresholdReached()
     return m_jitExecuteCounter.checkIfThresholdCrossedAndSet(this);
 }
 
-#if ENABLE(DFG_JIT)
-auto CodeBlock::updateOSRExitCounterAndCheckIfNeedToReoptimize(DFG::OSRExitState& exitState) -> OptimizeAction
-{
-    DFG::OSRExitBase& exit = exitState.exit;
-    if (!exitKindMayJettison(exit.m_kind)) {
-        // FIXME: We may want to notice that we're frequently exiting
-        // at an op_catch that we didn't compile an entrypoint for, and
-        // then trigger a reoptimization of this CodeBlock:
-        // https://bugs.webkit.org/show_bug.cgi?id=175842
-        return OptimizeAction::None;
-    }
-
-    exit.m_count++;
-    m_osrExitCounter++;
-
-    CodeBlock* baselineCodeBlock = exitState.baselineCodeBlock;
-    ASSERT(baselineCodeBlock == baselineAlternative());
-    if (UNLIKELY(baselineCodeBlock->jitExecuteCounter().hasCrossedThreshold()))
-        return OptimizeAction::ReoptimizeNow;
-
-    // We want to figure out if there's a possibility that we're in a loop. For the outermost
-    // code block in the inline stack, we handle this appropriately by having the loop OSR trigger
-    // check the exit count of the replacement of the CodeBlock from which we are OSRing. The
-    // problem is the inlined functions, which might also have loops, but whose baseline versions
-    // don't know where to look for the exit count. Figure out if those loops are severe enough
-    // that we had tried to OSR enter. If so, then we should use the loop reoptimization trigger.
-    // Otherwise, we should use the normal reoptimization trigger.
-
-    bool didTryToEnterInLoop = false;
-    for (InlineCallFrame* inlineCallFrame = exit.m_codeOrigin.inlineCallFrame(); inlineCallFrame; inlineCallFrame = inlineCallFrame->directCaller.inlineCallFrame()) {
-        if (inlineCallFrame->baselineCodeBlock->ownerExecutable()->didTryToEnterInLoop()) {
-            didTryToEnterInLoop = true;
-            break;
-        }
-    }
-
-    uint32_t exitCountThreshold = didTryToEnterInLoop
-        ? exitCountThresholdForReoptimizationFromLoop()
-        : exitCountThresholdForReoptimization();
-
-    if (m_osrExitCounter > exitCountThreshold)
-        return OptimizeAction::ReoptimizeNow;
-
-    // Too few fails. Adjust the execution counter such that the target is to only optimize after a while.
-    baselineCodeBlock->m_jitExecuteCounter.setNewThresholdForOSRExit(exitState.activeThreshold, exitState.memoryUsageAdjustedThreshold);
-    return OptimizeAction::None;
-}
-#endif
-
 void CodeBlock::optimizeNextInvocation()
 {
     dataLogLnIf(Options::verboseOSR(), *this, ": Optimizing next invocation.");

Source/JavaScriptCore/bytecode/CodeBlock.h

@@ -80,7 +80,6 @@ namespace JSC {
 #if ENABLE(DFG_JIT)
 namespace DFG {
 class JITData;
-struct OSRExitState;
 } // namespace DFG
 #endif
 
@@ -706,11 +705,6 @@ class CodeBlock : public JSCell {
 
     void countOSRExit() { m_osrExitCounter++; }
 
-    enum class OptimizeAction { None, ReoptimizeNow };
-#if ENABLE(DFG_JIT)
-    OptimizeAction updateOSRExitCounterAndCheckIfNeedToReoptimize(DFG::OSRExitState&);
-#endif
-
     static ptrdiff_t offsetOfOSRExitCounter() { return OBJECT_OFFSETOF(CodeBlock, m_osrExitCounter); }
 
     uint32_t adjustedExitCountThreshold(uint32_t desiredThreshold);

Source/JavaScriptCore/bytecode/MethodOfGettingAValueProfile.cpp

@@ -35,82 +35,52 @@
 
 namespace JSC {
 
-MethodOfGettingAValueProfile MethodOfGettingAValueProfile::fromLazyOperand(
-    CodeBlock* codeBlock, const LazyOperandValueProfileKey& key)
+void MethodOfGettingAValueProfile::emitReportValue(CCallHelpers& jit, CodeBlock* optimizedCodeBlock, JSValueRegs regs, GPRReg tempGPR, TagRegistersMode mode) const
 {
-    MethodOfGettingAValueProfile result;
-    result.m_kind = LazyOperand;
-    result.u.lazyOperand.codeBlock = codeBlock;
-    result.u.lazyOperand.bytecodeOffset = key.bytecodeIndex();
-    result.u.lazyOperand.operand = key.operand();
-    return result;
-}
+    if (m_kind == Kind::None)
+        return;
 
-void MethodOfGettingAValueProfile::emitReportValue(CCallHelpers& jit, JSValueRegs regs, GPRReg tempGPR, TagRegistersMode mode) const
-{
+    CodeBlock* baselineCodeBlock = optimizedCodeBlock->baselineAlternative();
+    CodeBlock* profiledBlock = baselineCodeBlockForOriginAndBaselineCodeBlock(m_codeOrigin, baselineCodeBlock);
     switch (m_kind) {
-    case None:
+    case Kind::None:
+        RELEASE_ASSERT_NOT_REACHED();
         return;
 
-    case Ready:
-        jit.storeValue(regs, u.profile->specFailBucket(0));
-        return;
-
-    case LazyOperand: {
-        LazyOperandValueProfileKey key(u.lazyOperand.bytecodeOffset, u.lazyOperand.operand);
+    case Kind::LazyOperandValueProfile: {
+        LazyOperandValueProfileKey key(m_codeOrigin.bytecodeIndex(), Operand::fromBits(m_rawOperand));
 
-        ConcurrentJSLocker locker(u.lazyOperand.codeBlock->m_lock);
-        LazyOperandValueProfile* profile =
-            u.lazyOperand.codeBlock->lazyOperandValueProfiles(locker).add(locker, key);
+        ConcurrentJSLocker locker(profiledBlock->m_lock);
+        LazyOperandValueProfile* profile = profiledBlock->lazyOperandValueProfiles(locker).add(locker, key);
         jit.storeValue(regs, profile->specFailBucket(0));
         return;
     }
 
-    case UnaryArithProfileReady: {
-        u.unaryArithProfile->emitObserveResult(jit, regs, tempGPR, mode);
+    case Kind::UnaryArithProfile: {
+        if (UnaryArithProfile* result = profiledBlock->unaryArithProfileForBytecodeIndex(m_codeOrigin.bytecodeIndex()))
+            result->emitObserveResult(jit, regs, tempGPR, mode);
         return;
     }
 
-    case BinaryArithProfileReady: {
-        u.binaryArithProfile->emitObserveResult(jit, regs, tempGPR, mode);
+    case Kind::BinaryArithProfile: {
+        if (BinaryArithProfile* result = profiledBlock->binaryArithProfileForBytecodeIndex(m_codeOrigin.bytecodeIndex()))
+            result->emitObserveResult(jit, regs, tempGPR, mode);
         return;
     }
-    }
-
-    RELEASE_ASSERT_NOT_REACHED();
-}
 
-void MethodOfGettingAValueProfile::reportValue(JSValue value)
-{
-    switch (m_kind) {
-    case None:
-        return;
-
-    case Ready:
-        *u.profile->specFailBucket(0) = JSValue::encode(value);
-        return;
-
-    case LazyOperand: {
-        LazyOperandValueProfileKey key(u.lazyOperand.bytecodeOffset, u.lazyOperand.operand);
-
-        ConcurrentJSLocker locker(u.lazyOperand.codeBlock->m_lock);
-        LazyOperandValueProfile* profile =
-            u.lazyOperand.codeBlock->lazyOperandValueProfiles(locker).add(locker, key);
-        *profile->specFailBucket(0) = JSValue::encode(value);
+    case Kind::ArgumentValueProfile: {
+        auto& valueProfile = profiledBlock->valueProfileForArgument(Operand::fromBits(m_rawOperand).toArgument());
+        jit.storeValue(regs, valueProfile.specFailBucket(0));
         return;
     }
 
-    case UnaryArithProfileReady: {
-        u.unaryArithProfile->observeResult(value);
-        return;
-    }
-
-    case BinaryArithProfileReady: {
-        u.binaryArithProfile->observeResult(value);
+    case Kind::BytecodeValueProfile: {
+        auto& valueProfile = profiledBlock->valueProfileForBytecodeIndex(m_codeOrigin.bytecodeIndex());
+        jit.storeValue(regs, valueProfile.specFailBucket(0));
         return;
     }
     }
-
+    
     RELEASE_ASSERT_NOT_REACHED();
 }
 

Source/JavaScriptCore/bytecode/MethodOfGettingAValueProfile.h

@@ -32,6 +32,7 @@
 #if ENABLE(DFG_JIT)
 
 #include "BytecodeIndex.h"
+#include "CodeOrigin.h"
 #include "GPRInfo.h"
 #include "Operands.h"
 #include "TagRegistersMode.h"
@@ -42,76 +43,74 @@ class UnaryArithProfile;
 class BinaryArithProfile;
 class CCallHelpers;
 class CodeBlock;
-class LazyOperandValueProfileKey;
 struct ValueProfile;
 
 class MethodOfGettingAValueProfile {
 public:
-    MethodOfGettingAValueProfile()
-        : m_kind(None)
+    MethodOfGettingAValueProfile() = default;
+
+    static MethodOfGettingAValueProfile unaryArithProfile(CodeOrigin codeOrigin)
     {
+        MethodOfGettingAValueProfile result;
+        result.m_kind = Kind::UnaryArithProfile;
+        result.m_codeOrigin = codeOrigin;
+        return result;
     }
-    
-    MethodOfGettingAValueProfile(ValueProfile* profile)
+
+    static MethodOfGettingAValueProfile binaryArithProfile(CodeOrigin codeOrigin)
     {
-        if (profile) {
-            m_kind = Ready;
-            u.profile = profile;
-        } else
-            m_kind = None;
+        MethodOfGettingAValueProfile result;
+        result.m_kind = Kind::BinaryArithProfile;
+        result.m_codeOrigin = codeOrigin;
+        return result;
     }
 
-    MethodOfGettingAValueProfile(UnaryArithProfile* profile)
+    static MethodOfGettingAValueProfile argumentValueProfile(CodeOrigin codeOrigin, Operand operand)
     {
-        if (profile) {
-            m_kind = UnaryArithProfileReady;
-            u.unaryArithProfile = profile;
-        } else
-            m_kind = None;
+        MethodOfGettingAValueProfile result;
+        result.m_kind = Kind::ArgumentValueProfile;
+        result.m_codeOrigin = codeOrigin;
+        result.m_rawOperand = operand.asBits();
+        return result;
     }
 
-    MethodOfGettingAValueProfile(BinaryArithProfile* profile)
+    static MethodOfGettingAValueProfile bytecodeValueProfile(CodeOrigin codeOrigin)
     {
-        if (profile) {
-            m_kind = BinaryArithProfileReady;
-            u.binaryArithProfile = profile;
-        } else
-            m_kind = None;
+        MethodOfGettingAValueProfile result;
+        result.m_kind = Kind::BytecodeValueProfile;
+        result.m_codeOrigin = codeOrigin;
+        return result;
     }
-
-    static MethodOfGettingAValueProfile fromLazyOperand(
-        CodeBlock*, const LazyOperandValueProfileKey&);
-
-    explicit operator bool() const { return m_kind != None; }
+
+    static MethodOfGettingAValueProfile lazyOperandValueProfile(CodeOrigin codeOrigin, Operand operand)
+    {
+        MethodOfGettingAValueProfile result;
+        result.m_kind = Kind::LazyOperandValueProfile;
+        result.m_codeOrigin = codeOrigin;
+        result.m_rawOperand = operand.asBits();
+        return result;
+    }
+
+    explicit operator bool() const { return m_kind != Kind::None; }
 
     // The temporary register is only needed on 64-bits builds (for testing BigInt32).
-    void emitReportValue(CCallHelpers&, JSValueRegs, GPRReg tempGPR, TagRegistersMode = HaveTagRegisters) const;
-    void reportValue(JSValue);
+    void emitReportValue(CCallHelpers&, CodeBlock* optimizedCodeBlock, JSValueRegs, GPRReg tempGPR, TagRegistersMode = HaveTagRegisters) const;
 
 private:
-    enum Kind {
+    enum class Kind : uint8_t {
         None,
-        Ready,
-        UnaryArithProfileReady,
-        BinaryArithProfileReady,
-        LazyOperand
+        UnaryArithProfile,
+        BinaryArithProfile,
+        BytecodeValueProfile,
+        ArgumentValueProfile,
+        LazyOperandValueProfile,
     };
-
-    Kind m_kind;
-    union Data {
-        Data()
-            : profile(nullptr)
-        { }
-
-        ValueProfile* profile;
-        UnaryArithProfile* unaryArithProfile;
-        BinaryArithProfile* binaryArithProfile;
-        struct {
-            CodeBlock* codeBlock;
-            BytecodeIndex bytecodeOffset;
-            Operand operand;
-        } lazyOperand;
-    } u;
+    static constexpr unsigned bitsOfKind = 3;
+    static_assert(static_cast<unsigned>(Kind::LazyOperandValueProfile) <= ((1U << bitsOfKind) - 1));
+
+    CodeOrigin m_codeOrigin;
+    uint64_t m_rawOperand : Operand::maxBits { 0 };
+    Kind m_kind : bitsOfKind { Kind::None };
 };
 
 } // namespace JSC