Structures should be swept after all other objects

https://bugs.webkit.org/show_bug.cgi?id=92679

Reviewed by Filip Pizlo.

In order to get rid of ClassInfo from our objects, we need to be able to safely get the
ClassInfo during the destruction of objects. We'd like to get the ClassInfo out of the
Structure, but currently it is not safe to do so because the order of destruction of objects
is not guaranteed to sweep objects before their corresponding Structure. We can fix this by
sweeping Structures after everything else.

* heap/Heap.cpp:
(JSC::Heap::isSafeToSweepStructures): Add a function that checks if it is safe to sweep Structures.
If the Heap's IncrementalSweeper member is null, that means we're shutting down this VM and it is
safe to sweep structures since we'll always do Structures last anyways due to the ordering of
MarkedSpace::forEachBlock.
(JSC):
(JSC::Heap::didStartVMShutdown): Add this intermediate function to the Heap that ~JSGlobalData now
calls rather than calling the two HeapTimer objects individually. This allows the Heap to null out
these pointers after it has invalidated them to prevent accidental use-after-free in the sweep()
calls during lastChanceToFinalize().
* heap/Heap.h:
(Heap):
* heap/HeapTimer.h:
(HeapTimer):
* heap/IncrementalSweeper.cpp:
(JSC::IncrementalSweeper::structuresCanBeSwept): Determines if it is currently safe to sweep Structures.
This decision is based on whether we have gotten to the end of the vector of blocks that need sweeping
the first time.
(JSC):
(JSC::IncrementalSweeper::doSweep): We add a second pass over the vector to sweep Structures after we
make our first pass. We now null out the slots as we sweep them so that we can quickly find the
Structures during the second pass.
(JSC::IncrementalSweeper::startSweeping): Initialize our new Structure sweeping index.
(JSC::IncrementalSweeper::willFinishSweeping): Callback that is called by MarkedSpace::sweep to notify
the IncrementalSweeper that we are going to sweep all of the remaining blocks in the Heap so it can
assume that everything is taken care of in the correct order. Since MarkedSpace::forEachBlock
iterates over the Structure blocks after all other blocks, the ordering property for sweeping Structures holds.
(JSC::IncrementalSweeper::IncrementalSweeper): Initialize Structure sweeping index.
* heap/IncrementalSweeper.h: Add declarations for new stuff.
(IncrementalSweeper):
* heap/MarkedAllocator.cpp:
(JSC::MarkedAllocator::tryAllocateHelper): We now check if the current block only contains structures and
if so and it isn't safe to sweep Structures according to the Heap, we just return early instead of doing
the normal lazy sweep. If this proves to be too much of a waste in the future we can add an extra clause that
will sweep some number of other blocks in place of the current block to mitigate the cost of the floating
Structure garbage.
(JSC::MarkedAllocator::addBlock):
* heap/MarkedAllocator.h:
(JSC::MarkedAllocator::zapFreeList): When we zap the free list in the MarkedAllocator, the current block is no
longer valid to allocate from, so we set the current block to null.
* heap/MarkedBlock.cpp:
(JSC::MarkedBlock::sweepHelper): Added a couple assertions to make sure that we weren't trying to sweep Structures
at an unsafe time.
* heap/MarkedSpace.cpp:
(JSC::MarkedSpace::sweep): Notify the IncrementalSweeper that the MarkedSpace will finish all currently remaining sweeping.
(JSC):
* heap/MarkedSpace.h:
(JSC):
* runtime/JSGlobalData.cpp:
(JSC::JSGlobalData::~JSGlobalData): Call the new Heap::didStartVMShutdown.


Canonical link: https://commits.webkit.org/110517@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@124123 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Source/JavaScriptCore/ChangeLog

@@ -1,3 +1,67 @@
+2012-07-30  Mark Hahnenberg  <mhahnenberg@apple.com>
+
+        Structures should be swept after all other objects
+        https://bugs.webkit.org/show_bug.cgi?id=92679
+
+        Reviewed by Filip Pizlo.
+
+        In order to get rid of ClassInfo from our objects, we need to be able to safely get the 
+        ClassInfo during the destruction of objects. We'd like to get the ClassInfo out of the 
+        Structure, but currently it is not safe to do so because the order of destruction of objects 
+        is not guaranteed to sweep objects before their corresponding Structure. We can fix this by 
+        sweeping Structures after everything else.
+
+        * heap/Heap.cpp:
+        (JSC::Heap::isSafeToSweepStructures): Add a function that checks if it is safe to sweep Structures.
+        If the Heap's IncrementalSweeper member is null, that means we're shutting down this VM and it is 
+        safe to sweep structures since we'll always do Structures last anyways due to the ordering of 
+        MarkedSpace::forEachBlock.
+        (JSC):
+        (JSC::Heap::didStartVMShutdown): Add this intermediate function to the Heap that ~JSGlobalData now
+        calls rather than calling the two HeapTimer objects individually. This allows the Heap to null out 
+        these pointers after it has invalidated them to prevent accidental use-after-free in the sweep() 
+        calls during lastChanceToFinalize().
+        * heap/Heap.h:
+        (Heap):
+        * heap/HeapTimer.h:
+        (HeapTimer):
+        * heap/IncrementalSweeper.cpp:
+        (JSC::IncrementalSweeper::structuresCanBeSwept): Determines if it is currently safe to sweep Structures.
+        This decision is based on whether we have gotten to the end of the vector of blocks that need sweeping
+        the first time.
+        (JSC):
+        (JSC::IncrementalSweeper::doSweep): We add a second pass over the vector to sweep Structures after we 
+        make our first pass. We now null out the slots as we sweep them so that we can quickly find the 
+        Structures during the second pass.
+        (JSC::IncrementalSweeper::startSweeping): Initialize our new Structure sweeping index.
+        (JSC::IncrementalSweeper::willFinishSweeping): Callback that is called by MarkedSpace::sweep to notify 
+        the IncrementalSweeper that we are going to sweep all of the remaining blocks in the Heap so it can 
+        assume that everything is taken care of in the correct order. Since MarkedSpace::forEachBlock 
+        iterates over the Structure blocks after all other blocks, the ordering property for sweeping Structures holds.
+        (JSC::IncrementalSweeper::IncrementalSweeper): Initialize Structure sweeping index.
+        * heap/IncrementalSweeper.h: Add declarations for new stuff.
+        (IncrementalSweeper):
+        * heap/MarkedAllocator.cpp:
+        (JSC::MarkedAllocator::tryAllocateHelper): We now check if the current block only contains structures and 
+        if so and it isn't safe to sweep Structures according to the Heap, we just return early instead of doing 
+        the normal lazy sweep. If this proves to be too much of a waste in the future we can add an extra clause that 
+        will sweep some number of other blocks in place of the current block to mitigate the cost of the floating 
+        Structure garbage.
+        (JSC::MarkedAllocator::addBlock):
+        * heap/MarkedAllocator.h:
+        (JSC::MarkedAllocator::zapFreeList): When we zap the free list in the MarkedAllocator, the current block is no 
+        longer valid to allocate from, so we set the current block to null.
+        * heap/MarkedBlock.cpp:
+        (JSC::MarkedBlock::sweepHelper): Added a couple assertions to make sure that we weren't trying to sweep Structures
+        at an unsafe time.
+        * heap/MarkedSpace.cpp:
+        (JSC::MarkedSpace::sweep): Notify the IncrementalSweeper that the MarkedSpace will finish all currently remaining sweeping.
+        (JSC): 
+        * heap/MarkedSpace.h:
+        (JSC):
+        * runtime/JSGlobalData.cpp:
+        (JSC::JSGlobalData::~JSGlobalData): Call the new Heap::didStartVMShutdown.
+
 2012-07-29  Filip Pizlo  <fpizlo@apple.com>
 
         PropertyNameArray::m_shouldCache is only assigned and never used

Source/JavaScriptCore/heap/Heap.cpp

@@ -830,4 +830,18 @@ void Heap::addCompiledCode(ExecutableBase* executable)
     m_compiledCode.append(executable);
 }
 
+bool Heap::isSafeToSweepStructures()
+{
+    return !m_sweeper || m_sweeper->structuresCanBeSwept();
+}
+
+void Heap::didStartVMShutdown()
+{
+    m_activityCallback->didStartVMShutdown();
+    m_activityCallback = 0;
+    m_sweeper->didStartVMShutdown();
+    m_sweeper = 0;
+    lastChanceToFinalize();
+}
+
 } // namespace JSC

Source/JavaScriptCore/heap/Heap.h

@@ -168,6 +168,8 @@ namespace JSC {
         void didAbandon(size_t);
 
         bool isPagedOut(double deadline);
+        bool isSafeToSweepStructures();
+        void didStartVMShutdown();
 
     private:
         friend class CodeBlock;

Source/JavaScriptCore/heap/IncrementalSweeper.cpp

@@ -69,10 +69,46 @@ void IncrementalSweeper::cancelTimer()
     CFRunLoopTimerSetNextFireDate(m_timer.get(), CFAbsoluteTimeGetCurrent() + s_decade);
 }
 
+bool IncrementalSweeper::structuresCanBeSwept()
+{
+    ASSERT(m_currentBlockToSweepIndex <= m_blocksToSweep.size());
+    return !m_blocksToSweep.size() || m_currentBlockToSweepIndex >= m_blocksToSweep.size();
+}
+
 void IncrementalSweeper::doSweep(double sweepBeginTime)
 {
     while (m_currentBlockToSweepIndex < m_blocksToSweep.size()) {
-        MarkedBlock* block = m_blocksToSweep[m_currentBlockToSweepIndex++];
+        MarkedBlock* block = m_blocksToSweep[m_currentBlockToSweepIndex];
+        if (block->onlyContainsStructures()) {
+            m_currentBlockToSweepIndex++;
+            continue;
+        }
+
+        m_blocksToSweep[m_currentBlockToSweepIndex++] = 0;
+
+        if (!block->needsSweeping())
+            continue;
+
+        block->sweep();
+        m_globalData->heap.objectSpace().freeOrShrinkBlock(block);
+
+        CFTimeInterval elapsedTime = WTF::monotonicallyIncreasingTime() - sweepBeginTime;
+        if (elapsedTime < sweepTimeSlice)
+            continue;
+
+        scheduleTimer();
+        return;
+    }
+
+    while (m_currentStructureBlockToSweepIndex < m_blocksToSweep.size()) {
+        MarkedBlock* block = m_blocksToSweep[m_currentStructureBlockToSweepIndex];
+        if (!block) {
+            m_currentStructureBlockToSweepIndex++;
+            continue;
+        }
+
+        m_blocksToSweep[m_currentStructureBlockToSweepIndex++] = 0;
+
         if (!block->needsSweeping())
             continue;
 
@@ -95,13 +131,23 @@ void IncrementalSweeper::startSweeping(const HashSet<MarkedBlock*>& blockSnapsho
 {
     WTF::copyToVector(blockSnapshot, m_blocksToSweep);
     m_currentBlockToSweepIndex = 0;
+    m_currentStructureBlockToSweepIndex = 0;
     scheduleTimer();
 }
 
+void IncrementalSweeper::willFinishSweeping()
+{
+    m_currentBlockToSweepIndex = m_currentStructureBlockToSweepIndex = 0;
+    m_blocksToSweep.clear();
+    if (m_globalData)
+        cancelTimer();
+}
+
 #else
 
 IncrementalSweeper::IncrementalSweeper(JSGlobalData* globalData)
     : HeapTimer(globalData)
+    , m_structuresCanBeSwept(false)
 {
 }
 
@@ -114,10 +160,21 @@ IncrementalSweeper* IncrementalSweeper::create(Heap* heap)
     return new IncrementalSweeper(heap->globalData());
 }
 
+bool IncrementalSweeper::structuresCanBeSwept()
+{
+    return m_structuresCanBeSwept;
+}
+
 void IncrementalSweeper::startSweeping(const HashSet<MarkedBlock*>&)
 {
+    m_structuresCanBeSwept = false;
 }
-
+
+void IncrementalSweeper::willFinishSweeping()
+{
+    m_structuresCanBeSwept = true;
+}
+
 #endif
 
 } // namespace JSC

Source/JavaScriptCore/heap/IncrementalSweeper.h

@@ -42,8 +42,11 @@ class IncrementalSweeper : public HeapTimer {
     static IncrementalSweeper* create(Heap*);
     void startSweeping(const HashSet<MarkedBlock*>& blockSnapshot);
     virtual void doWork();
+    bool structuresCanBeSwept();
+    void willFinishSweeping();
 
 private:
+
 #if USE(CF)
     IncrementalSweeper(Heap*, CFRunLoopRef);
 
@@ -52,11 +55,14 @@ class IncrementalSweeper : public HeapTimer {
     void cancelTimer();
 
     unsigned m_currentBlockToSweepIndex;
+    unsigned m_currentStructureBlockToSweepIndex;
     Vector<MarkedBlock*> m_blocksToSweep;
 #else
 
     IncrementalSweeper(JSGlobalData*);
-
+
+    bool m_structuresCanBeSwept;
+
 #endif
 };
 

Source/JavaScriptCore/heap/MarkedAllocator.cpp

@@ -3,6 +3,7 @@
 
 #include "GCActivityCallback.h"
 #include "Heap.h"
+#include "IncrementalSweeper.h"
 #include "JSGlobalData.h"
 #include <wtf/CurrentTime.h>
 
@@ -29,6 +30,14 @@ bool MarkedAllocator::isPagedOut(double deadline)
 inline void* MarkedAllocator::tryAllocateHelper()
 {
     if (!m_freeList.head) {
+        if (m_onlyContainsStructures && !m_heap->isSafeToSweepStructures()) {
+            if (m_currentBlock) {
+                m_currentBlock->didConsumeFreeList();
+                m_currentBlock = 0;
+            }
+            return 0;
+        }
+
         for (MarkedBlock*& block = m_blocksToSweep; block; block = static_cast<MarkedBlock*>(block->next())) {
             m_freeList = block->sweep(MarkedBlock::SweepToFreeList);
             if (m_freeList.head) {
@@ -104,7 +113,6 @@ MarkedBlock* MarkedAllocator::allocateBlock()
 void MarkedAllocator::addBlock(MarkedBlock* block)
 {
     ASSERT(!m_currentBlock);
-    ASSERT(!m_blocksToSweep);
     ASSERT(!m_freeList.head);
 
     m_blockList.append(block);

Source/JavaScriptCore/heap/MarkedAllocator.h

@@ -101,6 +101,7 @@ inline void MarkedAllocator::zapFreeList()
     }
 
     m_currentBlock->zapFreeList(m_freeList);
+    m_currentBlock = 0;
     m_freeList = MarkedBlock::FreeList();
 }
 

Source/JavaScriptCore/heap/MarkedBlock.cpp

@@ -26,6 +26,7 @@
 #include "config.h"
 #include "MarkedBlock.h"
 
+#include "IncrementalSweeper.h"
 #include "JSCell.h"
 #include "JSObject.h"
 #include "ScopeChain.h"
@@ -140,10 +141,12 @@ MarkedBlock::FreeList MarkedBlock::sweepHelper(SweepMode sweepMode)
         ASSERT_NOT_REACHED();
         return FreeList();
     case Marked:
+        ASSERT(!m_onlyContainsStructures || heap()->isSafeToSweepStructures());
         return sweepMode == SweepToFreeList
             ? specializedSweep<Marked, SweepToFreeList, destructorCallNeeded>()
             : specializedSweep<Marked, SweepOnly, destructorCallNeeded>();
     case Zapped:
+        ASSERT(!m_onlyContainsStructures || heap()->isSafeToSweepStructures());
         return sweepMode == SweepToFreeList
             ? specializedSweep<Zapped, SweepToFreeList, destructorCallNeeded>()
             : specializedSweep<Zapped, SweepOnly, destructorCallNeeded>();

Source/JavaScriptCore/heap/MarkedSpace.cpp

@@ -21,6 +21,7 @@
 #include "config.h"
 #include "MarkedSpace.h"
 
+#include "IncrementalSweeper.h"
 #include "JSGlobalObject.h"
 #include "JSLock.h"
 #include "JSObject.h"
@@ -108,6 +109,12 @@ void MarkedSpace::lastChanceToFinalize()
     forEachBlock<LastChanceToFinalize>();
 }
 
+void MarkedSpace::sweep()
+{
+    m_heap->sweeper()->willFinishSweeping();
+    forEachBlock<Sweep>();
+}
+
 void MarkedSpace::resetAllocators()
 {
     for (size_t cellSize = preciseStep; cellSize <= preciseCutoff; cellSize += preciseStep) {

Source/JavaScriptCore/heap/MarkedSpace.h

@@ -235,11 +235,6 @@ inline void MarkedSpace::clearMarks()
     forEachBlock<ClearMarks>();
 }
 
-inline void MarkedSpace::sweep()
-{
-    forEachBlock<Sweep>();
-}
-
 inline size_t MarkedSpace::objectCount()
 {
     return forEachBlock<MarkCount>();

Source/JavaScriptCore/runtime/JSGlobalData.cpp

@@ -223,9 +223,7 @@ JSGlobalData::JSGlobalData(GlobalDataType globalDataType, ThreadStackType thread
 JSGlobalData::~JSGlobalData()
 {
     ASSERT(!m_apiLock.currentThreadIsHoldingLock());
-    heap.activityCallback()->didStartVMShutdown();
-    heap.sweeper()->didStartVMShutdown();
-    heap.lastChanceToFinalize();
+    heap.didStartVMShutdown();
 
     delete interpreter;
 #ifndef NDEBUG