Web Inspector: [Meta] Implement Timelines Film Strip

https://bugs.webkit.org/show_bug.cgi?id=239350

Patch by Anjali Kumar <anjalik_22@apple.com> on 2022-05-13
Reviewed by Devin Rousso and Patrick Angle.

Source/JavaScriptCore:

* inspector/protocol/Timeline.json:

Source/WebCore:

Test: inspector/timeline/timeline-event-screenshots.html

* inspector/TimelineRecordFactory.cpp:
(WebCore::TimelineRecordFactory::createScreenshotData):
* inspector/TimelineRecordFactory.h:
* inspector/agents/InspectorTimelineAgent.cpp:
(WebCore::InspectorTimelineAgent::didComposite):
(WebCore::InspectorTimelineAgent::willPaint):
(WebCore::InspectorTimelineAgent::didPaint):
(WebCore::InspectorTimelineAgent::toggleInstruments):
(WebCore::InspectorTimelineAgent::captureScreenshot):
(WebCore::toProtocol):
(WebCore::InspectorTimelineAgent::createRecordEntry):
(WebCore::InspectorTimelineAgent::pushCurrentRecord):
* inspector/agents/InspectorTimelineAgent.h:

Source/WebInspectorUI:

Add the ability to see screenshots taken of the viewport within the Timelines tab. The purpose of the
screenshots is to provide more context to the other data presented within the Timelines tab, so that
developers can improve the efficiency of their page loading times. They can see what is painting on their
pages in addition to when the paints are occuring.

The screenshots presented are taken immediately after each composite. They are designed to be layered
on top of one another as opposed to being presented in a non-overlapping fashion in order to provide developers
with the exact screenshot that occured during a particular point in time on the timeline. This allows developers
to zoom in and pinpoint the exact moment the page looked like that particularly rendered screenshot.

When a screenshot is clicked on, the details section opens up to an enlarged view of that particular image.
Developers can utilize this to view each screenshot in greater detail.

This feature is named "Screenshots" (one word), as in the UI we currently have existing strings for "Capture
Screenshot" and "Could not capture screenshot".

* Localizations/en.lproj/localizedStrings.js:
* UserInterface/Base/Setting.js:
* UserInterface/Controllers/TimelineManager.js:
(WI.TimelineManager.defaultTimelineTypes):
(WI.TimelineManager.prototype._processRecord):
(WI.TimelineManager.prototype._updateAutoCaptureInstruments):
* UserInterface/Images/IdentifierIcons.svg:
* UserInterface/Main.html:
* UserInterface/Models/Instrument.js:
(WI.Instrument.createForTimelineType):
* UserInterface/Models/ScreenshotsInstrument.js: Added.
(WI.ScreenshotsInstrument):
(WI.ScreenshotsInstrument.supported):
(WI.ScreenshotsInstrument.prototype.get timelineRecordType):
* UserInterface/Models/ScreenshotsTimelineRecord.js: Added.
(WI.ScreenshotsTimelineRecord):
(WI.ScreenshotsTimelineRecord.async fromJSON):
(WI.ScreenshotsTimelineRecord.prototype.toJSON):
(WI.ScreenshotsTimelineRecord.prototype.get imageData):
(WI.ScreenshotsTimelineRecord.prototype.get width):
(WI.ScreenshotsTimelineRecord.prototype.get height):
* UserInterface/Models/TimelineRecord.js:
(WI.TimelineRecord.async fromJSON):
* UserInterface/Models/TimelineRecording.js:
(WI.TimelineRecording.prototype.addRecord):
* UserInterface/Test.html:
* UserInterface/Views/ContentView.js:
(WI.ContentView.createFromRepresentedObject):
* UserInterface/Views/ScreenshotsTimelineOverviewGraph.css: Added.
(body .sidebar > .panel.navigation.timeline > .timelines-content li.item.screenshots,):
(.timeline-overview-graph.screenshots > img):
(.timeline-overview-graph.screenshots > img.selected):
* UserInterface/Views/ScreenshotsTimelineOverviewGraph.js: Added.
(WI.ScreenshotsTimelineOverviewGraph):
(WI.ScreenshotsTimelineOverviewGraph.prototype.get height):
(WI.ScreenshotsTimelineOverviewGraph.prototype.layout):
(WI.ScreenshotsTimelineOverviewGraph.prototype.updateSelectedRecord):
(WI.ScreenshotsTimelineOverviewGraph.prototype._visibleRecords):
* UserInterface/Views/ScreenshotsTimelineView.css: Added.
(.timeline-view.screenshots):
(.timeline-view.screenshots > img):
(.timeline-view.screenshots > img.selected):
* UserInterface/Views/ScreenshotsTimelineView.js: Added.
(WI.ScreenshotsTimelineView):
(WI.ScreenshotsTimelineView.prototype.reset):
(WI.ScreenshotsTimelineView.prototype.clear):
(WI.ScreenshotsTimelineView.prototype.get showsFilterBar):
(WI.ScreenshotsTimelineView.prototype.layout):
(WI.ScreenshotsTimelineView.prototype.selectRecord):
(WI.ScreenshotsTimelineView.prototype._selectTimelineRecord):
(WI.ScreenshotsTimelineView.prototype._visibleRecords):
* UserInterface/Views/SettingsTabContentView.js:
(WI.SettingsTabContentView.prototype._createExperimentalSettingsView):
* UserInterface/Views/TimelineIcons.css:
(.screenshots-icon .icon):
(@media (prefers-color-scheme: dark) .screenshots-icon .icon):
* UserInterface/Views/TimelineOverviewGraph.js:
(WI.TimelineOverviewGraph.createForTimeline):
* UserInterface/Views/TimelineTabContentView.js:
(WI.TimelineTabContentView.displayNameForTimelineType):
(WI.TimelineTabContentView.iconClassNameForTimelineType):
(WI.TimelineTabContentView.genericClassNameForTimelineType):
(WI.TimelineTabContentView.iconClassNameForRecord):
(WI.TimelineTabContentView.displayNameForRecord):

LayoutTests:

* inspector/timeline/resources/timeline-event-utilities.js:
(TestPage.registerInitializer.InspectorTest.TimelineEvent.captureTimelineWithScript):
(TestPage.registerInitializer):
* inspector/timeline/timeline-event-screenshots-expected.txt: Added.
* inspector/timeline/timeline-event-screenshots.html: Added.
* inspector/timeline/timeline-recording-expected.txt:

Canonical link: https://commits.webkit.org/250535@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@294166 268f45cc-cd09-0410-ab3c-d52691b4dbfc

LayoutTests/ChangeLog

@@ -1,3 +1,17 @@
+2022-05-13  Anjali Kumar  <anjalik_22@apple.com>
+
+        Web Inspector: [Meta] Implement Timelines Film Strip
+        https://bugs.webkit.org/show_bug.cgi?id=239350
+
+        Reviewed by Devin Rousso and Patrick Angle.
+
+        * inspector/timeline/resources/timeline-event-utilities.js:
+        (TestPage.registerInitializer.InspectorTest.TimelineEvent.captureTimelineWithScript):
+        (TestPage.registerInitializer):
+        * inspector/timeline/timeline-event-screenshots-expected.txt: Added.
+        * inspector/timeline/timeline-event-screenshots.html: Added.
+        * inspector/timeline/timeline-recording-expected.txt:
+
 2022-05-13  Tim Nguyen  <ntim@apple.com>
 
         Clean up some html/semantics/forms/ test expectations

LayoutTests/inspector/timeline/resources/timeline-event-utilities.js

@@ -5,7 +5,7 @@ function savePageData(data) {
 TestPage.registerInitializer(() => {
     InspectorTest.TimelineEvent = {};
 
-    InspectorTest.TimelineEvent.captureTimelineWithScript = function({expression, eventType}) {
+    InspectorTest.TimelineEvent.captureTimelineWithScript = function({expression, eventType, timelineType}) {
         let savePageDataPromise = InspectorTest.awaitEvent("SavePageData").then((event) => {
             return event.data;
         });
@@ -15,14 +15,14 @@ TestPage.registerInitializer(() => {
         let listener = WI.timelineManager.addEventListener(WI.TimelineManager.Event.CapturingStateChanged, (event) => {
             if (WI.timelineManager.capturingState === WI.TimelineManager.CapturingState.Active) {
                 let recording = WI.timelineManager.activeRecording;
-                let scriptTimeline = recording.timelines.get(WI.TimelineRecord.Type.Script);
+                let timeline = recording.timelines.get(timelineType ?? WI.TimelineRecord.Type.Script);
 
-                let recordAddedListener = scriptTimeline.addEventListener(WI.Timeline.Event.RecordAdded, (recordAddedEvent) => {
+                let recordAddedListener = timeline.addEventListener(WI.Timeline.Event.RecordAdded, (recordAddedEvent) => {
                     let {record} = recordAddedEvent.data;
-                    if (record.eventType !== eventType)
+                    if (eventType && record.eventType !== eventType)
                         return;
 
-                    scriptTimeline.removeEventListener(WI.Timeline.Event.RecordAdded, recordAddedListener);
+                    timeline.removeEventListener(WI.Timeline.Event.RecordAdded, recordAddedListener);
 
                     InspectorTest.log("Stopping Capture...");
                     WI.timelineManager.stopCapturing();

LayoutTests/inspector/timeline/timeline-event-screenshots-expected.txt

@@ -0,0 +1,13 @@
+Tests 'Screenshot' Timeline event records.
+
+
+== Running test suite: TimelineEvent.FireScreenshots
+-- Running test case: TimelineEvent.FireScreenshots.requestScreenshots
+Starting Capture...
+Evaluating...
+Stopping Capture...
+PASS: Should have at least 1 Screenshot record.
+PASS: Screenshot record should contain image data.
+PASS: Screenshot record width should be non-zero.
+PASS: Screenshot record height should be non-zero.
+

LayoutTests/inspector/timeline/timeline-event-screenshots.html

@@ -0,0 +1,55 @@
+<!DOCTYPE html>
+<html>
+<head>
+<script src="../../http/tests/inspector/resources/inspector-test.js"></script>
+<script src="./resources/timeline-event-utilities.js"></script>
+<script>
+
+function testRequestScreenshots() {
+    document.getElementById("test").style.width = "200px";
+
+    savePageData({invalidatedLayout: true});
+}
+
+function test()
+{
+    let suite = InspectorTest.createAsyncSuite("TimelineEvent.FireScreenshots");
+
+    suite.addTestCase({
+        name: "TimelineEvent.FireScreenshots.requestScreenshots",
+        async test() {
+            WI.timelineManager.enabledTimelineTypes = [WI.TimelineRecord.Type.Screenshots];
+            let pageRecordingData = await InspectorTest.TimelineEvent.captureTimelineWithScript({
+                expression: `testRequestScreenshots()`,
+                timelineType: WI.TimelineRecord.Type.Screenshots,
+            });
+            InspectorTest.assert(pageRecordingData.invalidatedLayout);
+
+            let recording = WI.timelineManager.activeRecording;
+            let screenshotTimeline = recording.timelines.get(WI.TimelineRecord.Type.Screenshots);
+            let records = screenshotTimeline.records;
+            InspectorTest.expectGreaterThan(records.length, 0, "Should have at least 1 Screenshot record.");
+
+            InspectorTest.expectGreaterThan(records[0].imageData.length, 0, "Screenshot record should contain image data.");
+            InspectorTest.expectGreaterThan(records[0].width, 0, "Screenshot record width should be non-zero.");
+            InspectorTest.expectGreaterThan(records[0].height, 0, "Screenshot record height should be non-zero.");
+        }
+    });
+
+    suite.runTestCasesAndFinish();
+}
+
+</script>
+<style>
+#test {
+    width: 100px;
+    height: 100px;
+    background-color: darkseagreen;
+}
+</style>
+</head>
+<body onload="runTest()">
+    <p>Tests 'Screenshot' Timeline event records.</p>
+    <div id="test"></div>
+</body>
+</html>

LayoutTests/inspector/timeline/timeline-recording-expected.txt

@@ -35,6 +35,7 @@ Export Data:
   "endTime": "<filtered>",
   "discontinuities": [],
   "instrumentTypes": [
+    "timeline-record-type-screenshots",
     "timeline-record-type-network",
     "timeline-record-type-layout",
     "timeline-record-type-script",

LayoutTests/platform/gtk/TestExpectations

@@ -1199,6 +1199,8 @@ webkit.org/b/147518 inspector/debugger/nested-inspectors.html [ Timeout ]
 
 webkit.org/b/147229 inspector/css/modify-rule-selector.html [ Skip ] # Timeout
 
+webkit.org/b/240349 inspector/timeline/timeline-event-screenshots.html [ Skip ]
+
 webkit.org/b/142292 fast/images/animated-gif-zooming.html [ Timeout ]
 
 webkit.org/b/144690 editing/spelling/context-menu-suggestions-multiword-selection.html [ Timeout ]

Source/JavaScriptCore/ChangeLog

@@ -1,3 +1,12 @@
+2022-05-13  Anjali Kumar  <anjalik_22@apple.com>
+
+        Web Inspector: [Meta] Implement Timelines Film Strip
+        https://bugs.webkit.org/show_bug.cgi?id=239350
+
+        Reviewed by Devin Rousso and Patrick Angle.
+
+        * inspector/protocol/Timeline.json:
+
 2022-05-13  Lauro Moura  <lmoura@igalia.com>
 
         Unreviewed, non-unified build fixes

Source/JavaScriptCore/inspector/protocol/Timeline.json

@@ -30,7 +30,8 @@
                 "RequestAnimationFrame",
                 "CancelAnimationFrame",
                 "FireAnimationFrame",
-                "ObserverCallback"
+                "ObserverCallback",
+                "Screenshot"
             ]
         },
         {
@@ -43,7 +44,8 @@
                 "CPU",
                 "Memory",
                 "Heap",
-                "Animation"
+                "Animation",
+                "Screenshot"
             ]
         },
         {

Source/WebCore/ChangeLog

@@ -1,3 +1,26 @@
+2022-05-13  Anjali Kumar  <anjalik_22@apple.com>
+
+        Web Inspector: [Meta] Implement Timelines Film Strip
+        https://bugs.webkit.org/show_bug.cgi?id=239350
+
+        Reviewed by Devin Rousso and Patrick Angle.
+
+        Test: inspector/timeline/timeline-event-screenshots.html
+
+        * inspector/TimelineRecordFactory.cpp:
+        (WebCore::TimelineRecordFactory::createScreenshotData):
+        * inspector/TimelineRecordFactory.h:
+        * inspector/agents/InspectorTimelineAgent.cpp:
+        (WebCore::InspectorTimelineAgent::didComposite):
+        (WebCore::InspectorTimelineAgent::willPaint):
+        (WebCore::InspectorTimelineAgent::didPaint):
+        (WebCore::InspectorTimelineAgent::toggleInstruments):
+        (WebCore::InspectorTimelineAgent::captureScreenshot):
+        (WebCore::toProtocol):
+        (WebCore::InspectorTimelineAgent::createRecordEntry):
+        (WebCore::InspectorTimelineAgent::pushCurrentRecord):
+        * inspector/agents/InspectorTimelineAgent.h:
+
 2022-05-13  Michael Catanzaro  <mcatanzaro@redhat.com>
 
         -Wstringop-overflow warning in DocumentWriter.cpp

Source/WebCore/inspector/TimelineRecordFactory.cpp

@@ -161,6 +161,15 @@ Ref<JSON::Object> TimelineRecordFactory::createPaintData(const FloatQuad& quad)
     return data;
 }
 
+Ref<JSON::Object> TimelineRecordFactory::createScreenshotData(const String& imageData, int width, int height)
+{
+    Ref<JSON::Object> data = JSON::Object::create();
+    data->setString("imageData"_s, imageData);
+    data->setInteger("width"_s, width);
+    data->setInteger("height"_s, height);
+    return data;
+}
+
 void TimelineRecordFactory::appendLayoutRoot(JSON::Object& data, const FloatQuad& quad)
 {
     data.setArray("root"_s, createQuad(quad));

Source/WebCore/inspector/TimelineRecordFactory.h

@@ -58,6 +58,7 @@ class TimelineRecordFactory {
     static Ref<JSON::Object> createAnimationFrameData(int callbackId);
     static Ref<JSON::Object> createObserverCallbackData(const String& callbackType);
     static Ref<JSON::Object> createPaintData(const FloatQuad&);
+    static Ref<JSON::Object> createScreenshotData(const String& imageData, int width, int height);
 
     static void appendLayoutRoot(JSON::Object& data, const FloatQuad&);
 

Source/WebCore/inspector/agents/InspectorTimelineAgent.cpp

@@ -55,6 +55,7 @@
 #include <JavaScriptCore/ConsoleMessage.h>
 #include <JavaScriptCore/InspectorScriptProfilerAgent.h>
 #include <JavaScriptCore/ScriptArguments.h>
+#include <wtf/SetForScope.h>
 #include <wtf/Stopwatch.h>
 
 #if PLATFORM(IOS_FAMILY)
@@ -423,15 +424,24 @@ void InspectorTimelineAgent::didComposite()
     if (m_startedComposite)
         didCompleteCurrentRecord(TimelineRecordType::Composite);
     m_startedComposite = false;
+
+    if (m_instruments.contains(Protocol::Timeline::Instrument::Screenshot))
+        captureScreenshot();
 }
 
 void InspectorTimelineAgent::willPaint(Frame& frame)
 {
+    if (m_isCapturingScreenshot)
+        return;
+
     pushCurrentRecord(JSON::Object::create(), TimelineRecordType::Paint, true, &frame);
 }
 
 void InspectorTimelineAgent::didPaint(RenderObject& renderer, const LayoutRect& clipRect)
 {
+    if (m_isCapturingScreenshot)
+        return;
+
     if (m_recordStack.isEmpty())
         return;
 
@@ -578,6 +588,9 @@ void InspectorTimelineAgent::toggleInstruments(InstrumentState state)
         case Protocol::Timeline::Instrument::Animation:
             toggleAnimationInstrument(state);
             break;
+        case Protocol::Timeline::Instrument::Screenshot: {
+            break;
+        }
         }
     }
 }
@@ -649,6 +662,20 @@ void InspectorTimelineAgent::toggleAnimationInstrument(InstrumentState state)
     }
 }
 
+void InspectorTimelineAgent::captureScreenshot()
+{
+    SetForScope isTakingScreenshot(m_isCapturingScreenshot, true);
+
+    auto snapshotStartTime = timestamp();
+    auto& frame = m_inspectedPage.mainFrame();
+    auto viewportRect = m_inspectedPage.mainFrame().view()->unobscuredContentRect();
+    if (auto snapshot = snapshotFrameRect(frame, viewportRect, { { }, PixelFormat::BGRA8, DestinationColorSpace::SRGB() })) {
+        auto snapshotRecord = TimelineRecordFactory::createScreenshotData(snapshot->toDataURL("image/png"_s), viewportRect.width(), viewportRect.height());
+        pushCurrentRecord(WTFMove(snapshotRecord), TimelineRecordType::Screenshot, false, &frame, snapshotStartTime);
+        didCompleteCurrentRecord(TimelineRecordType::Screenshot);
+    }
+}
+
 void InspectorTimelineAgent::didRequestAnimationFrame(int callbackId, Frame* frame)
 {
     appendRecord(TimelineRecordFactory::createAnimationFrameData(callbackId), TimelineRecordType::RequestAnimationFrame, true, frame);
@@ -737,6 +764,9 @@ static Protocol::Timeline::EventType toProtocol(TimelineRecordType type)
 
     case TimelineRecordType::ObserverCallback:
         return Protocol::Timeline::EventType::ObserverCallback;
+
+    case TimelineRecordType::Screenshot:
+        return Protocol::Timeline::EventType::Screenshot;
     }
 
     return Protocol::Timeline::EventType::TimeStamp;
@@ -813,16 +843,16 @@ void InspectorTimelineAgent::sendEvent(Ref<JSON::Object>&& event)
     m_frontendDispatcher->eventRecorded(WTFMove(recordChecked));
 }
 
-InspectorTimelineAgent::TimelineRecordEntry InspectorTimelineAgent::createRecordEntry(Ref<JSON::Object>&& data, TimelineRecordType type, bool captureCallStack, Frame* frame)
+InspectorTimelineAgent::TimelineRecordEntry InspectorTimelineAgent::createRecordEntry(Ref<JSON::Object>&& data, TimelineRecordType type, bool captureCallStack, Frame* frame, std::optional<double> startTime)
 {
-    Ref<JSON::Object> record = TimelineRecordFactory::createGenericRecord(timestamp(), captureCallStack ? m_maxCallStackDepth : 0);
+    Ref<JSON::Object> record = TimelineRecordFactory::createGenericRecord(startTime.value_or(timestamp()), captureCallStack ? m_maxCallStackDepth : 0);
     setFrameIdentifier(&record.get(), frame);
     return TimelineRecordEntry(WTFMove(record), WTFMove(data), JSON::Array::create(), type);
 }
 
-void InspectorTimelineAgent::pushCurrentRecord(Ref<JSON::Object>&& data, TimelineRecordType type, bool captureCallStack, Frame* frame)
+void InspectorTimelineAgent::pushCurrentRecord(Ref<JSON::Object>&& data, TimelineRecordType type, bool captureCallStack, Frame* frame, std::optional<double> startTime)
 {
-    pushCurrentRecord(createRecordEntry(WTFMove(data), type, captureCallStack, frame));
+    pushCurrentRecord(createRecordEntry(WTFMove(data), type, captureCallStack, frame, startTime));
 }
 
 } // namespace WebCore

Source/WebCore/inspector/agents/InspectorTimelineAgent.h

@@ -79,6 +79,8 @@ enum class TimelineRecordType {
     FireAnimationFrame,
 
     ObserverCallback,
+
+    Screenshot,
 };
 
 class InspectorTimelineAgent final : public InspectorAgentBase , public Inspector::TimelineBackendDispatcherHandler , public JSC::Debugger::Observer {
@@ -155,6 +157,8 @@ class InspectorTimelineAgent final : public InspectorAgentBase , public Inspecto
     void disableBreakpoints();
     void enableBreakpoints();
 
+    void captureScreenshot();
+
     friend class TimelineRecordStack;
 
     struct TimelineRecordEntry {
@@ -178,10 +182,10 @@ class InspectorTimelineAgent final : public InspectorAgentBase , public Inspecto
 
     void sendEvent(Ref<JSON::Object>&&);
     void appendRecord(Ref<JSON::Object>&& data, TimelineRecordType, bool captureCallStack, Frame*);
-    void pushCurrentRecord(Ref<JSON::Object>&&, TimelineRecordType, bool captureCallStack, Frame*);
+    void pushCurrentRecord(Ref<JSON::Object>&&, TimelineRecordType, bool captureCallStack, Frame*, std::optional<double> startTime = std::nullopt);
     void pushCurrentRecord(const TimelineRecordEntry& record) { m_recordStack.append(record); }
 
-    TimelineRecordEntry createRecordEntry(Ref<JSON::Object>&& data, TimelineRecordType, bool captureCallStack, Frame*);
+    TimelineRecordEntry createRecordEntry(Ref<JSON::Object>&& data, TimelineRecordType, bool captureCallStack, Frame*, std::optional<double> startTime = std::nullopt);
 
     void setFrameIdentifier(JSON::Object* record, Frame*);
 
@@ -216,6 +220,7 @@ class InspectorTimelineAgent final : public InspectorAgentBase , public Inspecto
     std::unique_ptr<RunLoop::Observer> m_runLoopObserver;
 #endif
     bool m_startedComposite { false };
+    bool m_isCapturingScreenshot { false };
 };
 
 } // namespace WebCore