Skip to content

[Docs] Fixed the incorrect elapsed calculation and inappropriate output format #1093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Aaron2516 wants to merge 1 commit into from
Closed

[Docs] Fixed the incorrect elapsed calculation and inappropriate output format #1093

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/examples/scraping_strategies_performance.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
import time, re
from crawl4ai.content_scraping_strategy import WebScrapingStrategy, LXMLWebScrapingStrategy
import time
import functools
import time
from collections import defaultdict

from crawl4ai.content_scraping_strategy import WebScrapingStrategy, LXMLWebScrapingStrategy

class TimingStats:
def __init__(self):
self.stats = defaultdict(lambda: defaultdict(lambda: {"calls": 0, "total_time": 0}))
Expand Down Expand Up @@ -95,7 +95,7 @@ def test_scraping():

# Time the scraping
print("\nStarting scrape...")
start_time = time.time()
start_time = time.perf_counter()
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot May 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Use time.perf_counter() in the decorator for consistency
The timing_decorator still uses time.time(), which is less precise and can be affected by system clock adjustments. To maintain consistency with the high-resolution timer used in test_scraping, switch to time.perf_counter():

 def wrapper(*args, **kwargs):
-    start = time.time()
+    start = time.perf_counter()
     result = func(*args, **kwargs)
-    elapsed = time.time() - start
+    elapsed = time.perf_counter() - start
     timing_stats.add(strategy_name, func.__name__, elapsed)
     return result


kwargs = {
"url": "http://example.com",
Expand All @@ -117,7 +117,7 @@ def test_scraping():
timing_stats.report()

# Print stats of LXML output
print("\Turbo Output:")
print("\nTurbo Output:")
print(f"\nExtracted links: {len(result_selected.links.internal) + len(result_selected.links.external)}")
print(f"Extracted images: {len(result_selected.media.images)}")
print(f"Clean HTML size: {len(result_selected.cleaned_html)/1024:.2f} KB")
Expand Down