[Feature Request]: Configurable batch_size and dispatcher for deep crawl strategies

[yashikam19]

What needs to be done?

Add a batch_size constructor parameter to BestFirstCrawlingStrategy (currently
a hardcoded module constant, BATCH_SIZE = 10, controlling how many URLs are
pulled from the priority queue per round), and a dispatcher constructor
parameter to both BestFirstCrawlingStrategy and BFSDeepCrawlStrategy
(currently neither forwards a dispatcher to their internal arun_many()
calls, so callers can't supply a custom SemaphoreDispatcher, or a
MemoryAdaptiveDispatcher with a custom RateLimiter/CrawlerMonitor).

What problem does this solve?

BestFirstCrawlingStrategy's per-round batch size is fixed at 10 with no way
to change it, and neither BestFirstCrawlingStrategy nor BFSDeepCrawlStrategy
lets a caller supply their own dispatcher (e.g. a custom SemaphoreDispatcher,
or a MemoryAdaptiveDispatcher with a custom RateLimiter/CrawlerMonitor) to
their internal arun_many() calls. There's currently no way to reach either of
these without patching the library directly.

Target users/beneficiaries

Anyone running BestFirstCrawlingStrategy or BFSDeepCrawlStrategy deep crawls
at scale who needs to tune concurrency behavior per target site — e.g. going
faster against a resilient site, or backing off more carefully against one
with strict rate limits (we hit this ourselves: the same crawl succeeded
cleanly on one real site but got rate-limited on another under crawl4ai's
default settings).

Current alternatives/workarounds

CrawlerRunConfig.semaphore_count covers basic concurrency tuning today, but
only as a plain count passed to arun_many()'s own default MemoryAdaptiveDispatcher.
There's no way to pass a different dispatcher class, custom RateLimiter, or
monitor, and no way to change BestFirstCrawlingStrategy's fixed 10-per-round
batch pull. The only current workaround is patching the library's source directly.

Proposed approach

• BestFirstCrawlingStrategy.__init__: add batch_size: int = 10 (default
  matches the current hardcoded constant), replacing BATCH_SIZE at its three
  usage sites.
• BestFirstCrawlingStrategy.__init__ and BFSDeepCrawlStrategy.__init__: add
  dispatcher: Optional[BaseDispatcher] = None, forwarded to their internal
  arun_many() calls only when explicitly set — so the call shape (and any
  existing test doubles built against the old signature) stays unchanged by
  default, and existing callers see no behavior change.

Have a working implementation + passing tests ready to open as a PR if this
direction looks right.

[yashikam19]

Opened a PR implementing this: #2246 - adds batch_size to
BestFirstCrawlingStrategy and dispatcher to both BestFirstCrawlingStrategy
and BFSDeepCrawlStrategy, both opt-in with no change to existing behavior.
Happy to adjust the parameter shape if the direction here differs from what's
in the PR.