Add LLM Token and Cost Estimation Pre-check Before Indexing

[khaledalam]

Description

This PR introduces a new feature that estimates the token usage and LLM cost before executing the indexing pipeline. This helps users anticipate costs and make informed decisions before proceeding.

Related Issues

#385 , #487

Proposed Changes

New CLI params:

• --estimate-cost : [True || False] (by default False)
• --average-output-tokens-per-chunk : (by default 500}

graphrag index \
   --root ./ragtest \
   --estimate-cost \
   --average-output-tokens-per-chunk 500

• Util file: estimate_cost.py added for token counting and pricing logic.
• Pricing dynamically fetched from a public JSON source.
• CLI flag --estimate-cost added for optional pre-check.
• Console summary with embedding/chat model breakdown and token counts.
• User confirmation prompt before continuing indexing.
  = Conservative cost estimation based on --average-output-tokens-per-chunk value.

Checklist

• I have tested these changes locally.
• I have reviewed the code changes.
• I have updated the documentation (if necessary).
• I have added appropriate unit tests (if applicable).

[khaledalam]

@microsoft-github-policy-service agree

[khaledalam]

Please review @natoverse