fix: gracefully degrade ingestion processing on tokenCount errors

[Steffen911]

Important

Enhance IngestionService to handle tokenization errors gracefully by logging and continuing without token counts.

• Behavior:
  • In IngestionService, wrap tokenization logic in a try-catch block to handle errors gracefully.
  • On tokenization error, log the error using traceException() and continue without token counts.
• Logging:
  • Add warning log for async tokenization failure and fallback to sync tokenization.
  • Add error log for tokenization failure with traceException() in getUsageUnits().
• Functions:
  • Modify getUsageUnits() in index.ts to include error handling for tokenization.

^{This description was created by for 21cd2aa. You can customize this summary. It will automatically update as commits are pushed.}