feat(ai_grouping): Send token length metrics on stacktraces sent to Seer

[yuvmen]

In preparation to making the switch to token length being considered instead of frame count of errors, we take metrics of the token length of stacktraces being sent to be able to map out the statistics and the impact that would make. Insturmented get_token_count to monitor how long it takes.

Introduces usage of tokenizers library for token count. Added the local tokenization model to Sentry to be used for tokenization without external dependencies.

Redo of #99873 which removed tiktoken dep by mistake. It is still used in getsentry and causes build errors if removed.

[seer-by-sentry]

Potential bug: get_token_count calls get_grouping_info_from_variants, which returns data with keys incompatible with the downstream get_stacktrace_string function, causing incorrect calculations.

• Description: The get_token_count function calls get_grouping_info_from_variants to generate grouping information when a cached stacktrace string is not available. This function creates a dictionary with keys like app_stacktrace. However, the downstream get_stacktrace_string function, which consumes this data, expects keys like app and system. This key mismatch causes get_stacktrace_string to find no relevant data, produce an empty string, and consequently makes get_token_count always return 0. This silently defeats the purpose of the new token count metrics collection.

• Suggested fix: In get_token_count, replace the call to get_grouping_info_from_variants(variants) with get_grouping_info_from_variants_legacy(variants). This will produce a data structure with the keys that get_stacktrace_string expects, allowing for correct token count calculation.
  _{severity: 0.65, confidence: 0.95}

_{Did we get this right? 👍 / 👎 to inform future reviews.}

[yuvmen]

This was actually correct, @lobsterkatie recently made a change to get_grouping_info_from_variants changing existing usages to get_grouping_info_from_variants_legacy, which needed to happen here as well

[yuvmen]

also realised my tests didnt cover this case. While I realize we previously said variants shouldn't be empty, I guess it doesnt hurt to protect from it since its a dict and we cant be sure theoretically.

[codecov]

Codecov Report

❌ Patch coverage is 94.64286% with 3 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/sentry/seer/similarity/utils.py	94.54%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #101477      +/-   ##
===========================================
+ Coverage   78.81%    81.29%   +2.48%     
===========================================
  Files        8699      8595     -104     
  Lines      385940    383404    -2536     
  Branches    24413     23858     -555     
===========================================
+ Hits       304162    311698    +7536     
+ Misses      81427     71363   -10064     
+ Partials      351       343       -8