Tokenizer trust remote code

[BobaZooba]

Description

When loading the tokenizer, there could have been issues, for example with the 01-ai/Yi-6B and 01-ai/Yi-34B models, as they have their own tokenizer. This is resolved by adding trust_remote_code to the tokenizer load.

Type of Change

• 📚 Examples / docs / tutorials / dependencies update
• 🔧 Bug fix (non-breaking change which fixes an issue)
• 🥂 Improvement (non-breaking change which improves an existing feature)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to change)
• 🔐 Security fix

Checklist

• I've updated the code style using make codestyle.
• I've written tests for all new methods and classes that I created.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (a1b50f0) 79.6% compared to head (99ad6ec) 79.6%.

Additional details and impacted files

@@          Coverage Diff          @@
##            main      #2   +/-   ##
=====================================
  Coverage   79.6%   79.6%           
=====================================
  Files         38      38           
  Lines       1329    1329           
=====================================
  Hits        1058    1058           
  Misses       271     271           

Flag	Coverage Δ
unit	79.6% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.