encoder.json/vocab.bpe show up in every project that uses SK

[stephentoub]

Bringing in the Microsoft.SemanticKernel nuget package causes these files to show up in the consuming application:

and they end up in the output directory for the application, regardless of whether the app is using the tokenizer or not.

I'd opened #1800 to turn them into assembly resources instead, so that they'd simply be part of the assembly and not separate files, but it was closed due to a lack of a decision about what to do with it.

[anthonypuppo]

Echoing my comment on #1800

Bringing up SharpToken as this would solve 1) embedding of the tokenization resource files 2) extracting the tokenization logic to a separate package (maybe wrapped as an official SK package in the future) and 3) bug #2334.

[lemillermicrosoft]

One other option could be to move the tokenizer into an extension package for OpenAI connector.