SentencePiece 0.1.97 changed API to take std::vector<string_view> "everywhere", breaking SetVocabulary here

[Micket]

here

Tokenizer/src/SentencePiece.cc

Lines 51 to 57 in 813ab83

	void SentencePiece::set_vocabulary(const std::vector<std::string>& vocabulary,
	const Tokenizer::Options* options)
	{
	if (options && (options->joiner_annotate || options->spacer_new))
	throw std::invalid_argument("SentencePiece vocabulary restriction requires the tokenization "
	"to use \"spacer_annotate\" (same as spm_encode)");
	auto status = _processor->SetVocabulary(vocabulary);

either a quick conversion

    auto status = _processor->SetVocabulary(ToPieceArray(vocabulary));

is needed or possible the string_view should switch should be propagated up the abstractions.

ToPieceArray was added in the same release that made this switch to views.
google/sentencepiece@631420b#diff-77e6a3b3bfda73d84fe1fef8205f2a2ec1d46b8f232100041f7135505f8adcefR52

[guillaumekln]

We still want to support version 0.1.96 (which is included as a Git submodule in the third_party directory) so we should wrap this code with a preprocessor condition #if ... #else ... #endif.

However, I'm not sure what the condition should be. SentencePiece exposes a version string but I don't know how to use it in a preprocessor comparison.

Do you have any suggestions?

[Micket]

Unfortunately, I just happened to encounter this helping @boegel with how to fix this when packaging Tokenizer for easybuild (where we just patch it for since we know the version used). I wanted to make change known to everyone else.