[QUESTION] How to load only the tokenizer in multilingual pipeline

[jplu]

Hello,

I struggle to find how to do not load all the processors in a multilingual context.

When I do:

from stanza.pipeline.multilingual import MultilingualPipeline

nlp = MultilingualPipeline()
text = "my piece of text in a random language."
doc = nlp(text)

I get the following processors for the detected language:

============================
| Processor    | Package   |
----------------------------
| tokenize     | combined  |
| pos          | combined  |
| lemma        | combined  |
| depparse     | combined  |
| sentiment    | sstplus   |
| constituency | wsj       |
| ner          | ontonotes |
============================

Bust I only need to use and load the "tokenize" processor. How can I avoid the usage of all the other processors?

I specify that I don't know the language of the input text in advance.

Thanks in advance for any help :)

[AngledLuffa]

You can use the lang_configs field to pass in a map from language to config:

lang_configs = { "en": {"processors"="tokenize"} }
MultilingualPipeline(lang_configs=lang_configs)

Unfortunately, this doesn't generalize well since you'd have to add the same thing for every language your pipeline will encounter. I'll add the ability to pass in a default map.

[AngledLuffa]

sorry, had to edit mistake in the above code - don't rely on the email

[jplu]

That's basically the workaround I found as well but as you said, it must be specified for each lang. Would be nice indeed to have this ability to pass a default list of processors :)

Should I open a feature request instead?

[AngledLuffa]

That's fine. I'm doing it right now

[AngledLuffa]

If you install the dev branch, you can now pass in a defaultdict:

    lang_configs = defaultdict(lambda: dict(processors="tokenize"))
    nlp = MultilingualPipeline(lang_configs=lang_configs)

Not sure it's the best solution, but passing in a defaultdict wouldn't have worked as expected previously

[jplu]

Perfect! Looks good to me 👍 Thanks very much!