Azimuth for French with language selection in config

[lindsaydbrin]

Resolve #265

New news

No longer a draft PR; should run acceptably on French. Language-switching from config, updated behavioral tests, and multilingual FAISS encoder now included. Also more consistent reference to language-specific defaults in docs.

Old news (for reference):

Draft PR because:

• Goal was to get it working on a branch / not meant to be merged.
• More work required or preferred:
  • Language-switching from config (not hard-coded French, as it is now) = required
  • Changes for behavioral tests
  • Changes for FAISS encoder

Description:

New news

This code makes Azimuth work on French data/pipelines, and the language can be selected in the config.

Main changes:

• Part of Speech tagging: French defaults for spaCy model and dependency tags (to detect subjects and objects)
• Behavioral tests: French neutral tokens; fixed removal of space before punctuation (also submitted nlpaug issue)
• Config: Dynamic selection of language-specific defaults in the config

Out of scope things that could improve French Azimuth:

• Multilingual FAISS encoder (English-trained option works acceptably but not optimally)
• Multilingual tokenizer (Only used for comparing average token count per utterance between train and eval, so should have minimal impact on any conclusions)

Old news (for reference)

Addressed one aspect of Azimuth running for French models: spaCy part-of-speech tagging. This branch loads a French model (instead of English) and the missing_xyz smart tag POS/dependency tag lists have been updated to include French tags.

Several issues came up:

1. Single quote issue: Dataset included text with a single quote instead of an apostrophe, which broke POS-tagging (in particular, j' was not tagged as a subject) and search from the control panel. I added a new function to replace single quotes with apostrophes and used it both for searching and filtering. It worked reasonably well; out of 370 instances of j' in the eval set, 326 were tagged missing_verb before the fix, and 22 afterward (most of which started with "j'aimerais").
2. So then there was a problem with "j'" in "j'aimerais" being tagged as a subject (sometimes). This was solved with the md spaCy model (instead of sm).
3. SpaCy sometimes failed to identify verbs when they were the first word (common, especially with the imperative tense, e.g. "aidez-moi...") and/or capitalized (i.e. even in the middle of the sentence, which is not typical). Changing to the md model was a substantial improvement and fixed the majority of the issue. The exceptions seem to all be cases when the verb is capitalized (but not all capitalized verbs have problems; most don't). The capitalization issue seems to be fixed with the lg model...but it's like 500 MB...so I'm not sure if that's worth it (and haven't done it yet).

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• PRE-COMMIT. You ran pre-commit on all commits, or else, you
  ran pre-commit run --all-files at the end.
• FRONTEND TYPES. Regenerate the front-ent types if you played with types and routes.
  Run cd webapp && yarn types while the back-end is running.
• USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact
  our users.
• DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

[gabegma]

Cool, this gives us a good idea of the changes required! Some comments.

[JosephMarinier]

Hooo yeah! 👍

[lindsaydbrin]

@gabegma I'll wait for you to read docs (at your leisure - hope you enjoy nonfiction 😄 📖 ) before merging!

[gabegma]

Awesome PR! And great work with the documentation. I have a few comments. I've left the regex review to @JosephMarinier ;)

[gabegma]

Awesome work!! Love the documentation :)

[JosephMarinier]

I'd change only one word to be clear that it won't automatically "cause" that, but it will "force" you to set the whole AzimuthConfig as the module's scope.

Suggested change

	# cause a module to include all other configs in its scope.
	# force the module to include all other configs in its scope.

[lindsaydbrin]

Suggested change

	# cause a module to include all other configs in its scope.
	# force a module to include all other configs in its scope.

Sounds good! Just going to use "a" instead of "the."

[JosephMarinier]

The reason why I used "the" was to indicate that I am referring to the same module as the one with a "dependence on an attribute in AzimuthConfig". I didn't want to sentence to be understood as "dependence on an attribute [...] will force [another random] module to include all other configs in its scope".

Maybe I should have modified the first line

Remember that if a module depends on an attribute in AzimuthConfig, it will need to include all other configs in its scope.

[lindsaydbrin]

Yes, I understand the intention exactly! When I read what I committed, I read it as exactly that, whereas with "the," the pronoun feels surprising. I'm having trouble figuring out how to explain the grammatical reasoning...

Currently:

Before adding attributes: Remember that dependence on an attribute in AzimuthConfig will cause a module to include all other configs in its scope.

It is clear to me that the first phrase ("dependence...AzimuthConfig") belongs to the "a module" mentioned later - that it's a specific module that is the topic (subject?) of the sentence.

With the:

Before adding attributes: Remember that dependence on an attribute in AzimuthConfig will cause the module to include all other configs in its scope.

...I'm like, wait, what? Which module were we talking about? Were we talking about a module already?

But, all that said, even if I had a sound grammatical argument, it doesn't matter if it is confusing to you (or any other user). So to be super clear, we could change it to what you suggested, with a slight rephrasing in line with your earlier suggestion, and one of mine to be concise:

Reminder: If a module depends on an attribute in AzimuthConfig, the module will be forced to include all other configs in its scope.

What do you think? I'm not opposed to doing a new PR for one comment, all in the name of clear communication. :)

[JosephMarinier]

That's perfect! PR: #310

[JosephMarinier]

Beautiful! Thank you!