Proposal: Multi-language Support for Dataset Generation

[jyotirmayc]

Enabling the dataset generator to support multiple languages for prompts and outputs. This would allow users to generate datasets in languages other than English (e.g., German, Spanish, French).

[lukehinds]

I love it, great idea. I wonder if this should just work by using the language in the prompt entries. Do you see there being anything else needed, perhaps validation, or support for different char-sets ?

[jyotirmayc]

I plan to create support for primarily French, German, Spanish for now. As Python uses UTF-8 character encoding by default, it will be easier to handle the character sets for these languages.

We should check the CLI input and create a list of supported codes (--lang fr, --lang pt_BR, etc.), to give a clear error if the input is invalid.
If the requested language or code is missing, control will fall back to English to prevent users from seeing a blank screen.

We could also write an additional script that compares the language files to the default English file to see if any of the keys are missing.

[lukehinds]

Hey @jyotirmayc , did you manage any progress, holla if you need any help.

[jyotirmayc]

Sorry, could not get to start on it due to school work. Will start soon, and yes help would really be appreciated.