Added multilingual support for the Wikipedia reader.

[anirbanbasu]

Description

Wikipedia is available in multiple languages, see https://en.wikipedia.org/wiki/List_of_Wikipedias. Some articles are available in languages apart from English. The current Wikipedia reader does not look for articles outside https://en.wikipedia.org/.

The ability to connect to non-English Wikipedia has been already available in the underlying Wikipedia parser, see starting line 22.

Fixes #12596. Same as mistakenly deleted/closed PR12597.

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

• Yes
• No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No

Type of Change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• Added new unit/integration tests
• Added new notebook (that tests end-to-end)
• I stared at the code and made sure it makes sense

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran make format; make lint to appease the lint gods

[anirbanbasu]

@garys-io thanks for approving the changes. A review seems to be still required for the PR to be merged.

[nerdai]

@anirbanbasu thanks for the contribution. Looks good, but can you please check if we can do validation of lang_code as I mentioned in my comment. 🙏

[nerdai]

can we add some validation here? It looks like we can (and maybe should) check and raise error if the supplied lang_code isn't actually supported via wikipedia.languages()

source

[anirbanbasu]

@nerdai thanks for your comment. That sounds like a good idea. How about the following?

if lang_code.lower() in wikipedia.languages().keys():
    wikipedia.set_lang(lang_code.lower())
else:
    raise SomeError(with_a_message)

What error would you like to raise? wikipedia.exceptions.WikipediaException? The error message should be something like "The provided language prefix for Wikipedia is not supported. Check supported languages at https://en.wikipedia.org/wiki/List_of_Wikipedias."

Alternatively, would you like it to silently fall back to "en" if the prefix is not supported instead of raising an exception?

Since I use the word prefix instead of code in the error message to be consistent with the wikipedia package, I will also change the function argument from lang_code to lang_prefix.

Looks good?

[nerdai]

Yea, we can raise a ValueError here. The message you suggested seems reasonable to me.

lang_prefix > lang_code 🙏

Thanks!

[anirbanbasu]

Thanks, I think the following is better than checking wikipedia.languages().keys():

if lang_prefix.lower() in wikipedia.languages():
      wikipedia.set_lang(lang_prefix.lower())
else:
       raise ValueError(
                    f"Language prefix '{lang_prefix}' for Wikipedia is not supported. Check supported languages at https://en.wikipedia.org/wiki/List_of_Wikipedias."
                )

[nerdai]

Agreed!

[anirbanbasu]

@nerdai thanks, code pushed.

[nerdai]

lgtm!

[nerdai]

Merging this now @anirbanbasu - thanks again for the contribution!