Wikiprojects parser sanity tests

[codez266]

• Add tests for methods used by WikiProjects parser
• Add testfiles/ with sample wikitext containing WikiProjects for comparison in tests
• Modify parser from iterative to recursive
• Minor improvements to regexes

[halfak]

functions should be snake case https://www.python.org/dev/peps/pep-0008/#function-names

[codez266]

fixed.

[halfak]

"Wikipedia:WikiProject Council/Directory" should be a parameter or a static variable

[codez266]

fixed. Only left to figure out how to parameterize the complex regex - wp_section_regex

[halfak]

Variable names should be snake case (lower_case_with_underscores) https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles

[halfak]

would for section in sections be better here?

You can also do something like:

for i, section in enumerate(sections):
    ...

[codez266]

I would've have done that but the recursion inherently depends on fast-forwarding the loop variable over parsed sections while exiting to avoid repeated calls to parse sections.

[halfak]

This parser is actually making an HTTP connection. That means the test could fail if you don't have a connection to the internet. It's better to download the page content and include the data in the test so that an internet connection is not necessary.

[codez266]

Only the test - test_get_sub_categories makes api calls while all others use offline data.
Even the above test has been made to cache data offline in case tests are repeated. However, I'll still think about how can we mock api calls, as the method tested by this test makes repeated api calls for sections rather than fetching a page in a single call.

[halfak]

Generally, I'm concerned that the WikiProjectsParser constructs a session in it's init and that the host/URL for the WikiProject page is hard coded in that method. It doesn't seem like the class is performing a very useful function here. It would be better to have a pattern like:

def extract_directory(session, directory_page="Wikipedia:WikiProject..."):
    ...

def parse_section(section_text):
    ...

def parse_section_intro(intro_text):
    ...

def parse_wikiproject_table(table_text):
    ...

session = mwapi.Session(...)

print(extract_directory(session))

Then you'd write tests for parse_section() but not extract_directory which would only be responsible for making calls to the api iteratively.

In the end, this works. I'd like to, at minimum, get it pep8 compliant before we merge. We can worry about a restructuring later.

[codez266]

I agree with your point that we should have a session variable in the top level function extract_directory and make api calls only there. But because of certain nesting in the structure of WikiProjects directory, api calls only in the extract_directory was becoming impossible. Will discuss this in detail and think about how to restructure.

The above approach would work if I had fetched the entire page once and called helper methods subsequently to parse sections. However that seemed difficult because:

• Fetching sections one by one using api and section number is better and more robust that using regexes to isolate sections in the full wikitext of page.
• Essentially I use the output of ?action=parse&prop=sections to get sections of the page and use this info to extract each section which guarantees that we don't miss any section.

[adamwight]

This looks good! I ran flake8 and it found a few more pep8 glitches, report is attached.

pep8.txt

[codez266]

Thanks Adam! for your suggestions regarding flake8.