feat: Extraction of headlines in markdown files

[bogdankostic]

Related Issues

• fixes Extract headings from Markdown files #3056

Proposed Changes:

This PR adds the possibility to extract headlines out of a markdown file. For this, it adds the parameter extract_headlines to the MarkdownConverter and adapts the PreProcessor to be able to keep only the relevant headlines for each Document split when splitting the original Document.
Headlines are stored as a list of dictionaries in the Document's meta field "headlines" and are structured as followed:

{
    "headline": <THE HEADLINE STRING>,
    "start_idx": <IDX OF HEADLINE START IN document.content, int or None>,
    "level": <LEVEL OF THE HEADLINE, int>
}

start_idx is set to None by the PreProcessor when the headline is relevant for the current split but it appears in a previous split.

Further changes:

• Add parameter remove_code_snippets to MarkdownConverter to allow the users to choose whether to remove the code snippets (previously, code snippets were always removed)
• Refactor PreProcessor's split method to make it (hopefully) more readable
• Adapt the SentenceTokenizer in _split_sentences to not remove whitespace characters when splitting a text into sentences

How did you test it?

I added a couple of unit tests, let me know if I should add more.

Notes for the reviewer

I'm not quite sure about the dictionary format for the headlines, especially about setting start_idx to None if the headline cannot be found in Document.content. Please let me know if you think this is okay or you see a better solution.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added tests that demonstrate the correct behavior of the change
• I've used the conventional commit convention for my PR title
• I documented my code
• I ran pre-commit hooks and fixed any issue

[agnieszka-m]

Can we also add a sentence about how they can fix it or what happens now?

[bogdankostic]

Nor sure what we should add. Nothing really happening here, just informing the user that they have at least one exceptionally long sentence (longer than their specified split_length) in their Document. The consequence of this is that they will have at least one Document consisting of a single sentence.

[vblagoje]

Make the depth level configurable?

[vblagoje]

Or even better - find the deepest tree level in one pass of regular expressions?

[bogdankostic]

These are pre-defined HTML tags as listed for example here.
Before converting a markdown file to text, we convert it to HTML. The headline_tags set here is just used to check if an HTML element is a headline.

[vblagoje]

Overall, LGTM @bogdankostic, especially the effort you put into unit tests. Some total newcomer views: would it be possible/beneficial for users to return the headlines as a tree, as that's the natural order of the headlines? Then they can order it any way they want (post/pre/in order). Great work overall

[bogdankostic]

would it be possible/beneficial for users to return the headlines as a tree, as that's the natural order of the headlines?

My initial idea was actually to represent the extracted headlines as a tree. I decided against it because this can result in quite nested, complex structures - especially for large Documents. I thought it would be much more understandable like this.

[vblagoje]

@bogdankostic another option is id_hash_keys = id_hash_keys or self.id_hash_keys, but I am undecided about which one is better.

[bogdankostic]

id_hash_keys = id_hash_keys or self.id_hash_keys would work in this case because id_hash_keys is supposed to be a list, but this doesn't work with optional boolean parameters.

For example, if we explicitly set remove_code_snippets to False when calling convert, remove_code_snippets or self.remove_code_snippets would evaluate to whatever value self.remove_code_snippets has.

For consistency, I would like all these lines to have the same pattern.

[vblagoje]

Great points

[vblagoje]

GTG now @bogdankostic