Sparknlp-1125 Implement Chunking Strategies

[danilojsl]

Description

This PR introduces Chunking Strategies, which enhance the Partition and PartitionTransformer components by dividing content into meaningful units based on the document's structure and content.

Incorporated strategies:

• basic: A general-purpose chunker that segments data into coherent chunks based on character limits
• by_title: A structure aware chunker tailored for documents with headings, tables, and mixed semantic elements

Motivation and Context

• Improved Retrieval Quality: Chunking respects natural topic boundaries, resulting in more semantically coherent and contextually relevant retrieval results.
• Reduced Noise: By preventing the inclusion of unrelated or partial fragments, the updated chunking logic minimizes irrelevant content in retrieval pipelines.

How Has This Been Tested?

• Local Tests
• Google Colab
• Databricks

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Code improvements with no or little impact
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING page.
• I have added tests to cover my changes.
• All new and existing tests passed.

[DevinTDHa]

just a small question, otherwise looks good to me!