Skip to content

Commit bb306b9

Browse files
danilojslDevinTDHa
danilojsl
authored and
DevinTDHa
committed
[SPARKNLP-1125] Adding partition with chunk demo notebook [skip test]
1 parent 3e0aa61 commit bb306b9

File tree

1 file changed

+362
-0
lines changed

1 file changed

+362
-0
lines changed

‎examples/python/data-preprocessing/SparkNLP_Partition_with_Chunking_Demo.ipynb

Lines changed: 362 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,362 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"id": "tzcU5p2gdak9"
7+
},
8+
"source": [
9+
"# Introducing Partition with Semantic Chunking SparkNLP\n",
10+
"This notebook showcases the newly added `Partition` component in Spark NLP\n",
11+
"providing a streamlined and user-friendly interface for interacting with Spark NLP readers"
12+
]
13+
},
14+
{
15+
"cell_type": "markdown",
16+
"metadata": {
17+
"id": "RFOFhaEedalB"
18+
},
19+
"source": [
20+
"## Setup and Initialization\n",
21+
"Let's keep in mind a few things before we start 😊\n",
22+
"\n",
23+
"Support for **Partitioning** files was introduced in Spark NLP 6.0.1 \n",
24+
"\n",
25+
"Chunking support was added in Spark NLP 6.0.3\n",
26+
"Please make sure you have upgraded to the latest Spark NLP release.\n",
27+
"\n",
28+
"For local files example we will download different files from Spark NLP Github repo:"
29+
]
30+
},
31+
{
32+
"cell_type": "markdown",
33+
"metadata": {
34+
"id": "ATDLz3Gws5ob"
35+
},
36+
"source": [
37+
"**Downloading Files**"
38+
]
39+
},
40+
{
41+
"cell_type": "code",
42+
"execution_count": 7,
43+
"metadata": {
44+
"id": "g7PMCOJo0ZlU"
45+
},
46+
"outputs": [],
47+
"source": [
48+
"!mkdir txt-files\n",
49+
"!mkdir html-files"
50+
]
51+
},
52+
{
53+
"cell_type": "code",
54+
"execution_count": 8,
55+
"metadata": {
56+
"colab": {
57+
"base_uri": "https://localhost:8080/"
58+
},
59+
"id": "AV-krG6Ps8pq",
60+
"outputId": "ea4c2484-6e83-4a7a-a000-537f38189ed0"
61+
},
62+
"outputs": [
63+
{
64+
"name": "stdout",
65+
"output_type": "stream",
66+
"text": [
67+
"--2025-06-06 15:19:01-- https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp/feature/SPARKNLP-1125-Implement-Chunking-Strategies/src/test/resources/reader/txt/long-text.txt\n",
68+
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...\n",
69+
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
70+
"HTTP request sent, awaiting response... 200 OK\n",
71+
"Length: 1032 (1.0K) [text/plain]\n",
72+
"Saving to: ‘txt-files/long-text.txt’\n",
73+
"\n",
74+
"long-text.txt 100%[===================>] 1.01K --.-KB/s in 0s \n",
75+
"\n",
76+
"2025-06-06 15:19:01 (58.1 MB/s) - ‘txt-files/long-text.txt’ saved [1032/1032]\n",
77+
"\n",
78+
"--2025-06-06 15:19:01-- https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp/feature/SPARKNLP-1125-Implement-Chunking-Strategies/src/test/resources/reader/html/fake-html.html\n",
79+
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...\n",
80+
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
81+
"HTTP request sent, awaiting response... 200 OK\n",
82+
"Length: 665 [text/plain]\n",
83+
"Saving to: ‘html-files/fake-html.html’\n",
84+
"\n",
85+
"fake-html.html 100%[===================>] 665 --.-KB/s in 0s \n",
86+
"\n",
87+
"2025-06-06 15:19:02 (26.7 MB/s) - ‘html-files/fake-html.html’ saved [665/665]\n",
88+
"\n"
89+
]
90+
}
91+
],
92+
"source": [
93+
"!wget https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp/feature/SPARKNLP-1125-Implement-Chunking-Strategies/src/test/resources/reader/txt/long-text.txt -P txt-files\n",
94+
"!wget https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp/feature/SPARKNLP-1125-Implement-Chunking-Strategies/src/test/resources/reader/html/fake-html.html -P html-files"
95+
]
96+
},
97+
{
98+
"cell_type": "markdown",
99+
"metadata": {
100+
"id": "EoFI66NAdalE"
101+
},
102+
"source": [
103+
"## Partitioning Documents with Chunking\n",
104+
"Use the `basic` chunking to segment data into coherent chunks based on character limits"
105+
]
106+
},
107+
{
108+
"cell_type": "code",
109+
"execution_count": 9,
110+
"metadata": {
111+
"colab": {
112+
"base_uri": "https://localhost:8080/"
113+
},
114+
"id": "bAkMjJ1vdalE",
115+
"outputId": "75831f62-c84a-4170-f87e-e70a6c1ef39d"
116+
},
117+
"outputs": [
118+
{
119+
"name": "stdout",
120+
"output_type": "stream",
121+
"text": [
122+
"Warning::Spark Session already created, some configs may not take.\n"
123+
]
124+
}
125+
],
126+
"source": [
127+
"from sparknlp.partition.partition import Partition\n",
128+
"\n",
129+
"partition_df = Partition(content_type = \"text/plain\", chunking_strategy = \"basic\").partition(\"./txt-files/long-text.txt\")"
130+
]
131+
},
132+
{
133+
"cell_type": "markdown",
134+
"metadata": {
135+
"id": "k6uvYxiVzGsG"
136+
},
137+
"source": [
138+
"Output without `basic` chunk:"
139+
]
140+
},
141+
{
142+
"cell_type": "code",
143+
"execution_count": 10,
144+
"metadata": {
145+
"colab": {
146+
"base_uri": "https://localhost:8080/"
147+
},
148+
"id": "3L-Tp017qgqb",
149+
"outputId": "98af5f84-5abc-4554-bab7-7dd9c5212612"
150+
},
151+
"outputs": [
152+
{
153+
"name": "stdout",
154+
"output_type": "stream",
155+
"text": [
156+
"+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
157+
"|col |\n",
158+
"+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
159+
"|Ukrainian forces reportedly advanced in the western Donetsk-eastern Zaporizhia Oblast border area and in western Zaporizhia Oblast amid Ukrainian counteroffensive operations in southern and eastern Ukraine. Tavriisk Group of Forces Spokesperson Oleksandr Shtupun reported that Ukrainian forces are advancing in the directions of Novoprokopivka (13km south of Orikhiv), Mala Tokmachka (9km southeast of Orikhiv), and Ocheretuvate (25km southeast of Orikhiv) in western Zaporizhia Oblast.[1] Shtupun also stated that Ukrainian forces advanced near Urozhaine (9km south of Velyka Novosilka) and Robotyne (10km south of Orikhiv) and achieved unspecified successes near Staromayorske (9km south of Velyka Novosilka) in the Berdyansk direction (western Donetsk-eastern Zaporizhia Oblast border area) and in an unspecified location in the Melitopol direction (western Zaporizhia Oblast).[2] Ukrainian Eastern Group of Forces Spokesperson Ilya Yevlash stated that Ukrainian forces continued offensive operations in the Bakhmut direction.[3]|\n",
160+
"+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
161+
"\n"
162+
]
163+
}
164+
],
165+
"source": [
166+
"from pyspark.sql.functions import explode, col\n",
167+
"\n",
168+
"result_df = partition_df.select(explode(col(\"txt.content\")))\n",
169+
"result_df.show(truncate=False)"
170+
]
171+
},
172+
{
173+
"cell_type": "markdown",
174+
"metadata": {
175+
"id": "EQJvQsnxzRg1"
176+
},
177+
"source": [
178+
"Output with `basic` chunk:"
179+
]
180+
},
181+
{
182+
"cell_type": "code",
183+
"execution_count": 11,
184+
"metadata": {
185+
"colab": {
186+
"base_uri": "https://localhost:8080/"
187+
},
188+
"id": "VlhnXCV5qr4J",
189+
"outputId": "cdaf98f1-3109-4770-adaf-f51c80a59ab9"
190+
},
191+
"outputs": [
192+
{
193+
"name": "stdout",
194+
"output_type": "stream",
195+
"text": [
196+
"+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
197+
"|col |\n",
198+
"+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
199+
"|Ukrainian forces reportedly advanced in the western Donetsk-eastern Zaporizhia Oblast border area and in western Zaporizhia Oblast amid Ukrainian counteroffensive operations in southern and eastern Ukraine. Tavriisk Group of Forces Spokesperson Oleksandr Shtupun reported that Ukrainian forces are advancing in the directions of Novoprokopivka (13km south of Orikhiv), Mala Tokmachka (9km southeast of Orikhiv), and Ocheretuvate (25km southeast of Orikhiv) in western Zaporizhia Oblast.[1] Shtupun|\n",
200+
"|also stated that Ukrainian forces advanced near Urozhaine (9km south of Velyka Novosilka) and Robotyne (10km south of Orikhiv) and achieved unspecified successes near Staromayorske (9km south of Velyka Novosilka) in the Berdyansk direction (western Donetsk-eastern Zaporizhia Oblast border area) and in an unspecified location in the Melitopol direction (western Zaporizhia Oblast).[2] Ukrainian Eastern Group of Forces Spokesperson Ilya Yevlash stated that Ukrainian forces continued offensive |\n",
201+
"|operations in the Bakhmut direction.[3] |\n",
202+
"+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
203+
"\n"
204+
]
205+
}
206+
],
207+
"source": [
208+
"result_df = partition_df.select(explode(col(\"chunks.content\")))\n",
209+
"result_df.show(truncate=False)"
210+
]
211+
},
212+
{
213+
"cell_type": "markdown",
214+
"metadata": {
215+
"id": "4YYTB7G6zbmN"
216+
},
217+
"source": [
218+
"Use `by_title` chunking to group sections in documents with headings, tables, and mixed semantic elements"
219+
]
220+
},
221+
{
222+
"cell_type": "code",
223+
"execution_count": 12,
224+
"metadata": {
225+
"colab": {
226+
"base_uri": "https://localhost:8080/"
227+
},
228+
"id": "PxTf0Ot23ZaO",
229+
"outputId": "9b02a493-b4d0-41fc-c5ee-9ed8ab2de194"
230+
},
231+
"outputs": [
232+
{
233+
"name": "stdout",
234+
"output_type": "stream",
235+
"text": [
236+
"Warning::Spark Session already created, some configs may not take.\n"
237+
]
238+
}
239+
],
240+
"source": [
241+
"partition_df = Partition(content_type = \"text/html\", chunking_strategy = \"by_title\", combineTextUnderNChars = 50).partition(\"./html-files/fake-html.html\")"
242+
]
243+
},
244+
{
245+
"cell_type": "markdown",
246+
"metadata": {
247+
"id": "YXMf3cBfz_2-"
248+
},
249+
"source": [
250+
"Output without `by_title` chunk:"
251+
]
252+
},
253+
{
254+
"cell_type": "code",
255+
"execution_count": 13,
256+
"metadata": {
257+
"colab": {
258+
"base_uri": "https://localhost:8080/"
259+
},
260+
"id": "O-_R-W86sFo-",
261+
"outputId": "6f07e491-c556-41af-89da-273e905d0e8b"
262+
},
263+
"outputs": [
264+
{
265+
"name": "stdout",
266+
"output_type": "stream",
267+
"text": [
268+
"+----------------------------------------------------------------------------------------------------------------------------------+\n",
269+
"|col |\n",
270+
"+----------------------------------------------------------------------------------------------------------------------------------+\n",
271+
"|My First Heading |\n",
272+
"|My Second Heading |\n",
273+
"|My first paragraph. lorem ipsum dolor set amet. if the cow comes home under the sun how do you fault the cow for it's worn hooves?|\n",
274+
"|A Third Heading |\n",
275+
"|Column 1 Column 2 Row 1, Cell 1 Row 1, Cell 2 Row 2, Cell 1 Row 2, Cell 2 |\n",
276+
"+----------------------------------------------------------------------------------------------------------------------------------+\n",
277+
"\n"
278+
]
279+
}
280+
],
281+
"source": [
282+
"result_df = partition_df.select(explode(col(\"html.content\")))\n",
283+
"result_df.show(truncate=False)"
284+
]
285+
},
286+
{
287+
"cell_type": "markdown",
288+
"metadata": {
289+
"id": "EhLOvpfe0JIe"
290+
},
291+
"source": [
292+
"Output with `by_title` chunk:"
293+
]
294+
},
295+
{
296+
"cell_type": "code",
297+
"execution_count": 14,
298+
"metadata": {
299+
"colab": {
300+
"base_uri": "https://localhost:8080/"
301+
},
302+
"id": "WhSWaeYGrvP-",
303+
"outputId": "8f5da326-029c-4ad6-c201-c5d2f2f8fa7d"
304+
},
305+
"outputs": [
306+
{
307+
"name": "stdout",
308+
"output_type": "stream",
309+
"text": [
310+
"+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
311+
"|col |\n",
312+
"+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
313+
"|My First Heading My Second Heading My first paragraph. lorem ipsum dolor set amet. if the cow comes home under the sun how do you fault the cow for it's worn hooves? A Third Heading|\n",
314+
"|Column 1 Column 2 Row 1, Cell 1 Row 1, Cell 2 Row 2, Cell 1 Row 2, Cell 2 |\n",
315+
"+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
316+
"\n"
317+
]
318+
}
319+
],
320+
"source": [
321+
"result_df = partition_df.select(explode(col(\"chunks.content\")))\n",
322+
"result_df.show(truncate=False)"
323+
]
324+
},
325+
{
326+
"cell_type": "markdown",
327+
"metadata": {
328+
"id": "BB2FEfegGuxl"
329+
},
330+
"source": [
331+
"You can also use DFS file systems like:\n",
332+
"- Databricks: `dbfs://`\n",
333+
"- HDFS: `hdfs://`\n",
334+
"- Microsoft Fabric OneLake: `abfss://`"
335+
]
336+
}
337+
],
338+
"metadata": {
339+
"colab": {
340+
"provenance": []
341+
},
342+
"kernelspec": {
343+
"display_name": "Python 3 (ipykernel)",
344+
"language": "python",
345+
"name": "python3"
346+
},
347+
"language_info": {
348+
"codemirror_mode": {
349+
"name": "ipython",
350+
"version": 3
351+
},
352+
"file_extension": ".py",
353+
"mimetype": "text/x-python",
354+
"name": "python",
355+
"nbconvert_exporter": "python",
356+
"pygments_lexer": "ipython3",
357+
"version": "3.10.12"
358+
}
359+
},
360+
"nbformat": 4,
361+
"nbformat_minor": 1
362+
}

0 commit comments

Comments
 (0)