Updated Notebook

tirthajyoti · tirthajyoti · commit 037bd89013e5 · 2018-10-20T12:44:25.000-07:00
diff --git a/SparkContext and RDD Basics.ipynb b/SparkContext and RDD Basics.ipynb
@@ -394,7 +394,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Get another RDD with only the `distinct` elements"
+    "### Removing duplicates: Get another RDD with only the `distinct` elements\n",
+    "\n",
+    "The method `RDD.distinct()` Returns a new dataset that contains the distinct elements of the source dataset.\n",
+    "\n",
+    "**NOTE**: This operation requires a **shuffle** in order to detect duplication across partitions. **So, it is a slow operation.**"
    ]
   },
   {

Original file line number	Diff line number	Diff line change
`@@ -394,7 +394,11 @@`
`394`	`394`	`"cell_type": "markdown",`
`395`	`395`	`"metadata": {},`
`396`	`396`	`"source": [`
`397`		- "### Get another RDD with only the `distinct` elements"
	`397`	+ "### Removing duplicates: Get another RDD with only the `distinct` elements\n",
	`398`	`+ "\n",`
	`399`	+ "The method `RDD.distinct()` Returns a new dataset that contains the distinct elements of the source dataset.\n",
	`400`	`+ "\n",`
	`401`	`+ "NOTE: This operation requires a shuffle in order to detect duplication across partitions. So, it is a slow operation."`
`398`	`402`	`]`
`399`	`403`	`},`
`400`	`404`	`{`