[Spark-15997][DOC][ML] Update user guide for HashingTF, QuantileVectorizer and CountVectorizer

[GayathriMurali]

What changes were proposed in this pull request?

Made changes to HashingTF,QuantileVectorizer and CountVectorizer

[mengxr]

add to whitelist

[mengxr]

Could you limit the line width to 100?

[SparkQA]

Test build #60752 has finished for PR 13745 at commit 01e4a08.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #60758 has finished for PR 13745 at commit 3b01f11.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

Space missing: .Then -> . Then

[MLnick]

@GayathriMurali could you update the title to be SPARK-15997 as opposed to Spark 15997

[MLnick]

@jkbradley we could force a single partition in the data with repartition in the example code (per your comment #13176 (comment)). Perhaps we can hide that from the doc with example off? It is not ideal to have that in the example code either really, but I agree it's better than relativeError(0).

The only other solution I can think of off-hand is to change the example input data to be large enough that it shouldn't matter about partitioning.

[GayathriMurali]

@jkbradley @MLnick I agree with repartition idea. Although I think that it may not be a bad idea to call out that approxquantile calcultion for smaller datasets may be different on different machines depending on underlying cores available and leave the example and code as is. Please let me know whats best and I can change the documentation accordingly.

[jkbradley]

+1 for hiding repartition using example off/on
I'd add a small comment next to the repartition to make it clear why it is there.

[GayathriMurali]

@jkbradley @MLnick repartition needs to be added along with the creation of the dataframe like this.
val df = spark.createDataFrame(data).toDF("id","hour").repartition(1) since df is of type val. we cannot hide this statement. I could convert df to mutable object, but that would seem inconsistent. Am i missing something here?

[jkbradley]

Does this work?

    val df = spark.createDataFrame(data).toDF("id", "hour")
    // $example off$
      .repartition(1)
    // $example on$

[GayathriMurali]

Oops! That works.

[SparkQA]

Test build #61092 has finished for PR 13745 at commit 1074351.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #61094 has finished for PR 13745 at commit bbc0868.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

Did you check this works? I think it will throw SyntaxError. You may need to do dataFrame = dataFrame.repartition(1)

[MLnick]

Also can we just call it df to match the other examples (i just noticed that)

[GayathriMurali]

@MLnick I ran all unit tests and also tested them manually. It works fine. But I guess, writing it df = df.repartition(1) in the next line makes it look better. I will modify that.

[MLnick]

This is what I get:

./bin/spark-submit ~/workspace/quantile_discretizer_example.py
  File "/Users/nick/workspace/quantile_discretizer_example.py", line 18
    .repartition(1)
    ^
SyntaxError: invalid syntax

[MLnick]

@GayathriMurali couple final comments, then I think it's good to go. Thanks!

[SparkQA]

Test build #61115 has finished for PR 13745 at commit 2225c0a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

LGTM. Merged to master/branch-2.0. Thanks!