Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/ml-advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ license: |
# Optimization of linear methods (developer)

## Limited-memory BFGS (L-BFGS)
[L-BFGS](http://en.wikipedia.org/wiki/Limited-memory_BFGS) is an optimization
[L-BFGS](https://en.wikipedia.org/wiki/Limited-memory_BFGS) is an optimization
algorithm in the family of quasi-Newton methods to solve the optimization problems of the form
`$\min_{\wv \in\R^d} \; f(\wv)$`. The L-BFGS method approximates the objective function locally as a
quadratic without evaluating the second partial derivatives of the objective function to construct the
Expand Down
26 changes: 13 additions & 13 deletions docs/ml-classification-regression.md
Original file line number Diff line number Diff line change
Expand Up @@ -436,7 +436,7 @@ Refer to the [R API docs](api/R/reference/spark.svmLinear.html) for more details

## One-vs-Rest classifier (a.k.a. One-vs-All)

[OneVsRest](http://en.wikipedia.org/wiki/Multiclass_classification#One-vs.-rest) is an example of a machine learning reduction for performing multiclass classification given a base classifier that can perform binary classification efficiently. It is also known as "One-vs-All."
[OneVsRest](https://en.wikipedia.org/wiki/Multiclass_classification#One-vs.-rest) is an example of a machine learning reduction for performing multiclass classification given a base classifier that can perform binary classification efficiently. It is also known as "One-vs-All."

`OneVsRest` is implemented as an `Estimator`. For the base classifier, it takes instances of `Classifier` and creates a binary classification problem for each of the k classes. The classifier for class i is trained to predict whether the label is i or not, distinguishing class i from all other classes.

Expand Down Expand Up @@ -474,7 +474,7 @@ Refer to the [Java API docs](api/java/org/apache/spark/ml/classification/OneVsRe

## Naive Bayes

[Naive Bayes classifiers](http://en.wikipedia.org/wiki/Naive_Bayes_classifier) are a family of simple
[Naive Bayes classifiers](https://en.wikipedia.org/wiki/Naive_Bayes_classifier) are a family of simple
probabilistic, multiclass classifiers based on applying Bayes' theorem with strong (naive) independence
assumptions between every pair of features.

Expand All @@ -483,7 +483,7 @@ it computes the conditional probability distribution of each feature given each
For prediction, it applies Bayes' theorem to compute the conditional probability distribution
of each label given an observation.

MLlib supports [Multinomial naive Bayes](http://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes),
MLlib supports [Multinomial naive Bayes](https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes),
[Complement naive Bayes](https://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf),
[Bernoulli naive Bayes](http://nlp.stanford.edu/IR-book/html/htmledition/the-bernoulli-model-1.html)
and [Gaussian naive Bayes](https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Gaussian_naive_Bayes).
Expand All @@ -498,7 +498,7 @@ Feature values for Multinomial and Bernoulli models must be *non-negative*. The
For document classification, the input feature vectors should usually be sparse vectors.
Since the training data is only used once, it is not necessary to cache it.

[Additive smoothing](http://en.wikipedia.org/wiki/Lidstone_smoothing) can be used by
[Additive smoothing](https://en.wikipedia.org/wiki/Lidstone_smoothing) can be used by
setting the parameter $\lambda$ (default to $1.0$).

**Examples**
Expand Down Expand Up @@ -1003,7 +1003,7 @@ Refer to the [R API docs](api/R/reference/spark.survreg.html) for more details.


## Isotonic regression
[Isotonic regression](http://en.wikipedia.org/wiki/Isotonic_regression)
[Isotonic regression](https://en.wikipedia.org/wiki/Isotonic_regression)
belongs to the family of regression algorithms. Formally isotonic regression is a problem where
given a finite set of real numbers `$Y = {y_1, y_2, ..., y_n}$` representing observed responses
and `$X = {x_1, x_2, ..., x_n}$` the unknown response values to be fitted
Expand All @@ -1018,7 +1018,7 @@ with respect to complete order subject to
The resulting function is called isotonic regression and it is unique.
It can be viewed as least squares problem under order restriction.
Essentially isotonic regression is a
[monotonic function](http://en.wikipedia.org/wiki/Monotonic_function)
[monotonic function](https://en.wikipedia.org/wiki/Monotonic_function)
best fitting the original data points.

We implement a
Expand Down Expand Up @@ -1136,7 +1136,7 @@ Refer to [the linear methods guide for the RDD-based API](mllib-linear-methods.h
details about implementation and tuning; this information is still relevant.

We also include a DataFrame API for [Elastic
net](http://en.wikipedia.org/wiki/Elastic_net_regularization), a hybrid
net](https://en.wikipedia.org/wiki/Elastic_net_regularization), a hybrid
of $L_1$ and $L_2$ regularization proposed in [Zou et al, Regularization
and variable selection via the elastic
net](http://users.stat.umn.edu/~zouxx019/Papers/elasticnet.pdf).
Expand All @@ -1150,10 +1150,10 @@ regularization as special cases. For example, if a [linear
regression](https://en.wikipedia.org/wiki/Linear_regression) model is
trained with the elastic net parameter $\alpha$ set to $1$, it is
equivalent to a
[Lasso](http://en.wikipedia.org/wiki/Least_squares#Lasso_method) model.
[Lasso](https://en.wikipedia.org/wiki/Least_squares#Lasso_method) model.
On the other hand, if $\alpha$ is set to $0$, the trained model reduces
to a [ridge
regression](http://en.wikipedia.org/wiki/Tikhonov_regularization) model.
regression](https://en.wikipedia.org/wiki/Tikhonov_regularization) model.
We implement Pipelines API for both linear regression and logistic
regression with elastic net regularization.

Expand Down Expand Up @@ -1195,7 +1195,7 @@ or bin the continuous features and one-hot encode them.

# Decision trees

[Decision trees](http://en.wikipedia.org/wiki/Decision_tree_learning)
[Decision trees](https://en.wikipedia.org/wiki/Decision_tree_learning)
and their ensembles are popular methods for the machine learning tasks of
classification and regression. Decision trees are widely used since they are easy to interpret,
handle categorical features, extend to the multiclass classification setting, do not require
Expand Down Expand Up @@ -1300,7 +1300,7 @@ All output columns are optional; to exclude an output column, set its correspond

# Tree Ensembles

The DataFrame API supports two major tree ensemble algorithms: [Random Forests](http://en.wikipedia.org/wiki/Random_forest) and [Gradient-Boosted Trees (GBTs)](http://en.wikipedia.org/wiki/Gradient_boosting).
The DataFrame API supports two major tree ensemble algorithms: [Random Forests](https://en.wikipedia.org/wiki/Random_forest) and [Gradient-Boosted Trees (GBTs)](https://en.wikipedia.org/wiki/Gradient_boosting).
Both use [`spark.ml` decision trees](ml-classification-regression.html#decision-trees) as their base models.

Users can find more information about ensemble algorithms in the [MLlib Ensemble guide](mllib-ensembles.html).
Expand All @@ -1315,7 +1315,7 @@ The main differences between this API and the [original MLlib ensembles API](mll

## Random Forests

[Random forests](http://en.wikipedia.org/wiki/Random_forest)
[Random forests](https://en.wikipedia.org/wiki/Random_forest)
are ensembles of [decision trees](ml-classification-regression.html#decision-trees).
Random forests combine many decision trees in order to reduce the risk of overfitting.
The `spark.ml` implementation supports random forests for binary and multiclass classification and for regression,
Expand Down Expand Up @@ -1396,7 +1396,7 @@ All output columns are optional; to exclude an output column, set its correspond

## Gradient-Boosted Trees (GBTs)

[Gradient-Boosted Trees (GBTs)](http://en.wikipedia.org/wiki/Gradient_boosting)
[Gradient-Boosted Trees (GBTs)](https://en.wikipedia.org/wiki/Gradient_boosting)
are ensembles of [decision trees](ml-classification-regression.html#decision-trees).
GBTs iteratively train decision trees in order to minimize a loss function.
The `spark.ml` implementation supports GBTs for binary classification and for regression,
Expand Down
8 changes: 4 additions & 4 deletions docs/ml-clustering.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,10 @@ about these algorithms.

## K-means

[k-means](http://en.wikipedia.org/wiki/K-means_clustering) is one of the
[k-means](https://en.wikipedia.org/wiki/K-means_clustering) is one of the
most commonly used clustering algorithms that clusters the data points into a
predefined number of clusters. The MLlib implementation includes a parallelized
variant of the [k-means++](http://en.wikipedia.org/wiki/K-means%2B%2B) method
variant of the [k-means++](https://en.wikipedia.org/wiki/K-means%2B%2B) method
called [kmeans||](http://theory.stanford.edu/~sergei/papers/vldb12-kmpar.pdf).

`KMeans` is implemented as an `Estimator` and generates a `KMeansModel` as the base model.
Expand Down Expand Up @@ -193,10 +193,10 @@ Refer to the [R API docs](api/R/reference/spark.bisectingKmeans.html) for more d

## Gaussian Mixture Model (GMM)

A [Gaussian Mixture Model](http://en.wikipedia.org/wiki/Mixture_model#Multivariate_Gaussian_mixture_model)
A [Gaussian Mixture Model](https://en.wikipedia.org/wiki/Mixture_model#Multivariate_Gaussian_mixture_model)
represents a composite distribution whereby points are drawn from one of *k* Gaussian sub-distributions,
each with its own probability. The `spark.ml` implementation uses the
[expectation-maximization](http://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm)
[expectation-maximization](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm)
algorithm to induce the maximum-likelihood model given a set of samples.

`GaussianMixture` is implemented as an `Estimator` and generates a `GaussianMixtureModel` as the base
Expand Down
2 changes: 1 addition & 1 deletion docs/ml-collaborative-filtering.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ license: |

## Collaborative filtering

[Collaborative filtering](http://en.wikipedia.org/wiki/Recommender_system#Collaborative_filtering)
[Collaborative filtering](https://en.wikipedia.org/wiki/Recommender_system#Collaborative_filtering)
is commonly used for recommender systems. These techniques aim to fill in the
missing entries of a user-item association matrix. `spark.ml` currently supports
model-based collaborative filtering, in which users and products are described
Expand Down
14 changes: 7 additions & 7 deletions docs/ml-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ This section covers algorithms for working with features, roughly divided into t

## TF-IDF

[Term frequency-inverse document frequency (TF-IDF)](http://en.wikipedia.org/wiki/Tf%E2%80%93idf)
[Term frequency-inverse document frequency (TF-IDF)](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)
is a feature vectorization method widely used in text mining to reflect the importance of a term
to a document in the corpus. Denote a term by `$t$`, a document by `$d$`, and the corpus by `$D$`.
Term frequency `$TF(t, d)$` is the number of times that term `$t$` appears in document `$d$`, while
Expand All @@ -61,7 +61,7 @@ In MLlib, we separate TF and IDF to make them flexible.

`HashingTF` is a `Transformer` which takes sets of terms and converts those sets into
fixed-length feature vectors. In text processing, a "set of terms" might be a bag of words.
`HashingTF` utilizes the [hashing trick](http://en.wikipedia.org/wiki/Feature_hashing).
`HashingTF` utilizes the [hashing trick](https://en.wikipedia.org/wiki/Feature_hashing).
A raw feature is mapped into an index (term) by applying a hash function. The hash function
used here is [MurmurHash 3](https://en.wikipedia.org/wiki/MurmurHash). Then term frequencies
are calculated based on the mapped indices. This approach avoids the need to compute a global
Expand Down Expand Up @@ -321,7 +321,7 @@ for more details on the API.

## Tokenizer

[Tokenization](http://en.wikipedia.org/wiki/Lexical_analysis#Tokenization) is the process of taking text (such as a sentence) and breaking it into individual terms (usually words). A simple [Tokenizer](api/scala/org/apache/spark/ml/feature/Tokenizer.html) class provides this functionality. The example below shows how to split sentences into sequences of words.
[Tokenization](https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization) is the process of taking text (such as a sentence) and breaking it into individual terms (usually words). A simple [Tokenizer](api/scala/org/apache/spark/ml/feature/Tokenizer.html) class provides this functionality. The example below shows how to split sentences into sequences of words.

[RegexTokenizer](api/scala/org/apache/spark/ml/feature/RegexTokenizer.html) allows more
advanced tokenization based on regular expression (regex) matching.
Expand Down Expand Up @@ -507,7 +507,7 @@ for more details on the API.

## PCA

[PCA](http://en.wikipedia.org/wiki/Principal_component_analysis) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. A [PCA](api/scala/org/apache/spark/ml/feature/PCA.html) class trains a model to project vectors to a low-dimensional space using PCA. The example below shows how to project 5-dimensional feature vectors into 3-dimensional principal components.
[PCA](https://en.wikipedia.org/wiki/Principal_component_analysis) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. A [PCA](api/scala/org/apache/spark/ml/feature/PCA.html) class trains a model to project vectors to a low-dimensional space using PCA. The example below shows how to project 5-dimensional feature vectors into 3-dimensional principal components.

**Examples**

Expand Down Expand Up @@ -541,7 +541,7 @@ for more details on the API.

## PolynomialExpansion

[Polynomial expansion](http://en.wikipedia.org/wiki/Polynomial_expansion) is the process of expanding your features into a polynomial space, which is formulated by an n-degree combination of original dimensions. A [PolynomialExpansion](api/scala/org/apache/spark/ml/feature/PolynomialExpansion.html) class provides this functionality. The example below shows how to expand your features into a 3-degree polynomial space.
[Polynomial expansion](https://en.wikipedia.org/wiki/Polynomial_expansion) is the process of expanding your features into a polynomial space, which is formulated by an n-degree combination of original dimensions. A [PolynomialExpansion](api/scala/org/apache/spark/ml/feature/PolynomialExpansion.html) class provides this functionality. The example below shows how to expand your features into a 3-degree polynomial space.

**Examples**

Expand Down Expand Up @@ -821,7 +821,7 @@ for more details on the API.

## OneHotEncoder

[One-hot encoding](http://en.wikipedia.org/wiki/One-hot) maps a categorical feature, represented as a label index, to a binary vector with at most a single one-value indicating the presence of a specific feature value from among the set of all feature values. This encoding allows algorithms which expect continuous features, such as Logistic Regression, to use categorical features. For string type input data, it is common to encode categorical features using [StringIndexer](ml-features.html#stringindexer) first.
[One-hot encoding](https://en.wikipedia.org/wiki/One-hot) maps a categorical feature, represented as a label index, to a binary vector with at most a single one-value indicating the presence of a specific feature value from among the set of all feature values. This encoding allows algorithms which expect continuous features, such as Logistic Regression, to use categorical features. For string type input data, it is common to encode categorical features using [StringIndexer](ml-features.html#stringindexer) first.

`OneHotEncoder` can transform multiple columns, returning an one-hot-encoded output vector column for each input column. It is common to merge these vectors into a single feature vector using [VectorAssembler](ml-features.html#vectorassembler).

Expand Down Expand Up @@ -1074,7 +1074,7 @@ for more details on the API.

## Normalizer

`Normalizer` is a `Transformer` which transforms a dataset of `Vector` rows, normalizing each `Vector` to have unit norm. It takes parameter `p`, which specifies the [p-norm](http://en.wikipedia.org/wiki/Norm_%28mathematics%29#p-norm) used for normalization. ($p = 2$ by default.) This normalization can help standardize your input data and improve the behavior of learning algorithms.
`Normalizer` is a `Transformer` which transforms a dataset of `Vector` rows, normalizing each `Vector` to have unit norm. It takes parameter `p`, which specifies the [p-norm](https://en.wikipedia.org/wiki/Norm_%28mathematics%29#p-norm) used for normalization. ($p = 2$ by default.) This normalization can help standardize your input data and improve the behavior of learning algorithms.

**Examples**

Expand Down
4 changes: 2 additions & 2 deletions docs/ml-frequent-pattern-mining.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ license: |
Mining frequent items, itemsets, subsequences, or other substructures is usually among the
first steps to analyze a large-scale dataset, which has been an active research topic in
data mining for years.
We refer users to Wikipedia's [association rule learning](http://en.wikipedia.org/wiki/Association_rule_learning)
We refer users to Wikipedia's [association rule learning](https://en.wikipedia.org/wiki/Association_rule_learning)
for more information.

**Table of Contents**
Expand All @@ -36,7 +36,7 @@ The FP-growth algorithm is described in the paper
[Han et al., Mining frequent patterns without candidate generation](https://doi.org/10.1145/335191.335372),
where "FP" stands for frequent pattern.
Given a dataset of transactions, the first step of FP-growth is to calculate item frequencies and identify frequent items.
Different from [Apriori-like](http://en.wikipedia.org/wiki/Apriori_algorithm) algorithms designed for the same purpose,
Different from [Apriori-like](https://en.wikipedia.org/wiki/Apriori_algorithm) algorithms designed for the same purpose,
the second step of FP-growth uses a suffix tree (FP-tree) structure to encode transactions without generating candidate sets
explicitly, which are usually expensive to generate.
After the second step, the frequent itemsets can be extracted from the FP-tree.
Expand Down
6 changes: 3 additions & 3 deletions docs/mllib-classification-regression.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ license: |
---

The `spark.mllib` package supports various methods for
[binary classification](http://en.wikipedia.org/wiki/Binary_classification),
[binary classification](https://en.wikipedia.org/wiki/Binary_classification),
[multiclass
classification](http://en.wikipedia.org/wiki/Multiclass_classification), and
[regression analysis](http://en.wikipedia.org/wiki/Regression_analysis). The table below outlines
classification](https://en.wikipedia.org/wiki/Multiclass_classification), and
[regression analysis](https://en.wikipedia.org/wiki/Regression_analysis). The table below outlines
the supported algorithms for each type of problem.

<table>
Expand Down
Loading