First pass at improving analyzer docs (elastic#18269)

* Docs: First pass at improving analyzer docs

I've rewritten the intro to analyzers plus the docs
for all analyzers to provide working examples.

I've also removed:

* analyzer aliases (see elastic#18244)
* analyzer versions (see elastic#18267)
* snowball analyzer (see elastic#8690)

Next steps will be tokenizers, token filters, char filters

* Fixed two typos

docs/reference/analysis.asciidoc

@@ -3,68 +3,113 @@
 
 [partintro]
 --
-The index analysis module acts as a configurable registry of Analyzers
-that can be used in order to both break indexed (analyzed) fields when a
-document is indexed and process query strings. It maps to the Lucene
-`Analyzer`.
 
+_Analysis_ is the process of converting text, like the body of any email, into
+_tokens_ or _terms_ which are added to the inverted index for searching.
+Analysis is performed by an <<analysis-analyzers,_analyzer_>> which can be
+either a built-in analyzer or a <<analysis-custom-analyzer,`custom`>> analyzer
+defined per index.
 
-Analyzers are composed of a single <<analysis-tokenizers,Tokenizer>> 
-and zero or more <<analysis-tokenfilters,TokenFilters>>. The tokenizer may 
-be preceded by one or more <<analysis-charfilters,CharFilters>>. The
-analysis module allows one to register `TokenFilters`, `Tokenizers` and
-`Analyzers` under logical names that can then be referenced either in
-mapping definitions or in certain APIs. The Analysis module
-automatically registers (*if not explicitly defined*) built in
-analyzers, token filters, and tokenizers.
+[float]
+== Index time analysis
+
+For instance at index time, the built-in <<english-analyzer,`english`>> _analyzer_ would
+convert this sentence:
+
+[source,text]
+------
+"The QUICK brown foxes jumped over the lazy dog!"
+------
 
-Here is a sample configuration:
+into these terms, which would be added to the inverted index.
+
+[source,text]
+------
+[ quick, brown, fox, jump, over, lazi, dog ]
+------
+
+[float]
+=== Specifying an index time analyzer
+
+Each <<text,`text`>> field in a mapping can specify its own
+<<analyzer,`analyzer`>>:
 
 [source,js]
---------------------------------------------------
-index :
-    analysis :
-        analyzer : 
-            standard : 
-                type : standard
-                stopwords : [stop1, stop2]
-            myAnalyzer1 :
-                type : standard
-                stopwords : [stop1, stop2, stop3]
-                max_token_length : 500
-            # configure a custom analyzer which is 
-            # exactly like the default standard analyzer
-            myAnalyzer2 :
-                tokenizer : standard
-                filter : [standard, lowercase, stop]
-        tokenizer :
-            myTokenizer1 :
-                type : standard
-                max_token_length : 900
-            myTokenizer2 :
-                type : keyword
-                buffer_size : 512
-        filter :
-            myTokenFilter1 :
-                type : stop
-                stopwords : [stop1, stop2, stop3, stop4]
-            myTokenFilter2 :
-                type : length
-                min : 0
-                max : 2000
---------------------------------------------------
+-------------------------
+PUT my_index
+{
+  "mappings": {
+    "my_type": {
+      "properties": {
+        "title": {
+          "type":     "text",
+          "analyzer": "standard"
+        }
+      }
+    }
+  }
+}
+-------------------------
+// CONSOLE
+
+At index time, if no `analyzer` has been specified, it looks for an analyzer
+in the index settings called `default`.  Failing that, it defaults to using
+the <<analysis-standard-analyzer,`standard` analyzer>>.
+
 
 [float]
-[[backwards-compatibility]]
-=== Backwards compatibility
+== Search time analysis
+
+This same analysis process is applied to the query string at search time in
+<<full-text-queries,full text queries>> like the
+<<query-dsl-match-query,`match` query>>
+to convert the text in the query string into terms of the same form as those
+that are stored in the inverted index.
+
+For instance, a user might search for:
 
-All analyzers, tokenizers, and token filters can be configured with a
-`version` parameter to control which Lucene version behavior they should
-use. Possible values are: `3.0` - `3.6`, `4.0` - `4.3` (the highest
-version number is the default option).
+[source,text]
+------
+"a quick fox"
+------
+
+which would be analysed by the same `english` analyzer into the following terms:
+
+[source,text]
+------
+[ quick, fox ]
+------
+
+Even though the exact words used in the query string don't appear in the
+original text (`quick` vs `QUICK`, `fox` vs `foxes`), because we have applied
+the same analyzer to both the text and the query string, the terms from the
+query string exactly match the terms from the text in the inverted index,
+which means that this query would match our example document.
+
+[float]
+=== Specifying a search time analyzer
+
+Usually the same analyzer should be used both at
+index time and at search time, and <<full-text-queries,full text queries>>
+like the  <<query-dsl-match-query,`match` query>> will use the mapping to look
+up the analyzer to use for each field.
+
+The analyzer to use to search a particular field is determined by
+looking for:
+
+* An `analyzer` specified in the query itself.
+* The <<search-analyzer,`search_analyzer`>> mapping parameter.
+* The <<analyzer,`analyzer`>> mapping parameter.
+* An analyzer in the index settings called `default_search`.
+* An analyzer in the index settings called `default`.
+* The `standard` analyzer.
 
 --
 
+include::analysis/anatomy.asciidoc[]
+
+include::analysis/testing.asciidoc[]
+
 include::analysis/analyzers.asciidoc[]
 
 include::analysis/tokenizers.asciidoc[]

docs/reference/analysis/analyzers.asciidoc

@@ -1,67 +1,60 @@
 [[analysis-analyzers]]
 == Analyzers
 
-Analyzers are composed of a single <<analysis-tokenizers,Tokenizer>> 
-and zero or more <<analysis-tokenfilters,TokenFilters>>. The tokenizer may 
-be preceded by one or more <<analysis-charfilters,CharFilters>>.
-The analysis module allows you to register `Analyzers` under logical
-names which can then be referenced either in mapping definitions or in
-certain APIs. 
-
-Elasticsearch comes with a number of prebuilt analyzers which are
-ready to use.  Alternatively, you can combine the built in
-character filters, tokenizers and token filters to create 
-<<analysis-custom-analyzer,custom analyzers>>.
+Elasticsearch ships with a wide range of built-in analyzers, which can be used
+in any index without further configuration:
 
-[float]
-[[default-analyzers]]
-=== Default Analyzers
-
-An analyzer is registered under a logical name. It can then be
-referenced from mapping definitions or certain APIs. When none are
-defined, defaults are used. There is an option to define which analyzers
-will be used by default when none can be derived.
-
-The `default` logical name allows one to configure an analyzer that will
-be used both for indexing and for searching APIs. The `default_search`
-logical name can be used to configure a default analyzer that will be
-used just when searching (the `default` analyzer would still be used for
-indexing).
-
-For instance, the following settings could be used to perform exact matching
-only by default:
-
-[source,js]
---------------------------------------------------
-index :
-  analysis :
-    analyzer :
-      default :
-        tokenizer : keyword
---------------------------------------------------
+<<analysis-standard-analyzer,Standard Analyzer>>::
 
-[float]
-[[aliasing-analyzers]]
-=== Aliasing Analyzers
+The `standard` analyzer divides text into terms on word boundaries, as defined
+by the Unicode Text Segmentation algorithm. It removes most punctuation,
+lowercases terms, and supports removing stop words.
+
+<<analysis-simple-analyzer,Simple Analyzer>>::
+
+The `simple` analyzer divides text into terms whenever it encounters a
+character which is not a letter.  It lowercases all terms.
+
+<<analysis-whitespace-analyzer,Whitespace Analyzer>>::
+
+The `whitespace` analyzer divides text into terms whenever it encounters any
+whitespace character.  It does not lowercase terms.
+
+<<analysis-stop-analyzer,Stop Analyzer>>::
+
+The `stop` analyzer is like the `simple` analyzer, but also supports removal
+of stop words.
+
+<<analysis-keyword-analyzer,Keyword Analyzer>>::
 
-Analyzers can be aliased to have several registered lookup names
-associated with them. For example, the following will allow 
-the `standard` analyzer to also be referenced with `alias1`
-and `alias2` values.
+The `keyword` analyzer is a ``noop'' analyzer that accepts whatever text it is
+given and outputs the exact same text as a single term.
 
+<<analysis-pattern-analyzer,Pattern Analyzer>>::
 
-[source,js]
---------------------------------------------------
-index :
-  analysis :
-    analyzer :
-      standard :
-        alias: [alias1, alias2]
-        type : standard
-        stopwords : [test1, test2, test3]
---------------------------------------------------
+The `pattern` analyzer uses a regular expression to split the text into terms.
+It supports lower-casing and stop words.
 
-Below is a list of the built in analyzers.
+<<analysis-lang-analyzer,Language Analyzers>>::
+
+Elasticsearch provides many language-specific analyzers like `english` or
+`french`.
+
+<<analysis-fingerprint-analyzer,Fingerprint Analyzer>>::
+
+The `fingerprint` analyzer is a specialist analyzer which creates a
+fingerprint which can be used for duplicate detection.
+
+[float]
+=== Custom analyzers
+
+If you do not find an analyzer suitable for your needs, you can create a
+<<analysis-custom-analyzer,`custom`>> analyzer which combines the appropriate
+<<analysis-charfilters, character filters>>,
+<<analysis-tokenizers,tokenizer>>, and <<analysis-tokenfilters,token filters>>.
+
+
+include::analyzers/configuring.asciidoc[]
 
 include::analyzers/standard-analyzer.asciidoc[]
 
@@ -77,8 +70,6 @@ include::analyzers/pattern-analyzer.asciidoc[]
 
 include::analyzers/lang-analyzer.asciidoc[]
 
-include::analyzers/snowball-analyzer.asciidoc[]
-
 include::analyzers/fingerprint-analyzer.asciidoc[]
 
 include::analyzers/custom-analyzer.asciidoc[]

docs/reference/analysis/analyzers/configuring.asciidoc

@@ -0,0 +1,66 @@
+[[configuring-analyzers]]
+=== Configuring built-in analyzers
+
+The built-in analyzers can be used directly without any configuration.  Some
+of them, however, support configuration options to alter their behaviour.  For
+instance, the <<analysis-standard-analyzer,`standard` analyzer>> can be configured
+to support a list of stop words:
+
+[source,js]
+--------------------------------
+PUT my_index
+{
+  "settings": {
+    "analysis": {
+      "analyzer": {
+        "std_english": { <1>
+          "type":      "standard",
+          "stopwords": "_english_"
+        }
+      }
+    }
+  },
+  "mappings": {
+    "my_type": {
+      "properties": {
+        "my_text": {
+          "type":     "text",
+          "analyzer": "standard", <2>
+          "fields": {
+            "english": {
+              "type":     "text",
+              "analyzer": "std_english" <3>
+            }
+          }
+        }
+      }
+    }
+  }
+}
+
+GET _cluster/health?wait_for_status=yellow
+
+POST my_index/_analyze
+{
+  "field": "my_text", <2>
+  "text": "The old brown cow"
+}
+
+POST my_index/_analyze
+{
+  "field": "my_text.english", <3>
+  "text": "The old brown cow"
+}
+
+--------------------------------
+// CONSOLE
+
+<1> We define the `std_english` analyzer to be based on the `standard`
+    analyzer, but configured to remove the pre-defined list of English stopwords.
+<2> The `my_text` field uses the `standard` analyzer directly, without
+    any configuration.  No stop words will be removed from this field.
+    The resulting terms are: `[ the, old, brown, cow ]`
+<3> The `my_text.english` field uses the `std_english` analyzer, so
+    English stop words will be removed.  The resulting terms are:
+    `[ old, brown, cow ]`
+