Add more info to 1.2 doc about configuring TLS

filebeat/docs/index.asciidoc

@@ -14,9 +14,13 @@ include::./configuring-howto.asciidoc[]
 
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
+include::./multiple-prospectors.asciidoc[]
+
+include::./securing-filebeat.asciidoc[]
+
 include::../../libbeat/docs/https.asciidoc[]
 
-include::./multiple-prospectors.asciidoc[]
+include::../../libbeat/docs/shared-tls-logstash-config.asciidoc[]
 
 include::./troubleshooting.asciidoc[]
 

filebeat/docs/securing-filebeat.asciidoc

@@ -0,0 +1,7 @@
+[[securing-filebeat]]
+== Securing Filebeat
+
+The following topics describe how to secure communication between Filebeat and other products in the Elastic stack:
+
+* <<securing-communication-elasticsearch>>
+* <<configuring-tls-logstash>>

libbeat/docs/https.asciidoc

@@ -10,7 +10,8 @@
 //// This content is structured to be included as a whole file.
 //////////////////////////////////////////////////////////////////////////
 
-=== Securing Communication
+[[securing-communication-elasticsearch]]
+=== Securing Communication With Elasticsearch
 
 To secure the communication between {beatname_uc} and Elasticsearch, you can use HTTPS
 and basic authentication. Here is a sample configuration:
@@ -39,8 +40,10 @@ https://www.elastic.co/guide/en/shield/current/certificate-authority.html[Settin
 appendix from the Shield guide.
 
 By default {beatname_uc} uses the list of trusted certificate authorities from the
-operating system where {beatname_uc} is running. You can configure a Beat to use a specific list of
-CA certificates instead of the list from the OS. Here is an example:
+operating system where {beatname_uc} is running. You can configure {beatname_uc} to use a specific list of
+CA certificates instead of the list from the OS. You can also configure it to use client authentication
+by specifying the certificate and key to use when the server requires the Beat to authenticate. Here is an example
+configuration:
 
 ["source","yaml",subs="attributes,callouts"]
 ----------------------------------------------------------------------
@@ -53,8 +56,12 @@ elasticsearch:
     certificate_authorities: <1>
       - /etc/pki/my_root_ca.pem
       - /etc/pki/my_other_ca.pem
+    certificate: "/etc/pki/client.pem" <2>
+    certificate_key: "/etc/pki/key.pem" <3>
 ----------------------------------------------------------------------
 <1> The list of CA certificates to trust
+<2> The path to the certificate for TLS client authentication
+<3> The client certificate key
 
 NOTE: For any given connection, the SSL/TLS certificates must have a subject
 that matches the value specified for `hosts`, or the TLS handshake fails.

libbeat/docs/shared-faq.asciidoc

@@ -68,3 +68,41 @@ telnet <hostname or IP> 5044
 TIP: For testing purposes only, you can set `insecure: true` to disable hostname checking.
 
 * Make sure that you have enabled SSL (set `ssl => true`) when configuring the https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash].
+
+==== Common Errors and Resolutions
+
+Here are some common errors and ways to fix them:
+
+* <<cannot-validate-certificate>>
+* <<getsockopt-no-route-to-host>>
+* <<getsockopt-connection-refused>>
+* <<target-machine-refused-connection>>
+
+[[cannot-validate-certificate]]
+===== x509: cannot validate certificate for <IP address> because it doesn't contain any IP SANs
+
+This happens because your certificate is only valid for the hostname present in the Subject field.
+
+To resolve this problem, try one of these solutions:
+
+* Create a DNS entry for the hostname mapping it to the server's IP.
+* Create an entry in `/etc/hosts` for the hostname. Or on Windows add an entry to
+`C:\Windows\System32\drivers\etc\hosts`.
+* Re-create the server certificate and add a SubjectAltName (SAN) for the IP address of the server. This make the
+server's certificate valid for both the hostname and the IP address.
+
+[[getsockopt-no-route-to-host]]
+===== getsockopt: no route to host
+
+This is not a TLS problem. It's a networking problem. Make sure the two hosts can communicate.
+
+[[getsockopt-connection-refused]]
+===== getsockopt: connection refused
+
+This is not a TLS problem. Make sure that Logstash is running and that there is no firewall blocking the traffic.
+
+[[target-machine-refused-connection]]
+===== No connection could be made because the target machine actively refused it
+
+A firewall is refusing the connection. Check if a firewall is blocking the traffic on the client, the network, or the
+destination host.

libbeat/docs/shared-tls-logstash-config.asciidoc

@@ -0,0 +1,142 @@
+//////////////////////////////////////////////////////////////////////////
+//// This content is shared by all Elastic Beats. Make sure you keep the
+//// descriptions here generic enough to work for all Beats that include
+//// this file. When using cross references, make sure that the cross
+//// references resolve correctly for any files that include this one.
+//// Use the appropriate variables defined in the index.asciidoc file to
+//// resolve Beat names: beatname_uc and beatname_lc.
+//// Use the following include to pull this content into a doc file:
+//// include::../../libbeat/docs/shared-tls-logstash-config.asciidoc[]
+//////////////////////////////////////////////////////////////////////////
+
+[[configuring-tls-logstash]]
+=== Securing Communication With Logstash by Using TLS
+
+You can use TLS mutual authentication to secure connections between {beatname_uc} and Logstash. This ensures that
+{beatname_uc} sends encrypted data to trusted Logstash servers only, and that the Logstash server receives data from
+trusted {beatname_uc} clients only.
+
+To use TLS mutual authentication:
+
+. Create a certificate authority (CA) and use it to sign the certificates that you plan to use for
+{beatname_uc} and Logstash. Creating a correct SSL/TLS infrastructure is outside the scope of this
+document. There are many online resources available that describe how to create certificates.
++
+NOTE: Certificates must be signed by your root CA. Intermediate CAs are currently not supported. 
+
+. Configure {beatname_uc} to use TLS. In the +{beatname_lc}.yml+ config file, specify the following settings under
+`tls`: 
++
+* `certificate_authorities`: Configures {beatname_uc} to trust any certificates signed by the specified CA. If
+`certificate_authorities` is empty or not set, the trusted certificate authorities of the host system are used.
+* `certificate` and `certificate_key`: Specifies the certificate and key that {beatname_uc} uses to authenticate with 
+Logstash.
++
+For example:
++
+[source,yaml]
+------------------------------------------------------------------------------
+output:
+  logstash:
+    hosts: ["logs.mycompany.com:5044"]
+    tls:
+      certificate_authorities: ["/etc/ca.crt"]
+      certificate: "/etc/client.crt"
+      certificate_key: "/etc/client.key"
+------------------------------------------------------------------------------
++
+For more information about these configuration options, see <<configuration-output-tls>>.
+
+. Configure Logstash to use TLS. In the Logstash config file, specify the following settings for the https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash]:
++
+* `ssl`: When set to true, enables Logstash to use SSL/TLS.
+* `ssl_certificate_authorities`: Configures Logstash to trust any certificates signed by the specified CA. 
+* `ssl_certificate` and `ssl_key`: Specify the certificate and key that Logstash uses to authenticate with the client. 
+* `ssl_verify_mode`: Specifies whether the Logstash server verifies the client certificate against the CA. You
+need to specify either `peer` or `force_peer` to make the server ask for the certificate and validate it. If you
+specify `force_peer`, and {beatname_uc} doesn't provide a certificate, the Logstash connection will be closed. 
++
+For example:
++
+[source,json]
+------------------------------------------------------------------------------
+input {
+  beats {
+    port => 5044
+    ssl => true
+    ssl_certificate_authorities => ["/etc/ca.crt"]
+    ssl_certificate => "/etc/server.crt"
+    ssl_key => "/etc/server.key"
+    ssl_verify_mode => "force_peer"
+  }
+}
+------------------------------------------------------------------------------
++
+For more information about these options, see the
+https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[documentation] for the Beats input plugin.
+
+[[testing-tls-logstash]]
+==== Validating the Logstash Server's Certificate
+
+Before running {beatname_uc}, you should validate the Logstash server's certificate. You can use `curl` to validate the certificate even though the protocol used to communicate with Logstash is not based on HTTP. For example:
+
+[source,shell]
+------------------------------------------------------------------------------
+curl -v --cacert ca.crt https://logs.mycompany.com:5044
+------------------------------------------------------------------------------
+
+If the test is successful, you'll receive an empty response error:
+
+[source,shell]
+------------------------------------------------------------------------------
+* Rebuilt URL to: https://logs.mycompany.com:5044/
+*   Trying 192.168.99.100...
+* Connected to logs.mycompany.com (192.168.99.100) port 5044 (#0)
+* TLS 1.2 connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
+* Server certificate: logs.mycompany.com
+* Server certificate: mycompany.com
+> GET / HTTP/1.1
+> Host: logs.mycompany.com:5044
+> User-Agent: curl/7.43.0
+> Accept: */*
+> 
+* Empty reply from server
+* Connection #0 to host logs.mycompany.com left intact
+curl: (52) Empty reply from server
+------------------------------------------------------------------------------
+
+The following example uses the IP address rather than the hostname to validate the certificate: 
+
+[source,shell]
+------------------------------------------------------------------------------
+curl -v --cacert ca.crt https://192.168.99.100:5044
+------------------------------------------------------------------------------
+
+Validation for this test fails because the certificate is not valid for the specified IP address. It's only valid for the `logs.mycompany.com`, the hostname that appears in the Subject field of the certificate.  
+
+[source,shell]
+------------------------------------------------------------------------------
+* Rebuilt URL to: https://192.168.99.100:5044/
+*   Trying 192.168.99.100...
+* Connected to 192.168.99.100 (192.168.99.100) port 5044 (#0)
+* WARNING: using IP address, SNI is being disabled by the OS.
+* SSL: certificate verification failed (result: 5)
+* Closing connection 0
+curl: (51) SSL: certificate verification failed (result: 5)
+------------------------------------------------------------------------------
+
+See the <<ssl-client-fails,troubleshooting docs>> for info about resolving this issue.
+
+==== Testing the Beats to Logstash Connection
+
+If you have {beatname_uc} running as a service, first stop the service. Then test your setup by running {beatname_uc} in 
+the foreground so you can quickly see any errors that occur: 
+
+["source","sh",subs="attributes,callouts"]
+------------------------------------------------------------------------------
+{beatname_lc} -c {beatname_lc}.yml -e -v
+------------------------------------------------------------------------------
+
+Any errors will be printed to the console. See the <<ssl-client-fails,troubleshooting docs>> for info about
+resolving common errors.
+

packetbeat/docs/index.asciidoc

@@ -16,14 +16,18 @@ include::./configuring-logstash.asciidoc[]
 
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
-include::../../libbeat/docs/https.asciidoc[]
-
 include::./capturing.asciidoc[]
 
 include::./thrift.asciidoc[]
 
 include::./maintaining-topology.asciidoc[]
 
+include::./securing-packetbeat.asciidoc[]
+
+include::../../libbeat/docs/https.asciidoc[]
+
+include::../../libbeat/docs/shared-tls-logstash-config.asciidoc[]
+
 include::./visualizing-data-packetbeat.asciidoc[]
 
 include::./filtering.asciidoc[]

packetbeat/docs/securing-packetbeat.asciidoc

@@ -0,0 +1,7 @@
+[[securing-packetbeat]]
+== Securing Packetbeat
+
+The following topics describe how to secure communication between Packetbeat and other products in the Elastic stack:
+
+* <<securing-communication-elasticsearch>>
+* <<configuring-tls-logstash>>

topbeat/docs/index.asciidoc

@@ -16,8 +16,12 @@ include::./configuring-logstash.asciidoc[]
 
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
+include::./securing-topbeat.asciidoc[]
+
 include::../../libbeat/docs/https.asciidoc[]
 
+include::../../libbeat/docs/shared-tls-logstash-config.asciidoc[]
+
 include::./troubleshooting.asciidoc[]
 
 include::./faq.asciidoc[]

topbeat/docs/securing-topbeat.asciidoc

@@ -0,0 +1,7 @@
+[[securing-topbeat]]
+== Securing Topbeat
+
+The following topics describe how to secure communication between Topbeat and other products in the Elastic stack:
+
+* <<securing-communication-elasticsearch>>
+* <<configuring-tls-logstash>>

winlogbeat/docs/index.asciidoc

@@ -14,8 +14,12 @@ include::./configuring-howto.asciidoc[]
 
 include::../../libbeat/docs/shared-env-vars.asciidoc[]
 
+include::./securing-winlogbeat.asciidoc[]
+
 include::../../libbeat/docs/https.asciidoc[]
 
+include::../../libbeat/docs/shared-tls-logstash-config.asciidoc[]
+
 include::./troubleshooting.asciidoc[]
 
 include::./faq.asciidoc[]

winlogbeat/docs/securing-winlogbeat.asciidoc

@@ -0,0 +1,7 @@
+[[securing-winlogbeat]]
+== Securing Winlogbeat
+
+The following topics describe how to secure communication between Winlogbeat and other products in the Elastic stack:
+
+* <<securing-communication-elasticsearch>>
+* <<configuring-tls-logstash>>