Update manual testing guide and docker-compose. (elastic#555)

README.md

@@ -58,11 +58,11 @@ For Testing check out the [testing guide](TESTING.md)
 ### Update
 
 Each beat has a template for the mapping in elasticsearch and a documentation for the fields
-which is automatically generated based on `etc/fields.yml`.
-To generate etc/apm-server.template.json and etc/apm-server.asciidoc
+which is automatically generated based on `fields.yml`.
+To generate required configuration files and templates run:
 
 ```
-make update
+make index-template update
 ```
 
 ### Cleanup
@@ -81,46 +81,49 @@ make clean
 
 For further development, check out the [beat developer guide](https://www.elastic.co/guide/en/beats/libbeat/current/new-beat.html).
 
-## Packaging
-
-The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires [docker](https://www.docker.com/) and vendoring as described above. To build packages of your beat, run the following command:
-
-```
-make package
-```
-
-This will fetch and create all images required for the build process. The whole process can take several minutes.
-
 ## Update Dependencies
 
-The `apm-server` has two types of dependencies:
-
-* Golang packages managed with `govendor`
-* Beats framework managed with `cd _beats && sh update.sh`
-
-It is recommended to keep the version of the beats framework and libbeat in sync. To make an update of both, run `make update-beats`.
+The `apm-server` has two types of dependencies, 
+the Golang packages managed with *Govendor* and a dependency to the *Beats Framework*.
 
 ### Govendor
 
-For details on [govendor](https://github.com/kardianos/govendor) check the docs [here](https://github.com/kardianos/govendor).
+Checkout the [govendor tool](https://github.com/kardianos/govendor).
 
 To update beats to the most recent version from your go path for example use: `govendor fetch github.com/elastic/beats/...`.
 Govendor will automatically pick the files needed.
 
-### Framework Update
+### Beats Framework Update
 
 To update the beats framework run `make update-beats`. This will fetch the most recent version of beats from master and copy
 the files which are needed for the framework part to the `_beats` directory. These are files like libbeat config files and
 scripts which are used for testing or packaging.
 
+It is recommended to keep the version of the beats framework and libbeat in sync.
+To make an update of both, run:
+
+```
+make update-beats
+```
+
 To update the dependency to a specific commit or branch run command as following:
 
 ```
 BEATS_VERSION=f240148065af94d55c5149e444482b9635801f27 make update-beats
 ```
 
+## Packaging
+
+The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires [docker](https://www.docker.com/) and vendoring as described above. To build packages of your beat, run the following command:
+
+```
+make package
+```
+
+This will fetch and create all images required for the build process. The whole process can take several minutes.
+
 ## Documentation
-The [Documentation](docs/index.asciidoc) for the Intake-API and Elasticsearch can be found in `docs/data`. 
+The [Documentation](https://www.elastic.co/guide/en/apm/server/current/index.html) for the APM Server can be found in the `docs` folder.
 
 ## Help
 

TESTING.md

@@ -28,15 +28,18 @@ Following workflow is intended:
 
 ## Manual Testing
 
-For manual testing with the elastic stack and the agents there is a test environment based on docker. To
-get this running execute the following commands.
-
-* Run `make start-env`
-* Run 3 times around your table, it takes ES quite a bit to startup completely with X-Pack
-* Go to `localhost:5000` or `localhost:5000/error` and press refresh a few times
-* Open `localhost:5601` in your browser and log into Kibana with `elastic` and `changeme`
-* Create the `apm-server-*` index pattern
-* In Kibana go to the Discovery tab and you should see data
-
-For manual testing with specific agents, check instructions at `tests/agent/[LANG]/README.md`
-
+For manual testing with the elastic stack there is a test environment based on docker. 
+To get this running execute the following commands.
+
+* Run `STACK_VERSION=6.1.2 make start-env` to start docker containers for the elastic stack. 
+Set the version you want to test with via the environment variable `STACK_VERSION`.
+* Run `make import-dashboards` to import the Elasticsearch template and the Kibana dashboards
+* Check instructions at the 
+[Elastic APM agents docs](https://www.elastic.co/guide/en/apm/agent/index.html) 
+to setup an agent of your choice and create test data.
+Alternatively you can send a `curl` request to the 
+[transaction](https://www.elastic.co/guide/en/apm/server/current/transaction-api.html#transaction-api-examples) or the 
+[error](https://www.elastic.co/guide/en/apm/server/current/error-api.html#transaction-api-examples) endpoint   
+* Open `localhost:5601` in your browser
+* In Kibana go to the Dashboards to see APM Data
+* In Kibana go to the Discovery tab to query for APM Data

tests/docker-compose.yml

@@ -1,33 +1,58 @@
-version: '2.1'
+version: "2.1"
 services:
-  apm-server:
-    depends_on:
-      - elasticsearch
-      - kibana
-      - agent-python
-    build: .
-    ports:
-      - "8200:8200"
-    command: ./apm-server -E output.elasticsearch.hosts=[http://elastic:changeme@elasticsearch:9200] -E apm-server.host=0.0.0.0:8200 -e
-
   elasticsearch:
-    extends:
-      file: ../_beats/testing/environments/snapshot.yml
-      service: elasticsearch
+    image: docker.elastic.co/elasticsearch/elasticsearch-platinum:${STACK_VERSION}
+    container_name: localtesting_${STACK_VERSION}_elasticsearch
+    environment:
+      - cluster.name=docker-cluster
+      - xpack.security.enabled=false
+      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+    mem_limit: 1g
     ports:
-      - "9200:9200"
+      - "127.0.0.1:${ELASTICSEARCH_PORT:-9200}:9200"
+    healthcheck:
+      test: ["CMD-SHELL", "curl -f http://localhost:9200/_cluster/health | grep -v '\"status\":\"red\"'"]
+    labels:
+      - "co.elatic.apm.stack-version=${STACK_VERSION}"
 
   kibana:
-    extends:
-      file: ../_beats/testing/environments/snapshot.yml
-      service: kibana
+    image: docker.elastic.co/kibana/kibana-x-pack:${STACK_VERSION}
+    container_name: localtesting_${STACK_VERSION}_kibana
+    environment:
+      SERVER_NAME: kibana.example.org
+      ELASTICSEARCH_URL: http://elasticsearch:9200
     ports:
-      - "5601:5601"
+      - "127.0.0.1:${KIBANA_PORT:-5601}:5601"
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://kibana:5601/"]
+    depends_on:
+      elasticsearch:
+        condition: service_healthy
+    labels:
+      - "co.elatic.apm.stack-version=${STACK_VERSION}"
 
-  agent-python:
-    build: ./agent/python
+  apm-server:
+    image: docker.elastic.co/apm/apm-server:${STACK_VERSION}
+    container_name: localtesting_${STACK_VERSION}_apm-server
     ports:
-      - "8000:8000"
+      - "127.0.0.1:${APM_SERVER_PORT:-8200}:8200"
     environment:
-      APM_SERVER: "http://apm-server:8200"
-      SPANS_SEND_FREQ_SECS: "1"
+      - output.elasticsearch.hosts=['http://elasticsearch:9200']
+      - apm-server.host="0.0.0.0:8200"
+      - setup.kibana.host="kibana:5601"
+    command: apm-server -e -d "*" -E apm-server.host="0.0.0.0:8200" -E apm-server.expvar.enabled=true
+    depends_on:
+      elasticsearch:
+        condition: service_healthy
+      kibana:
+        condition: service_healthy
+    healthcheck:
+      test: ["CMD", "curl", "-f" ,"http://apm-server:8200/healthcheck"]
+    labels:
+      - "co.elatic.apm.stack-version=${STACK_VERSION}"
+