This plugin provides API plugin for Apache CloudStack to process and view virtual machine logs which are handled by ELK and delivered by Filebeat. The version of the plugin matches Apache CloudStack version that it is build for.
The plugin is developed and tested only with Apache CloudStack 4.11.2
Download the plugin jar with dependencies file from OSS Nexus (https://oss.sonatype.org/content/groups/public/com/bwsw/cloud-plugin-vm-logs/) which corresponds to your ACS version (e.g. 4.11.2.0), put it to lib directory and restart Management server. In Ubuntu installation which is based on deb package:
cd /usr/share/cloudstack-management/lib/
wget --trust-server-names "https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=com.bwsw&a=cloud-plugin-vm-logs&c=jar-with-dependencies&v=4.11.2.0-SNAPSHOT"
service cloudstack-management stop
service cloudstack-management start
| Name | Description | Default value |
|---|---|---|
| vm.log.elasticsearch.list | comma separated list of ElasticSearch HTTP hosts; e.g. http://localhost,http://localhost:9201 | |
| vm.log.elasticsearch.username | Elasticsearch username for authentication; should be empty if authentication is disabled | |
| vm.log.elasticsearch.password | Elasticsearch password for authentication; should be empty if authentication is disabled | |
| vm.log.page.size.default | the default page size for VM log listing | 100 |
| vm.log.usage.timeout | Timeout in seconds to send VM log statistics | 3600 |
default.page.size is used as a default value for pagesize parameter in listVmLogFiles command. Its value should be less or equal to Elasticsearch index.max_result_window otherwise listVmLogFiles requests without pagesize parameter will fail.
Following components should be deployed:
Version recommended: 6.2.4
The official documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/6.2/index.html
Once ElasticSearch is deployed following actions must be done:
- to create VM log template
- to create
vmlog-registryindex using settings
If customization for log and file tags in responses for getVmLogs command is required a new template based on VM log template for an index pattern vmlog-* with an adjusted mapping for message and source properties correspondingly should be created.
Version recommended: 6.3.2
The official documentation can be found at https://www.elastic.co/guide/en/logstash/6.3/index.html.
The log pipeline should be used for VM log processing.
In the template above following placeholders should be replaced with real values:
| Name | Description |
|---|---|
| %PORT% | the port to process incoming beats from virtual machines |
| %ELASTICSEARCH_HOSTS% | Elasticsearch hosts to store VM logs |
| %VMLOG_REGISTRY_QUERY_TEMPLATE% | file path to Elasticsearch query template |
If SSL or user authentification are required Elasticsearch output plugin should be adjusted (see https://www.elastic.co/guide/en/logstash/6.2/plugins-outputs-elasticsearch.html).
If throttling for VM logs are required Throttle filter plugin should be used (see https://www.elastic.co/guide/en/logstash/6.2/plugins-filters-throttle.html).
If the token specified in Filebeat configuration is invalid or Elasticsearch is unavailable VM logs will be dropped.
Configuration example:
input {
beats {
port => 5045
}
}
filter {
if [vm_uuid] {
drop {
}
}
if ![token] {
drop {
}
}
elasticsearch {
hosts => "localhost:9200"
index => "vmlog-registry"
ssl => false
tag_on_failure => ["token_failure"]
fields => {
"vm_uuid" => "vm_uuid"
}
query_template => "/usr/share/logstash/config/vmlog-registry-query-template.json"
}
if ![vm_uuid] {
drop {
}
}
mutate {
remove_field => ["token"]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "vmlog-%{[vm_uuid]}-%{+YYYY-MM-dd}"
ssl => false
}
}
Version recommended: 6.3.2
Filebeat should be used in virtual machines for log processing.
The official documentation can be found at https://www.elastic.co/guide/en/beats/filebeat/6.3/index.html
Filebeat configuration should contain a field token that is the token obtained via CloudStack (see createVmLogToken), fields_under_root equal to true and Logstash output.
Filebeat configuration must not contain a field vm_uuid otherwise VM logs will be dropped.
A configuration example can be find here.
Version recommended: 5.5.2
Curator is used to delete old virtual machine logs.
The official documentation can be found at https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/index.html
The action file should be used for VM log processing.
In the template following placeholders should be replaced with real values:
| Name | Description |
|---|---|
| %TIMEOUT% | a client timeout in seconds |
| %DAYS% | a number of days to store VM logs |
The plugin provides following API commands to view virtual machine logs:
Lists available log files for the virtual machine.
Request parameters
| Parameter Name | Description | Required |
|---|---|---|
| id | the ID of the virtual machine | true |
| startdate | the start date/time in UTC, yyyy-MM-ddTHH:mm:ss | false |
| enddate | the end date/time in UTC, yyyy-MM-ddTHH:mm:ss | false |
| page | the requested page of the result listing | false |
| pagesize | the size for result listing | false |
Response tags
| Response Name | Description |
|---|---|
| file | the log file name |
Retrieves logs for the virtual machine.
Request parameters
| Parameter Name | Description | Required |
|---|---|---|
| id | the ID of the virtual machine | true |
| startdate | the start date/time in UTC, yyyy-MM-ddTHH:mm:ss | false |
| enddate | the end date/time in UTC, yyyy-MM-ddTHH:mm:ss | false |
| keywords | comma separated list of keywords (AND logical operator is used if multiple keywords are specified) | false |
| logfile | the log file | false |
| sort | comma separated list of response tags optionally prefixed with - for descending order | false |
| page | the requested page of the result listing | false |
| pagesize | the size for result listing | false |
| scroll | timeout in ms for subsequent scroll requests | false |
If both page/pagesize and scroll parameters are specified scroll is used.
Sorting and filtering for file and log tags in responses is applied to 256 first characters. The information how to change the limit can be found at deployment section.
Response tags
See VM log response tags.
Retrieves next batch of logs for the virtual machine.
Request parameters
| Parameter Name | Description | Required |
|---|---|---|
| scrollid | the tag to request next batch of logs | true |
| timeout | timeout in ms for subsequent scroll requests | true |
Response tags
See VM log response tags.
Creates a token to publish VM logs.
Request parameters
| Parameter Name | Description | Required |
|---|---|---|
| id | the ID of the virtual machine | true |
Response tags
| Response Name | Description |
|---|---|
| vmlogtoken | the token response |
| token | the token |
Request parameters
| Parameter Name | Description | Required |
|---|---|---|
| token | the token to publish VM logs | true |
Response tags
| Response Name | Description |
|---|---|
| vmlogtokenresult | success response |
| success | true if the token |
| Response Name | Description |
|---|---|
| vmlogs | the log listing |
| count | the total number of log entries |
| items(*) | log entries |
| id | the log id |
| timestamp | the date/time of log event registration |
| file | the log file |
| log | the log data |
| scrollid | the tag to request next batch of logs |