Skip to content
This repository has been archived by the owner. It is now read-only.
Permalink
Browse files
AMBARI-23456. Add more markdown docs
  • Loading branch information
oleewere committed Dec 11, 2018
1 parent 146a176 commit 2e218a8670c41354df5eed2952f281f57c4f9b13
Showing 18 changed files with 365 additions and 29 deletions.
@@ -28,6 +28,7 @@
<packaging>jar</packaging>
<url>http://maven.apache.org</url>
<name>Ambari Logsearch Docs</name>
<description>Ambari Logsearch Docs</description>
<build>
<resources>
<resource>
@@ -13,4 +13,14 @@ distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-->

## Contribution Guide

https://cwiki.apache.org/confluence/display/AMBARI/How+to+Contribute

(That is the ambari contribution guide, everything is the same here except use ambari-logsearch repository instead of ambari)

## License

[Apache 2.0 LICENSE](https://www.apache.org/licenses/LICENSE-2.0)
@@ -13,4 +13,96 @@ distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-->

## Adding New Logs to Monitor

#### Input / Filter (shipper) descriptors

For adding a new log that will be monitored by Log Feeder, first, it is needed to understand how Log Feeder registers and use those logs as inputs.
All the input log descriptors are stored in Log Feeder configuration directory with `input.config-<service>.json` naming pattern. (see: [Shipper configurations](shipper_configurations.md))

Based on these descriptors, the Log Feeder will start to monitor the inputs that are defined inside the descriptor as inputs.
After the configs are first processed, those will be handled/uploaded by a configuration handler, that should be an implementation of the Config API.

#### Config API

The Config API is responsible for to load and handle input (and filter) descriptors for Log Feeder. The idea is to store those configurations in a shared system
and Log Feeders should be notified about any of the input configuration changes.
It has different implementations, the default one is ZooKeeper (there is 2 other one right now: Solr and Local as well). Also with using/enabling the Config API,
the user will be able to edit / test / add input descriptors.

#### Support new input descriptor(s) for Ambari components

- Create a new jinja template: `<SERVICE_LOCATION>/package/templates/input.config-<service_name>.json.j2`

```jinja2
{
"input":[
{
"type":"my_log",
"rowtype":"service",
"path":"{{default('/configurations/myservice-env/log_dir', '/var/log/myservice')}}/my_log*.log"
},
{
"type":"myother_log",
"rowtype":"service",
"path":"{{default('/configurations/myservice-env/log_dir', '/var/log/myservice')}}/myother_log*.log"
}
],
"filter": [
...
]
```

- Change the stack code to write the input descriptor (add this to the start script):

```python
from resource_management.core.source import Template
from resource_management.libraries.functions.default import default
from resource_management.libraries.functions.generate_logfeeder_input_config import generate_logfeeder_input_config
# ...
# that will write input.config.myservice.json file to /usr/lib/ambari-logsearch-logfeeder/conf folder
generate_logfeeder_input_config('myservice', Template("input.config-myservice.json.j2", extra_imports=[default]))
# ...
```
- Edit the `metainfo.xml`: add logId mapping for components (logIds are log types in input descriptors)
```xml
<!-- ... -->
<components>
<component>
<name>MYCOMPONENT</name>
<logs>
<log>
<logId>my_log</logId>
<primary>true</primary>
</log>
<log>
<logId>myother_log</logId>
</log>
</logs>
<!-- ... -->
<component>
<components>
<!-- ... -->
```
From that point - during service start - the input config desriptor will be written into `/usr/lib/ambari-logsearch-logfeeder/conf`. The Log Feeder applications monitor the files in that folder, if there is a new one, it will start to monitor those inputs. Note that if the Config API is configured to use ZooKeeper, these configurations are only used from bootstrapping, after these files are uploaded to ZooKeeper, you will be onlyable to edit the input descriptos by Log Search Portal (or manually edit the znodes)

#### Add / Edit / Test input desriptors by Log Search Portal

On the Log Search Portal (if you go into `Configuration Editor`) you can edit (or add) existing input / filter configurations:

![Add / Edit Shipper Configs](images/edit_configs.jpg)

Also you can add test samples that you can validate:

![Test Log Samples](images/test_sample.jpg)

If the validation is successful, you should got a map response with the created fields:

![Test Log Samples Result](images/test_sample_result.jpg)

### Add input descriptors without Ambari

As you need the `input.config-<service>.json` files in `/usr/lib/ambari-logsearch-logfeeder/conf` folder (or where your configuration is located), it is enough to just create a new file at that location.
@@ -13,4 +13,8 @@ distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-->

### Log Feeder: cloud mode

TODO
@@ -0,0 +1,62 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

### Collections

By default there are 3 main storage layer abstractions in Log Search:

- Service logs (default name: `hadoop_logs`)
- Audit logs (default name: `audit_logs`)
- Metadata (default name: `logsearch_metadata`)

Service logs collection is responsible to store most of the logs (by type), except audit related data, for those use the audit logs collection.
The metadata collection is used to store Log Search UI related (dynamic) configurations/settings.

### Schema fields

Minimal required service log fields in Solr to make it work with the UI:

- id (string, unique - identifier for Solr doc)
- log_message
- type (string - log type)
- logtime (timestamp)
- seq_num (numeric - sequence number for log events, useful to not sort only by date)
- level (string - log level for logs, e.g.: WARN)
- host (string)
- cluster (string)

see more: [Service logs schema](https://github.com/apache/ambari-logsearch/blob/master/ambari-logsearch-server/src/main/configsets/hadoop_logs/conf/managed-schema)

Minimal required audit log fields in Solr to make it work with the UI:

- id (string, unique - identifier for Solr doc)
- evtTime (timestamp)
- repo (string - represents the audit source type)
- seq_num (numeric - sequence number for log events, useful to not sort only by date)

see more: [Audit logs schema](https://github.com/apache/ambari-logsearch/blob/master/ambari-logsearch-server/src/main/configsets/audit_logs/conf/managed-schema)

Fields for metadata:
- id (string, unique - identifier for Solr doc)
- name (string - metadata identifier)
- username (string - for identify user related data)
- type (string - type of the metadata)
- value (string - can be anything)

### Customize field names on the Log Search UI

Field name labels on the UI can be customized in `logsearch.properties`, see: [AMBARI-22842](https://issues.apache.org/jira/browse/AMBARI-22842)
@@ -15,8 +15,10 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

## Installation
### Backend development

## Build
TODO

## Requirements
### UI development

TODO
@@ -13,4 +13,16 @@ distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-->

### Rest API doc

Swagger specification: [logsearch-swagger.yaml](api-docs/logsearch-swagger.yaml)

### Javadoc

Generate javadoc with the following command from the project root:

```bash
make javadoc
```
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@@ -18,4 +18,31 @@ limitations under the License.

Log aggregation, analysis, and visualization for Ambari managed (or any other) services.

## Architecture
## Features

- Parse / aggregate and ship logs
- Send and index logs in Solr
- Store logs (structured or un-sructured format) in Cloud Storage (S3 / GCS / ADLS / WASB)
- Fultext Search in logs (if the logs are shipped to Solr)
- JWT/SSO support
- Support testing the log parsing on the UI

## Architecture

- Log Feeder: agent component on all hosts to monitor and shipping logs.
- Log Search Portal: REST API + UI for rendering logs
- Solr (Optional - default): storage for logs, used by both Log Search Portal and Log Feeder
- ZooKeeper (Optional - default): configuration service for Solr, Log Search and Log Feeder
- HDFS / S3 / GCS / ADLS: storage for logs (write only), used by Log Feeder [cloud mode](cloud_mode.md)

![Log Search Architecture Overview](images/architecture_overview.jpg)

## Contents

- [1, Installation](installation.md)
- [2. Collections](collections.md)
- [3. Adding new logs to monitor](add_new_input.md)
- [4. Development guide](development.md)
- [5. Using Log Feeder in Cloud mode](cloud_mode.md)
- [6. Contribution Guide and License](about.md)

0 comments on commit 2e218a8

Please sign in to comment.