AMBARI-23456. Add more markdown docs

ambari-logsearch-docs/pom.xml

@@ -28,6 +28,7 @@
   <packaging>jar</packaging>
   <url>http://maven.apache.org</url>
   <name>Ambari Logsearch Docs</name>
+  <description>Ambari Logsearch Docs</description>
   <build>
     <resources>
       <resource>

docs/about.md

@@ -13,4 +13,14 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
--->
+-->
+
+## Contribution Guide
+
+https://cwiki.apache.org/confluence/display/AMBARI/How+to+Contribute
+
+(That is the ambari contribution guide, everything is the same here except use ambari-logsearch repository instead of ambari)
+
+## License
+
+[Apache 2.0 LICENSE](https://www.apache.org/licenses/LICENSE-2.0)

docs/add_new_input.md

@@ -13,4 +13,96 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
--->
+-->
+
+## Adding New Logs to Monitor
+
+#### Input / Filter (shipper) descriptors
+
+For adding a new log that will be monitored by Log Feeder, first, it is needed to understand how Log Feeder registers and use those logs as inputs.
+All the input log descriptors are stored in Log Feeder configuration directory with `input.config-<service>.json` naming pattern. (see: [Shipper configurations](shipper_configurations.md))
+
+Based on these descriptors, the Log Feeder will start to monitor the inputs that are defined inside the descriptor as inputs. 
+After the configs are first processed, those will be handled/uploaded by a configuration handler, that should be an implementation of the Config API.
+
+#### Config API
+
+The Config API is responsible for to load and handle input (and filter) descriptors for Log Feeder. The idea is to store those configurations in a shared system 
+and Log Feeders should be notified about any of the input configuration changes. 
+It has different implementations, the default one is ZooKeeper (there is 2 other one right now: Solr and Local as well). Also with using/enabling the Config API, 
+the user will be able to edit / test / add input descriptors.
+
+#### Support new input descriptor(s) for Ambari components
+
+- Create a new jinja template: `<SERVICE_LOCATION>/package/templates/input.config-<service_name>.json.j2`
+
+```jinja2
+{
+  "input":[
+    {
+      "type":"my_log",
+      "rowtype":"service",
+      "path":"{{default('/configurations/myservice-env/log_dir', '/var/log/myservice')}}/my_log*.log"
+    },
+    {
+      "type":"myother_log",
+      "rowtype":"service",
+      "path":"{{default('/configurations/myservice-env/log_dir', '/var/log/myservice')}}/myother_log*.log"
+    }
+  ],
+  "filter": [
+    ...
+  ]
+```
+
+- Change the stack code to write the input descriptor (add this to the start script):
+
+```python
+from resource_management.core.source import Template
+from resource_management.libraries.functions.default import default
+from resource_management.libraries.functions.generate_logfeeder_input_config import generate_logfeeder_input_config
+
+# ...
+    # that will write input.config.myservice.json file to /usr/lib/ambari-logsearch-logfeeder/conf folder
+    generate_logfeeder_input_config('myservice', Template("input.config-myservice.json.j2", extra_imports=[default]))
+# ...
+```
+- Edit the `metainfo.xml`: add logId mapping for components (logIds are log types in input descriptors)
+```xml
+<!--  ... -->
+     <components>
+        <component>
+          <name>MYCOMPONENT</name>
+          <logs>
+            <log>
+              <logId>my_log</logId>
+              <primary>true</primary>
+            </log>
+            <log>
+              <logId>myother_log</logId>
+            </log>
+          </logs>
+          <!--  ... -->
+        <component>
+     <components>
+     <!--  ... -->
+```
+From that point - during service start - the input config desriptor will be written into `/usr/lib/ambari-logsearch-logfeeder/conf`. The Log Feeder applications monitor the files in that folder, if there is a new one, it will start to monitor those inputs. Note that if the Config API is configured to use ZooKeeper, these configurations are only used from bootstrapping, after these files are uploaded to ZooKeeper, you will be onlyable to edit the input descriptos by Log Search Portal (or manually edit the znodes)
+
+#### Add / Edit / Test input desriptors by Log Search Portal
+
+On the Log Search Portal (if you go into `Configuration Editor`) you can edit (or add) existing input / filter configurations:
+
+![Add / Edit Shipper Configs](images/edit_configs.jpg)
+
+Also you can add test samples that you can validate:
+
+![Test Log Samples](images/test_sample.jpg)
+
+If the validation is successful, you should got a map response with the created fields:
+
+![Test Log Samples Result](images/test_sample_result.jpg)
+
+### Add input descriptors without Ambari
+
+As you need the `input.config-<service>.json` files in `/usr/lib/ambari-logsearch-logfeeder/conf` folder (or where your configuration is located), it is enough to just create a new file at that location.

docs/cloud_mode.md

@@ -13,4 +13,8 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
--->
+-->
+
+### Log Feeder: cloud mode
+
+TODO

docs/collections.md

@@ -0,0 +1,62 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+### Collections
+
+By default there are 3 main storage layer abstractions in Log Search:
+
+- Service logs (default name: `hadoop_logs`)
+- Audit logs (default name: `audit_logs`)
+- Metadata (default name: `logsearch_metadata`)
+
+Service logs collection is responsible to store most of the logs (by type), except audit related data, for those use the audit logs collection. 
+The metadata collection is used to store Log Search UI related (dynamic) configurations/settings. 
+
+### Schema fields
+
+Minimal required service log fields in Solr to make it work with the UI:
+
+- id (string, unique - identifier for Solr doc)
+- log_message
+- type (string - log type)
+- logtime (timestamp)
+- seq_num (numeric - sequence number for log events, useful to not sort only by date)
+- level (string - log level for logs, e.g.: WARN)
+- host (string)
+- cluster (string)
+
+see more: [Service logs schema](https://github.com/apache/ambari-logsearch/blob/master/ambari-logsearch-server/src/main/configsets/hadoop_logs/conf/managed-schema)
+
+Minimal required audit log fields in Solr to make it work with the UI:
+
+- id (string, unique - identifier for Solr doc)
+- evtTime (timestamp)
+- repo (string - represents the audit source type)
+- seq_num (numeric - sequence number for log events, useful to not sort only by date)
+
+see more: [Audit logs schema](https://github.com/apache/ambari-logsearch/blob/master/ambari-logsearch-server/src/main/configsets/audit_logs/conf/managed-schema)
+
+Fields for metadata:
+- id (string, unique - identifier for Solr doc)
+- name (string - metadata identifier)
+- username (string - for identify user related data)
+- type (string - type of the metadata)
+- value (string - can be anything)
+
+### Customize field names on the Log Search UI
+
+Field name labels on the UI can be customized in `logsearch.properties`, see: [AMBARI-22842](https://issues.apache.org/jira/browse/AMBARI-22842)

docs/getting_started.md → docs/development.md

@@ -15,8 +15,10 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-## Installation
+### Backend development
 
-## Build
+TODO
 
-## Requirements
+### UI development
+
+TODO

docs/docs.md

@@ -13,4 +13,16 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
--->
+-->
+
+### Rest API doc
+
+Swagger specification: [logsearch-swagger.yaml](api-docs/logsearch-swagger.yaml)
+
+### Javadoc
+
+Generate javadoc with the following command from the project root:
+
+```bash
+make javadoc
+```

docs/index.md

@@ -18,4 +18,31 @@ limitations under the License.
 
 Log aggregation, analysis, and visualization for Ambari managed (or any other) services.
 
-## Architecture
+## Features
+
+- Parse / aggregate and ship logs
+- Send and index logs in Solr
+- Store logs (structured or un-sructured format) in Cloud Storage (S3 / GCS / ADLS / WASB)
+- Fultext Search in logs (if the logs are shipped to Solr)
+- JWT/SSO support
+- Support testing the log parsing on the UI
+
+## Architecture
+
+- Log Feeder: agent component on all hosts to monitor and shipping logs.
+- Log Search Portal: REST API + UI for rendering logs
+- Solr (Optional - default): storage for logs, used by both Log Search Portal and Log Feeder
+- ZooKeeper (Optional - default): configuration service for Solr, Log Search and Log Feeder
+- HDFS / S3 / GCS / ADLS: storage for logs (write only), used by Log Feeder [cloud mode](cloud_mode.md)
+
+![Log Search Architecture Overview](images/architecture_overview.jpg)
+
+## Contents
+
+- [1, Installation](installation.md)
+- [2. Collections](collections.md)
+- [3. Adding new logs to monitor](add_new_input.md) 
+- [4. Development guide](development.md)
+- [5. Using Log Feeder in Cloud mode](cloud_mode.md)
+- [6. Contribution Guide and License](about.md)
+