Merge pull request #237 from guilhemmarchand/testing

Version 1.2.32
guilhemmarchand · Feb 9, 2021 · 8bb54eb · 8bb54eb
2 parents f18d9b3 + 920ee68
commit 8bb54eb
Show file tree

Hide file tree

Showing 31 changed files with 1,525 additions and 70 deletions.
diff --git a/.gitignore b/.gitignore
@@ -6,9 +6,6 @@
 # local metadata
 */metadata/local.meta
 
-# lookups dir
-*/lookups
-
 # Compiled Python files
 *.pyo
 *.pyc
@@ -39,3 +36,7 @@ backups
 
 # appinspect reports
 appinspect_report*.html
+
+# backup directory
+trackme/backup
+
diff --git a/build.sh b/build.sh
@@ -12,7 +12,7 @@ rm -f *.tgz
 find . -name "*.pyc" -type f | xargs rm -f
 find . -name "*.py" -type f | xargs chmod go-x
 find trackme/lib -name "*.py" -type f | xargs chmod a-x
-tar -czf ${app}_${version}.tgz --exclude=trackme/local --exclude=trackme/metadata/local.meta --exclude=trackme/lookups/lookup_file_backups --exclude=trackme/default.old* --exclude='./.*'  --exclude='.[^/]*' --exclude="._*" trackme
+tar -czf ${app}_${version}.tgz --exclude=trackme/local --exclude=trackme/backup --exclude=trackme/metadata/local.meta --exclude=trackme/lookups/lookup_file_backups --exclude=trackme/default.old* --exclude='./.*'  --exclude='.[^/]*' --exclude="._*" trackme
 echo "Wrote: ${app}_${version}.tgz"
 
 exit 0
diff --git a/docs/img/backup_and_restore/backup_list.png b/docs/img/backup_and_restore/backup_list.png
diff --git a/docs/img/backup_and_restore/backup_on_demand.png b/docs/img/backup_and_restore/backup_on_demand.png
diff --git a/docs/img/backup_and_restore/backup_purge.png b/docs/img/backup_and_restore/backup_purge.png
diff --git a/docs/img/backup_and_restore/restore1.png b/docs/img/backup_and_restore/restore1.png
diff --git a/docs/img/backup_and_restore/restore2.png b/docs/img/backup_and_restore/restore2.png
diff --git a/docs/img/backup_and_restore/restore3.png b/docs/img/backup_and_restore/restore3.png
diff --git a/docs/img/backup_and_restore/restore4.png b/docs/img/backup_and_restore/restore4.png
diff --git a/docs/releasenotes.rst b/docs/releasenotes.rst
@@ -1,6 +1,28 @@
 Release notes
 #############
 
+Version 1.2.32
+==============
+
+**CAUTION:**
+
+This is a new main release branch, TrackMe 1.2.x requires the deployment of the following dependencies:
+
+- Semicircle Donut Chart Viz, Splunk Base: https://splunkbase.splunk.com/app/4378
+- Splunk Machine Learning Toolkit, Splunk Base: https://splunkbase.splunk.com/app/2890
+- Splunk Timeline - Custom Visualization, Splunk Base: https://splunkbase.splunk.com/app/3120
+
+TrackMe requires a summary index (defaults to trackme_summary) and a metric index (defaults to trackme_metrics):
+https://trackme.readthedocs.io/en/latest/configuration.html
+
+- Enhancement - Issue #230 - data host over time and single search performance improvements
+- Enhancement - Issue #222 - Automatically Backup Main KV Store collections, provide endpoints for backup and restore operations
+- Enhancement - Issue #232 - REST API and tooling - Provide a new app nav menu and a new dashboard to demonstrate the REST API endpoints and the usage of the trackme API in SPL commands
+- Fix - Issue #231 - UI - reduce the max number of entries in the tag policies screen (goes beyond the modal limitation)
+- Fix - Issue #233 - Smart Status - orange state due to week days monitoring is not properly handled
+- Fix - Issue #235 - Data sources - Week days monitoring rules are not honoured if triggering due to dcount host
+- Fix - Issue #236 - Data sources - status message is inaccurate if data source is in data sampling alert but week days monitoring rules are not met
+
 Version 1.2.31
 ==============
 
@@ -113,7 +135,7 @@ https://trackme.readthedocs.io/en/latest/configuration.html
 
 *Data sources changes:*
 
-- Feature: Issue #196 Data sources - Provides distinct count threshold capabilities to turn a data source red if the number of hosts goes bellow a static threshold, provides chart visibility in Overview screen of the data source
+- Feature: Issue #196 Data sources - Provides distinct count threshold capabilities to turn a data source red if the number of hosts goes below a static threshold, provides chart visibility in Overview screen of the data source
 
 *Others:*
 

diff --git a/docs/rest_api_reference.rst b/docs/rest_api_reference.rst
@@ -41,6 +41,8 @@ These resource groups are accessible by specific endpoint paths as following:
 +----------------------------------------------+----------------------------------------------+
 | :ref:`Smart Status endpoints`                | /services/trackme/v1/smart_status            |
 +----------------------------------------------+----------------------------------------------+
+| :ref:`Backup and Restore endpoints`          | /services/trackme/v1/backup_and_restore      |
++----------------------------------------------+----------------------------------------------+
 
 These endpoints can be used to interract with TrackMe in a programmatic fashion, for instance to perform integration tasks with automation systems.
 
@@ -4524,3 +4526,133 @@ mh_smart_status / Run Smart Status for a metric host
     }
 
 *The API response depends on the smart status results.*
+
+Backup and Restore endpoints
+----------------------------
+
+**Resources summary:**
+
++---------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
+| Resource                                                                                          | API Path                                                                     | 
++===================================================================================================+==============================================================================+
+| :ref:`backup / Get backup archive files available`                                                | /services/trackme/v1/backup_and_restore/backup                               |
++---------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
+| :ref:`backup / Run backup KVstore collections`                                                    | /services/trackme/v1/backup_and_restore/backup                               |
++---------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
+| :ref:`backup / Purge older backup archive files`                                                  | /services/trackme/v1/backup_and_restore/backup                               |
++---------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
+| :ref:`restore / Perform a restore of KVstore collections`                                         | /services/trackme/v1/backup_and_restore/restore                              |
++---------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
+
+
+backup / Get backup archive files available
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+**This endpoint lists all the backup files available on the search head, files are stored in the backup directory of the application, it requires a GET call with no arguments:**
+
+*External:*
+
+::
+
+    curl -k -u admin:'ch@ngeM3' -X GET https://localhost:8089/services/trackme/v1/backup_and_restore/backup
+
+*SPL query:*
+
+::
+
+    | trackme url="/services/trackme/v1/backup_and_restore/backup" mode="get"
+
+*JSON response:*
+
+::
+
+    {"backup_files": "['/opt/splunk/etc/apps/trackme/backup/trackme-backup-20210205-142635.tgz', '/opt/splunk/etc/apps/trackme/backup/trackme-backup-20210205-142607.tgz']"}
+
+backup / Run backup KVstore collections
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+**This endpoint performs a backup of all TrackMe collections in a compressed tarball file stored in the backup directory of the application, it requires a POST call with no arguments:**
+
+*External:*
+
+::
+
+    curl -k -u admin:'ch@ngeM3' -X POST https://localhost:8089/services/trackme/v1/backup_and_restore/backup
+
+*SPL query:*
+
+::
+
+    | trackme url="/services/trackme/v1/backup_and_restore/backup" mode="post"
+
+*JSON response:*
+
+::
+
+    { "backup_archive": "/opt/splunk/etc/apps/trackme/backup/trackme-backup-20210205-142505.tgz", "report": "23 collections backed up / 5 collections empty"}
+
+backup / Purge older backup archive files
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+**This endpoint performs a purge of backup archive files older than x days, it requires a DELETE call with the following arguments:**
+
+- ``retention_days: (integer) OPTIONAL: the maximal retention for backup archive files in days, if not specified defaults to 7 days``
+
+*External:*
+
+::
+
+    curl -k -u admin:'ch@ngeM3' -X DELETE https://localhost:8089/services/trackme/v1/backup_and_restore/backup -d '{"metric_host": "telegraf-node1"}'
+
+*SPL query:*
+
+::
+
+    | trackme url="/services/trackme/v1/backup_and_restore/backup" mode="delete" body="{\"retention_days\": \"7\"}"
+
+*JSON response:*
+
+::
+
+    {"status": "There were no backup archive files older than 7 days to be purged"}
+
+restore / Perform a restore of KVstore collections
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+**This endpoint performs a backup of all TrackMe collections in compressed tarball file stored in the backup directory of the application, it requires a POST call with thre following arguments:**
+
+- ``backup_archive``:
+
+The archive file to be restoring from, the tarball compressed file must be located in the backup directory of the trackMe application.
+
+- ``dry_run``: 
+
+(true / false) OPTIONAL: if true, the endpoint will only verify that the archive can be found and successfully extracted, there will be no modifications at all. (default to true)
+
+- ``target:``
+
+(all / name of the KVstore json file) OPTIONAL: restore all available KVstore collection files (all) or choose a specific KVstore json file target to restore a unique collection. (default to all)
+
+*External:*
+
+::
+
+    curl -k -u admin:'ch@ngeM3' -X GET https://localhost:8089/services/trackme/v1/backup_and_restore/backup -d '{"backup_archive": "trackme-backup-20210205-142635.tgz", "dry_run": "false", "target": "all"}'
+
+*SPL query:*
+
+::
+
+    | trackme url="/services/trackme/v1/backup_and_restore/restore" mode="post" body="{\"backup_archive\": \"trackme-backup-20210205-142635.tgz\", \"dry_run\": \"false\", \"target\": \"all\"}"
+
+*JSON response in dry_run: true:*
+
+::
+
+    {"response": "Success: the archive /opt/splunk/etc/apps/trackme/backup/trackme-backup-20210205-142635.tgz could be successfully extracted, the following KVstore collections can be restored (empty collections are not backed up)", "collections": "['kv_trackme_data_source_monitoring_blacklist_sourcetype.json', 'kv_trackme_maintenance_mode.json', 'kv_trackme_data_host_monitoring_blacklist_host.json', 'kv_trackme_tags_policies.json', 'kv_trackme_metric_lagging_definition.json', 'kv_trackme_data_sampling.json', 'kv_trackme_data_source_monitoring_blacklist_index.json', 'kv_trackme_custom_lagging_definition.json', 'kv_trackme_summary_investigator_volume_outliers.json', 'kv_trackme_host_monitoring.json', 'kv_trackme_data_sampling_custom_models.json', 'kv_trackme_logical_group.json', 'kv_trackme_elastic_sources.json', 'kv_trackme_data_source_monitoring.json', 'kv_trackme_metric_host_monitoring.json', 'kv_trackme_data_source_monitoring_blacklist_host.json', 'kv_trackme_metric_host_monitoring_blacklist_host.json', 'kv_trackme_metric_host_monitoring_blacklist_metric_category.json', 'kv_trackme_data_host_monitoring_blacklist_sourcetype.json', 'kv_trackme_audit_changes.json', 'kv_trackme_metric_host_monitoring_blacklist_index.json', 'kv_trackme_data_host_monitoring_blacklist_index.json', 'kv_trackme_elastic_sources_dedicated.json']"}
+
+*JSON response in dry_run: false:*
+
+::
+
+    { "backup_archive": "/opt/splunk/etc/apps/trackme/backup/trackme-backup-20210205-142635.tgz", "status": "restore is now complete, please reload TrackMe","collections_files_restored": "['kv_trackme_data_source_monitoring_blacklist_sourcetype.json', 'kv_trackme_maintenance_mode.json', 'kv_trackme_data_host_monitoring_blacklist_host.json', 'kv_trackme_tags_policies.json', 'kv_trackme_metric_lagging_definition.json', 'kv_trackme_data_sampling.json', 'kv_trackme_data_source_monitoring_blacklist_index.json', 'kv_trackme_custom_lagging_definition.json', 'kv_trackme_summary_investigator_volume_outliers.json', 'kv_trackme_host_monitoring.json', 'kv_trackme_data_sampling_custom_models.json', 'kv_trackme_logical_group.json', 'kv_trackme_elastic_sources.json', 'kv_trackme_data_source_monitoring.json', 'kv_trackme_metric_host_monitoring.json', 'kv_trackme_data_source_monitoring_blacklist_host.json', 'kv_trackme_metric_host_monitoring_blacklist_host.json', 'kv_trackme_metric_host_monitoring_blacklist_metric_category.json', 'kv_trackme_data_host_monitoring_blacklist_sourcetype.json', 'kv_trackme_audit_changes.json', 'kv_trackme_metric_host_monitoring_blacklist_index.json', 'kv_trackme_data_host_monitoring_blacklist_index.json', 'kv_trackme_elastic_sources_dedicated.json']"}