YACHT-895: Updated documentation, config, queue and test_configuration

README.md

@@ -130,12 +130,12 @@ To install BBQ in GCP, click button below or follow [Setup.md](./SETUP.md) doc.
 # Usage
 
 ## How to run backups?
-Backup process is scheduled periodically for all specified projects (check [config/prd/config.yaml](./config/prd/config.yaml) to specify which projects to backup and [config/cron.yaml](./config/cron.yaml) to configure schedule time).
+Backup process is scheduled periodically for all specified projects (check [config.yaml](./config/config.yaml) to specify which projects to backup and [config/cron.yaml](./config/cron.yaml) to configure schedule time).
 
 However, you may also invoke backup process manually from [cron jobs](https://console.cloud.google.com/appengine/taskqueues/cron).
 
 It's worth to underline that:
-* Backups for partitions are scheduled randomly within the range of time specified in [config.yaml](./config/prd/config.yaml),
+* Backups for partitions are scheduled randomly within the range of time specified in [config.yaml](./config/config.yaml),
 * It is possible to check the progress via [Task Queues](https://console.cloud.google.com/appengine/taskqueues).
 
 ## How to list already created backups?

SETUP.md

@@ -19,9 +19,9 @@ Then you could follow below steps:
       export PROJECT_ID="<your-project-id>"
       ```
 
-1. Change all **\<your-project-id\>** to your previously created project in [./config/prd/config.yaml](./config/prd/config.yaml) config file.
+1. Change all **\<your-project-id\>** to your previously created project in [config.yaml](./config/config.yaml) config file.
       ```bash
-      sed -i -e "s/<your-project-id>/${PROJECT_ID}/g" config/prd/config.yaml
+      sed -i -e "s/<your-project-id>/${PROJECT_ID}/g" config/config.yaml
       ```
 
 1. Install dependency requirements
@@ -30,7 +30,7 @@ Then you could follow below steps:
       ```
 1.  Deploy App Engine application
       ```bash
-      gcloud app deploy --project ${PROJECT_ID} app.yaml config/cron.yaml config/prd/queue.yaml config/index.yaml
+      gcloud app deploy --project ${PROJECT_ID} app.yaml config/cron.yaml config/queue.yaml config/index.yaml
       ```
 
     Note: If it is your first App Engine deploy, App Engine needs to be initialised and you will need to choose [region/location](https://cloud.google.com/appengine/docs/locations).
@@ -86,7 +86,7 @@ All backups that was invoked on local application will end up in this project.
       export PROJECT_ID="<your-project-id>"
       ```
 
-1. Change all **\<your-project-id\>** to your previously created project id in [./config/local/config.yaml](./config/local/config.yaml).
+1. Change all **\<your-project-id\>** to your previously created project id in [config.yaml](./config/config.yaml).
       ```bash
       sed -i -e "s/<your-project-id>/${PROJECT_ID}/g" config/local/config.yaml
       ```
@@ -98,7 +98,7 @@ All backups that was invoked on local application will end up in this project.
 
 1. Link config files to main application folder (due to lack of possibility to pass full path to dev_appserver.py)
       ```bash
-      ln -s config/local/queue.yaml queue.yaml
+      ln -s config/queue.yaml queue.yaml
       ln -s config/cron.yaml cron.yaml
       ln -s config/index.yaml index.yaml
       ```

config/config.yaml

@@ -10,7 +10,7 @@ project_settings:
   backup_project_id: '<your-project-id>'
   restoration_project_id: '<your-project-id>'
   authorized_requestor_service_accounts: ['sa-eligible-to-read-data@your-project-id.iam.gserviceaccount.com']
-  debug_mode: True
+  debug_mode: False
 
 tests:
   restoration_daily_test_results_gcs_bucket: 'a-gcs-bucket'

config/queue.yaml

@@ -5,36 +5,39 @@ queue:
   max_concurrent_requests: 100
   bucket_size: 20
   retry_parameters:
-    task_retry_limit: 5
-    task_age_limit: 1h
+    task_age_limit: 12h
+    min_backoff_seconds: 60
+    max_backoff_seconds: 3600
 - name: table-retention-scheduler
   rate: 10/s
   max_concurrent_requests: 20
   bucket_size: 20
   retry_parameters:
     task_retry_limit: 5
-    task_age_limit: 1h
+    task_age_limit: 3h
 - name: table-retention
   rate: 100/s
   max_concurrent_requests: 100
   bucket_size: 100
   retry_parameters:
     task_retry_limit: 5
-    task_age_limit: 1h
+    task_age_limit: 6h
 - name: backup-scheduler
   rate: 10/s
   max_concurrent_requests: 20
   bucket_size: 20
   retry_parameters:
     task_retry_limit: 5
-    task_age_limit: 1h
+    task_age_limit: 3h
 - name: backup-worker
   rate: 100/s
   max_concurrent_requests: 100
   bucket_size: 20
   retry_parameters:
     task_retry_limit: 5
-    task_age_limit: 1h
+    task_age_limit: 12h
+    min_backoff_seconds: 500
+    max_backoff_seconds: 3600
 
 - name: backups-post-copy-action
   rate: 5/s
@@ -58,20 +61,19 @@ queue:
     task_age_limit: 1h
 
 - name: restore-copy-job
-  rate: 5/s
+  rate: 20/s
   bucket_size: 1
   retry_parameters:
     task_retry_limit: 5
     task_age_limit: 1h
-
 - name: restore-result-check
   rate: 20/s
   retry_parameters:
     task_retry_limit: 5
     task_age_limit: 1h
 
 - name: restore-post-copy-action
-  rate: 5/s
+  rate: 20/s
   bucket_size: 1
   retry_parameters:
     task_retry_limit: 5

tests/config/test_configuration.py

@@ -8,19 +8,27 @@ class TestPrdConfiguration(unittest.TestCase):
     configuration = Configuration("config/config.yaml")
 
     def test_should_be_able_to_read__copy_job_result_check_countdown_in_sec(self):
-        self.assertEqual(self.configuration.copy_job_result_check_countdown_in_sec, 5)
+        self.__instance_of(self.configuration.copy_job_result_check_countdown_in_sec, int)
 
     def test_should_be_able_to_read__backup_worker_max_countdown_in_sec(self):
-        self.assertEqual(self.configuration.backup_worker_max_countdown_in_sec, 0)
+        self.__instance_of(self.configuration.backup_worker_max_countdown_in_sec, int)
 
     def test_should_be_able_to_read__backup_project_id(self):
-        self.assertEqual(self.configuration.backup_project_id, '<your-project-id>')
+        self.__instance_of(self.configuration.backup_project_id, str)
 
     def test_should_be_able_to_read__restoration_project_id(self):
-        self.assertEqual(self.configuration.restoration_project_id, '<your-project-id>')
+        self.__instance_of(self.configuration.restoration_project_id, str)
 
     def test_should_be_able_to_read__projects_to_skip(self):
-        self.assertEqual(self.configuration.projects_to_skip, ['<your-project-id>'])
+        self.__is_list_and_each_item_instance_of(self.configuration.projects_to_skip, str)
 
     def test_should_be_able_to_read__custom_project_list(self):
-        self.assertEqual(self.configuration.backup_settings_custom_project_list, [])
+        self.__is_list_and_each_item_instance_of(self.configuration.projects_to_skip, str)
+
+    def __instance_of(self, obj, expected_type):
+        self.assertTrue(isinstance(obj, expected_type))
+
+    def __is_list_and_each_item_instance_of(self, obj, expected_type):
+        self.__instance_of(obj, list)
+        for item in obj:
+            self.__instance_of(item, expected_type)