Merge 2760a20 into 9868567

README.rst

@@ -19,72 +19,74 @@ develop:
     :target: https://coveralls.io/github/unfoldingWord-dev/tx-manager?branch=develop
 
 
+tX Overview
+===========
 
-**NOTE: High level Architecture documentation is here\:** `tX Architecture`_.
-
-
-
-tx-manager
-==========
-
-Project description at `tX Manager Module`_.
+tX (translationConverter) is a conversion tool for the content in the `Door43 Conent Service (DCS) <https://git.door43.org/>`__. The goal is to support several different input formats, output formats, and resource types.
 
-Issue for its creation at https://github.com/unfoldingWord-dev/door43.org/issues/53.
+See the documentation at https://tx-manager.readthedocs.io/en/latest.
 
+Issue queue maintained at https://github.com/unfoldingWord-dev/door43.org/issues.
 
 Setting Up a Developer Environment (CLI Version)
-================================================
+------------------------------------------------
 
 Satisfy basic depedencies:
 
-.. highlight:: bash
+.. code-block:: bash
 
     git clone git@github.com:unfoldingWord-dev/tx-manager.git
     sudo apt-get install libssl1.0.0 python-pip
     pip install virtualenv
 
 We recommend you create a Python virtual environment to help manage Python package dependencies:
 
-.. highlight:: bash
+.. code-block:: bash
 
     cd tx-manager
     virtualenv venv
 
 Now load that virtual environment and install dependencies:
 
-.. highlight:: bash
+.. code-block:: bash
 
     source venv/bin/activate
     pip install -r requirements.txt
 
 Set AWS variables:
 
-.. highlight:: bash
+.. code-block:: bash
 
     export AWS_DEFAULT_REGION=us-west-2
     export AWS_REGION=us-west-2
 
 Run the test suite:
 
-.. highlight:: bash
+.. code-block:: bash
 
     python test-setup.py test
 
+Or, to run a single test, run:
+
+.. code-block:: bash
+
+    python -m unittest tests.client_tests.test_client_webhook
+
 Optionally to do Integration tests on 'test' site, first deploy tx-manager to test:
 
-.. highlight:: bash
+.. code-block:: bash
 
     apex deploy -p test -e test
 
 Now you can run the integration test(s) (see run_integration_tests for setup steps and help):
 
-.. highlight:: bash
+.. code-block:: bash
 
     ./scripts/run_integration_tests.sh test_ts_mat_conversion
 
 
 tX Pipeline
-===========
+-----------
 
 1. Gogs
 2. Webhook
@@ -95,15 +97,16 @@ tX Pipeline
 7. Door43 Deploy
 
 Definitions
-===========
+-----------
+
 The following placeholders are used in examples in this document:
 
 * <repo> - the machine name of a Gog's repo. This is used in the URL for the repo, such as en-obs
 * <user> - the user or organization that the Gog's repo belongs to, such as richmahn (user) or door43 (org)
 * <commit> - The 10 character hash string that represents the commit (revision) that is being processed
 
 How it Works
-============
+------------
 
 Request Conversion Job
 ----------------------
@@ -194,23 +197,19 @@ the following variables:
 See `tx-md2html_register Lambda function <https://github.com/unfoldingWord-dev/tx-md2html/blob/develop/functions/register/main.py>`_. for an example of a module registering itself.
 
 
-.. _tX Architecture:
-
-.. include:: README-tXArchitecture.rst
-
-
 Setting up as deployed in virtual environment
-=============================================
+---------------------------------------------
 
 In IntelliJ terminal, switch to virtual environment and install requirements.
 
-.. highlight:: bash
+.. code-block:: bash
 
     source ~/venv/txml/bin/activate
     ./install-requirements.sh
 
 Deploying your branch of tx-manager to AWS
-==========================================
+------------------------------------------
+
 For developing the tx-manager library which this repo uses for every function, you can deploy your code to a test AWS
 environment with apex by doing the following:
 

README-tXArchitecture.rst → docs/architecture.rst

@@ -1,5 +1,5 @@
-tX Development Architecture
-===========================
+tX Architecture
+===============
 
 This document explains the layout of the translationConvertor (tX)
 conversion platform and how the components of the system should interact
@@ -63,6 +63,7 @@ Modules MUST all present an API endpoint that the other components of
 the system can use. Modules MAY present API endpoints that the public
 can use.
 
+Background: Issue for tX creation at https://github.com/unfoldingWord-dev/door43.org/issues/53.
 
 Separating Production from Development and Test
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

docs/index.rst

@@ -1,16 +1,15 @@
-tx-manager documentation!
-=========================
+tX Documentation
+================
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents:
 
    readme
+   architecture
    license
 
 Indices and tables
 ==================
 
 * :ref:`genindex`
 * :ref:`modindex`
-* :ref:`search`

libraries/client/client_webhook.py

@@ -43,23 +43,34 @@ def process_webhook(self):
 
         # Check that the user token is valid
         if not App.gogs_user_token:
-            raise Exception('DSC user token not given in Payload.')
+            raise Exception('DCS user token not given in Payload.')
         user = App.gogs_handler().get_user(App.gogs_user_token)
         if not user:
             raise Exception('Invalid DCS user token given in Payload')
 
-        # Get the commit_id, and see if there are multiple commits to this webhook payload
+        # Check that the URL to the DCS repo is valid
+        if not self.commit_data['repository']['html_url'].startswith(App.gogs_url):
+            raise Exception('Repos can only belong to {0} to use this webhook client.'.format(App.gogs_url))
+
+        # Check that commit is on repo's default branch, else quit
+        try:
+            commit_branch = self.commit_data['ref'].split('/')[2]
+        except IndexError:
+            raise Exception('Could not determine commit branch, exiting.')
+        except KeyError:
+            Exception('This does not appear to be a push, exiting.')
+        if commit_branch != self.commit_data['repository']['default_branch']:
+            raise Exception('Commit branch: {0} is not the default branch, exiting.'.format(commit_branch))
+
+        # Get the commit_id, commit_url
         commit_id = self.commit_data['after']
         commit = None
         for commit in self.commit_data['commits']:
             if commit['id'] == commit_id:
                 break
         commit_id = commit_id[:10]  # Only use the short form
-
-        # Check that the URL to the DCS repo is valid
         commit_url = commit['url']
-        if not commit_url.startswith(App.gogs_url):
-            raise Exception('Repos can only belong to {0} to use this webhook client.'.format(App.gogs_url))
+
 
         # Gather other details from the commit that we will note for the job(s)
         user_name = self.commit_data['repository']['owner']['username']

tests/client_tests/test_client_webhook.py

@@ -116,6 +116,12 @@ def test_process_webhook(self, mock_download_file):
         self.assertEqual(tx_manifest.lang_code, 'kpb')
         self.assertEqual(tx_manifest.id, tx_job.manifests_id)
 
+    @mock.patch('libraries.client.client_webhook.download_file')
+    def test_process_webhook_wrong_branch(self, mock_download_file):
+        # Verifies that webhook bails if we are not on default_branch for the repo
+        self.assertRaises(Exception, self.setup_client_webhook_mock(
+            'kpb_mat_text_udb_repo', mock_download_file, 'refs/heads/test_branch'))
+
     @mock.patch('libraries.client.client_webhook.download_file')
     def test_process_webhook_no_converter_error(self, mock_download_file):
         # given
@@ -255,12 +261,12 @@ def get_last_uploaded_file(self, match):
                 return upload['file']
         return None
 
-    def setup_client_webhook_mock(self, repo_name, mock_download_file):
-        App.gogs_url = self.resources_dir
+    def setup_client_webhook_mock(self, repo_name, mock_download_file, ref='refs/heads/master'):
+        App.gogs_url = 'https://git.door43.org'
         App.gogs_user_token = mock_utils.valid_token
         mock_download_file.side_effect = self.mock_download_file
         source = os.path.join(self.resources_dir, repo_name)
-        commit_data = self.get_commit_data(source)
+        commit_data = self.get_commit_data(source, ref)
         self.cwh = ClientWebhook(commit_data)
         self.cwh.send_payload_to_converter = self.mock_send_payload_to_converter
         self.cwh.send_payload_to_linter = self.mock_send_payload_to_linter
@@ -298,7 +304,7 @@ def mock_cdn_get_json(self, project_json_key):
     def mock_clear_commit_directory_in_cdn(self, s3_results_key):
         return
 
-    def get_commit_data(self, source_path):
+    def get_commit_data(self, source_path, ref):
         base_url = 'https://git.door43.org'
         commit_id = '22f3d09f7a33d2496db6993648f0cd967a9006f6'
         commit_path = '/tx-manager-test-data/en-ulb/commit/22f3d09f7a33d2496db6993648f0cd967a9006f6'
@@ -310,6 +316,7 @@ def get_commit_data(self, source_path):
 
         commit_data = {
             'after': commit_id,
+            'ref': ref,
             'commits': [
                 {
                     'id': commit_id,
@@ -319,6 +326,8 @@ def get_commit_data(self, source_path):
             'compare_url': '',
             'repository': {
                 'name': repo,
+                'html_url': 'https://git.door43.org/unfoldingWord/en_ulb',
+                'default_branch': 'master',
                 'owner': {
                     'id': '1234567890',
                     'username': user,