Merge 4b359f6 into 1a78bce

.github/workflows/ci.yml

@@ -18,7 +18,7 @@ jobs:
     name: Python ${{ matrix.python-version }} for ES ${{ matrix.elasticsearch-version }}
     strategy:
       matrix:
-        python-version: [3.6, 3.7, 3.8]
+        python-version: [3.7, 3.8]
         elasticsearch-version: [6.8.6]
 
     steps:
@@ -44,9 +44,10 @@ jobs:
 
     - name: Install dependencies
       run: |
-        pip install --upgrade pip==18.1
-        pip install --upgrade setuptools==49.6.0
+        pip install --upgrade pip
+        pip install --upgrade setuptools
         pip install --upgrade wheel
+        pip install "numpy<=1.18.3"
         pip install -r "requirements.txt"
         pip install -r "requirements_tests.txt"
         pip install flake8 coveralls

Getting-Started.md

@@ -182,7 +182,7 @@ The final results should be something similar to the image below.
 
 Now that you have the ElasticSearch, Kibiter and MariaDB running on your system and the project configured in the PyCharm, we can execute micro-mordred/sirmordred. 
 
-To execute micro-mordred, define a [setup.cfg](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/utils/setup.cfg) and [projects.json](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/utils/projects.json), and
+To execute micro-mordred, define a [setup.cfg](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/setup.cfg) and [projects.json](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/projects.json), and
 run the following commands, which will collect and enrich the data coming from the git sections and upload the corresponding panels to Kibiter:
 ```
 micro.py --raw --enrich --cfg ./setup.cfg --backends git cocom
@@ -286,7 +286,7 @@ Following is a list of common problems encountered while setting up GrimoireLab
 #### Empty Index [↑](#troubleshooting-)
 
 * Indications and Diagnosis:
-  Check for the following error after executing [Micro Mordred](https://github.com/chaoss/grimoirelab-sirmordred/tree/master/utils/micro.py)
+  Check for the following error after executing [Micro Mordred](https://github.com/chaoss/grimoirelab-sirmordred/tree/master/sirmordred/utils/micro.py)
   using ```micro.py --raw --enrich --panels --cfg ./setup.cfg --backends git```(Here, using git as backend)
   ```
   [git] Problem executing study enrich_areas_of_code:git, RequestError(400, 'search_phase_execution_exception', 'No mapping 
@@ -299,7 +299,7 @@ Following is a list of common problems encountered while setting up GrimoireLab
 
   There are 2 methods to solve this problem:
 
-  Method 1: Disable the param [latest-items](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/utils/setup.cfg#L78) by setting it to false.
+  Method 1: Disable the param [latest-items](https://github.com/chaoss/grimoirelab-sirmordred/blob/master/sirmordred/utils/setup.cfg#L78) by setting it to false.
 
   Method 2: Delete the local clone of the repo (which is stored in ```~/.perceval/repositories```).
 

README.md

@@ -1434,7 +1434,7 @@ studies = [enrich_demography:weblate] (optional)
 
 Micro Mordred is a simplified version of Mordred which omits the use of its scheduler. Thus, Micro Mordred allows running single Mordred tasks (e.g., raw collection, enrichment) per execution.
 
-Micro Mordred is located in the [/utils](https://github.com/chaoss/grimoirelab-sirmordred/tree/master/utils/micro.py) folder of this same repository. It can be executed via command line, its parameters are summarized below:
+Micro Mordred is located in the [sirmordred/utils](https://github.com/chaoss/grimoirelab-sirmordred/tree/master/sirmordred/utils/micro.py) folder of this same repository. It can be executed via command line, its parameters are summarized below:
 ```
 --debug: execute Micro Mordred in debug mode
 
@@ -1455,7 +1455,7 @@ Micro Mordred is located in the [/utils](https://github.com/chaoss/grimoirelab-s
 
 Examples of possible executions are shown below:
 ```
-cd .../grimoirelab-sirmordred/utils/
+cd .../grimoirelab-sirmordred/sirmordred/utils/
 micro.py --raw --enrich --cfg ./setup.cfg --backends git # execute the Raw and Enrich tasks for the Git cfg section
 micro.py --panels # execute the Panels task to load the Sigils panels to Kibiter
 micro.py --raw --enrich --debug --cfg ./setup.cfg --backends groupsio --logs-dir logs # execute the raw and enriched tasks for the groupsio cfg section with debug mode on and logs being saved in the folder logs in the same directory as micro.py

pyproject.toml

@@ -0,0 +1,67 @@
+[tool.poetry]
+name = "sirmordred"
+version = "0.2.43"
+description = "Drive GrimoireLab tools to produce a dashboard"
+authors = [
+    "GrimoireLab Developers"
+]
+license = "GPL-3.0+"
+
+readme = "README.md"
+
+homepage = "https://chaoss.github.io/grimoirelab/"
+repository = "https://github.com/chaoss/grimoirelab-sirmordred"
+
+keywords = [
+    "development",
+    "grimoirelab"
+]
+
+packages = [
+    { include = "sirmordred" },
+    { include = "tests", format = "sdist" },
+]
+
+classifiers = [
+   "Development Status :: 4 - Beta",
+   "Intended Audience :: Developers",
+   "Intended Audience :: Science/Research",
+   "Topic :: Software Development",
+   "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
+   "Programming Language :: Python :: 3"
+]
+
+[tool.poetry.urls]
+"Bug Tracker" = "https://github.com/chaoss/grimoirelab-sirmordred/issues"
+
+[tool.poetry.scripts]
+'sirmordred' = 'sirmordred.bin.sirmordred:main'
+'healthcheck.py' = 'sirmordred.utils.healthcheck:main'
+'micro.py' = 'sirmordred.utils.micro:main'
+'panels_config.py' = 'sirmordred.utils.panels_config:main'
+
+[tool.poetry.dependencies]
+python = "^3.7"
+
+colorlog = "4.1.0"
+elasticsearch = "6.3.1"
+elasticsearch-dsl = "6.3.1"
+file-read-backwards = "2.0.0"
+grimoirelab-toolkit = {git = "https://github.com/chaoss/grimoirelab-toolkit"}
+sortinghat = {git = "https://github.com/chaoss/grimoirelab-sortinghat"}
+kidash = {git = "https://github.com/chaoss/grimoirelab-kidash"}
+grimoireelk = {git = "https://github.com/chaoss/grimoirelab-elk"}
+perceval = {git = "https://github.com/chaoss/grimoirelab-perceval"}
+perceval-puppet = {git = "https://github.com/chaoss/grimoirelab-perceval-puppet"}
+perceval-opnfv = {git = "https://github.com/chaoss/grimoirelab-perceval-opnfv"}
+perceval-mozilla = {git = "https://github.com/chaoss/grimoirelab-perceval-mozilla"}
+perceval-finos = {git = "https://github.com/Bitergia/grimoirelab-perceval-finos"}
+perceval-weblate = {git = "https://github.com/chaoss/grimoirelab-perceval-weblate"}
+grimoirelab-panels = {git = "https://github.com/chaoss/grimoirelab-sigils"}
+
+[tool.poetry.dev-dependencies]
+httpretty = "^1.1.4"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"

requirements.txt

@@ -2,15 +2,15 @@ colorlog==4.1.0
 elasticsearch==6.3.1
 elasticsearch-dsl==6.3.1
 file-read-backwards==2.0.0
--e git+https://github.com/chaoss/grimoirelab-toolkit/#egg=grimoirelab-toolkit
--e git+https://github.com/chaoss/grimoirelab-sortinghat/#egg=grimoirelab-sortinghat
--e git+https://github.com/chaoss/grimoirelab-kidash/#egg=grimoirelab-kidash
--e git+https://github.com/chaoss/grimoirelab-elk/#egg=grimoirelab-elk
--e git+https://github.com/chaoss/grimoirelab-graal/#egg=grimoirelab-graal
--e git+https://github.com/chaoss/grimoirelab-perceval/#egg=grimoirelab-perceval
--e git+https://github.com/chaoss/grimoirelab-perceval-puppet/#egg=grimoirelab-perceval-puppet
--e git+https://github.com/chaoss/grimoirelab-perceval-opnfv/#egg=grimoirelab-perceval-opnfv
--e git+https://github.com/chaoss/grimoirelab-perceval-mozilla/#egg=grimoirelab-perceval-mozilla
--e git+https://github.com/Bitergia/grimoirelab-perceval-finos/#egg=grimoirelab-perceval-finos
--e git+https://github.com/chaoss/grimoirelab-perceval-weblate/#egg=grimoirelab-perceval-weblate
--e git+https://github.com/chaoss/grimoirelab-sigils/#egg=grimoirelab-sigils
+grimoirelab-toolkit @ git+https://github.com/chaoss/grimoirelab-toolkit@master
+sortinghat @ git+https://github.com/chaoss/grimoirelab-sortinghat@master
+kidash @ git+https://github.com/chaoss/grimoirelab-kidash@master
+grimoireelk @ git+https://github.com/chaoss/grimoirelab-elk@master
+graal @ git+https://github.com/chaoss/grimoirelab-graal@master
+perceval @ git+https://github.com/chaoss/grimoirelab-perceval@master
+perceval-puppet @ git+https://github.com/chaoss/grimoirelab-perceval-puppet@master
+perceval-opnfv @ git+https://github.com/chaoss/grimoirelab-perceval-opnfv@master
+perceval-mozilla @ git+https://github.com/chaoss/grimoirelab-perceval-mozilla@master
+perceval-finos @ git+https://github.com/Bitergia/grimoirelab-perceval-finos@master
+perceval-weblate @ git+https://github.com/chaoss/grimoirelab-perceval-weblate@master
+grimoirelab-panels @ git+https://github.com/chaoss/grimoirelab-sigils@master

setup.py

@@ -58,7 +58,9 @@
       ],
       keywords="development repositories analytics git github bugzilla jira jenkins",
       packages=[
-          'sirmordred'
+          'sirmordred',
+          'sirmordred.bin',
+          'sirmordred.utils',
       ],
       install_requires=[
           'grimoire-elk>=0.32',
@@ -70,10 +72,12 @@
           'file-read-backwards==2.0.0',
           'colorlog==4.1.0'
       ],
-      scripts=[
-          'bin/sirmordred',
-          'utils/healthcheck.py',
-          'utils/micro.py',
-          'utils/panels_config.py'
-      ],
+      entry_points={
+          'console_scripts': [
+              'sirmordred=sirmordred.bin.sirmordred:main',
+              'healthcheck.py=sirmordred.utils.healthcheck:main',
+              'micro.py=sirmordred.utils.micro:main',
+              'panels_config.py=sirmordred.utils.panels_config:main',
+          ]
+      },
       zip_safe=False)

bin/sirmordred → sirmordred/bin/sirmordred.py

@@ -43,7 +43,39 @@
 SLEEPFOR_ERROR = """Error: You may be Arthur, King of the Britons. But you still """ + \
                  """need the 'sleep_for' variable in sortinghat section\n - Sir Mordred said."""
 
-logger = logging.getLogger(__name__)
+Logger = logging.getLogger(__name__)
+
+
+def main():
+    args = parse_args()
+
+    if args.config_template_file is not None:
+        Config.create_config_file(args.config_template_file)
+        Logger.info("Sample config file created in {}".format(args.config_template_file))
+        return 0
+    elif args.config_files is None:
+        Logger.error("Option -t or -c is required")
+        return 1
+
+    try:
+        config = Config(args.config_files[0], args.config_files[1:])
+        config_dict = config.get_conf()
+        logs_dir = config_dict['general']['logs_dir']
+        debug_mode = config_dict['general']['debug']
+        logger = setup_logs(logs_dir, debug_mode)
+    except RuntimeError as error:
+        print("Error while consuming configuration: {}".format(error))
+        return 1
+
+    if args.phases:
+        logger.info("Executing sirmordred for phases: {}".format(args.phases))
+        # HACK: the internal dict of Config is modified directly
+        # In manual phases execute sirmordred as an script
+        config_dict['general']['update'] = False
+        for phase in config_dict['phases']:
+            config_dict['phases'][phase] = True if phase in args.phases else False
+
+    SirMordred(config).start()
 
 
 def setup_logs(logs_dir, debug_mode):
@@ -101,31 +133,14 @@ def parse_args():
 
 
 if __name__ == '__main__':
-    args = parse_args()
-    if args.config_template_file is not None:
-        Config.create_config_file(args.config_template_file)
-        logger.info("Sample config file created in {}".format(args.config_template_file))
-        sys.exit(0)
-    elif args.config_files is None:
-        logger.error("Option -t or -c is required")
-        sys.exit(1)
-
     try:
-        config = Config(args.config_files[0], args.config_files[1:])
-        config_dict = config.get_conf()
-        logs_dir = config_dict['general']['logs_dir']
-        debug_mode = config_dict['general']['debug']
-        logger = setup_logs(logs_dir, debug_mode)
-    except RuntimeError as error:
-        print("Error while consuming configuration: {}".format(error))
+        code = main()
+        sys.exit(code)
+    except KeyboardInterrupt:
+        s = "\n\nReceived Ctrl-C or other break signal. Exiting.\n"
+        sys.stderr.write(s)
+        sys.exit(0)
+    except RuntimeError as e:
+        s = "Error: %s\n" % str(e)
+        sys.stderr.write(s)
         sys.exit(1)
-
-    if args.phases:
-        logger.info("Executing sirmordred for phases: {}".format(args.phases))
-        # HACK: the internal dict of Config is modified directly
-        # In manual phases execute sirmordred as an script
-        config_dict['general']['update'] = False
-        for phase in config_dict['phases']:
-            config_dict['phases'][phase] = True if phase in args.phases else False
-
-    SirMordred(config).start()

utils/micro.py → sirmordred/utils/micro.py

@@ -40,6 +40,17 @@
 LOG_COLORS = {'DEBUG': 'white', 'INFO': 'cyan', 'WARNING': 'yellow', 'ERROR': 'red', 'CRITICAL': 'red,bg_white'}
 
 
+def main():
+    args = get_params()
+    config_logging(args.debug, args.logs_dir)
+    micro_mordred(args.cfg_path, args.backend_sections,
+                  args.repos_to_check, args.raw,
+                  args.identities_load,
+                  args.identities_merge,
+                  args.enrich,
+                  args.panels)
+
+
 def micro_mordred(cfg_path, backend_sections, repos_to_check, raw, identities_load, identities_merge, enrich, panels):
     """Execute the Mordred tasks using the configuration file (`cfg_path`).
 
@@ -238,11 +249,4 @@ def get_params():
 
 
 if __name__ == '__main__':
-    args = get_params()
-    config_logging(args.debug, args.logs_dir)
-    micro_mordred(args.cfg_path, args.backend_sections,
-                  args.repos_to_check, args.raw,
-                  args.identities_load,
-                  args.identities_merge,
-                  args.enrich,
-                  args.panels)
+    main()

tests/test_healthcheck.py

@@ -20,17 +20,12 @@
 #     Luis Cañas-Díaz <lcanas@bitergia.com>
 
 
-import sys
 import unittest
 
 from datetime import datetime
 from unittest.mock import patch, mock_open
 
-# Hack to make sure that tests import the right packages
-# due to setuptools behaviour
-sys.path.insert(0, '..')
-
-from utils.healthcheck import match_error_string, read_cache_file
+from sirmordred.utils.healthcheck import match_error_string, read_cache_file
 
 
 class TestHealthCheck(unittest.TestCase):