Feature/pm #4 (#8)

* PM #4 - The table TEST_METRICS now has a session field that links to a row in the (brand new) TEST_SESSIONS table. Global values are stored in that table (run date, scm reference, session description) in order to avoid repetitive information to be stored.
Documentation updated accordingly.

.circleci/config.yml

@@ -65,6 +65,10 @@ workflows:
                 filters:
                     tags:
                         only: /.*/
+            - pytestmonitor-py36-pytest54:
+                  filters:
+                      tags:
+                          only: /.*/
             - pytestmonitor-py37-pytest44:
                 filters:
                     tags:
@@ -93,6 +97,10 @@ workflows:
                 filters:
                     tags:
                         only: /.*/
+            - pytestmonitor-py37-pytest54:
+                  filters:
+                      tags:
+                          only: /.*/
             - pytestmonitor-py38-pytest46:
                 filters:
                     tags:
@@ -102,9 +110,13 @@ workflows:
                     tags:
                         only: /.*/
             - pytestmonitor-py38-pytest53:
-                filters:
-                    tags:
-                        only: /.*/
+              filters:
+                  tags:
+                      only: /.*/
+            - pytestmonitor-py38-pytest54:
+                  filters:
+                      tags:
+                          only: /.*/
             - deploy:
                 requires:
                     - pytestmonitor-py36-pytest44

docs/sources/changelog.rst

@@ -2,8 +2,10 @@
 Changelog
 =========
 
-* :release:`1.1.0 <2020-03-27>`
+* :release:`1.1.0 <2020-03-30>`
+* :feature:`5` Extend item information and separate item from its variants.
 * :feature:`3` Compute user time and kernel time on a per test basis for clarity and ease of exploitation.
+* :feature:`4` Added an option to add a description to a pytest run
 
 * :release:`1.0.1 <2020-03-18>`
 * :bug:`2` pytest-monitor hangs infinitely when a pytest outcome (skip, fail...) is issued.

docs/sources/configuration.rst

@@ -79,3 +79,14 @@ Disable monitoring
 If you need for some reason to disable the monitoring, pass the *\-\-no-trace* option.
 
 
+Adding a description to a run
+-----------------------------
+
+Sometimes, you might want to compare identical state of your code. In such cases, relying only on the scm
+references and the run date of the session. For that, `pytest-monitor` can assist you by adding a
+description field to your session.
+Setting a description is as simple as this:
+
+.. code-block:: shell
+
+    bash $> pytest --description "Any run description you want"

docs/sources/exploitation.rst

@@ -26,16 +26,19 @@ your Metrics and Execution Context (see below):
 
     pytest --remote server:port
 
-Execution Context and Metrics
------------------------------
+Execution Context, Metrics and Session
+--------------------------------------
 
 We distinguish two kinds of measures:
 
     - those related to the **Execution Context**. This is related to your machine (node name, CPU, memory…),
     - the **Metrics** related to the tests themselves (this can be the memory used, the CPU usage…).
 
-Each test is linked to an Execution Context so that comparisons between runs is possible. 
+Regarding tests related **metrics**, one can see metrics which are tests independent and those which
+are session independent (session start date, scm reference). For this reason, `pytest-monitor` uses
+a notion of session metrics to which each tests are linked to.
 
+Additionally, each test is linked to an Execution Context so that comparisons between runs is possible.
 
 
 Model
@@ -75,19 +78,32 @@ ENV_H (TEXT 64 CHAR)
 
 In the local database, Execution Contexts are stored in table `EXECUTION_CONTEXTS`.
 
+
+Sessions
+--------
+SESSION_H (TEXT 64 CHAR)
+    Hash string used to uniquely identify a session run.
+RUN_DATE (TEXT 64 CHAR)
+    Time at which the `pytest` session was started. The full format is
+    'YYYY-MM-DDTHH:MM:SS.uuuuuu' (ISO 8601 format with UTC time). The fractional second part is omitted if it is zero.
+SCM_ID (TEXT 128 CHAR)
+    Full reference to the source code management system if any.
+RUN_DESCRIPTION (TEXT 1024 CHAR)
+    A free text field that you can use to describe a session run.
+
+In the local database, Sessions are stored under the table `TEST_SESSIONS`.
+
+
 Metrics
 ~~~~~~~
 
 Metrics are collected at test, class and/or module level. For both classes and modules, some of the
 metrics can be skewed due to the technical limitations described earlier. 
 
-RUN_DATE (TEXT 64 CHAR)
-    Time at which the `pytest` session was started. The full format is
-    'YYYY-MM-DDTHH:MM:SS.uuuuuu' (ISO 8601 format with UTC time). The fractional second part is omitted if it is zero.
+SESSION_H (TEXT 64 CHAR)
+    Session context used for this test.
 ENV_H (TEXT 64 CHAR)
     Execution Context used for this test.
-SCM_ID (TEXT 128 CHAR)
-    Full reference to the source code management system if any.
 ITEM_START_TIME (TEXT 64 CHAR)
     Time at which the item test was launched. The full format is
     'YYYY-MM-DDTHH:MM:SS.uuuuuu' (ISO 8601 format with UTC time). The fractional second part is omitted if it is zero.

pytest_monitor/handler.py

@@ -5,20 +5,27 @@ class DBHandler:
     def __init__(self, db_path):
         self.__db = db_path
         self.__cnx = sqlite3.connect(self.__db) if db_path else None
+        self.prepare()
 
     def query(self, what, bind_to, many=False):
         cursor = self.__cnx.cursor()
         cursor.execute(what, bind_to)
         return cursor.fetchall() if many else cursor.fetchone()
 
-    def insert_metric(self, run_date, item_start_date, env_id, scm_id, item, item_path, item_variant,
+    def insert_session(self, h, run_date, scm_id, description):
+        with self.__cnx:
+            self.__cnx.execute(f'insert into TEST_SESSIONS(SESSION_H, RUN_DATE, SCM_ID, RUN_DESCRIPTION)'
+                               f' values (?,?,?,?)',
+                               (h, run_date, scm_id, description))
+
+    def insert_metric(self, session_id, env_id, item_start_date, item, item_path, item_variant,
                       item_loc, kind, component, total_time, user_time, kernel_time, cpu_usage, mem_usage):
         with self.__cnx:
-            self.__cnx.execute(f'insert into TEST_METRICS(RUN_DATE,ITEM_START_TIME,ENV_H,SCM_ID,ITEM,'
+            self.__cnx.execute(f'insert into TEST_METRICS(SESSION_H,ENV_H,ITEM_START_TIME,ITEM,'
                                f'ITEM_PATH,ITEM_VARIANT,ITEM_FS_LOC,KIND,COMPONENT,TOTAL_TIME,'
                                f'USER_TIME,KERNEL_TIME,CPU_USAGE,MEM_USAGE) '
-                               f'values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)',
-                               (run_date, item_start_date, env_id, scm_id, item, item_path,
+                               f'values (?,?,?,?,?,?,?,?,?,?,?,?,?,?)',
+                               (session_id, env_id, item_start_date, item, item_path,
                                 item_variant, item_loc, kind, component, total_time, user_time,
                                 kernel_time, cpu_usage, mem_usage))
 
@@ -35,10 +42,16 @@ def insert_execution_context(self, exc_context):
     def prepare(self):
         cursor = self.__cnx.cursor()
         cursor.execute('''
-CREATE TABLE IF NOT EXISTS TEST_METRICS (
+CREATE TABLE IF NOT EXISTS TEST_SESSIONS(
+    SESSION_H varchar(64) primary key not null unique, -- Session identifier
     RUN_DATE varchar(64), -- Date of test run
+    SCM_ID varchar(128), -- SCM change id 
+    RUN_DESCRIPTION varchar(1024)
+);''')
+        cursor.execute('''
+CREATE TABLE IF NOT EXISTS TEST_METRICS (
+    SESSION_H varchar(64), -- Session identifier
     ENV_H varchar(64), -- Environment description identifier
-    SCM_ID varchar(128),
     ITEM_START_TIME varchar(64), -- Effective start time of the test
     ITEM_PATH varchar(4096), -- Path of the item, following Python import specification
     ITEM varchar(2048), -- Name of the item
@@ -51,7 +64,8 @@ def prepare(self):
     KERNEL_TIME float, -- time spent in kernel space
     CPU_USAGE float, -- cpu usage
     MEM_USAGE float, -- Max resident memory used.
-    FOREIGN KEY (ENV_H) REFERENCES EXECUTION_CONTEXTS(ENV_H)
+    FOREIGN KEY (ENV_H) REFERENCES EXECUTION_CONTEXTS(ENV_H),
+    FOREIGN KEY (SESSION_H) REFERENCES TEST_SESSIONS(SESSION_H)
 );''')
         cursor.execute('''
 CREATE TABLE IF NOT EXISTS EXECUTION_CONTEXTS (

pytest_monitor/pytest_monitor.py

@@ -1,11 +1,9 @@
-import os
 # -*- coding: utf-8 -*-
 import memory_profiler
 import pytest
 import time
 import warnings
 
-from pytest_monitor.sys_utils import ExecutionContext
 from pytest_monitor.session import PyTestMonitorSession
 
 # These dictionaries are used to compute members set on each items.
@@ -44,6 +42,8 @@ def pytest_addoption(parser):
     group.addoption('--component-prefix', action='store',
                     help='Prefix each found components with the given value (applies to all tests'
                          ' run in this session).')
+    group.addoption('--description', action='store', default='',
+                    help='Use this option to provide a small summary about this run.')
 
 
 def pytest_configure(config):
@@ -165,8 +165,10 @@ def pytest_sessionstart(session):
         component = '{user_component}'
     db = None if (session.config.option.mtr_none or session.config.option.no_db) else session.config.option.mtr_db_out
     remote = None if session.config.option.mtr_none else session.config.option.remote
-    session.pytest_monitor = PyTestMonitorSession(db=db, remote=remote, component=component)
-    session.pytest_monitor.set_environment_info(ExecutionContext())
+    session.pytest_monitor = PyTestMonitorSession(db=db, remote=remote,
+                                                  component=component,
+                                                  scope=session.config.option.mtr_scope)
+    session.pytest_monitor.compute_info(session.config.option.description)
     yield
 
 
@@ -183,7 +185,7 @@ def prf_module_tracer(request):
     pypath = request.module.__name__[:-len(item)-1]
     request.session.pytest_monitor.add_test_info(item, pypath, '',
                                                  request.node._nodeid,
-                                                 'module', request.config.option.mtr_scope,
+                                                 'module',
                                                  component, t_a, t_z - t_a,
                                                  ptimes_b.user - ptimes_a.user,
                                                  ptimes_b.system - ptimes_a.system,
@@ -200,7 +202,7 @@ def prf_tracer(request):
         item_loc = getattr(request.node, PYTEST_MONITOR_ITEM_LOC_MEMBER)[0]
         request.session.pytest_monitor.add_test_info(item_name, request.module.__name__,
                                                      request.node.name, item_loc,
-                                                     'function', request.config.option.mtr_scope,
+                                                     'function',
                                                      request.node.monitor_component,
                                                      request.node.test_effective_start_time,
                                                      request.node.test_run_duration,

pytest_monitor/session.py

@@ -1,26 +1,26 @@
 import datetime
+import hashlib
 import json
 import memory_profiler
 import os
 import psutil
 import requests
-import subprocess
 import warnings
 
 from pytest_monitor.handler import DBHandler
+from pytest_monitor.sys_utils import ExecutionContext, determine_scm_revision
 
 
 class PyTestMonitorSession(object):
-    def __init__(self, db=None, remote=None, component=''):
-        self.__run_date = datetime.datetime.now().isoformat()
+    def __init__(self, db=None, remote=None, component='', scope=None):
         self.__db = DBHandler(db) if db else None
         self.__remote = remote
         self.__component = component
-        self.__scm = ''
+        self.__session = ''
+        self.__scope = scope or []
         self.__eid = (None, None)
         self.__mem_usage_base = None
         self.__process = psutil.Process(os.getpid())
-        self.prepare()
 
     @property
     def remote_env_id(self):
@@ -50,13 +50,21 @@ def get_env_id(self, env):
                     remote = None
         return db, remote
 
-    def determine_scm_revision(self):
-        for cmd in [r'git rev-parse HEAD', r'p4 changes -m1 \#have']:
-            p = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
-            p_out, _ = p.communicate()
-            if p.returncode == 0:
-                self.__scm = p_out.decode().split('\n')[0]
-                return
+    def compute_info(self, description):
+        run_date = datetime.datetime.now().isoformat()
+        scm = determine_scm_revision()
+        h = hashlib.md5()
+        h.update(scm.encode())
+        h.update(run_date.encode())
+        h.update(description.encode())
+        self.__session = h.hexdigest()
+        # Now get memory usage base and create the database
+        self.prepare()
+        self.set_environment_info(ExecutionContext())
+        if self.__db:
+            self.__db.insert_session(self.__session, run_date, scm, description)
+        if self.__remote:
+            warnings.warn('todo')
 
     def set_environment_info(self, env):
         self.__eid = self.get_env_id(env)
@@ -73,19 +81,16 @@ def set_environment_info(self, env):
             else:
                 remote_id = json.loads(r.text)['h']
         self.__eid = db_id, remote_id
-        self.determine_scm_revision()
 
     def prepare(self):
         def dummy():
             return True
 
         self.__mem_usage_base = memory_profiler.memory_usage((dummy,), max_usage=True)
-        if self.__db:
-            self.__db.prepare()
 
-    def add_test_info(self, item, item_path, item_variant, item_loc, kind, allowed_scope, component,
+    def add_test_info(self, item, item_path, item_variant, item_loc, kind, component,
                       item_start_time, total_time, user_time, kernel_time, mem_usage):
-        if kind not in allowed_scope:
+        if kind not in self.__scope:
             return
         mem_usage = float(mem_usage) - self.__mem_usage_base
         cpu_usage = (user_time + kernel_time) / total_time
@@ -95,12 +100,12 @@ def add_test_info(self, item, item_path, item_variant, item_loc, kind, allowed_s
             final_component = final_component[:-1]
         item_variant = item_variant.replace('-', ', ')  # No choice
         if self.__db and self.db_env_id is not None:
-            self.__db.insert_metric(self.__run_date, item_start_time, self.db_env_id, self.__scm, item,
+            self.__db.insert_metric(self.__session, self.db_env_id, item_start_time, item,
                                     item_path, item_variant, item_loc, kind, final_component, total_time, user_time,
                                     kernel_time, cpu_usage, mem_usage)
         if self.__remote and self.remote_env_id is not None:
             r = requests.post(f'{self.__remote}/metrics/',
-                              json=dict(run_date=self.__run_date, context_h=self.remote_env_id, scm_ref=self.__scm,
+                              json=dict(context_h=self.remote_env_id,
                                         item=item, kind=kind, component=final_component, total_time=total_time,
                                         item_start_time=item_start_time, user_time=user_time,
                                         kernel_time=kernel_time,

pytest_monitor/sys_utils.py

@@ -8,6 +8,15 @@
 import sys
 
 
+def determine_scm_revision():
+    for cmd in [r'git rev-parse HEAD', r'p4 changes -m1 \#have']:
+        p = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
+        p_out, _ = p.communicate()
+        if p.returncode == 0:
+            return p_out.decode().split('\n')[0]
+    return ''
+
+
 def _get_cpu_string():
     if platform.system().lower() == "darwin":
         old_path = os.environ['PATH']