updates from paper analyses for HIV, TB and malaria (#1273)

* updates from paper analyses for HIV, TB and malaria

* remove unused import statements

* fix imports

* update filepath for malaria resource file

* remove test_hiv_tb_scenarios.py

* updated test_healthsystem.py: test_manipulation_of_service_availability as small population size over 7 days will not schedule VMMC through HIV module. Remove HIV_Prevention_Circumcision in assert statement for HIV services delivered in one week

* change ipt_coverage in TB logger as conflicts with existing parameter

* updated ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx with updated NTP2019 TB data

* remove test code

* remove test code

* delete tmp resourcefiles

* malaria code use person_id consistently instead of individual_id

* use individual_id for demography.do_death()

* style change to avoid conflicts with master

* style change to avoid conflicts with master

* fix conflicts with master

* fix conflicts with master

* merge in master

* check property hv_date_last_ART correctly set

* Manually add PostnatalCare_Comprehensive to policy priorities

* edit fix

* add schisto high infection as conditional predictor for bladder cancer

* fix conditional predictor for active TB - should check presence of CardioMetabolicDisorders

* add 'ss' prefix to properties from schisto module referenced in bladder_cancer.py

* edit praziquantel code in schisto.py to use value from CMST in place of donated

* add parameter rr_depr_hiv for risk of depression with HIV infection

* tidy up linear models in depression, include conditional predictors for hiv infection and add comments

* move hv_inf into conditional predictor for depression in initial population

* convert lm for incident cancer (site_confined) to model with conditional predictors. Include HIV as risk factor.

* add parameter rr_site_confined_hiv to other_adult_cancers.py

* update other_adult_cancers write-up to include HIV as risk factor

* update Depression.docx to include HIV as risk factor for depression

* edit HIV in depression to include only HIV cases not virally suppressed

* update other_adult_cancers.py linear model to include HIV as risk factor only if not virally suppressed

* edit: HIV remains as risk factor for depression independent of treatment status

* include HIV as risk factor for low grade dysplasia (oesophageal cancer). Update ResourceFile_Oesophageal_Cancer.xlsx

* update linear model for low grade dysplasia to include HIV as conditional risk factor

* update OesophagealCancer.docx write-up to include HIV risk

* add condition hiv diagnosed for increased risk of depression

* remove hiv as risk factor for oesophageal cancer

* remove parameter for hiv as risk factor for oesophageal cancer

* update OesophagealCancer.docx to remove hiv as risk factor

* update value for weighted risk of other_adult_cancers with unsuppressed HIV

* add rr_hiv to linear model. update ResourceFile_cmd_condition_onset.xlsx with rr_hiv, leave value=1.0 if no effect of hiv

* update resourcefiles for CMD include rr_hiv for all, no effect of majority of processes

* add diabetes as risk for active TB and relapse. add params to ResourceFile_TB.xlsx
replace linearmodels dict which was accidentally removed in depression.py

* add diabetes as risk factor for tb death

* add diabetes as risk factor for PLHIV with active TB and on TB treatment

* add diabetes as risk factor for PLHIV with active TB and on TB treatment

* set up run to check calibration of deaths and disability

* add predictor high-intensity S. haematobium infection to risk of bladder cancer in initial population

* add predictor high-intensity S. haematobium infection to risk of HIV acquisition

* fix indenting in HSI_Hiv_StartOrContinueTreatment

* add hv_date_treated abd hv_date_last_ART to baseline_art

* convert linear model in CMD to include conditional predictors

* delete resourcefile created in error

* comment out path-specific changes to analysis_cause_of_death_and_disability_calibrations.py

* fix CMD error if Hiv not registered

* remove parameter rr_bcg_inf from tb.py

* edit comment in initialise_simulation

* fix parameter name error

* update parameters

* test runs

* edit and fix flake8 errors

* fix failing test

* update ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx

* update ResourceFile_PriorityRanking_ALLPOLICIES.xlsx

* updated ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx

* edit check for last ART when new dispensation occurs

* update schisto risk on HIV to only include women

* Update tests/test_healthsystem.py

Co-authored-by: Tim Hallett <39991060+tbhallett@users.noreply.github.com>

* address comments in PR

* address comments in PR

* check changes in PR and rollback

* check changes in PR and rollback

* fix failing tests

* merge in latest files from master

* merge in latest files from master

* baseline AIDS/death scheduling should happen for anyone on ART and not VL suppressed

* fix failing tests in test_hiv.py

* check tests_hiv passing locally

* update enhanced_lifestyle.py due to failing test

* remove json file committed accidentally

* roll back incidental changes in test_healthsystem.py

* roll back inicdental changes in malaria

* roll back incidental changes in test_healthsystem.py

* roll back incidental changes

* Update src/tlo/methods/hiv.py

Co-authored-by: Tim Hallett <39991060+tbhallett@users.noreply.github.com>

* check the usage of '' versus ""
address comments in PR #1273 review

* set VMMC HSI to level 1b otherwise consumables never available

* check events using isinstance not string check (string could change)

* linting

* roll back incidental (formatting) changes in malaria.py

* reinstate checks to avoid division by zero

* isort

---------

Co-authored-by: Tim Hallett <39991060+tbhallett@users.noreply.github.com>

src/scripts/hiv/projections_jan2023/analysis_full_model.py

@@ -23,8 +23,8 @@
 
 # %% Run the simulation
 start_date = Date(2010, 1, 1)
-end_date = Date(2020, 1, 1)
-popsize = 5000
+end_date = Date(2012, 1, 1)
+popsize = 500
 # scenario = 0
 
 # set up the log config

src/scripts/hiv/projections_jan2023/analysis_logged_deviance.py

@@ -34,8 +34,8 @@
 
 # %% Run the simulation
 start_date = Date(2010, 1, 1)
-end_date = Date(2015, 1, 1)
-popsize = 5000
+end_date = Date(2014, 1, 1)
+popsize = 1000
 
 # scenario = 1
 

src/tlo/methods/bladder_cancer.py

@@ -45,7 +45,7 @@ def __init__(self, name=None, resourcefilepath=None):
 
     INIT_DEPENDENCIES = {'Demography', 'Lifestyle', 'HealthSystem', 'SymptomManager'}
 
-    OPTIONAL_INIT_DEPENDENCIES = {'HealthBurden'}
+    OPTIONAL_INIT_DEPENDENCIES = {'HealthBurden', 'Schisto'}
 
     METADATA = {
         Metadata.DISEASE_MODULE,
@@ -242,17 +242,23 @@ def initialise_population(self, population):
         # check parameters are sensible: probability of having any cancer stage cannot exceed 1.0
         assert sum(p['init_prop_bladder_cancer_stage']) <= 1.0
 
-        lm_init_bc_status_any_stage = LinearModel(
-            LinearModelType.MULTIPLICATIVE,
-            sum(p['init_prop_bladder_cancer_stage']),
+        predictors = [
             Predictor('li_tob').when(True, p['rp_bladder_cancer_tobacco']),
-            # todo: add line when schisto is merged
-            # Predictor('sh_infection_status').when('High-infection', p['rp_bladder_cancer_schisto_h']),
             Predictor('age_years', conditions_are_mutually_exclusive=True)
             .when('.between(30,49)', p['rp_bladder_cancer_age3049'])
             .when('.between(50,69)', p['rp_bladder_cancer_age5069'])
             .when('.between(70,120)', p['rp_bladder_cancer_agege70'])
             .when('.between(0,14)', 0.0)
+        ]
+
+        conditional_predictors = [
+            Predictor('ss_sh_infection_status').when('High-infection', p['rp_bladder_cancer_schisto_h']),
+        ] if "Schisto" in self.sim.modules else []
+
+        lm_init_bc_status_any_stage = LinearModel(
+            LinearModelType.MULTIPLICATIVE,
+            sum(p['init_prop_bladder_cancer_stage']),
+            *(predictors + conditional_predictors)
         )
 
         bc_status_any_stage = lm_init_bc_status_any_stage.predict(df.loc[df.is_alive], self.rng)
@@ -397,20 +403,26 @@ def initialise_simulation(self, sim):
         p = self.parameters
         lm = self.linear_models_for_progession_of_bc_status
 
-        lm['tis_t1'] = LinearModel(
-            LinearModelType.MULTIPLICATIVE,
-            p['r_tis_t1_bladder_cancer_none'],
-            # todo: add in when schisto is in
-            # Predictor('sh_infection_status').when('High-infection', p['rp_bladder_cancer_schisto_h']),
+        predictors = [
             Predictor('age_years', conditions_are_mutually_exclusive=True)
             .when('.between(30,49)', p['rp_bladder_cancer_age3049'])
             .when('.between(50,69)', p['rp_bladder_cancer_age5069'])
             .when('.between(70,120)', p['rp_bladder_cancer_agege70'])
             .when('.between(0,14)', 0.0),
             Predictor('li_tob').when(True, p['rr_tis_t1_bladder_cancer_none_tobacco']),
-            # todo: add in when schisto module in master
-            # Predictor('sh_').when(True, p['rr_tis_t1_bladder_cancer_none_ex_alc']),
-            Predictor('bc_status').when('none', 1.0).otherwise(0.0)
+            # todo:
+            # Predictor('tmp_').when(True, p['rr_tis_t1_bladder_cancer_none_ex_alc']),
+            Predictor('bc_status').when('none', 1.0).otherwise(0.0),
+        ]
+
+        conditional_predictors = [
+            Predictor('ss_sh_infection_status').when('High-infection', p['rp_bladder_cancer_schisto_h']),
+        ] if "Schisto" in self.sim.modules else []
+
+        lm["tis_t1"] = LinearModel(
+            LinearModelType.MULTIPLICATIVE,
+            p['r_tis_t1_bladder_cancer_none'],
+            *(predictors + conditional_predictors)
         )
 
         lm['t2p'] = LinearModel(

src/tlo/methods/cardio_metabolic_disorders.py

@@ -61,7 +61,7 @@ class CardioMetabolicDisorders(Module):
 
     INIT_DEPENDENCIES = {'Demography', 'Lifestyle', 'HealthSystem', 'SymptomManager'}
 
-    OPTIONAL_INIT_DEPENDENCIES = {'HealthBurden'}
+    OPTIONAL_INIT_DEPENDENCIES = {'HealthBurden', 'Hiv'}
 
     ADDITIONAL_DEPENDENCIES = {'Depression'}
 
@@ -598,9 +598,7 @@ def build_linear_model(self, condition, interval_between_polls, lm_type):
         # LinearModel expects native python types - if it's numpy type, convert it
         baseline_annual_probability = float(baseline_annual_probability)
 
-        linearmodel = LinearModel(
-            LinearModelType.MULTIPLICATIVE,
-            baseline_annual_probability,
+        predictors = [
             Predictor('sex').when('M', p['rr_male']),
             Predictor(
                 'age_years',
@@ -693,6 +691,17 @@ def build_linear_model(self, condition, interval_between_polls, lm_type):
                 'rr_chronic_ischemic_heart_disease_on_medication']),
             Predictor('nc_ever_stroke_on_medication').when(True, p['rr_stroke_on_medication']),
             Predictor('nc_ever_heart_attack_on_medication').when(True, p['rr_heart_attack_on_medication'])
+        ]
+
+        conditional_predictors = [
+            Predictor().when('hv_inf & '
+                             '(hv_art != "on_VL_suppressed")', p['rr_hiv']),
+        ] if "Hiv" in self.sim.modules else []
+
+        linearmodel = LinearModel(
+            LinearModelType.MULTIPLICATIVE,
+            baseline_annual_probability,
+            *(predictors + conditional_predictors)
         )
 
         return linearmodel

src/tlo/methods/depression.py

@@ -39,7 +39,7 @@ def __init__(self, name=None, resourcefilepath=None):
         'Demography', 'Contraception', 'HealthSystem', 'Lifestyle', 'SymptomManager'
     }
 
-    OPTIONAL_INIT_DEPENDENCIES = {'HealthBurden'}
+    OPTIONAL_INIT_DEPENDENCIES = {'HealthBurden', 'Hiv'}
 
     # Declare Metadata
     METADATA = {
@@ -145,6 +145,8 @@ def __init__(self, name=None, resourcefilepath=None):
 
         'rr_depr_agege60': Parameter(Types.REAL, 'Relative rate of depression associated with age > 60'),
 
+        'rr_depr_hiv': Parameter(Types.REAL, 'Relative rate of depression associated with HIV infection'),
+
         'depr_resolution_rates': Parameter(
             Types.LIST,
             'Risk of depression resolving in 3 months if no chronic conditions and no treatments.'
@@ -227,11 +229,13 @@ def read_parameters(self, data_folder):
         )
         p = self.parameters
 
-        # Build the Linear Models:
+        # Build the Linear Models
+
+        # ----- Initialisation of population -----
         self.linearModels = dict()
-        self.linearModels['Depression_At_Population_Initialisation'] = LinearModel(
-            LinearModelType.MULTIPLICATIVE,
-            self.parameters['init_pr_depr_m_age1519_no_cc_wealth123'],
+
+        # risk of depression in initial population
+        predictors = [
             Predictor('de_cc').when(True, p['init_rp_depr_cc']),
             Predictor('li_wealth').when('.isin([4,5])', p['init_rp_depr_wealth45']),
             Predictor().when('(sex=="F") & de_recently_pregnant', p['init_rp_depr_f_rec_preg']),
@@ -244,44 +248,63 @@ def read_parameters(self, data_folder):
             .when('.between(0, 14)', 0)
             .when('.between(15, 19)', 1.0)
             .when('.between(20, 59)', p['init_rp_depr_age2059'])
-            .when('>= 60', p['init_rp_depr_agege60'])
+            .when('>= 60', p['init_rp_depr_agege60']),
+        ]
+
+        conditional_predictors = [
+            Predictor().when(
+                'hv_inf & hv_diagnosed',
+                p["rr_depr_hiv"]),
+        ] if "Hiv" in self.sim.modules else []
+
+        self.linearModels["Depression_At_Population_Initialisation"] = LinearModel(
+            LinearModelType.MULTIPLICATIVE,
+            p['init_pr_depr_m_age1519_no_cc_wealth123'],
+            *(predictors + conditional_predictors)
         )
 
+        # risk of ever having depression in initial population
         self.linearModels['Depression_Ever_At_Population_Initialisation_Males'] = LinearModel.multiplicative(
             Predictor('age_years').apply(
                 lambda x: (x if x > 15 else 0) * self.parameters['init_rp_ever_depr_per_year_older_m']
             )
         )
 
+        # risk of ever having depression in initial population (female)
         self.linearModels['Depression_Ever_At_Population_Initialisation_Females'] = LinearModel.multiplicative(
             Predictor('age_years').apply(lambda x: (x if x > 15 else 0) * p['init_rp_ever_depr_per_year_older_f'])
         )
 
+        # risk of ever having diagnosed depression in initial population
         self.linearModels['Depression_Ever_Diagnosed_At_Population_Initialisation'] = LinearModel.multiplicative(
             Predictor('de_ever_depr').when(True, p['init_pr_ever_diagnosed_depression'])
                                      .otherwise(0.0)
         )
 
+        # risk of currently using anti-depressants in initial population
         self.linearModels['Using_AntiDepressants_Initialisation'] = LinearModel.multiplicative(
             Predictor('de_depr').when(True, p['init_pr_antidepr_curr_depr']),
             Predictor().when('~de_depr & de_ever_diagnosed_depression', p['init_rp_antidepr_ever_depr_not_curr'])
         )
 
+        # risk of ever having talking therapy in initial population
         self.linearModels['Ever_Talking_Therapy_Initialisation'] = LinearModel(
             LinearModelType.MULTIPLICATIVE,
             p['init_pr_ever_talking_therapy_if_diagnosed'],
             Predictor('de_ever_diagnosed_depression').when(False, 0)
         )
 
+        # risk of ever having self-harmed in initial population
         self.linearModels['Ever_Self_Harmed_Initialisation'] = LinearModel(
             LinearModelType.MULTIPLICATIVE,
             p['init_pr_ever_self_harmed_if_ever_depr'],
             Predictor('de_ever_depr').when(False, 0)
         )
 
-        self.linearModels['Risk_of_Depression_Onset_per3mo'] = LinearModel(
-            LinearModelType.MULTIPLICATIVE,
-            p['base_3m_prob_depr'],
+        # ----- Recurring events -----
+
+        # risk of depression every 3 months
+        predictors = [
             Predictor('de_cc').when(True, p['rr_depr_cc']),
             Predictor('age_years', conditions_are_mutually_exclusive=True)
             .when('.between(0, 14)', 0)
@@ -291,26 +314,42 @@ def read_parameters(self, data_folder):
             Predictor('sex').when('F', p['rr_depr_female']),
             Predictor('de_recently_pregnant').when(True, p['rr_depr_pregnancy']),
             Predictor('de_ever_depr').when(True, p['rr_depr_prev_epis']),
-            Predictor('de_on_antidepr').when(True, p['rr_depr_on_antidepr'])
+            Predictor('de_on_antidepr').when(True, p['rr_depr_on_antidepr']),
+        ]
+
+        conditional_predictors = [
+            Predictor().when(
+                'hv_inf & hv_diagnosed',
+                p["rr_depr_hiv"]),
+        ] if "Hiv" in self.sim.modules else []
+
+        self.linearModels["Risk_of_Depression_Onset_per3mo"] = LinearModel(
+            LinearModelType.MULTIPLICATIVE,
+            p['base_3m_prob_depr'],
+            *(predictors + conditional_predictors)
         )
 
+        # risk of depression resolution every 3 months
         self.linearModels['Risk_of_Depression_Resolution_per3mo'] = LinearModel.multiplicative(
             Predictor('de_intrinsic_3mo_risk_of_depr_resolution').apply(lambda x: x),
             Predictor('de_cc').when(True, p['rr_resol_depr_cc']),
             Predictor('de_on_antidepr').when(True, p['rr_resol_depr_on_antidepr']),
             Predictor('de_ever_talk_ther').when(True, p['rr_resol_depr_current_talk_ther'])
         )
 
+        # risk of stopping anti-depressants every 3 months
         self.linearModels['Risk_of_Stopping_Antidepressants_per3mo'] = LinearModel.multiplicative(
             Predictor('de_depr').when(True, p['prob_3m_default_antidepr'])
                                 .when(False, p['prob_3m_stop_antidepr'])
         )
 
+        # risk of self-harm every 3 months
         self.linearModels['Risk_of_SelfHarm_per3mo'] = LinearModel(
             LinearModelType.MULTIPLICATIVE,
             p['prob_3m_selfharm_depr']
         )
 
+        # risk of suicide every 3 months
         self.linearModels['Risk_of_Suicide_per3mo'] = LinearModel(
             LinearModelType.MULTIPLICATIVE,
             p['prob_3m_suicide_depr_m'],
@@ -519,7 +558,7 @@ def _check_for_suspected_depression(
     ):
         """
         Returns True if any signs of depression are present, otherwise False.
-        
+
         Raises an error if the treatment type cannot be identified.
         """
         if treatment_id == "FirstAttendance_NonEmergency":
@@ -574,7 +613,7 @@ def do_when_suspected_depression(
         """
         This is called by any HSI event when depression is suspected or otherwise investigated.
 
-        At least one of the diagnosis_function or hsi_event arguments must be provided; if both 
+        At least one of the diagnosis_function or hsi_event arguments must be provided; if both
         are provided, the hsi_event argument is ignored.
         - If the hsi_event argument is provided, that event is used to access the diagnosis
         manager and run diagnosis tests.