-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
updates from paper analyses for HIV, TB and malaria #1273
Conversation
@tbhallett I have an error occurring in test_analysis: test_get_parameter_functions I can see it's down to the data contained in ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking but I don't want to edit this without Bingling's input. Let me know what you'd prefer me to do. |
…ty as small population size over 7 days will not schedule VMMC through HIV module. Remove HIV_Prevention_Circumcision in assert statement for HIV services delivered in one week
…x with updated NTP2019 TB data
# Conflicts: # resources/malaria/ResourceFile_malaria.xlsx # src/tlo/methods/hiv.py # src/tlo/methods/malaria.py
# Conflicts: # resources/ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx # resources/healthsystem/priority_policies/ResourceFile_PriorityRanking_ALLPOLICIES.xlsx # src/tlo/methods/malaria.py # tests/test_malaria.py
# Conflicts: # resources/ResourceFile_HIV.xlsx # src/tlo/methods/hiv.py
@tbhallett Hi Tim, this error is now occurring but I'm not sure what the correct fix would be:
I had updated the spreadsheet resources/ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx Is it possible that the parameter CareOfWomenDuringPregnancy:prob_intervention_delivered_ifa was removed at some point but still exists in ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx? Should it still be there? |
@tbhallett one test is failing in test_healthsystem:test_all_treatment_ids_defined_in_priority_policies |
That's ok, we can still go ahead and merge this branch as the fix being developed targets master |
Co-authored-by: Tim Hallett <39991060+tbhallett@users.noreply.github.com>
address comments in PR #1273 review
@tbhallett This is ready for review now |
* updates from paper analyses for HIV, TB and malaria * remove unused import statements * fix imports * update filepath for malaria resource file * remove test_hiv_tb_scenarios.py * updated test_healthsystem.py: test_manipulation_of_service_availability as small population size over 7 days will not schedule VMMC through HIV module. Remove HIV_Prevention_Circumcision in assert statement for HIV services delivered in one week * change ipt_coverage in TB logger as conflicts with existing parameter * updated ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx with updated NTP2019 TB data * remove test code * remove test code * delete tmp resourcefiles * malaria code use person_id consistently instead of individual_id * use individual_id for demography.do_death() * style change to avoid conflicts with master * style change to avoid conflicts with master * fix conflicts with master * fix conflicts with master * merge in master * check property hv_date_last_ART correctly set * Manually add PostnatalCare_Comprehensive to policy priorities * edit fix * add schisto high infection as conditional predictor for bladder cancer * fix conditional predictor for active TB - should check presence of CardioMetabolicDisorders * add 'ss' prefix to properties from schisto module referenced in bladder_cancer.py * edit praziquantel code in schisto.py to use value from CMST in place of donated * add parameter rr_depr_hiv for risk of depression with HIV infection * tidy up linear models in depression, include conditional predictors for hiv infection and add comments * move hv_inf into conditional predictor for depression in initial population * convert lm for incident cancer (site_confined) to model with conditional predictors. Include HIV as risk factor. * add parameter rr_site_confined_hiv to other_adult_cancers.py * update other_adult_cancers write-up to include HIV as risk factor * update Depression.docx to include HIV as risk factor for depression * edit HIV in depression to include only HIV cases not virally suppressed * update other_adult_cancers.py linear model to include HIV as risk factor only if not virally suppressed * edit: HIV remains as risk factor for depression independent of treatment status * include HIV as risk factor for low grade dysplasia (oesophageal cancer). Update ResourceFile_Oesophageal_Cancer.xlsx * update linear model for low grade dysplasia to include HIV as conditional risk factor * update OesophagealCancer.docx write-up to include HIV risk * add condition hiv diagnosed for increased risk of depression * remove hiv as risk factor for oesophageal cancer * remove parameter for hiv as risk factor for oesophageal cancer * update OesophagealCancer.docx to remove hiv as risk factor * update value for weighted risk of other_adult_cancers with unsuppressed HIV * add rr_hiv to linear model. update ResourceFile_cmd_condition_onset.xlsx with rr_hiv, leave value=1.0 if no effect of hiv * update resourcefiles for CMD include rr_hiv for all, no effect of majority of processes * refactoring: * put the helper function for switching scenario into same file a ScenarioSwitcher class * put tests for class and helper function together (next step will be to rename and mock-up extended functionality) * add diabetes as risk for active TB and relapse. add params to ResourceFile_TB.xlsx replace linearmodels dict which was accidentally removed in depression.py * add diabetes as risk factor for tb death * add diabetes as risk factor for PLHIV with active TB and on TB treatment * add diabetes as risk factor for PLHIV with active TB and on TB treatment * set up run to check calibration of deaths and disability * add predictor high-intensity S. haematobium infection to risk of bladder cancer in initial population * add predictor high-intensity S. haematobium infection to risk of HIV acquisition * fix indenting in HSI_Hiv_StartOrContinueTreatment * add hv_date_treated abd hv_date_last_ART to baseline_art * convert linear model in CMD to include conditional predictors * delete resourcefile created in error * comment out path-specific changes to analysis_cause_of_death_and_disability_calibrations.py * fix CMD error if Hiv not registered * initial sketch of structure * tidy-up and fix tests * commments * remove parameter rr_bcg_inf from tb.py * edit comment in initialise_simulation * fix parameter name error * update parameters * test runs * edit and fix flake8 errors * fix failing test * update ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx * update ResourceFile_PriorityRanking_ALLPOLICIES.xlsx * updated ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx * scenario file * starting work on the scenario file * design and HR scenarios * scenario for HRH * roll back changes to ScenarioSwitcher * Revert "roll back changes to ScenarioSwitcher" This reverts commit 49d14e7. * sketch out of prposed changed for scenario_switcher * sketch out of prposed changed for scenario_switcher * sketch out of prposed changed for scenario_switcher * sketch out of prposed changed for scenario_switcher * sketch out of prposed changed for scenario_switcher * Add ResourceFile_Consumables_Item_Designations.csv * create special scenarios for consumables availability based on the designation of the consumable item * comment and fix imports * draft of scenario file * linting * increase length of scenarios and reduce reps * rename * correct error in specifiction of dynamic scaling scenario * remove trailing commas that casts return as tuple * Initially only consider baseline and perfect healthcare seeking scenarios, to get upper limit on RAM requirements * Submit remaining scenarios * direct to self.module (rather than self) * edit check for last ART when new dispensation occurs * update schisto risk on HIV to only include women * add scale-up parameter for htm programs add event to hiv module to switch parameters include new keywords in hiv module for scale-up separately for hiv, tb and malaria add new sheet in ResourceFile_HIV.xlsx containing new scale-up parameter values * add catch for malaria rdt_testing_rates post-2024 * malaria parameters scale-up included * restore `scenario_comparison_of_horizontal_and_vertical_programs` (removing comments used to select certain scenarios to be run). * first draft of figures * set up test for HTM scenario scale-up * add tests for HTM scale-up * check resourcefiles updated * check the usage of '' versus "" * reset to single quotes to match PR #1273 * remove unneeded resource files * remove unneeded resource files * edit filepaths * isort for imports * edit ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx to remove 2024 missing value in malaria Rate_rdt_testing * fix filename for test_HTMscaleup.py * update scenario file * update scenario file * include "scale_to_effective_capabilities": True * linting * linting * reduce number of draws * set scaleup parameters separately in each module use parameters to select scaleup instead of module arguments update resourcefiles * set scaleup parameters separately in each module use parameters to select scaleup instead of module arguments update resourcefiles * update resourcefiles * update resourcefiles * set up script to test scenarios - scale-up of HTM programs * test runs * add scenario switch to malaria.py * cherry-pick inadvertently updated files and revert * isort fixes * Delete resources/~$ResourceFile_HIV.xlsx * Delete resources/~$ResourceFile_TB.xlsx * Delete resources/malaria/~$ResourceFile_malaria.xlsx * fix failing tests * rollback changes to calibration_analyses/scenarios/long_run_all_diseases.py * fix error in filename * revert timedelta format to 'M' * add todos * merge in master * set up test runs for scale-up * edit scenario start date * change parameter scaleup_start_date to an integer value used in DateOffset, as timestamp is not json serializable for analysis runs * change scale-up event scheduling to use new DateOffset parameter * test runs * set up test runs * set up test runs * fix failing tests * Update tests/test_HTMscaleup.py Co-authored-by: Tim Hallett <39991060+tbhallett@users.noreply.github.com> * address comments on PR review * isort fixes * isort fixes * change np.timedelta in enhanced_lifestyle.py back to original * remove json file * call it 'htm_scenario_analysis' rather than just 'scenario_analysis' * update comment * roll back change in test_tb.py * remove .py extension for clarity * roll back incidental change * linting and editing string for clarity * roll back incidental changes * defaults for healthsystem ok -- no need to step through each option * remove inadvertent duplication in code * remove comment * use dict for ease of accessing * parameter to be the YEAR (int) of the change to fit with the convention used in other modules (instead of years since the beginning of the simulation) * remove comment * refactor module method for clarity * refactor to prevent same name being used for events specific to different modules * specify year for scale-up in analysis file * rename; add note; remove comments for HTM scenarios * refacotring * rename scenario class for hss elements * new scenario for combinations of vertical and horizontal programs * minor refactor * update docstring * renaming * add @Property decorator * plot DALYS averted relative to Baseline - broken down by major cause (HIV, TB, MALARIA) * remove unused parameter prob_start_art_after_hiv_test * set up test runs * update linear models for tb and malaria after changing parameters * set up test runs * set up test runs * remove unused parameter from ResourceFile_HIV.xlsx * remove annual_testing_rate_children as not used * add scale-up parameter probability_of_being_retained_on_art_every_3_months, increase to 1 * set cons_availability to all and set off larger runs * test runs * change spec. of runs to be smaller pop size and fewer runs * roll back changes that are not related to this PR * delete file added by accident * roll back change (I think this is mistake) * refactor * remove clause `if p["do_scale_up"]` as not neccessary * make "_build_linear_models" function, called from initialise_population and update_parameter_for_program_scaleup --------- Co-authored-by: tdm32 <t.mangal@imperial.ac.uk> Co-authored-by: Tara <37845078+tdm32@users.noreply.github.com> Co-authored-by: Margherita Molaro <48129834+marghe-molaro@users.noreply.github.com>
…s needed for the "Horizontal vs Vertical"-type Analysis, (#1288) * updates from paper analyses for HIV, TB and malaria * remove unused import statements * fix imports * update filepath for malaria resource file * remove test_hiv_tb_scenarios.py * updated test_healthsystem.py: test_manipulation_of_service_availability as small population size over 7 days will not schedule VMMC through HIV module. Remove HIV_Prevention_Circumcision in assert statement for HIV services delivered in one week * change ipt_coverage in TB logger as conflicts with existing parameter * updated ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx with updated NTP2019 TB data * remove test code * remove test code * delete tmp resourcefiles * malaria code use person_id consistently instead of individual_id * use individual_id for demography.do_death() * style change to avoid conflicts with master * style change to avoid conflicts with master * fix conflicts with master * fix conflicts with master * merge in master * check property hv_date_last_ART correctly set * Manually add PostnatalCare_Comprehensive to policy priorities * edit fix * add schisto high infection as conditional predictor for bladder cancer * fix conditional predictor for active TB - should check presence of CardioMetabolicDisorders * add 'ss' prefix to properties from schisto module referenced in bladder_cancer.py * edit praziquantel code in schisto.py to use value from CMST in place of donated * add parameter rr_depr_hiv for risk of depression with HIV infection * tidy up linear models in depression, include conditional predictors for hiv infection and add comments * move hv_inf into conditional predictor for depression in initial population * convert lm for incident cancer (site_confined) to model with conditional predictors. Include HIV as risk factor. * add parameter rr_site_confined_hiv to other_adult_cancers.py * update other_adult_cancers write-up to include HIV as risk factor * update Depression.docx to include HIV as risk factor for depression * edit HIV in depression to include only HIV cases not virally suppressed * update other_adult_cancers.py linear model to include HIV as risk factor only if not virally suppressed * edit: HIV remains as risk factor for depression independent of treatment status * include HIV as risk factor for low grade dysplasia (oesophageal cancer). Update ResourceFile_Oesophageal_Cancer.xlsx * update linear model for low grade dysplasia to include HIV as conditional risk factor * update OesophagealCancer.docx write-up to include HIV risk * add condition hiv diagnosed for increased risk of depression * remove hiv as risk factor for oesophageal cancer * remove parameter for hiv as risk factor for oesophageal cancer * update OesophagealCancer.docx to remove hiv as risk factor * update value for weighted risk of other_adult_cancers with unsuppressed HIV * add rr_hiv to linear model. update ResourceFile_cmd_condition_onset.xlsx with rr_hiv, leave value=1.0 if no effect of hiv * update resourcefiles for CMD include rr_hiv for all, no effect of majority of processes * refactoring: * put the helper function for switching scenario into same file a ScenarioSwitcher class * put tests for class and helper function together (next step will be to rename and mock-up extended functionality) * add diabetes as risk for active TB and relapse. add params to ResourceFile_TB.xlsx replace linearmodels dict which was accidentally removed in depression.py * add diabetes as risk factor for tb death * add diabetes as risk factor for PLHIV with active TB and on TB treatment * add diabetes as risk factor for PLHIV with active TB and on TB treatment * set up run to check calibration of deaths and disability * add predictor high-intensity S. haematobium infection to risk of bladder cancer in initial population * add predictor high-intensity S. haematobium infection to risk of HIV acquisition * fix indenting in HSI_Hiv_StartOrContinueTreatment * add hv_date_treated abd hv_date_last_ART to baseline_art * convert linear model in CMD to include conditional predictors * delete resourcefile created in error * comment out path-specific changes to analysis_cause_of_death_and_disability_calibrations.py * fix CMD error if Hiv not registered * initial sketch of structure * tidy-up and fix tests * commments * remove parameter rr_bcg_inf from tb.py * edit comment in initialise_simulation * fix parameter name error * update parameters * test runs * edit and fix flake8 errors * fix failing test * update ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx * update ResourceFile_PriorityRanking_ALLPOLICIES.xlsx * updated ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx * scenario file * starting work on the scenario file * design and HR scenarios * scenario for HRH * roll back changes to ScenarioSwitcher * Revert "roll back changes to ScenarioSwitcher" This reverts commit 49d14e7. * sketch out of prposed changed for scenario_switcher * sketch out of prposed changed for scenario_switcher * sketch out of prposed changed for scenario_switcher * sketch out of prposed changed for scenario_switcher * sketch out of prposed changed for scenario_switcher * Add ResourceFile_Consumables_Item_Designations.csv * create special scenarios for consumables availability based on the designation of the consumable item * comment and fix imports * draft of scenario file * linting * increase length of scenarios and reduce reps * rename * correct error in specifiction of dynamic scaling scenario * remove trailing commas that casts return as tuple * Initially only consider baseline and perfect healthcare seeking scenarios, to get upper limit on RAM requirements * Submit remaining scenarios * direct to self.module (rather than self) * edit check for last ART when new dispensation occurs * update schisto risk on HIV to only include women * add scale-up parameter for htm programs add event to hiv module to switch parameters include new keywords in hiv module for scale-up separately for hiv, tb and malaria add new sheet in ResourceFile_HIV.xlsx containing new scale-up parameter values * add catch for malaria rdt_testing_rates post-2024 * malaria parameters scale-up included * restore `scenario_comparison_of_horizontal_and_vertical_programs` (removing comments used to select certain scenarios to be run). * first draft of figures * set up test for HTM scenario scale-up * add tests for HTM scale-up * check resourcefiles updated * check the usage of '' versus "" * reset to single quotes to match PR #1273 * remove unneeded resource files * remove unneeded resource files * edit filepaths * isort for imports * edit ResourceFile_Improved_Healthsystem_And_Healthcare_Seeking.xlsx to remove 2024 missing value in malaria Rate_rdt_testing * fix filename for test_HTMscaleup.py * update scenario file * update scenario file * include "scale_to_effective_capabilities": True * linting * linting * reduce number of draws * set scaleup parameters separately in each module use parameters to select scaleup instead of module arguments update resourcefiles * set scaleup parameters separately in each module use parameters to select scaleup instead of module arguments update resourcefiles * update resourcefiles * update resourcefiles * set up script to test scenarios - scale-up of HTM programs * test runs * add scenario switch to malaria.py * cherry-pick inadvertently updated files and revert * isort fixes * Delete resources/~$ResourceFile_HIV.xlsx * Delete resources/~$ResourceFile_TB.xlsx * Delete resources/malaria/~$ResourceFile_malaria.xlsx * fix failing tests * rollback changes to calibration_analyses/scenarios/long_run_all_diseases.py * fix error in filename * revert timedelta format to 'M' * add todos * merge in master * set up test runs for scale-up * edit scenario start date * change parameter scaleup_start_date to an integer value used in DateOffset, as timestamp is not json serializable for analysis runs * change scale-up event scheduling to use new DateOffset parameter * test runs * set up test runs * set up test runs * fix failing tests * Update tests/test_HTMscaleup.py Co-authored-by: Tim Hallett <39991060+tbhallett@users.noreply.github.com> * address comments on PR review * isort fixes * isort fixes * change np.timedelta in enhanced_lifestyle.py back to original * remove json file * call it 'htm_scenario_analysis' rather than just 'scenario_analysis' * update comment * roll back change in test_tb.py * remove .py extension for clarity * roll back incidental change * linting and editing string for clarity * roll back incidental changes * defaults for healthsystem ok -- no need to step through each option * remove inadvertent duplication in code * remove comment * use dict for ease of accessing * parameter to be the YEAR (int) of the change to fit with the convention used in other modules (instead of years since the beginning of the simulation) * remove comment * refactor module method for clarity * refactor to prevent same name being used for events specific to different modules * specify year for scale-up in analysis file * rename; add note; remove comments for HTM scenarios * refacotring * rename scenario class for hss elements * new scenario for combinations of vertical and horizontal programs * minor refactor * update docstring * renaming * add @Property decorator * plot DALYS averted relative to Baseline - broken down by major cause (HIV, TB, MALARIA) * define mini run scenario * rename scenario * lining isort * linting ruff * delete not-yet-used file * isort again! --------- Co-authored-by: tdm32 <t.mangal@imperial.ac.uk> Co-authored-by: Tara <37845078+tdm32@users.noreply.github.com> Co-authored-by: Margherita Molaro <48129834+marghe-molaro@users.noreply.github.com>
Updates include:
palliative care HSI for HIV
palliative care HSI for TB
priority ranking spreadsheet updated to include new EndOfLifeCare HSIs
updated TB diagnostic referral pathway, moving all xpert testing to facility level 2 (as consumables for xpert not available at level 1a and 1b)
remove unused parameters
update data in ResourceFiles
remove shorter paediatric treatment for TB as not commonly used in practice
add consistent repeat healthcare seeking behaviour cap for HIV, TB and malaria HSIs
remove parameters and code relating to scenario setting (not required for master, only for analyses)
include data_parameter for TB to match updated data (ending 2022)
new TbDecideDeath feature schedules hospital stay prior to death and determines probability of death via. linear model
HSI_Tb_ScreeningAndRefer accepts argument facility_level
HSI_ClinicaDiagnosis is now separate (was previously just a diagnostic test) meaning we can track the HR requirements
additional properties in HIV: hv_on_cotrimoxazole, hv_date_treated
schedule hospital stay prior to AIDS death
These changes were all implemented for branches used for the analysis in the recent two papers and so outputs have been checked and are all consistent with expectations.