Cervical cancer #1287

andrew-phillips-1 · 2024-03-04T17:54:49Z

Here's a first draft of this module. I need to do another search to try to find additional data for calibration.

I couldn't upload the draft write as it was asking me to use the command line and Git LFS.

…odule

andrew-phillips-1

CervicalCancer.docx
Here is the draft write-up

src/tlo/methods/cervical_cancer.py

andrew-phillips-1 · 2024-06-10T17:47:32Z

src/tlo/methods/cervical_cancer.py

+        # this was not assigned here at outset because baseline value of hv_inf was not accessible - it is assigned
+        # st start of main polling event below


I think maybe Tara has changed this in the HIV module as I remember discussing with her.

andrew-phillips-1 · 2024-06-11T17:39:23Z

src/tlo/methods/cervical_cancer.py

+
+        # ----- SCHEDULE LOGGING EVENTS -----
+        # Schedule logging event to happen immediately
+        sim.schedule_event(CervicalCancerLoggingEvent(self), sim.date + DateOffset(months=0))


I don't know. I think I copied this from another module. Shall I change both to immediate ?

andrew-phillips-1 · 2024-06-11T17:39:54Z

src/tlo/methods/cervical_cancer.py

+        sim.schedule_event(CervicalCancerLoggingEvent(self), sim.date + DateOffset(months=0))
+
+        # ----- SCHEDULE MAIN POLLING EVENTS -----
+        # Schedule main polling event to happen immediately


or shall I schedule it immediately ?

andrew-phillips-1 · 2024-06-13T15:53:10Z

src/tlo/methods/tb.py

@@ -20,7 +20,7 @@
 from tlo.util import random_date

 logger = logging.getLogger(__name__)
-logger.setLevel(logging.INFO)
+logger.setLevel(logging.CRITICAL)


andrew-phillips-1 · 2024-06-13T15:55:18Z

src/tlo/simulation.py

@@ -16,7 +16,7 @@
 from tlo.progressbar import ProgressBar

 logger = logging.getLogger(__name__)
-logger.setLevel(logging.INFO)
+logger.setLevel(logging.CRITICAL)


andrew-phillips-1 · 2024-06-13T15:55:59Z

src/tlo/simulation.py

@@ -82,7 +82,7 @@ def __init__(self, *, start_date: Date, seed: int = None, log_config: dict = Non
        self.rng = np.random.RandomState(np.random.MT19937(self._seed_seq))

    def configure_logging(self, filename: str = None, directory: Union[Path, str] = "./outputs",
-                          custom_levels: Dict[str, int] = None, suppress_stdout: bool = False):
+                          custom_levels: Dict[str, int] = None, suppress_stdout: bool = True):


andrew-phillips-1 · 2024-06-13T15:57:02Z

src/tlo/simulation.py

+#           print(stats_dict)
+


yes I will keep in while I am still thinking of using it.

andrew-phillips-1 · 2024-06-13T15:57:46Z

tests/test_cervical_cancer.py

+# todo: not sure what is wrong with this assert as I am fairly certain the intended assert is true
+
+#   assert set(sim.modules['SymptomManager'].who_has('vaginal_bleeding')).issubset(
+#       df.index[df.ce_cc_ever])


So how do I change that ?

thewati

Perhaps logging must start when polling starts or a month after polling judging by this parameter r_vaginal_bleeding_cc_stage1.

# Conflicts: # resources/ResourceFile_Bladder_Cancer.xlsx # src/tlo/methods/enhanced_lifestyle.py # src/tlo/methods/hsi_generic_first_appts.py # src/tlo/simulation.py

tbhallett · 2024-06-20T09:29:37Z

I'm merging in master on this.
I've rolled back the updates the simulation.py and hsi_events.py and enhanced_lifestyle.py and ResourceFile_BladderCancer.xlsx

and refactored to include the logic intended when presenting at first appointment (non-emergency), namely. (@andrew-phillips-1 - this is because since you started work on this, we've refactored this so that the actions to do in that appointment and coded into the module itself.)

            # If the symptoms include vaginal bleeding:
            if 'vaginal_bleeding' in symptoms:
                schedule_hsi(
                    HSI_CervicalCancerPresentationVaginalBleeding(
                        person_id=person_id,
                        module=sim.modules['CervicalCancer']
                    ),
                    priority=0,
                    topen=sim.date,
                    tclose=None)

            if 'chosen_via_screening_for_cin_cervical_cancer' in symptoms:
                schedule_hsi(
                    HSI_CervicalCancer_AceticAcidScreening(
                        person_id=person_id,
                        module=sim.modules['CervicalCancer']
                    ),
                    priority=0,
                    topen=sim.date,
                    tclose=None)


            if 'chosen_xpert_screening_for_hpv_cervical_cancer' in symptoms:
                schedule_hsi(
                    HSI_CervicalCancer_XpertHPVScreening(
                        person_id=person_id,
                        module=sim.modules['CervicalCancer']
                    ),
                    priority=0,
                    topen=sim.date,
                    tclose=None)

…cancer_

src/tlo/methods/cervical_cancer.py

tbhallett · 2024-06-20T09:49:22Z

src/tlo/methods/cervical_cancer.py

+                    odds_ratio_health_seeking_in_adults=1.00)
+        )
+
+        # in order to implement screening for cervical cancer creating a dummy symptom - likely there is a better way


The comment is right here -- this is having the desired effect, but we would rather not use the 'symptoms' in this way.

Instead, for the individuals that are chosen to be screened, create and schedule the HSI event directly.

e.g. for each indivisual to be screened... make an HSI_Event_CervicalCancer_Screening..... and in that event, do whatever is required for the sceening. (might be the same as happens in the generic appointment, in which case point them both to the same function)

Thanks - I will see if I can make that work.

tbhallett · 2024-06-20T09:51:57Z

src/tlo/methods/cervical_cancer.py

+
+        # ----- HSI FOR PALLIATIVE CARE -----
+        on_palliative_care_at_initiation = df.index[df.is_alive & ~pd.isnull(df.ce_date_palliative_care)]
+        for person_id in on_palliative_care_at_initiation:


for scheduling the same class of HSI_Event to multiple people, more efficient to use schedule_batch_of_individual_hsi_events

OK I will look into how to do that

tbhallett · 2024-06-20T09:56:14Z

src/tlo/methods/cervical_cancer.py

+        p = self.sim.modules['CervicalCancer'].parameters
+
+        # ------------------- SET INITIAL CE_HPV_CC_STATUS -------------------------------------------------------------------
+        # this was done here and not at outset because baseline value of hv_inf was not accessible


this is interesting. As HIV is declared a dependency, it should be registered first, and so these properties be available by the time of running initialise_simulation and initialise_population. Could Wati/Emmanuel have a look at this, please?

If a one-off job is essential (though not sure it should be )... I think neater to put this into its own non-recurring event, and schedule it to occur on first day of simulation only. (That way it doesnt clutter up the main polling event)

tbhallett · 2024-06-20T09:58:09Z

src/tlo/methods/cervical_cancer.py

+
+        # -------------------- ACQUISITION AND PROGRESSION OF CANCER (ce_hpv_cc_status) -----------------------------------
+
+        df.ce_new_stage_this_month = False


this is being broadcast. it should be lmited to those with is_alive: ie. df.loc[df.is_alive, 'cc_new_stage_this_month'] = False

As I expect this is going to be over-written (further down) it would be more efiicent to not write it into the main sim.population.props df yet (reading/writing there is time-consuming), and instead do one write to it at the end of the event, when everything is settled.

ok will try to amend

tbhallett · 2024-06-20T09:58:43Z

src/tlo/methods/cervical_cancer.py

+
+            idx_gets_new_stage = gets_new_stage[gets_new_stage].index
+
+#           print(stage, lm, gets_new_stage, idx_gets_new_stage)


Suggested change

# print(stage, lm, gets_new_stage, idx_gets_new_stage)

tbhallett · 2024-06-20T10:00:45Z

src/tlo/methods/cervical_cancer.py

+            df.loc[idx_gets_new_stage, 'ce_hpv_cc_status'] = stage
+            df.loc[idx_gets_new_stage, 'ce_new_stage_this_month'] = True
+
+        df['ce_cc_ever'] = ((df.ce_hpv_cc_status == 'stage1') | (df.ce_hpv_cc_status == 'stage2a')


this is also broadcasting to all dataframe (including dead peple and never alive people, potentially).

Also, it will over-write to False those people not in any of those categories. I can see that this will not violate the logic, but the safest thing would be to also include in the chanied union statement the current value, in order to absolute prevent reversions... i.e. add in ce_cc_ever on the end of this line.

tbhallett · 2024-06-20T10:01:02Z

src/tlo/methods/cervical_cancer.py

+            disease_module=self.module
+        )
+
+        self.sim.modules['SymptomManager'].change_symptom(


see comment above about symptoms

tbhallett · 2024-06-20T11:48:50Z

src/tlo/methods/cervical_cancer.py

+            .when(2, p['rr_hpv_vaccinated']),
+            Predictor('age_years', conditions_are_mutually_exclusive=True)
+            .when('.between(0,15)', 0.0)
+            .when('.between(50,110)', p['rr_hpv_age50plus']),


Suggested change

.when('.between(50,110)', p['rr_hpv_age50plus']),

.when('>50'),p['rr_hpv_age50plus'])```

done thanks

tbhallett · 2024-06-20T11:49:16Z

src/tlo/methods/cervical_cancer.py

+            LinearModelType.MULTIPLICATIVE,
+            p['r_cin1_hpv'],
+            Predictor('ce_hpv_cc_status').when('hpv', 1.0).otherwise(0.0),
+            Predictor('hv_inf', conditions_are_mutually_exclusive=True)


lots of these are also 'conditions_are_exhaustive=True' too

tbhallett · 2024-06-20T11:52:14Z

src/tlo/methods/cervical_cancer.py

+
+        # Assign daly_wt to those with cancer stages before stage4 and have either never been treated or are no longer
+        # in the stage in which they were treated
+        disability_series_for_alive_persons.loc[


I'm a bit surprised this works, because the masks being used are wrt to df, but the indexing into a series with a difference index. Maybe it only works as long as everyone is alive!?

src/tlo/methods/cervical_cancer.py

tbhallett · 2024-06-20T11:56:13Z

src/tlo/methods/cervical_cancer.py

+            disease_module=self.module
+        )
+
+# vaccinating 9 year old girls - this only uncommented for testing - vaccination is controlled by epi


I think we don't need that code as vaccination is controlled elsewhere

tbhallett · 2024-06-20T11:58:28Z

src/tlo/methods/cervical_cancer.py

+
+        for person_id in selected_to_die:
+            self.sim.schedule_event(
+                InstantaneousDeath(self.module, person_id, "CervicalCancer"), self.sim.date


ideally, we'd distribute in time these deaths so that they don;t clump on first of the month (but this is a not a big deal).

The utilility function random_date (in utils) can be used

tbhallett · 2024-06-20T12:01:25Z

src/tlo/methods/cervical_cancer.py

+    def __init__(self, module, person_id):
+        super().__init__(module, person_id=person_id)
+
+        self.TREATMENT_ID = "CervicalCancer_XpertHPVScreening"


I think some equipment declaration needed for this (and others?) HSI_Events?

yes I will be talking to Joe about that

tbhallett · 2024-06-20T12:04:25Z

src/tlo/methods/cervical_cancer.py

+        assert not pd.isnull(df.at[person_id, "ce_date_diagnosis"])
+        assert not pd.isnull(df.at[person_id, "ce_date_treatment"])
+
+        days_threshold_365 = 365


could use pd.Dateoffset(years =...) instead of the number of days for ease for reading/comprehension

tbhallett · 2024-06-20T12:06:08Z

src/tlo/methods/cervical_cancer.py

+        out.update({"n_diagnosed_1_year_ago": n_diagnosed_1_year_ago})
+        out.update({"n_diagnosed_1_year_ago_died": n_diagnosed_1_year_ago_died})
+
+        print(self.sim.date, 'total_none:', out['total_none'], 'total_hpv:', out['total_hpv'], 'total_cin1:',out['total_cin1'],


move to using the logger:
i.e. logger.info(key='cervical_cancer_stats_every_month', description='XX', data=out)

…to cervical_cancer_

andrew-phillips-1 added 30 commits October 16, 2023 16:26

.

4299067

.

d6bdece

.

533357a

.

116f241

.

4bc722b

.

9a3b48a

.

0393e17

.

bc1ac59

first pass at cervical cancer module based on editing breast cancer m…

5a66e5d

…odule

first pass at cervical cancer module based on editing breast cancer m…

b24c6bd

…odule

first pass at cervical cancer module based on editing breast cancer m…

cc488bd

…odule

first pass at cervical cancer module based on editing breast cancer m…

144644a

…odule

first pass at cervical cancer module based on editing breast cancer m…

f1015b5

…odule

first pass at cervical cancer module based on editing breast cancer m…

f2b44b0

…odule

first pass at cervical cancer module based on editing breast cancer m…

0d06e44

…odule

first pass at cervical cancer module based on editing breast cancer m…

c964058

…odule

first pass at cervical cancer module based on editing breast cancer m…

1b0226b

…odule

first pass at cervical cancer module based on editing breast cancer m…

fdcea86

…odule

first pass at cervical cancer module based on editing breast cancer m…

7f13653

…odule

first pass at cervical cancer module based on editing breast cancer m…

356973c

…odule

first pass at cervical cancer module based on editing breast cancer m…

91efced

…odule

first pass at cervical cancer module based on editing breast cancer m…

443401b

…odule

first pass at cervical cancer module based on editing breast cancer m…

9e60e5c

…odule

first pass at cervical cancer module based on editing breast cancer m…

8f5e8f0

…odule

first pass at cervical cancer module based on editing breast cancer m…

5464169

…odule

first pass at cervical cancer module based on editing breast cancer m…

86a503f

…odule

first pass at cervical cancer module based on editing breast cancer m…

242de2c

…odule

first pass at cervical cancer module based on editing breast cancer m…

77a2808

…odule

HSIs

41b9743

HSIs

0fe0ee1

.

ca8a1f5

andrew-phillips-1 commented Jun 19, 2024

View reviewed changes

tbhallett marked this pull request as ready for review June 19, 2024 19:54

.

01398e9

thewati reviewed Jun 20, 2024

View reviewed changes

Merge branch 'refs/heads/master' into cervical_cancer_

3e4508b

# Conflicts: # resources/ResourceFile_Bladder_Cancer.xlsx # src/tlo/methods/enhanced_lifestyle.py # src/tlo/methods/hsi_generic_first_appts.py # src/tlo/simulation.py

tbhallett and others added 7 commits June 20, 2024 10:34

refactor to use do_at_generic_first_appt on the module

e30044f

roll back changes to healthsystem.py

2c53eb2

roll back changes to hiv.py

9138a01

cleaning up files

73682ea

cleaning up files

445b6b3

.

d727856

Merge remote-tracking branch 'origin/cervical_cancer_' into cervical_…

8dcc0e6

…cancer_

tbhallett reviewed Jun 20, 2024

View reviewed changes

andrew-phillips-1 and others added 13 commits June 21, 2024 11:45

Merge remote-tracking branch 'origin/master' into cervical_cancer_

6f25468

HSIs restructured with Screening HSI

910a080

HSIs restructured with Screening HSI

e81857e

.

890b245

Merge branch 'cervical_cancer_' of https://github.com/UCL/TLOmodel in…

eef200f

…to cervical_cancer_

.

00b5944

Merge branch 'master' into cervical_cancer_

381a307

restructured HSIs again within module

23724e8

.

5373f3a

.

2d96458

Accessing consumables and analyses using logger

7673ed0

Accessing consumables from cancer_consumables.py

0e60aa3

Rollback to using csv file in analyses

519ca33

		# this was not assigned here at outset because baseline value of hv_inf was not accessible - it is assigned
		# st start of main polling event below


		# -------------------- ACQUISITION AND PROGRESSION OF CANCER (ce_hpv_cc_status) -----------------------------------

		df.ce_new_stage_this_month = False


		idx_gets_new_stage = gets_new_stage[gets_new_stage].index

		# print(stage, lm, gets_new_stage, idx_gets_new_stage)

	.when('.between(50,110)', p['rr_hpv_age50plus']),
	.when('>50'),p['rr_hpv_age50plus'])```

Cervical cancer #1287

Are you sure you want to change the base?

Cervical cancer #1287

Conversation

andrew-phillips-1 commented Mar 4, 2024

andrew-phillips-1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thewati left a comment

Choose a reason for hiding this comment

tbhallett commented Jun 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment