Gamma discussion

[SimonBiggs]

@kingrpaul certainly don't have to discuss it now, but just leaving an issue here so that it is not forgotten

@kingrpaul said in #32 (comment)

I would like to have a conversation, at some point, about "gamma analysis" as an approach. It has, at times, seemed to me that we, as a profession, are perhaps not doing this as well as it might be done.

[kingrpaul]

• Comments inserted here were lost. Twice now that GitHub has dumped what I've written

Shorter version

3/3/10 -> x% is what we do and is very non-linear, hides agreement quality information

100% -> x/y/10 would be more linear and perhaps more meaningful in comparing 2 dose distributions, especially in evaluating the quality of a TPS beam-model

[SimonBiggs]

You would have to have one of x and one y fixed. Or it could produce a plot in x, y space that draws the line where x and y hit 100%.

I like it. It would require a bit of a refactor of the current gamma code, but I can envisage this not being too difficult.

What has caused GitHub to dump your text? Have you left the page and come back again?

[SimonBiggs]

Or even a contour plot, which presents 90% through to 100% in 1% intervals.

[SimonBiggs]

You know, we could make this happen relatively quick, and you might be able to submit an abstract on it to ICCR in Montreal. Abstracts are due on the 15th of Feb though:

https://iccr-mcma.org/abstracts/

[kingrpaul]

GitHub dropped me once on a pull request, that conflicted(?) with yours and another on a comment that I'm not sure what happened,

[kingrpaul]

Might be too quick for me right now on the abstract, but with enough time, I'm for it.

[kingrpaul]

I wonder if there would be a unique minimal dose and distance pair such that pass rate == 100%?

[SimonBiggs]

It would not be unique, for any one distance there would be a corresponding dose.

…

On Sat, 2 Feb 2019 at 09:01, Paul King ***@***.***> wrote: I wonder if there would be a unique minimal dose and distance pair such that pass rate == 100%? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGQVe-l5K-SWn7CTIsPwUbMtuNrpH0UFks5vJLlUgaJpZM4adkWl> .

[SimonBiggs]

What browser are you using?

…

On Sat, 2 Feb 2019 at 09:45, Simon Biggs ***@***.***> wrote: It would not be unique, for any one distance there would be a corresponding dose. On Sat, 2 Feb 2019 at 09:01, Paul King ***@***.***> wrote: > I wonder if there would be a unique minimal dose and distance pair such > that pass rate == 100%? > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#36 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AGQVe-l5K-SWn7CTIsPwUbMtuNrpH0UFks5vJLlUgaJpZM4adkWl> > . >

[Matthew-Jennings]

I wonder if there would be a unique minimal dose and distance pair such that pass rate == 100%?

It would not be unique, for any one distance there would be a corresponding
dose.

I think average gamma value - rather than percent of gamma points < 1 - captures the essence of Paul's suggestion. The gamma index itself already seems to match Paul's preferred format, it's just normalised to the %dd and DTA criteria (with a mishmash of units, haha). Where we deviate from traditional error handling is the pass rate, not so much the index.

Agreed that gamma is not ideal, at least when using 3%, 3 mm criteria. There is a reasonable amount of literature on this very point. But I do think investigating a new alternative would be quite an undertaking. It's not so easy to prove a metric is better, unless you use imperfect baselines like TPS calculated DVH values (which oncologists tend to use, along with coverage).

Unless this is a passion of yours and you can dedicate the time, I can't help but think PyMedPhys is better off just supplying already-suggested alternatives. I would prefer that PyMedPhys didn't promote an unpublished metric, at least not in the main library.

The obvious addition that could be made is DVH based comparison. Admittedly this needs a structure set, but it at least correlates more closely to how oncologists evaluate plans.

[kingrpaul]

Firefox.
I'm somewhat familiar with the DVH based approach, but see it as adding complexity more so than clarity.
Less a passion than a nagging dissatisfaction.

[SimonBiggs]

This is using a toy gamma data set, so the results aren't very interesting. But this is sort of what I was thinking:

And for a range of different thresholds:

The slow part of the gamma calc can be reused between different dose/distance threshold choices. So a grid like this could be quite easily created and a contour map of the distance/dose/pass rate could be created.

@Centrus007 I know personally I would find something like this a helpful plot. What do you think?

[SimonBiggs]

When it is a full plot of the full gamma space of interest, and then a contour plot can be drawn then I think it would be quite a neat visualisation.

[kingrpaul]

Me too.

[SimonBiggs]

So, here and there today (in between cleaning the house and playing a bit of astrobot with the kids) I tinkered around within the following pull request:

#38

Within that pull request the following code, produces the following image:

imports ...

dose_thresholds = np.arange(0.5,3,0.1)
distance_thresholds = np.arange(0.5,3,0.1)

gamma = gamma_shell(
    coords_reference, dose_reference,
    coords_evaluation, dose_evaluation,
    dose_thresholds, distance_thresholds, 
    max_gamma=1.0001, skip_once_passed=True)

pass_rates = []

for key, value in gamma.items():
    pass_rates.append(convert_to_percent_pass(value))
    
pass_rates = np.reshape(
    pass_rates, 
    (len(dose_thresholds), len(distance_thresholds)))
    
plt.contourf(
    distance_thresholds, dose_thresholds, 
    pass_rates, 100, 
    vmin=90, vmax=100, cmap='coolwarm_r')

plt.title('Gamma pass-rate space')
plt.xlabel('Distance Threshold (mm)')
plt.ylabel('Dose Threshold (%)')
plt.colorbar(label='Gamma pass rate (gamma <= 1)')

[SimonBiggs]

@kingrpaul is the above figure something like what you were thinking of?

Also, it barely adds any time at all to the calculation of a single gamma value :) win.

[SimonBiggs]

@Centrus007 pretty cool? What do you think?

[kingrpaul]

Very cool. Wonder what happens if you expand the size of the graph and make all points for which the pass rate is <100% to be uniformly white?

[kingrpaul]

I suggest that the lower left 100% iso line of this plot is useful.

I suggest that among the points where the pass % = 100, the one "closest" to (0,0) is a unique and useful figure of merit. It's better to know that (1.9 mm, 0.7%) yields 100% pass than that 100% pass at (3,3). Both yield a binary "pass".
You can do this analysis with SNC software, but it's slow.

I'd see this as an application of gamma analysis, not a redirection.

It might merit a poster.

[SimonBiggs]

Well once this merges in you'll be free to experiment. You can essentially pass multiple distance and dose threshold values to gamma in the form of a list. From there I believe you'd be equipped to do what you're after? I do agree, it would make a nice poster.

…

On Sun., 3 Feb. 2019, 4:07 am Paul King ***@***.*** wrote: I suggest that the lower left 100% iso line of this plot is useful. I suggest that among the points where the pass % = 100, the one "closest" to (0,0) is a unique and useful figure of merit. It's better to know that (1.9 mm, 0.7%) yields 100% pass than that 100% pass at (3,3). Both yield a binary "pass". You can do this analysis with SNC software, but it's slow. I'd see this as an application of gamma analysis, not a redirection. It might merit a poster. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGQVe2ESMZtrmXWFv9F699Mp5lRdLE_Dks5vJcXcgaJpZM4adkWl> .

[SimonBiggs]

One issue I do draw is how to weight the axis. By searching for a smallest distance there is an implicit weighting equality being drawn between the two axis. One is essentially saying that a change in 1% dose is equivalent to 1mm distance.

Because those axis are different units to each other there is no reason one could not be scaled by a scaling factor. (For example if instead of using mm we used some arbitrary fractions of inches).

This also carries through to the fact that our distance criterion is informed by the uncertainty budget in positioning. For example, let's say the margins chosen for the patient have built in to them a given distance for the differences that occur between the beam model and the machine delivery not including patient positioning effects and other positioning effects. Let's say that the common 3% 3mm threshold is informed by that.

If I found that 0% 4 mm passes, with the closest distance of "4" to the origin. That would be concerning. At that point I don't care if I can get every point to dosimetrically agree at 4 mm away. But even though this metric has potentially led me down the garden path, at a further "distance" of 4.2 is also a 100% pass of 3%/3mm.

For your idea to work I think there would need be a region of allowable tolerances. An example set of allowable tolerances might be all those <= 3% and 3 mm. That could inform the thresholds that are sent to gamma in the first place.

From there, I'm still not convinced a distance to the origin of 100% is a meaningful parameter, due to the implicit value parity being placed between the two axes by doing so.

[Matthew-Jennings]

Well, it certainly looks cool! But I'm not really convinced of its utility just yet. I haven't constructed a scenario in my head for which I'd personally opt to use this. It gives a swathe of data whose usefulness is dubious.

When looking at multiple dose comparisons, maybe this could be a good tool to evaluate which is best. E.g. you have some measured data and multiple instances of calculated data that differ by slight changes in a certain TPS model parameter. I.e. you are trying to tune this model parameter. Maybe a simple 3%, 3 mm or 2%, 2 mm pass rate shows negligible difference, but you can visually identify which is best using the above plot, since it includes a vast "pass rate space".

Clutching at straws a bit though. I don't think it's really suitable for routine QA. I see the added information as unnecessarily detailed, confusing for the interpreter (what do I do with this information) and perhaps even providing the false sense of security that heaps of "data" is wont to provide.

---

In my opinion, the major issue with gamma pass rates is lack of spatial information; failures in certain positions matter more than in others. It seems to me that the solution to this is not to show more pass rates, it's to find a way to weight scores based on their locations in the dataset. Visual scrutiny of a gamma distribution (spatial, not histogram) is a subjective way of achieving this. Alternatively, DVH comparison does this nicely and is easily automatable. There could be other ways.

[kingrpaul]

Beam modeling is one area where I think this would be useful. It would also make detector drift visible in IMRT QA.

The argument for 1% equating to 1mm, in terms of axes, would be that I choose to use 3/3/10. So, consensus.

[SimonBiggs]

Well either way, I have in the past wanted to run gamma for multiple threshold values. In the past I would have done it in series taking not always a small amount of time. What is there now in the pull request allows for efficiently running multiple threshold values at once. Plotting or not, up to the user of the library, but I think the ability to pass multiple threshold values is a useful utility.

…

On Sun., 3 Feb. 2019, 10:45 am Paul King ***@***.*** wrote: Beam modeling is one area where I think this would be useful. It would also make detector drift visible in IMRT QA. The argument for 1% equating to 1mm, in terms of axes, would be that I choose to use 3/3/10. So, consensus. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGQVe_9fOZVip430reaXiXpPnWlwIys2ks5vJiMYgaJpZM4adkWl> .

[SimonBiggs]

I also do agree by the way that DVHs ideally is where we should be heading
for patient specific QA. At the end of all this we want to be able to talk
to RTs and Doctors in a language that they understand. They want to know
"is this effect you're telling me about clinically significant?". To answer
that, all the features used by the planner to create the plan and the
features used by the doctor to approve it should be the language we speak
in.

I have nevertheless conceded that there is a usefulness for gamma, it just
doesn't belong in a vacuum.

Even or beam modelling I have found only half helpful, and a simple dose
difference on profiles I have found can be more insightful. But it it
situation dependent. Making tools is not the same as declaring when and how
those tools should be used. That is up to the professional judgement of the
user.

Presenting tools at conferences and saying this is how we use them in our
clinic, now that is where we begin to make claims on how we believe they
should be used.

...so in that I think we should definitely discuss further before putting
it in a poster if there is doubt about whether or not it should be promoted
for that given purpose. But I am happy to have it merged in as an expanded
feature of pymedphys.gamma. As even simply being able to efficiently check
a couple of different thresholds is of value to me.

[Matthew-Jennings]

Beam modeling is one area where I think this would be useful. It would also make detector drift visible in IMRT QA.

What is there now in the pull request allows for efficiently running
multiple threshold values at once. Plotting or not, up to the user of the
library, but I think the ability to pass multiple threshold values is a
useful utility.

But I am happy to have it merged in as an expanded
feature of pymedphys.gamma. As even simply being able to efficiently check
a couple of different thresholds is of value to me.

Fair enough :)

Beam modeling is one area where I think this would be useful.

Even or beam modelling I have found only half helpful, and a simple dose
difference on profiles I have found can be more insightful.

Our department wouldn't use gamma for beam profiles, but for TG119 and TG244; specifically film and 3D detector results. We actually do our final tuning of a few of our parameters using TG119. It gives a better representation of the likely clinical effects of parameter tweaks. So perhaps that is one use case.

[SimonBiggs]

This was implemented

[SimonBiggs]

Hi @kingrpaul,

You might be interested to know that this was recently used to run a comparison to a separate library called "flashgamma". See samuelpeet/flashgamma#7

And also, I wanted to say, I was recently having a look at the logo, and absolutely loving the sync waveform design. Thanks for the idea 🙂. It was great iterating on ideas for the logo with you :).