Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

overrideReasons for 1.1 #513

Merged
merged 17 commits into from
Apr 1, 2020
Merged

overrideReasons for 1.1 #513

merged 17 commits into from
Apr 1, 2020

Conversation

isaacvetter
Copy link
Member

@kpshek kpshek temporarily deployed to cds-hooks-docs-pr-513 November 26, 2019 22:50 Inactive
@isaacvetter isaacvetter changed the title add ackReason documentation acknowledgement reasons for 1.1 Nov 26, 2019
docs/specification/1.0.md Outdated Show resolved Hide resolved
@jmandel
Copy link
Member

jmandel commented Nov 27, 2019

I've read through the text here but I can't figure out what's going on. An end-to-end example might help.

Question I have, that I hope this PR can address:

  • Is "acknowledgement" purely about indicating why a card's advice was not taken? Or is this just one use case? ("MAY" is confusing me, and the general English-language concept seems to imply positive and negative acknowledgements.)

  • Is "ack" about rejecting / ignoring a whole card? It's not clear to me that card-level is the right idea here, but an example with multiple suggestions + some info would probably help.

  • When/how is an acknowledgement sent from the client to the service? Is this part of an analytics mechanism?

Co-Authored-By: Josh Mandel <jmandel@alum.mit.edu>
@kpshek kpshek temporarily deployed to cds-hooks-docs-pr-513 November 27, 2019 19:22 Inactive
@isaacvetter
Copy link
Member Author

isaacvetter commented Nov 27, 2019

Josh,

Your lack of understanding reflects poorly on my spec authoring. :( Particularly, if you've read the recently created Acknowledgement-reasons-for-1.1 wiki page. (Have you read this page?)

Regardless, the spec needs to clearly set expectation for this feature.

Is "acknowledgement" purely about indicating why a card's advice was not taken? Or is this just one use case? ("MAY" is confusing me, and the general English-language concept seems to imply positive and negative acknowledgements.)

Yes, an "acknowledgement" is only used to indicate why a card's advice was not taken. I was initially calling these reject reasons, we got feedback during the CDS Hooks breakout that a clinician would rather acknowledge cds than reject it. Also, acknowledge is an existing term and reject would be a new term.

I added a new sentence to the ackReason row within the Card table to try to better explain this.

Is "ack" about rejecting / ignoring a whole card? It's not clear to me that card-level is the right idea here, but an example with multiple suggestions + some info would probably help.

Yes, an acknowledgement rejects/ignores the whole card. I think that the card is the right level. While an acknowledgement reason almost behaves in the workflow as an alternate suggestion (i.e. the user can pick suggestion A, or suggestion B or the ackReason); it is not a suggestion. An ackReason is the rejection of the provided suggestions. (This design is influenced by existing art).

I struggled to determine how much of the wiki page could be forced into the spec. Are you suggesting that an example should be in the spec?

When/how is an acknowledgement sent from the client to the service? Is this part of an analytics mechanism?

Yes, I haven't written the PR content for the "feedback service" yet. Here's the draft wiki writeup: Feedback-endpoint-for-CDS-Hooks-1.1.

Isaac

@isaacvetter
Copy link
Member Author

Following conversation with @dennispatterson , @brettmarquard, we should:

  1. Change name of ackReason to overrideReason for simplicity in language and readability.
  2. Consider if a standardized or example set of override reasons is possible or reasonable for the spec to suggest. We think this isn't reasonable today, but should be something to work towards int he future.
  3. Dennis: Change 'key to id. Brett thinks this should be a FHIR code, with an example set. (Are there overrides in clinical reasoning?)

Copy link
Contributor

@brynrhodes brynrhodes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, just a few comments throughout

docs/specification/1.0.md Outdated Show resolved Hide resolved
docs/specification/1.0.md Outdated Show resolved Hide resolved
docs/specification/1.0.md Outdated Show resolved Hide resolved
docs/specification/1.0.md Outdated Show resolved Hide resolved
docs/specification/1.0.md Outdated Show resolved Hide resolved
docs/specification/1.0.md Outdated Show resolved Hide resolved
docs/specification/1.0.md Outdated Show resolved Hide resolved
docs/specification/1.0.md Outdated Show resolved Hide resolved
@brettmarquard brettmarquard mentioned this pull request Jan 22, 2020
docs/specification/1.0.md Outdated Show resolved Hide resolved

#### OverrideReason

An **OverrideReason** is described by the following attributes. This specification does not prescribe a standard set of override reasons; implementers are encouraged to submit suggestions for standardization. To enable analysis of a card's usage, the CDS service SHALL send the same `code`, `system`, and `label` across CDS Hooks requests for the same CDS client.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"...across CDS Hooks responses for the same..."

the CDS service SHALL send the same code, system, and label across CDS Hooks requests for the same CDS client.

I want to make sure this doesn't get read as each card must have the same code(s), but that the codes/systems/labels are from a set of codes/systems/labels that is consistently drawn from across replies. What about:

The CDS Service SHALL use a consistent set of override reasons across CDS Hooks responses for the same CDS Client

Copy link
Member

@jmandel jmandel Jan 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the confusion; your proposed language here isn't helping me, though. I think we're trying to say:

  1. for a given CDS Client and label the CDS Service SHALL send the same code and system
  2. for a given CDS Client, code, and system, the CDS Service SHALL send the same label

Is that right? i.e., both (1) and (2) are desired? Why is this important? What is the connection to analytics? Why not just do analytics based on the code/system? Also, why is all of thsi "for a given CDS Client"?

@dennispatterson
Copy link
Collaborator

I recently had some e-mail conversations with Anna Dover at First Databank to get CDS Service perspective on this, which some comments bleeding into the territory of the feedback endpoint. I wanted to share what has been discussed so far:

Yes, we would return a set of override reasons if possible. We would very much like to have the ping back of how a user responded to a hook...it might be good to know what action(s) was taken as part of the response.

The key is the display to the end-user to help illicit the appropriate action. If they are too specific, then we will have many. If they are less specific, we will need the ability to overwrite the display but store the ping back with the approved reason.

As for the types of reasons, she shares some but also draws a distinction that not all reasons are override reasons, per se.

I think it could vary a bit from card to card and some vendors will want to have many reasons to help better gain insight in the workflow. The challenge with calling it an override is that they may be interacting with the card and making a change that is not directly from the card. Some examples I would want to have available: Lab Ordered, Will Review Orders, Follow-up with Attending, etc. Some that are overrides are Not Clinically Relevant, New Information Available, etc.

An example would be “Will Review Clinical Data”, a reason we have with our current integration. Very often a clinical end-user will not have all of the information they need to process to make a decision about the information in the card. This allows them to move past the card and review the patient holistically, which may or may not require the intervention based on risk/benefit to the patient.

Override implies it was not a meaningful alert. Perhaps we call it a response vs a reason. It is a way to approximate their response to the alert, even if they cannot take an action directly from the hook. Very often a clinician may review the patient based on an alert, but not make a change. The alert is meaningful but no change is needed at that time. Monitoring can continue and the review was warranted, even if a change was not.

Copy link
Member

@jmandel jmandel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks -- my questions from up-thread have been addressed. I do it's still worth trying to nail down the language of what these things are called (@dennispatterson suggests some generic terms like response). But with that caveat, from my perspective, this is good to go.

Copy link
Contributor

@mattvarghese mattvarghese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As Josh also mentions in the card.source.topic PR, can we update the overrideReason object to contain 'display' instead of 'label' to better match FHIR Coding?

Copy link
Contributor

@brynrhodes brynrhodes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've suggested a minor change, but don't think it should block approval. Nice work!

docs/specification/current.md Outdated Show resolved Hide resolved
for consistency with FHIR and card.topic
per Bryn, use:
`"code":"reason-code-provided-by-service",`
and not:
`"code":"reason-id-provided-by-service",`
@isaacvetter isaacvetter changed the title acknowledgement reasons for 1.1 overrideReasons for 1.1 Mar 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants