Home
In this public wiki, we forge the scientific community to use it as a reference for the concepts and instructions of the project. If you are new to the project and framework, you must follow this wiki with attention. Although it is an important piece of documentation, this wiki is not supposed to replace the official documentation of each tool, prototype or used library. Moreover, we also provide a private wiki on the meta-private repository for team usage. Unfortunately, you need to be a member of our team to access the restricted information.
MIMBCD-UI project deals with the use of a recently proposed technique in literature: Deep Convolutional Neural Networks (CNNs). These deep networks will incorporate information from several modes: Magnetic Resonance Imaging (MRI) volumes, UltraSound (US) images, MammoGraphic (MG) images (both views CC and MLO) and text. The proposed algorithm, called for multimodality CNNs (MMCNNs) will have the ability to process multimodal information in a unified and sustained manner. This methodology needs to "learn" what are the masses and calcifications. So, that is necessary to collect the ground-truth or notes of the masses and calcifications provided by medical experts. For the collection of these notes, the design, and development of an interface is necessary. The purpose of this interface is to allow users (in this case, medical specialists) to display various types of image (i.e., MG, US, and MRI). Moreover, the interface allows the clinician for user interaction, particularly in providing the notes of the masses and calcifications. For these reasons, it is crucial for the development of this project, cooperation with experts providing the above notes.
Involved in medical imaging analysis, our research team is using specific Machine Learning (ML) frameworks to perform the classification and lesion detection of breast cancer. One of the problems that are common in the medical imaging field, is that the number of images (i.e., exams) is exponentially growing while the number of radiologists remains the same. This is a common scenario in current clinical setups. The consequence is that the radiologists have to perform the examination in numerous exams that are constantly growing, making their task cumbersome, painful, and time-consuming. One way to alleviate this is to provide a "second reader" to speed up their work. This second reader can be obtained by an automatic means, i.e., an Artificial Intelligence (AI) agent.
The difficulty of breast cancer is one of such challenges that fulfills the above scenario. A research topic that we are involved since 2015.
To accomplish these, three steps were mandatory:
1 - To collect a large amount of multimodal breast data;
2 - To implement new AI models for detection, classification, and segmentation of breast lesions;
3 - To develop new interfaces and put this problem in an HCI context, given that the radiologist interaction is of primordial importance;
From (1) it is possible to have a larger number of images required to train Deep Neural Networks (DNNs), to perform classification or the lesion detection. Indeed, the team has been collecting data since 2015, and now we already have about 700 classified (by a radiologist) exams. This is a slow process because each exam needs to have a BIRADS classification score. Furthermore, a multimodal dataset must be built. This embraces MG, US, and MRI. Due to the current clinical setups, the radiologist performs a sort of "loop inspection" [1] within these three modalities - where radiologist cannot see some lesions in MG (specially in dense breast), but the lesions are visible in US or MRI modalities. Thus, cross inspection is mandatory. As such, our intelligent system should also work under this multimodal condition.
For (2), the implementation of new AI models is therefore important and related with the (1) first step. We build on a growing interest in studying and implementing novel AI methods, such as aspects of model transparency, and conduct a deep exploration of these issues within the domain of decision-making in medical imaging. For that, we incorporate human feedback in the model training process to create better ML models.
In (3) and since this classification is possible to be automatically obtained, it is mandatory to confront these classification results to the medical team for two primary reasons:
(a) The problem of black-boxes raised in the context of deep neural networks.
(b) To ascertain if indeed the classification results conform with the perceived classification of radiologists.
We have noticed how radiologists are resistant to the introduction of such intelligent agents in their workflow. As such, (a) must be addressed. In (b), we expect to overcome this "roadblock" of trust in (a), by confronting the high accuracies in DNN classification results with the radiologists. We observed that these two issues are interconnected, and have to be jointly addressed.
Medical imaging diagnosis is a routine effort performed by radiologists to help diagnose or monitor a patient's medical condition. Medical imaging diagnosis allows physicians to identify pathologies by decoding tissue characteristics while examining visual properties in medical imaging. It plays a central role in modern medicine, particularly in the prevention and diagnosis of breast cancer, one of the leading causes of mortality worldwide. Breast cancer is the most common cancer in women worldwide, with nearly 1.7 million new cases diagnosed in 2012. However, proper classification, location, detection, targeting and registration of tumors requires the use of different imaging modalities that contribute to diagnostic reliability.
With the hype of new Artificial Intelligence (AI) methods such as Machine Learning (ML) and Deep Learning (DL) algorithms, automated agents (assisted by AI) are closer to the radiology room than ever before. This proximity becomes highly relevant for the medical community, which is interested in solutions to support clinical diagnosis. Greater stakeholder engagement requires sufficient validation and continuously monitored emerging AI tools as well as available breast cancer screening data. That said, national health systems are increasingly in need of methods, such as our solutions, for ongoing clinical diagnostic support to support medical decision-making.
Our goal is to provide AI solutions for radiology. We bring radiology experts and software engineers together to optimize healthcare professionals workload through the development of high-performance AI tools.
The aim of this project is to develop a system supported by intelligent agents for diagnosing medical images in the field of breast cancer. To address the effects of lack of clarity in AI results, we provide a service where our intelligent agents are applied to the breast cancer domain at different levels of medical expertise and multiple clinical workflows. Despite the promise of assisting clinicians in the decision-making process, there are two initial challenges that our services aim to address: (i) the lack of available and cured medical data to be consumed by AI algorithms; and (ii) the fact that medical professionals often find it difficult to understand how an AI system turns its outcome into a final medical decision. The goal is to develop a disruptive platform with the introduction of various AI techniques using an intelligent agent to communicate with the various physicians in an interpretable way. Specifically, focusing on how the multimodality and assistance of an AI model can add value to the medical workflow.
In our solution, we need to consider several factors when making decisions about how to add AI to medical practice. For instance, bias is a familiar concept for clinicians, who are already trained to practice evidence-based medicine. Our solution was designed concerning the interaction of clinicians and AI in a real-world setting. Hence, we provide several explainability techniques supported by summarizing the reasons of the AI behavior, gaining the user's trust, or producing insights about the decision-making causes. These explainability techniques were verified at top scientific publications, where evidence of effectiveness and efficiency was demonstrated. A key factor to our solution is pairing control functionalities across the AI outputs with several visual explainability techniques that empower the clinicians' choice and sense of control.
With this solution, clinicians reduced about 20% of the medical error. Specifically, the developed solution provides an approximate decrease of 20% for False-Positives and a decrease of 2% for False-Negatives. Moreover, this solution is reducing 35% of the time for clinicians to fully diagnose a patient. Due to these results, we are providing evidence [1] of an immense reduction of healthcare costs for governments and private institutions.
For radiologists who need an immediate second reader, BreastScreening-AI is a product that uses AI to aid breast cancer diagnosis, mitigating the clinical error and offering improvements in terms of clinical performance. Unlike the market alternatives, our product differentiates itself by using a user-centered approach, analyzing characteristics of medical behavior so that the assistant can promote better decision-making and trust from clinicians. An example of this is our developed explainability and intelligibility techniques, in which our assistant is summarizing the reasons of the AI suggestions visually. In this work, we are developing a system that communicates the suggestive recommendation depending on medical characteristics or accuracy of the AI model.
A significant number of technologies under development and in prototype or clinical trials, suggest that AI-powered diagnostic departments will feature in many future clinical institutions [2]. Diagnostic institutions are leveraging pattern recognition and DL to reduce diagnosis turnabout time [4], and improve pathology workflow accuracy and efficiency of the diagnostic.
Our solution is being used in more than nine public and private health institutions [1] in Portugal. We are currently scaling our services and solutions outside Portugal, where we already have the collaboration of more than 300 international physicians. From our networking of physicians, many belong to excellent radiology associations and institutions, such as the American College of Radiology (ACR) with collaboration with this project. Within the ACR (among other institutions), we will be able to further validate and promote our solution. Nevertheless, we already have an invention [2, 3] registered as a national patent, with the number 116801 and reference DP/01/2021/74923, as well as, registered internationally, as the number PCT/PT2021/050029.
As already mentioned, we are scaling our services and solutions outside Portugal, where we already have the collaboration of more than 300 international doctors. Of these physicians, many belong to excellent radiology associations and institutions, such as the ACR. Within the ACR (among other institutions), we will be able to further validate and promote our solution.
An approach of using AI to simulate clinical trials before human trials have also been seen, leaving plenty of scope available for what AI can create. From medical recommendations and lesion detection, to experimental clinical trials, the scope of this technology is rapidly expanding. Additionally, we aim to integrate our solution into the scope of the AI Portugal 2030 initiative, one of the main strategies of the INCoDe.2030 program.
Our AI system has been shown to reduce errors in human observation, as well as being as good as expert radiologists in the process of detecting cancer. The solution represents a breakthrough in early detection of breast cancer. Thus, saving many lives or reducing costs for the patient by developing an AI system able to identify cancers with a degree of precision similar to radiologists. Specifically, AI can lead clinicians to reduce the number of False-Positives and False-Negatives.
As the influence and value of this technology grows, so too does the importance of developing effective patent strategies for AI inventions. In this project, we need to address the key questions faced by us in this space of intellectual property. Moreover, we need to undrstand the best strategies and considerations for protecting AI-implemented inventions for healthcare products and services.
Currently, our research work follows a TRL6, meaning that the system prototype was already demonstrated in a relevant environment. In fact, we already studied our solution with 45 clinicians recruited on a volunteer basis from nine clinical institutions in Portugal.
By using the Innovation Maturity Level (IML), defined by CIMIT, will be applied as a matrix system to measure the maturity of four domains: Technology, Regulatory, Marketing/Business, and Clinical. Currently, we have fully completed the invention cycle and are dealing with translation. For the translation cycle, we already surpassed the IML 5 (Proof of Value), as we already studied and demonstrated the potential of the solution to work and create value for all stakeholders. Now, we are resolving the IML 6 (Initial Clinical Trials), as we are surpassing the regulated production of the prototypes and collection of clinical, as well as economic data.
At the moment, we have a registered patent at national and international level. However, this only represents a tiny slice of the potential inventions in our laboratory. For now, we will be present at the next edition of Lab2Market organized by the Technology Transfer Area at Instituto Superior Técnico. Our aim is to develop and register more patents, as well as further to create an organization that promotes services and products aimed at these solutions.
Getting real-world evidence concerning AI to be paid for, we need data that shows our solution is making a difference. Hence, our project provides evidence from our findings [1] to support the argument that AI can improve the medical imaging workflow.
Every year, about 30% to 50% of diagnosed cases result in False Positives [2]. The vast majority of these numbers are translated into biopsies resulting in an enormous cost to the national health system, as well as physically and psychologically harming the patient. Other than that, about 8% to 10% of diagnosed cases are False-Negatives, which often cause the patient's death. To improve both False Positives and False Negatives, we are able to reduce costs and improve doctors' lives [1], bringing better healthcare to patients. Further, AI will become a $2 billion industry worldwide. Making it essential to have a solution, as proposed in this invention.
[1] Francisco Maria Calisto, Carlos Santiago, Nuno Nunes, Jacinto C. Nascimento, Introduction of human-centric AI assistant to aid radiologists for multimodal breast image classification, International Journal of Human-Computer Studies, Volume 150, 2021, 102607, ISSN 1071-5819, https://doi.org/10.1016/j.ijhcs.2021.102607
[2] Calisto, F. M. (2020). Breast Cancer Medical Imaging Multimodality Lesion Contours Annotating Method. Instituto Superior Técnico. https://doi.org/10.13140/RG.2.2.14792.55049
[3] Francisco Maria Calisto, Nuno Nunes, and Jacinto C. Nascimento. 2020. BreastScreening: On the Use of Multi-Modality in Medical Imaging Diagnosis. In Proceedings of the International Conference on Advanced Visual Interfaces (AVI '20). Association for Computing Machinery, New York, NY, USA, Article 49, 1–5. DOI:https://doi.org/10.1145/3399715.3399744
[4] Francisco M. Calisto, Alfredo Ferreira, Jacinto C. Nascimento, and Daniel Gonçalves. 2017. Towards Touch-Based Medical Image Diagnosis Annotation. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). Association for Computing Machinery, New York, NY, USA, 390–395. DOI:https://doi.org/10.1145/3132272.3134111