Skip to content

Commit

Permalink
style: pre-commit fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
@pre-commit-ci
pre-commit-ci[bot] committed Oct 6, 2023
1 parent d637cc3 commit 613db67
Show file tree
Hide file tree
Showing 5 changed files with 219 additions and 94 deletions.
33 changes: 17 additions & 16 deletions site/docs/about.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -95,28 +95,29 @@ building and sharing an assurance case can be more involved. As such, a
significant element of the TEA platform is the user guidance and documentation
that helps establish best practices and enhance the capabilities of teams.

Our user guidance and documentation is continuously evolving, and the following tables provides an overview of what is currently available.[^gh_issues]
Our user guidance and documentation is continuously evolving, and the following
tables provides an overview of what is currently available.[^gh_issues]

#### Introductory Resources and User Guidance

| Title | Description | Type |
| --- | ------- | ---- |
| What is Trustworthy and Ethical Assurance? | A general introduction to trustworthy and ethical assurance as an ethical concept, rather than the methodology. | `Introductory Resource` |
| The Assurance Ecosystem—A Brief Overview | An overview of the assurance ecosystem, including key actors, responsibilites, and challenges. | `Introductory Resource` |
| An Introduction to Argument-Based Assurance | A simple introduction to argument-based assurance, including its history and motivation. | `Introductory Resource` |
| Operationalising Ethical Principles—Putting TEA into Practice | How to operationalise ethical principles | `User Guidance` |
| TEA and the Project Lifecycle | How to embed trustworthy and ethical assurance over the course of a project's lifecycle | `User Guidance` |
| The Role of Standards | An introduction to standards as they apply to trustworthy and ethical assurance. | `User Guidance` |
| Open Challenges in Assurance | An overview of open challenges and research questions, including links to further resources. | `User Guidance` |
| Title | Description | Type |
| ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ----------------------- |
| What is Trustworthy and Ethical Assurance? | A general introduction to trustworthy and ethical assurance as an ethical concept, rather than the methodology. | `Introductory Resource` |
| The Assurance Ecosystem—A Brief Overview | An overview of the assurance ecosystem, including key actors, responsibilites, and challenges. | `Introductory Resource` |
| An Introduction to Argument-Based Assurance | A simple introduction to argument-based assurance, including its history and motivation. | `Introductory Resource` |
| Operationalising Ethical Principles—Putting TEA into Practice | How to operationalise ethical principles | `User Guidance` |
| TEA and the Project Lifecycle | How to embed trustworthy and ethical assurance over the course of a project's lifecycle | `User Guidance` |
| The Role of Standards | An introduction to standards as they apply to trustworthy and ethical assurance. | `User Guidance` |
| Open Challenges in Assurance | An overview of open challenges and research questions, including links to further resources. | `User Guidance` |

#### Technical Documentation

| Title | Description |
| --- | ----- |
| About the Platform | An overview of the technical details of the platform. |
| Installation | Instructions on how to install the platform locally. |
| Cloud Deployment | Instructions for deploying the platform to a cloud environment, such as Azure. |
| API Details | Details of the platform's API. |
| Title | Description |
| ------------------ | ------------------------------------------------------------------------------ |
| About the Platform | An overview of the technical details of the platform. |
| Installation | Instructions on how to install the platform locally. |
| Cloud Deployment | Instructions for deploying the platform to a cloud environment, such as Azure. |
| API Details | Details of the platform's API. |

[^gh_issues]:
You can also see our GitHub issues for further details of other modules or
Expand Down
59 changes: 35 additions & 24 deletions site/docs/introductory-resources/what-is-tea.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,12 @@ local dealership. One of the sales advisors convinces you to buy a second hand
car that later turns out to have an issue with the engine. Frustrated, you take
the car back and the sales advisor apologises. They explain that all their
second hand cars undergo a thorough assessment before they are placed on the
market but, nevertheless, go on to process a return and get you a different car. You are
reassured and happy, but only for a short period of time. Yet again, the car
turns out to have a problem with the engine—the same problem as before! The
market but, nevertheless, go on to process a return and get you a different car.
You are reassured and happy, but only for a short period of time. Yet again, the
car turns out to have a problem with the engine—the same problem as before! The
sales advisor tries to convince you that this is just a series of unlucky
incidents, but without clear evidence to support their claims, this time around you do not trust
them and take your business elsewhere.
incidents, but without clear evidence to support their claims, this time around
you do not trust them and take your business elsewhere.

> Assurance involves communicating reasons and evidence that help people
> understand and evaluate the _trustworthiness_ of a claim (or series of claims)
Expand All @@ -29,8 +29,8 @@ In the above example, the sales advisor needed to provide assurance that their
cars were _safe_ or _reliable_, but the claims they made about the assessment
process were undermined by the contrary evidence (i.e. two unreliable cars).

In a different scenario, things may go differently. For instance, you may have
a higher level of trust with friends or other professionals (e.g. doctors,
In a different scenario, things may go differently. For instance, you may have a
higher level of trust with friends or other professionals (e.g. doctors,
teachers) than with car sales persons. And, as such, you may be more likely to
accept claims in the absence of evidence or in spite of repeated instances of
contrary evidence. However, the relationship between trust and assurance is
Expand All @@ -43,23 +43,25 @@ the design, development, and deployment of data-driven technologies.

## Building Trust and Communicating Trustworthiness for Data-Driven Technologies

Trustworthy and Ethical Assurance could apply to many systems, but the TEA platform specifically addresses data-driven technologies, such as artificial intelligence or digital twins.
Trustworthy and Ethical Assurance could apply to many systems, but the TEA
platform specifically addresses data-driven technologies, such as artificial
intelligence or digital twins.

There are many benefits and risks associated with the design, development, and
deployment of data-driven technologies. And, therefore, many organisations and companies
find themselves in a situation of needing to communicate to customers, users, or
stakeholders how they have maximised the benefits and minimised the risks
associated with their product, service, or system.
deployment of data-driven technologies. And, therefore, many organisations and
companies find themselves in a situation of needing to communicate to customers,
users, or stakeholders how they have maximised the benefits and minimised the
risks associated with their product, service, or system.

For example, an organisation building an autonomous vehicle may need to explain how their system is safe,
secure, fair, explainable, among other goals. How they achieve this will depend
on myriad contextual factors, including who they are communicating with (e.g.
regulators, potential customers).
For example, an organisation building an autonomous vehicle may need to explain
how their system is safe, secure, fair, explainable, among other goals. How they
achieve this will depend on myriad contextual factors, including who they are
communicating with (e.g. regulators, potential customers).

Consider the goal of _safety_ with respect to the following questions:

- _How was the performance of the autonomous vehicle evaluated, and how will it be
monitored?_ There are many metrics that can be used to evaluate the
- _How was the performance of the autonomous vehicle evaluated, and how will it
be monitored?_ There are many metrics that can be used to evaluate the
performance of an autonomous vehicle, including metrics that assess the
performance of components of the vehicle such as the object recognition system
(e.g. its accuracy, robustness, interpretability) as well as metrics that
Expand All @@ -86,13 +88,19 @@ the assurance of data-driven technologies, both within and between different
domains. There are, for instance, a recurring set of goals (or, principles) that
people emphasise when asked about the _ethical_ or _societal_ issues related to
data-driven technologies (e.g. fairness and bias, transparency and
explainability)[^principles]. And, furthermore, there is a growing set of techniques
and practices in place for building trust through open, transparent and accessible
forms of communication [^ttw].
explainability)[^principles]. And, furthermore, there is a growing set of
techniques and practices in place for building trust through open, transparent
and accessible forms of communication [^ttw].

[^principles]: Floridi, L., & Cowls, J. (2019). A Unified Framework of Five Principles for AI in Society. Harvard Data Science Review, 1(1). https://doi.org/10.1162/99608f92.8cd550d1
[^principles]:
Floridi, L., & Cowls, J. (2019). A Unified Framework of Five Principles for
AI in Society. Harvard Data Science Review, 1(1).
https://doi.org/10.1162/99608f92.8cd550d1

[^ttw]: The Turing Way Community. (2022). The Turing Way: A handbook for reproducible, ethical and collaborative research. Zenodo. doi: 10.5281/zenodo.3233853.
[^ttw]:
The Turing Way Community. (2022). The Turing Way: A handbook for
reproducible, ethical and collaborative research. Zenodo. doi:
10.5281/zenodo.3233853.

Trustworthy and ethical assurance is a framework that is anchored in these
similarities and existing techniques, but also recognises the importance of
Expand All @@ -103,4 +111,7 @@ design, development, or deployment of a data-driven technology. The methodology
serves as a guide for developing the cases, while the platform helps to build
and communicate them with the wider community or stakeholders.

The remaining parts of this section provide further context to help situate and motivate trustworthy and ethical assurance. However, if you'd prefer to jump straight in, you can jump to our [user guidance](user-guidance/index.md) section instead.
The remaining parts of this section provide further context to help situate and
motivate trustworthy and ethical assurance. However, if you'd prefer to jump
straight in, you can jump to our [user guidance](user-guidance/index.md) section
instead.
51 changes: 41 additions & 10 deletions site/drafts/argument-based-assurance.md
View file
Original file line number Diff line number Diff line change
@@ -1,23 +1,54 @@
# An Introduction to Argument-Based Assurance

The TEA platform is built around a methodology known as argument-based assurance (ABA).
The TEA platform is built around a methodology known as argument-based assurance
(ABA).

The general approach of ABA is to provide a reasoned and justified argument regarding some top-level claim (i.e. the goal). Historically, ABA has been used in safety-critical domains, such as aviation or energy, and so the top-level claims typically refer to goals such as reliability, safety, or security of a system. The main argument sets out the claims made about the system (including aspects of the project governance or design), the evidence supporting those claims, and the rationale that links the evidence to the claims.
The general approach of ABA is to provide a reasoned and justified argument
regarding some top-level claim (i.e. the goal). Historically, ABA has been used
in safety-critical domains, such as aviation or energy, and so the top-level
claims typically refer to goals such as reliability, safety, or security of a
system. The main argument sets out the claims made about the system (including
aspects of the project governance or design), the evidence supporting those
claims, and the rationale that links the evidence to the claims.

!!! note "From safety assurance to ethical assurance"

The TEA platform instantiates a type of argument-based assurance that focuses on ethical goals and principles (e.g. fairness, explainability), which help establish justified trust in data-driven technologies, such as artificial intelligence or digital twins.

ABA follows a process critical reasoning to create a convincing "story" or "case" that articulates why a system can be trusted to operate within certain contexts. The final artefact is known as an assurance case, and is typically presented in a visually intuitive form that supports accessible communication and assists critical engagement (e.g. identifying gaps in the argument, evaluating the strength of supporting evidence for linked claims). As such, ABA helps teams and stakeholders consider both positive evidence as well as possible counterarguments, gaps, and uncertainties, offering mitigations for those when possible.
ABA follows a process critical reasoning to create a convincing "story" or
"case" that articulates why a system can be trusted to operate within certain
contexts. The final artefact is known as an assurance case, and is typically
presented in a visually intuitive form that supports accessible communication
and assists critical engagement (e.g. identifying gaps in the argument,
evaluating the strength of supporting evidence for linked claims). As such, ABA
helps teams and stakeholders consider both positive evidence as well as possible
counterarguments, gaps, and uncertainties, offering mitigations for those when
possible.

!!! info "History of argument-based assurance"

Add a short history of ABA.

One of the key strengths of this approach is its ability to facilitate clear communication among stakeholders, including researchers, developers, regulators, and system users. By making assumptions explicit and providing a structured framework for critical reasoning, ABA facilitates a transparent, understandable and reproducible assurance process. It also offers a flexible and extensible way to integrate various types of evidence (and [standards](standards.md)), such as empirical data, expert opinion, and formal methods, into a cohesive argument.

However, ABA is not without its challenges[^habli]. Constructing a rigorous argument requires significant expertise, can be time-consuming, and the quality of the argument is heavily dependent on the strength and sufficiency of the underlying evidence. Furthermore, there are also concerns regarding how to update or modify assurance arguments as systems evolve or when new information becomes available.

To address some of these challenges, the TEA platform takes a community-driven approach to increasing capabilities and best practices within the [assurance ecosystem](assurance-ecosystem.md).

[^habli]: Habli, I., Alexander, R., & Hawkins, R. D. (2021). Safety Cases: An Impending Crisis? In Safety-Critical Systems Symposium (SSS’21). https://core.ac.uk/download/pdf/363148691.pdf
One of the key strengths of this approach is its ability to facilitate clear
communication among stakeholders, including researchers, developers, regulators,
and system users. By making assumptions explicit and providing a structured
framework for critical reasoning, ABA facilitates a transparent, understandable
and reproducible assurance process. It also offers a flexible and extensible way
to integrate various types of evidence (and [standards](standards.md)), such as
empirical data, expert opinion, and formal methods, into a cohesive argument.

However, ABA is not without its challenges[^habli]. Constructing a rigorous
argument requires significant expertise, can be time-consuming, and the quality
of the argument is heavily dependent on the strength and sufficiency of the
underlying evidence. Furthermore, there are also concerns regarding how to
update or modify assurance arguments as systems evolve or when new information
becomes available.

To address some of these challenges, the TEA platform takes a community-driven
approach to increasing capabilities and best practices within the
[assurance ecosystem](assurance-ecosystem.md).

[^habli]:
Habli, I., Alexander, R., & Hawkins, R. D. (2021). Safety Cases: An
Impending Crisis? In Safety-Critical Systems Symposium (SSS’21).
https://core.ac.uk/download/pdf/363148691.pdf

0 comments on commit 613db67

Please sign in to comment.