5.5 Introduction to Algorithmic Auditing

Note: Content adapted from Atenas, J., & Caldwell, N. H. M. (2026). AI, data and information literacy for the digital student. In The Digital Student (pp. 197–235). Elsevier. https://doi.org/10.1016/B978-0-443-34057-4.00008-0

Introduction to Algorithmic Auditing

Algorithmic auditing is the structured interrogation of socio-technical systems to verify that model behaviour, data pipelines, and deployment contexts conform to declared legal, ethical, and performance requirements. It draws on a wide range of interdisciplinary perspectives, including computer science, statistics, law, ethics, and social sciences, to examine how algorithms operate in practice and what impacts they produce in real-world settings.

At its core, algorithmic auditing aims to evaluate whether automated decision-making systems are fair, transparent, accountable, and aligned with societal values. It involves systematic investigation of data inputs, model design, training processes, and outputs, as well as the broader organisational and institutional contexts in which these systems are embedded.

Auditing practices can take different forms, including internal audits conducted by developers or organisations, and external audits carried out by regulators, independent researchers, or civil society groups. These approaches may include technical testing of model behaviour, analysis of datasets for bias or representational issues, and evaluation of governance structures and compliance with regulatory frameworks.

More broadly, algorithmic auditing also engages with questions of power, responsibility, and trust in data-driven systems. It recognises that algorithms are not neutral tools, but sociotechnical systems that reflect and reproduce human decisions, assumptions, and inequalities.

As algorithmic systems increasingly shape decisions in domains such as finance, healthcare, education, and public policy, auditing has become a key mechanism for ensuring accountability and protecting public interest in automated environments.

Algorithmic Auditing workflow

flowchart TD

%% ================================
%% LEVEL 1 — FOUNDATION (PASTEL PINK)
%% ================================
subgraph L1C["Level 1 — Foundation"]
direction TB

L1([Foundation Level])

F1[Sociotechnical Problem Framing]
F2[Data & Documentation Literacy]
F3[Critical Interpretation]

G1{Foundation<br/>Gate Review}

L1 --> F1 --> F2 --> F3 --> G1
G1 -- Fail --> R1["Remediation:<br/>training & supervision"]
R1 --> F1
G1 -- Pass --> C1["[Foundation Certified]"]

end

%% ================================
%% LEVEL 2 — INTERMEDIATE (PASTEL BLUE)
%% ================================
subgraph L2C["Level 2 — Evaluation"]
direction TB

L2([Intermediate Level])

I1["Empirical Measurement<br/>and Metrics"]
I2["Testing & Validation<br/>(Correctness & Harm)"]
I3["Traceability &<br/>System Analysis"]

G2{Evaluation<br/>Gate Review}

C1 --> L2
L2 --> I1 --> I2 --> I3 --> G2
G2 -- Fail --> R2["Remediation:<br/>restricted practice,<br/>guided audits"]
R2 --> I1
G2 -- Pass --> C2["[Evaluation Certified]"]

end

%% ================================
%% LEVEL 3 — ADVANCED (PASTEL GREEN)
%% ================================
subgraph L3C["Level 3 — Assurance"]
direction TB

L3([Advanced Level])

A1["Assurance Design<br/>and Lifecycle Reasoning"]
A2["Legal & Regulatory<br/>Alignment"]
A3["Independence,<br/>Ethics & Reflexivity"]
A4["Strategic Risk<br/>Governance"]

G3{Assurance<br/>Board Review}

C2 --> L3
L3 --> A1 --> A2 --> A3 --> A4 --> G3
G3 -- Fail --> R3["Remediation:<br/>governance or<br/>ethical reassessment"]
R3 --> A1
G3 -- Pass --> C3["[Assurance Authorised]"]

end

%% ================================
%% AUTHORISED PRACTICE
%% ================================
C3 --> P([Permission to Issue<br/>Public or Regulatory Assurance])

%% ================================
%% LEVEL BACKGROUND COLOURS
%% ================================
style L1C fill:#FFF1F3,stroke:#E5A3B3,stroke-width:2px
style L2C fill:#F2F7FF,stroke:#9DBCEB,stroke-width:2px
style L3C fill:#F2FBF5,stroke:#9CD3AE,stroke-width:2px

Loading