Skip to content

5.1 Examining AI Ethics

Jump to bottom
javieraatenas-pixel edited this page Jun 15, 2026 · 3 revisions

Content adapted from:

  • Atenas, J., Picasso, F., Nerantzi, C., Havemann, L., Serbati, A., Agostini, D., & Caldwell, N. (2026). Critical data and Artificial Intelligence literacies in academic practice. In W. Holmes (Ed.), Handbook of critical studies of Artificial Intelligence and education (pp. 265–283). Edward Elgar Publishing. https://doi.org/10.4337/9781035335879.00030
  • Atenas, J., & Caldwell, N. H. M. (2026). AI, data and information literacy for the digital student. In The Digital Student (pp. 197–235). Elsevier. https://doi.org/10.1016/B978-0-443-34057-4.00008-0
  • Atenas, Javiera. (2021). The Datafied Present and Future. Zenodo. https://doi.org/10.5281/ZENODO.4698609
  • Atenas, J., Havemann, L., & Timmermann, C. (2023). Reframing data ethics in research methods education: A pathway to critical data literacy. International Journal of Educational Technology in Higher Education, 20(1), 11. https://doi.org/10.1186/s41239-023-00380-y

Introduction

Data ethics refers to the principles that guide what is considered right or wrong throughout the entire data lifecycle, from its collection and production to its use and dissemination. It recognises that not all data—whether public or private—should be published or used without consideration, as data practices can have harmful consequences. Ethical data use is shaped by regulations designed to balance benefits and risks for individuals, organisations, and society, ensuring data is used responsibly and without causing harm. A key concern is how human-generated data is handled, particularly in ways that protect individuals and vulnerable communities from intrusive or exploitative uses, while still enabling data to contribute positively to social, educational, and economic contexts.

To design teaching and learning activities regarding the ethical boundaries of AI, algorithms, and machine learning, it is essential to understand how their opacity affects us all directly and indirectly. Students, as citizens, need to develop the awareness and competencies to participate in democratic discussions and contribute to legal frameworks that prevent misuses or unethical applications of AI. Accordingly, UNESCO stresses the importance of “educating algorithms,” while citizens must be capable of understanding potential problems and challenging them.

flowchart TD

    A[AI System Design]
    --> B[Data Collection]
    --> C[Algorithm Development]
    --> D[Deployment]
    --> E[Impact on Individuals and Society]

    %% Ethics checkpoints
    B --> B1[Bias and Data Quality]
    C --> C1[Fairness and Transparency]
    D --> D1[Accountability and Governance]
    E --> E1[Social Impact and Justice]

    %% Rights and safeguards
    E --> F[User Rights and Protections]
    F --> F1[Privacy]
    F --> F2[Explainability]
    F --> F3[Contestability]

    %% Styles (pastel + black font)
    style A fill:#FFE4E1,color:#000
    style B fill:#E6F2FF,color:#000
    style C fill:#E6FFE6,color:#000
    style D fill:#FFF5E1,color:#000
    style E fill:#F3E6FF,color:#000

    style B1 fill:#FDEDEC,color:#000
    style C1 fill:#EBF5FB,color:#000
    style D1 fill:#E8F8F5,color:#000
    style E1 fill:#FEF9E7,color:#000

    style F fill:#E0F7FA,color:#000
    style F1 fill:#F5EEF8,color:#000
    style F2 fill:#F5EEF8,color:#000
    style F3 fill:#F5EEF8,color:#000
Loading

Data ethics refers to the principles that guide what is considered right or wrong throughout the entire data lifecycle, from its collection and production to its use and dissemination. It recognises that not all data—whether public or private—should be published or used without consideration, as data practices can have harmful consequences. Ethical data use is shaped by regulations designed to balance benefits and risks for individuals, organisations, and society, ensuring data is used responsibly and without causing harm. A key concern is how human-generated data is handled, particularly in ways that protect individuals and vulnerable communities from intrusive or exploitative uses, while still enabling data to contribute positively to social, educational, and economic contexts.

Given these risks, it is critical to protect vulnerable populations from predatory and harmful uses of AI (Nature), ensuring that technological interventions do not exacerbate existing inequalities. Education and policy frameworks must address these challenges to safeguard equity and justice.

Safiya Umoja Noble has highlighted how algorithms can function as tools of oppression (YouTube), sparking discussions about unethical or illegal uses of AI. Some of the key categories of impact include:

Racism

The opacity of algorithms creates “black boxes” (Michigan Law Review), which have led to discriminatory outcomes such as:

  • Consumer lending discrimination and barriers to obtaining visas (BBC)
  • Educational inequities through unfair learning analytics (UNESCO PDF) and student surveillance (EFF)
  • Facial recognition bias, disproportionately affecting Black people (ACLU, Primer PDF)
  • Racist predictive policing, contributing to longer incarceration sentences (Brennan Center, YouTube)

These examples underline the urgent need for regulatory frameworks to prevent systemic discrimination through AI.

  • Sexism: Gender bias in AI arises partly because 78% of AI professionals are men (World Economic Forum 2018), meaning male experiences often dominate algorithmic design. This has significant consequences for women:

  • Health outcomes: Misdiagnosis (Women in Global Health), incorrect treatment prescriptions (BI Health), and damaged health (Forbes)

  • Financial discrimination: Unequal access to loans or financial services (BBC)

  • Employment bias: Targeted lower-paying job ads (Euronews), biased HR tests (Quartz), and discriminatory hiring algorithms (Reuters)

  • Impact on Queer and Trans Communities: AI can harm non-binary and trans individuals by portraying queer and trans people in negative or stereotypical ways (Real Life Magazine, Vice, The Guardian)

  • Reinforcing societal biases through algorithmic predictions

Ethical AI development must address the needs of non-binary and trans people (Model View Culture) to protect them from potential harm and ensure equitable treatment.

  • Socioeconomic Discrimination: Algorithms can disproportionately affect individuals from lower-income households and disadvantaged neighborhoods. Examples include:

  • Education: AI systems have been shown to lower school grades for students from disadvantaged backgrounds (The Guardian). This phenomenon is often referred to as automating poverty or automating inequality (Data & Society).

  • Welfare and Public Services: AI is used to assign or remove unemployment benefits, child support, housing, and food subsidies (The Guardian). In extreme cases, this has led to death (ENewsroom) or severe health issues.

  • Systemic Oppression: Automated systems may request biometric data to access essential services such as school meals, reinforcing structural inequalities (Privacy International).

  • Children Rights and AI: UNICEF emphasises the need to protect children's rights, particularly as AI-driven decisions affect poor families in welfare systems (UNICEF PDF). Automated systems determine eligibility for benefits and map poverty (The Guardian, The Guardian, ADB, Electronics Specifier).

These systems may reinforce negative stereotypes depending on school attended, residential location (OD4D Education, DSL Redlining Map), and socio-economic status, and even predict poverty outcomes (Stanford Sustain).

  • Surveillance : Businesses, employers, educational organisations, and governments increasingly use surveillance mechanisms to monitor and influence behaviour:

  • Retail: Shops track customer movements and behaviours using AI-powered cameras (Security Info Watch).

  • Workplaces: Companies monitor employees’ activities, productivity, and online interactions (The Guardian).

  • Education: Schools monitor children’s engagement (Taylor & Francis Online) and universities use online proctoring systems for exam supervision (Washington Post).

Shoshana Zuboff terms this pervasive tracking and monitoring surveillance capitalism (YouTube).

  • State-Sponsored AI Surveillance: The Carnegie Endowment for International Peace highlights the global expansion of AI surveillance, with many states deploying advanced tools to monitor citizens for policy objectives—some lawful, some violating human rights, and many in a grey zone (Carnegie Endowment). They developed the AI Global Surveillance (AIGS) Index (Carnegie AIGS) to track this proliferation worldwide.

  • Calls for Regulation: International organisations stress the need for ethical frameworks and regulations to prevent misuse of AI surveillance:

  • United Nations: Advocates for responsible AI deployment and ethical safeguards (UN)

  • UNESCO: Highlights ethical risks of AI in society (UNESCO)

  • European Council: Provides legal and policy recommendations on AI and human rights (European Council PDF)

  • OECD: Issues principles for AI, including fairness, accountability, transparency, and privacy (OECD AI Principles)

These efforts aim to ensure that AI surveillance does not infringe on human rights or enable abusive monitoring practices.

  • Manipulation: AI has increasingly been used for social influence and behaviour manipulation (Emerj), particularly via social media, targeting individuals’ political views and opinions (Mind Matters).

Key concerns include:

  • Propaganda and targeted messaging: AI can be used to spread content designed to radicalise or polarise groups (Open University, Council of Europe).
  • Threats to democracy: Algorithmic manipulation can undermine democratic processes and public trust (CFR, Medium).

A regulatory framework for targeted information in political campaigns is necessary to mitigate these risks (European Parliament PDF).

Data ethics in practice

Data ethics is a branch of applied ethics concerned with evaluating the moral implications of data, algorithms, and their associated practices across the entire data lifecycle. It extends beyond technical considerations to examine how data is collected, interpreted, and used within social, political, and economic contexts. Unlike information ethics, which focuses on meaning and interpretation, data ethics engages at a broader level, interrogating the assumptions, power structures, and human decisions that shape data systems. As a core component of data literacy, it requires individuals not only to handle and analyse data, but also to critically assess its origins, biases, and impacts, recognising that data is constructed rather than neutral or inevitable.

In contemporary datafied societies, ethical concerns are intensified by the rise of algorithmic decision-making, surveillance capitalism, and systemic bias embedded in datasets and AI systems. These technologies can reproduce and amplify inequalities across domains such as education, employment, and healthcare. Higher education therefore plays a crucial role in fostering critical data literacy, equipping learners to understand and challenge these dynamics. A critical and care-based pedagogical approach emphasises social justice, collective responsibility, and active participation, encouraging students to move beyond technical competence to ethical awareness and civic engagement. Ultimately, data ethics aims to empower individuals to act as informed, responsible agents capable of questioning and reshaping data practices for the public good.

flowchart TD

A[Data Generation] --> B[Data Collection]
B --> C[Data Processing & Cleaning]
C --> D[Analysis & Algorithms]
D --> E[Interpretation & Meaning-Making]
E --> F[Decision-Making]
F --> G[Social Impact]

%% Critical layers
C --> H[Bias & Inequality]
H --> D

D --> I[Assumptions & Design Choices]
I --> E

F --> J[Institutional Use]
J --> G

%% Feedback loop
G --> K[Public Response & Awareness]
K --> A

%% Ethical and educational intervention
G --> L[Ethical Reflection]
L --> M[Critical Data Literacy]
M --> N[Responsible Action]
N --> B

%% Styling pastel colours with black fonts
style A fill:#FDE2E4,color:#000000
style B fill:#E2ECE9,color:#000000
style C fill:#FFF1E6,color:#000000
style D fill:#E0FBFC,color:#000000
style E fill:#F1E3F3,color:#000000
style F fill:#FFE5D9,color:#000000
style G fill:#EDE7B1,color:#000000

style H fill:#FFD6A5,color:#000000
style I fill:#E4C1F9,color:#000000
style J fill:#CDEAC0,color:#000000

style K fill:#BDE0FE,color:#000000
style L fill:#FAD2E1,color:#000000
style M fill:#D0F4DE,color:#000000
style N fill:#FFC8DD,color:#000000
Loading

Recommended Activity

To explore discrimination and manipulation through algorithms:

  1. Predictive Policing Simulation:
    Students can interact with the Pre-crime Calculator to understand how predictive policing systems classify potential suspects or victims and how location data influences risk perception.

  2. Personality Test Experiment:
    Ask students to take an online test like the Business Personality Profile using male and female identities with similar traits. Compare the results to observe potential biases in algorithmic evaluation.

  3. Class Discussion:
    Have students share their findings and reflect on how algorithms can reinforce bias or influence behaviour, considering the ethical and societal implications.

Clone this wiki locally