Define unwanted data categories in datasets

[deanwampler]

There is a discussion in the now-archived agents-wg project about "unwanted data catagories in datasets": https://github.com/The-AI-Alliance/agents-wg/discussions/29. This discussion copies over that content, as it's a relevant topic for OTDI.

@billstout-snc posted the following to start the discussion:

Suggestion: Define categories of unwanted data in datasets, declare those categories in dataset cards

The AI supply chain starts with data. Datasets are a collection of assets and liabilities. Assets include information useful for a model, and there's debate on if liabilities are useful for a model to know and recognize, or if a model should not have been trained on unwanted data.

Example categories of potentially unwanted data listed below:

Category	Description of the Problem	Impacted Party	Evidence (Links)
Personal Identifiable Information (PII)	Can violate privacy laws (e.g., GDPR/CCPA) if scraped or shared without consent; risk of regulatory fines and reputational damage.	Producer & Deployer	Clearview AI scraping case
Sensitive Health Data	Sharing or using patient data without proper consent can violate HIPAA and other healthcare privacy regulations; risk of lawsuits and heavy fines.	Producer & Deployer	Google’s “Project Nightingale”
Copyrighted or Licensed Content	Ingesting protected text or images without permission can lead to infringement claims; potential lawsuits, takedowns, or cease-and-desist orders.	Producer & Deployer	Getty Images lawsuit vs. Stability AI
Defamatory or Misinformation	Models trained on false or defamatory data may reproduce harmful statements, exposing deployers to defamation claims; can erode trust in AI systems.	Producer & Deployer	OpenAI sued for defamation over ChatGPT “hallucinations”
Hate Speech or Extremist Content	Risk of amplifying hateful or violent ideologies; can damage brand reputation and invite regulatory scrutiny.	Deployer	YouTube algorithm controversy
Unlabeled / Poorly Labeled Data	Incorrect, biased, or offensive labels can propagate errors, biases, or harmful outcomes in downstream models; may require costly dataset rework.	Producer	MIT’s withdrawal of “80 Million Tiny Images” dataset
Malware or Malicious Code	Trojans or backdoors hidden in training data or dependencies can compromise models, infrastructure, or end-user systems.	Producer	PyTorch-nightly dependency compromise
Regionally Restricted or Illegal Content	Material that is banned or heavily regulated in certain countries (e.g., Nazi symbols in Germany, certain religious or political content in other jurisdictions). Violations can result in fines, local bans, or legal actions.	Producer & Deployer	Germany’s ban on Nazi symbols in games
Child Sexual Exploitation (CSE) Material	Universally illegal. Even inadvertent inclusion in datasets can lead to severe criminal penalties; immediate takedown notices, public outcry, and possible prosecution.	Producer & Deployer	Platform struggles with CSE detection
Locally Censored or Regulated Content	Many regions impose censorship on specific political, religious, or cultural information (e.g., speech critical of government, certain religious references). Failing to comply can lead to shutdowns or blocking within a region.	Deployer	Google facilitated Russia and China’s censorship requests (coverage of broader content restrictions)

@julietshen replied with this comment:

A few good sources to cross-reference definitions and categories with include the TSPA (trust and safety professional association)'s abuse types page: https://www.tspa.org/curriculum/ts-fundamentals/policy/abuse-types/.

CSE, for example, may have different degrees of feasibility depending on if it's text or multimodal. (not a lawyer and based on my best guess, I'd recommend checking with a lawyer here) Grooming text is not necessarily treated the same way as CSAM media and thus may be somewhat easier to detect without having to get licenses to access databases of known CSAM hashes.

@roostorg is working on some tooling to make hash matching against known illegal content easier, building on the work done in https://github.com/facebook/ThreatExchange/tree/main/hasher-matcher-actioner.

And then, of course, the need to reflect these categories of unwanted data in a global perspective and ensure any filtering is effective for non-english content.

[billbrietstout]

Potentially social media brain rot is a category.

Supply chain related - Social media rots LLM brains as well
https://arxiv.org/abs/2510.13928

https://llm-brain-rot.github.io/

[hexsyro]

This is a solid taxonomy. One gap in practice: most unwanted data categories (PII, hate speech, spam, bots) are only detectable at the content level, but social media datasets also need engagement-based signals — bot likelihood, author verification, cross-platform dedup — to catch adversarial content that looks clean in isolation.

We built a deterministic enrichment pipeline at Social Intel that scores every post across 25 quality checks before delivery, including bot detection, toxicity scoring, and PII filtering. Every category in your table maps to a check we run upstream, so the dataset ships clean rather than requiring downstream filtering.