From 37694da59f60ea0986b947b3a57b060045ec1500 Mon Sep 17 00:00:00 2001 From: Jari Kasandiredjo Date: Sat, 10 Feb 2024 13:11:46 +0100 Subject: [PATCH 1/2] fix typo disproportionately --- introduction/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/introduction/index.md b/introduction/index.md index e46aa5b6..258fd091 100644 --- a/introduction/index.md +++ b/introduction/index.md @@ -1220,7 +1220,7 @@ slides: true
-

This aspect of the Xinjiang case makes it very unusual. It's much more common that designers try to avoid using ethnicity, race or other sensitive attributes, and that information finds its way into the system anyway. In such cases the effects that are considered harmful are not intended.

In this article, the organization ProPublica broke the news that a system called COMPAS, used nation-wide in the United States to aid parole decisions was considerably more likely to deny black people parole than white people, even when all other factors were accounted for.

This was not an explicit design choice of the makers of the system (a company called NorthPointe). In fact, they explicitly excluded race as a feature. However, even if we exclude sensitive attributes as features, we often can still infer them from other features. For instance, we may include a feature like a subject's postcode. This is usually strongly correlated with race, and so the system can still make the classification it would have made if race had been available.

Contrast this with the previous situation. As before, the makers of the system deny the allegations of causing harm. Here, however, there is agreement on whether classifying by race is harmful. Both parties, ProPublica and Northpoint, presumably agree that a system would cause harm if it disproportionally denied black people parole. What they disagree on is whether this system does that. The effect that ProPublica alleges (and provides credible evidence for) in unintentional.

source: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

For references, see the social impact dossier.

+

This aspect of the Xinjiang case makes it very unusual. It's much more common that designers try to avoid using ethnicity, race or other sensitive attributes, and that information finds its way into the system anyway. In such cases the effects that are considered harmful are not intended.

In this article, the organization ProPublica broke the news that a system called COMPAS, used nation-wide in the United States to aid parole decisions was considerably more likely to deny black people parole than white people, even when all other factors were accounted for.

This was not an explicit design choice of the makers of the system (a company called NorthPointe). In fact, they explicitly excluded race as a feature. However, even if we exclude sensitive attributes as features, we often can still infer them from other features. For instance, we may include a feature like a subject's postcode. This is usually strongly correlated with race, and so the system can still make the classification it would have made if race had been available.

Contrast this with the previous situation. As before, the makers of the system deny the allegations of causing harm. Here, however, there is agreement on whether classifying by race is harmful. Both parties, ProPublica and Northpoint, presumably agree that a system would cause harm if it disproportionately denied black people parole. What they disagree on is whether this system does that. The effect that ProPublica alleges (and provides credible evidence for) in unintentional.

source: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

For references, see the social impact dossier.

click image for animation @@ -1234,7 +1234,7 @@ slides: true
-

A system like COMPAS, that disproportionally denies black people parole is said to have a bias. This kind of bias can come from different places.

One important source of bias is the distribution of the training data. Where we get our data has a tremendous impact on what the model learns. Since machine learning often requires large amounts of data, we usually can’t afford to control the gathering of data very carefully: unlike studies in life sciences, medicine and so on, we rarely make sure that all variables are carefully controlled.

The result is that systems have unexpected biases. This is a picture of Joy Buolamwini. As a PhD student, she worked on existing face recognition systems. She found that if she tested them on her own face, they would not recognize her, and she needed to wear a light-colored mask to be recognized at all.

One aspect of this problem is the bias in the data that face recognition systems are trained on. If, for instance, such data is gathered carelessly, we end up inheriting whatever biases our source has. If white people are overrepresented, then we end up training a system that works less well on non-white people.

image source: https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html

+

A system like COMPAS, that disproportionately denies black people parole is said to have a bias. This kind of bias can come from different places.

One important source of bias is the distribution of the training data. Where we get our data has a tremendous impact on what the model learns. Since machine learning often requires large amounts of data, we usually can’t afford to control the gathering of data very carefully: unlike studies in life sciences, medicine and so on, we rarely make sure that all variables are carefully controlled.

The result is that systems have unexpected biases. This is a picture of Joy Buolamwini. As a PhD student, she worked on existing face recognition systems. She found that if she tested them on her own face, they would not recognize her, and she needed to wear a light-colored mask to be recognized at all.

One aspect of this problem is the bias in the data that face recognition systems are trained on. If, for instance, such data is gathered carelessly, we end up inheriting whatever biases our source has. If white people are overrepresented, then we end up training a system that works less well on non-white people.

image source: https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html

From f9e2ea8e5dc0f25ad9fa8b0653ffdcc56d96c73e Mon Sep 17 00:00:00 2001 From: Jari Kasandiredjo Date: Sat, 10 Feb 2024 13:12:03 +0100 Subject: [PATCH 2/2] fix typo hanging a --- introduction/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/introduction/index.md b/introduction/index.md index 258fd091..1585df71 100644 --- a/introduction/index.md +++ b/introduction/index.md @@ -1393,7 +1393,7 @@ slides: true
-

So, let’s return to our gender classifier, and ask some of these questions. Are sex and gender a sensitive attributes and if so, what should we do about gender classification?

We've already seen, in the translation example, that data bias is an important problem when dealing with gender in data. Even if genders are carefully represented in your data, they may be associated in a biased way, such as associating doctors with men and nurses with women. As we saw, even if these biases are an accurate reflection of the state of society, we may still be in danger of amplifying them.

Still, that does not in itself preclude us from using sex or gender as a target attribute for classification. To understand the controversy, we need to look at different questions.

+

So, let’s return to our gender classifier, and ask some of these questions. Are sex and gender sensitive attributes and if so, what should we do about gender classification?

We've already seen, in the translation example, that data bias is an important problem when dealing with gender in data. Even if genders are carefully represented in your data, they may be associated in a biased way, such as associating doctors with men and nurses with women. As we saw, even if these biases are an accurate reflection of the state of society, we may still be in danger of amplifying them.

Still, that does not in itself preclude us from using sex or gender as a target attribute for classification. To understand the controversy, we need to look at different questions.