Exponentiated gradient implementation for bounded group loss by dgiova · Pull Request #442 · fairlearn/fairlearn

dgiova · 2020-05-29T19:59:09Z

Summary of contributions:

added signed_weights method for default objective.
implemented logic to compute the best response estimator for bounded group loss.
developed method to produce randomized predictions, given the set of estimators produced by the exponentiated gradient optimization.

closes #380

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

…rolutz/exponentiated_gradient_tests Signed-off-by: Roman Lutz <rolutz@microsoft.com>

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

…rolutz/exponentiated_gradient_tests

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

gamma and eps + test codes in BGL

Rolutz/exponentiated gradient tests

Lauren dev

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

…davide/exponentiated_gradient-BGL

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

romanlutz · 2020-07-09T07:04:57Z

CHANGES.md has this already from the previous PR, documentation (user guide) exists now for the moment, although we still need to write the expgrad user guide (separate PR), eps removed from lagrangian constructor, but tests for BGL still missing.

…c format Signed-off-by: Roman Lutz <rolutz@microsoft.com>

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

…davide/exponentiated_gradient-BGL

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

…davide/exponentiated_gradient-BGL

…all moments Signed-off-by: Roman Lutz <rolutz@microsoft.com>

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

…davide/exponentiated_gradient-BGL

riedgar-ms

I'm not sure there aren't two separate changesets here, but I'm not sure it's worth the trouble to separate them.

…davide/exponentiated_gradient-BGL

MiroDudik · 2020-07-29T15:08:31Z

-            y = self.flipped_y
-        else:
-            y = self.y
+    smoke_test_data_regression = [


Two issues with the current tests:

you also need to check expgrad.weights_ not just expgrad.predictors_

it would be good to try setting upper_bound to a value that's different from eps and something that would make the problem feasible--currently all your examples have infeasible constraints... so you could maybe try upper_bound=0.05, which would make some of the examples feasible and some infeasible; i'd also suggest setting nu=1e-6 or 1e-3 and max_iter=20 in these runs (since the automatic setting of nu needs to be double-checked and might change in the future)

yes! Updated.

Good point. I thought that's an issue since the gap doesn't go to 0. Will add a few more tests, although I'll leave the existing ones there as well. In practice that can happen, too, and we should have tests covering that. I'm adding the tests for upper_bound=0.05 with best_gap=0.0. Regarding nu and max_iter restrictions: The number of iterations is never larger than 5 or 6 so that wouldn't do anything, and that applies even when I set nu to 1e-6. I've added it anyway for now, but just so you're aware.

MiroDudik · 2020-07-29T16:09:07Z

-labels =     [int(x) for x in '0110100' '0010111' '001111']  # noqa: E222
-X1 =         [int(x) for x in '0110101' '0111101' '001011']  # noqa: E222
-X2 =         [int(x) for x in '0000100' '0000011' '111111']  # noqa: E222
-X3 =         [int(x) for x in '1111111' '1111111' '111111']  # noqa: E222


this looks buggy, you probably mean float(x)/10 or int(x)/10.0?

Nope.

>>> int(7)/10 0.7

It's all identical.

MiroDudik · 2020-07-29T16:15:17Z

+labels =             [int(x) for x in '0110100' '0010111' '001111']  # noqa: E222
+X1 =                 [int(x) for x in '0110101' '0111101' '001011']  # noqa: E222
+X2 =                 [int(x) for x in '0000100' '0000011' '111111']  # noqa: E222
+X3 =                 [int(x) for x in '1111111' '1111111' '111111']  # noqa: E222


I don't love this calling convention. The issue is that if you sat y_as_scores to True, it's not just a matter of returning y as real values, you are actually returning a completely different y. But... I don't necessarily want to be overly pedantic here... so okay to leave as is.

Yeah, just a test utility, not an API that we expect anyone to use.

…davide/exponentiated_gradient-BGL

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

Roman Lutz and others added 30 commits April 8, 2020 15:17

replace A with sensitive_features

9b5a8df

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

remove variable pred_h

de8263f

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

add _MIN_T assertion

2e70be2

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

ensure that input data is not affected by input validation

ed4f544

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

add tests for _call_oracle and _eval

43c7d1a

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

flake8

e6db936

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

move pickle import to top of file

5a6f5aa

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

document why mocks are used

819678d

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

Merge branch 'master' of https://github.com/fairlearn/fairlearn into …

377fe3e

…rolutz/exponentiated_gradient_tests Signed-off-by: Roman Lutz <rolutz@microsoft.com>

add separate test to ensure input X data type is not changed

184133a

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

Merge branch 'master' of https://github.com/fairlearn/fairlearn into …

93023f5

…rolutz/exponentiated_gradient_tests

remove dependence on lagrangian object in tests, reorder checks

ab2c863

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

add callable h case back, and include it in tests

175d42c

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

h -> Q where appropriate

02c5b89

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

opt_lambda doc

a0c4abf

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

adding self.eps in load_data and gamma

7a66bb9

gamma and eps + test codes in BGL

a395ed2

Merge pull request #1 from dgiova/andra_dev

1468177

gamma and eps + test codes in BGL

Merge pull request #2 from dgiova/rolutz/exponentiated_gradient_tests

b33f6c7

Rolutz/exponentiated gradient tests

Update bounded_group_loss.py

12226c6

Update conditional_selection_rate.py

cbaf091

Update moment.py

9ba4801

added bound for bgl but cant test

cf029f9

updated bgl args, moment args

b2bc79e

update test

9a00d8b

Merge pull request #4 from dgiova/lauren_dev

52bb084

Lauren dev

adding test script for BGL in GPA application

778f771

replace A with sensitive_features

793285a

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

add tests for _call_oracle and _eval

300187d

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

add callable h case back, and include it in tests

539c347

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

romanlutz added enhancement New feature or request feature-request labels Jun 15, 2020

romanlutz added this to the Fair Regression with ExponentiatedGradient milestone Jun 15, 2020

romanlutz mentioned this pull request Jun 15, 2020

ENH - ExponentiatedGradient: use only classifiers with non-zero weights for predictions #380

Closed

romanlutz linked an issue Jun 15, 2020 that may be closed by this pull request

ENH - ExponentiatedGradient: use only classifiers with non-zero weights for predictions #380

Closed

MiroDudik removed the feature-request label Jun 22, 2020

romanlutz removed this from the Fair Regression with ExponentiatedGradient milestone Jun 22, 2020

Roman Lutz added 2 commits July 8, 2020 19:37

Merge branch 'master' of https://github.com/fairlearn/fairlearn into …

5270c1a

…davide/exponentiated_gradient-BGL

add documentation for BGL

a33eb59

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

Roman Lutz added 8 commits July 9, 2020 00:10

flake8, remove eps from lagrangian, and reformat API doc into numpydo…

fe25916

…c format Signed-off-by: Roman Lutz <rolutz@microsoft.com>

remove eps from code as arg to Lagrangian

e157c93

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

Merge branch 'master' of https://github.com/fairlearn/fairlearn into …

cba0c22

…davide/exponentiated_gradient-BGL

extend lagrangian tests for all constraints

f93edd6

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

Merge branch 'master' of https://github.com/fairlearn/fairlearn into …

d5c3559

…davide/exponentiated_gradient-BGL

add label check in FPRP and TPRP, update lagrangian tests to include …

938210d

…all moments Signed-off-by: Roman Lutz <rolutz@microsoft.com>

add smoke tests for BGL in expgrad and extend existing tests

8f72c01

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

Merge branch 'master' of https://github.com/fairlearn/fairlearn into …

d9e8a21

…davide/exponentiated_gradient-BGL

riedgar-ms reviewed Jul 27, 2020

View reviewed changes

Comment thread fairlearn/reductions/_moments/utility_parity.py Outdated

riedgar-ms approved these changes Jul 27, 2020

View reviewed changes

Merge branch 'master' of https://github.com/fairlearn/fairlearn into …

fb32b20

…davide/exponentiated_gradient-BGL

MiroDudik reviewed Jul 29, 2020

View reviewed changes

Merge branch 'master' of https://github.com/fairlearn/fairlearn into …

438bdd6

…davide/exponentiated_gradient-BGL

romanlutz mentioned this pull request Aug 3, 2020

FalsePositiveRateParity and TruePositiveRateParity can't handle absence of positives/negatives, respectively #543

Open

Roman Lutz added 2 commits August 3, 2020 17:24

update tests to not expect exception, assert on weights

6ba97a1

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

remove debug print

d640435

Signed-off-by: Roman Lutz <rolutz@microsoft.com>

MiroDudik approved these changes Aug 4, 2020

View reviewed changes

MiroDudik merged commit 924ec4e into fairlearn:master Aug 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Exponentiated gradient implementation for bounded group loss#442

Exponentiated gradient implementation for bounded group loss#442
MiroDudik merged 72 commits into
fairlearn:masterfrom
dgiova:davide/exponentiated_gradient-BGL

dgiova commented May 29, 2020 •

edited by romanlutz

Loading

Uh oh!

romanlutz commented Jul 9, 2020

Uh oh!

Uh oh!

riedgar-ms left a comment

Uh oh!

MiroDudik Jul 29, 2020

Uh oh!

romanlutz Aug 4, 2020

Uh oh!

MiroDudik Jul 29, 2020

Uh oh!

romanlutz Aug 3, 2020

Uh oh!

romanlutz Aug 3, 2020

Uh oh!

MiroDudik Jul 29, 2020

Uh oh!

romanlutz Aug 3, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

dgiova commented May 29, 2020 • edited by romanlutz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

romanlutz commented Jul 9, 2020

Uh oh!

Uh oh!

riedgar-ms left a comment

Choose a reason for hiding this comment

Uh oh!

MiroDudik Jul 29, 2020

Choose a reason for hiding this comment

Uh oh!

romanlutz Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

MiroDudik Jul 29, 2020

Choose a reason for hiding this comment

Uh oh!

romanlutz Aug 3, 2020

Choose a reason for hiding this comment

Uh oh!

romanlutz Aug 3, 2020

Choose a reason for hiding this comment

Uh oh!

MiroDudik Jul 29, 2020

Choose a reason for hiding this comment

Uh oh!

romanlutz Aug 3, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dgiova commented May 29, 2020 •

edited by romanlutz

Loading