Declare 0.4.0 release #193
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Richard Edgar <riedgar@microsoft.com>
romanlutz
approved these changes
Dec 5, 2019
romanlutz
pushed a commit
that referenced
this pull request
Feb 10, 2020
Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com>
romanlutz
pushed a commit
that referenced
this pull request
Feb 10, 2020
* take changes from other branch that touches all modules Signed-off-by: Roman Lutz <rolutz@microsoft.com> * get all tests working again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Dashboard for Census Notebook (#171) Update the existing Census notebook for grid search to use the new dashboard. The bulk of the notebook is unchanged (including the fictional motivating scenario). Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Stop installing old dashboard (#176) Have moved notebooks off the old dashboard. Remove dependency from pipelines Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update ReadMe with Yarn instructions (#177) Now that the dashboard tarball is no longer checked in, provide instructions on creating it in a cloned repo Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Standardise input convertors for test (#178) Create and use a standard set of convertors for use with our 'argument type' tests. This has required adding several `__init__.py` files to the `test` directory to enable `pytest` to find the common code. Also add an 'argument type' test to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Small fixes to get the documentation appearing (#179) Fix issues in getting documentation to appear in Sphinx. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * law school notebook (#169) * law school notebook Signed-off-by: Miro Dudik <mdudik@gmail.com> * Remove hypens from filename Some copy edit fixes Correct suspected bug in ExponentiatedGradient section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Didn't quite undo all my temporary changes Signed-off-by: Richard Edgar <riedgar@microsoft.com> * address some of the comments Signed-off-by: Miro Dudik <mdudik@gmail.com> * Fix typo in name Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Improve spacing and add a comment in expgrad section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * add AUC explanation Signed-off-by: Miro Dudik <mdudik@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Code cleanups (#181) Fix some minor things: - Make the dashboard use the same copyright notice as the rest of the code - Some renaming of `expgrad` to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Build the widget (#185) Add a job template which builds the widget to the PR-Gate, Nightly and Nightly-Fixed builds. Note that this does not run any tests, but just ensures that the widget builds successfully Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update logging to use FileHandler instead of basicConfig (#175) Signed-off-by: Ilya Matiach <ilmat@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Enable ReadTheDocs (#182) Change how the documentation is done slightly, so that our documentation can show up on ReadTheDocs. Some additional copy-editing of the in-code documentation has been done as a result of this. The docs should appear at: https://fairlearn.readthedocs.io/en/latest/ Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Pin scikit-learn (#189) The recent update to scikit-learn is causing a break in one of the Notebooks. Until this is debugged, pin the version Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add more flake8 checks (#187) Add a number of extra flake8 checks: - flake8-blind-except - flake8-builtins - flake8-docstrings - flake8-logging-format - flake8-rst-docstrings Since these create a huge number of issues, suppress a lot of these for now in `setup.cfg` (plus a handful of special cases done inline). Put in fixes for the simpler complaints, such as: - Separate summaries in docstrings - Spacing within and around docstrings - Deferring string interpolation in `logging` calls Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Rename files and update license and docs (#183) * rename files * update comment * update license * address comments Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix for Law School Notebook (#191) Tweak the Law School notebook so that it works with the latest `scikit-learn` This enables us to unpin the version of `scikit-learn` in our `requirements.txt` file Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Markdown updates based on doc bash (#186) * address feedback from doc bash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex updates Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo latex changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove commas Signed-off-by: Roman Lutz <rolutz@microsoft.com> * rephrasing postprocessing constructor requirements Signed-off-by: Roman Lutz <rolutz@microsoft.com> * feedback from Miro Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Reorganise documentation (#192) Reorganising how the documentation is presented, since the default style from `sphinx-apidoc` assumed we had lots of individual modules rather than larger packages Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Declare 0.4.0 release (#193) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Workaround Python 3.5 issue with Linux (#194) An issue with the `pip` install of `shap` has appeared on the Linux agents under Python 3.5. Reasons are currently obscure, but this is blocking a release. Since Python 3.5 continues to work on Windows, rely on that (pending further debugging) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix classification classification bug (#201) * Update readme for v0.4.0 (#196) Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Pin troublesome package (#198) During our release process, a new version of `colorama` (required by one of our dependencies) was released. This has issues with the Windows/3.7 build. Unblock the release by pinning the version Signed-off-by: Richard Edgar <riedgar@microsoft.com> * fix classification classification bug Signed-off-by: Roman Lutz <rolutz@microsoft.com> * version change to address security bug (#203) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix accidental merge (#205) Some portions of the v0.4.0 release branch were accidentally merged into master - Making the ReadMe version suitable for PyPI - Pinning the `colorama` version to unblock the release train This changeset undoes these fixes in master Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ReadMe Processor for Releases (#206) Create a python script to translate `ReadMe.md` from GitHub to PyPI. This will avoid the need to create a branch to do a release. This script is slightly dependent on the structure of the file, so if there are substantial changes to that, this script will require updating. It also assumes that a tag `v(fairlearn.__version__)` exists in the repo. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Track pip dependencies (#208) We've had trouble with our dependencies updating and breaking our builds Augment the build pipelines so that they publish the output of `pip freeze` to an artifact. This will aid debugging these issues. The name of both the artifact itself and the file therein can be specified. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Remove some flake8 global suppressions (#209) After adding more `flake8` analysers, we were obliged to put in some global suppressions to keep the number of issues manageable. Start the process of removing these with D102, D103 and D401. Some of these just move the suppression to file-level, while others tweak documentation blocks to suit. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Re-enable Linux 3.5 (#210) Roman figured out a workaround for getting `shap` installed with Linux and Python 3.5. Put this into `fairlearn` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Expand Notebook testing (#212) Increase the variety of platforms used for testing our Jupyter Notebooks. Unable to test on MacOS at present, due to some problem installing `lightgbm`. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Improvements for pinning requirements (#213) A better way of running our tests with pinned requirements. Rather than have a separate `requirements-fixed.txt` file, have a script to turn the `requirements.txt` file into the former. Update builds accordingly. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Standardise ML argument documentation (#214) Make our documentation of fit(), X, predict() etc. more standard between our various submodules. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf test through Azure ML (#180) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Documentation and flake8 (#215) Various updates for the documentation: - Remove another `flake8` global suppression - Add explanations for remaining `flake8` suppressions - Replace the `:any:` references in the documentation with appropriate ones - Make some file-level suppressions (which may have actually turned `flake8` off entirely on the file) specific to the appropriate lines Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update Release pipeline after KV move (#216) The KeyVault containing the PyPI secrets has been moved to a more appropriate subscription. As a result, the Release pipeline needs to be updated with the correct service connection Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Basic unit tests for EqualizedOdds and DemographicParity moments (#217) Some very basic unit tests for the `EqualizedOdds` and `DemographicParity` moment classes. These are 'pinning' tests two establish the behaviour of these classes. The `gamma` method is not yet included in these tests, since that requires a trained model. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add more time metrics to performance tests (#219) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add additional time-based metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustments to fix syntax errors and logical issues in the calculation of metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add oracle calls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * custom metrics for executions times, min, max, mean Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo sphinx special docs for test/perf Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update description of oracle execution time properties Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix perf tests by logging lists through log_list instead of log (#221) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add additional time-based metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustments to fix syntax errors and logical issues in the calculation of metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add oracle calls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * custom metrics for executions times, min, max, mean Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo sphinx special docs for test/perf Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update description of oracle execution time properties Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * bug fix for list logging Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * make metric logging a lot more readable and provide additional metrics to show the overhead fairlearn adds (#228) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Convert Notebook tests to papermill (#223) Rather than using `nbval`, convert our notebook tests to use `papermill`. With the help of `nteract-scrapbook` we can then examine the contents of particular variables from the notebooks to ensure that we're getting the expected results. Explicit `scrapbook` commands are required to save out values for future examination, but we don't want to include these when our users look at the notebooks. Accordingly, we include machinery for adding the necessary cells to the notebooks dynamically. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Bump dashboard npm package to match source code (#229) * publish latest version * update docs for push Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Remove unused ReST files (#233) Two of the ReST files generated by sphinx-autodoc weren't actually used. Remove them to get rid of a warning. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Basic Moments documentation (#241) Add some basic documentation of the `Moment` class and its subclasses. Also: - Turn the `n` field of the `Moment` object into a `total_samples` property - Add `intersphinx` hook for `pandas` documentation Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * [WIP] create extensions to install custom plots separately & check in generated files (#240) * check in generated javascript files and split into package with extensions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add installation tests, move yml files to templates directory if appropriate, delete unused and broken yml file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * separate directories per package, composition with minimal fairlearn package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * script updates to get doc and wheel builds in shape Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update yml files and scripts to enable wheel upload per package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address feedback from PR by adding documentation to the pipeline definition yml files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove "templates/" as location prefix for files in the templates directory itself Signed-off-by: Roman Lutz <rolutz@microsoft.com> * first version of widget build validation script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * corrections in widget build validation Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo adjustments to completely split up packages Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * reverse code coverage build changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ignore install tests when necessary Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add macos python 3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add exceptions module back to documentaiton Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add name for job Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix characters in job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add image label Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct installation path Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo other changes to rst file Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add logging variant of numpy.all_close (#246) The `numpy` package provides an `all_close` routine for comparing two arrays. Unfortunately, there's no mechanism for showing which elements failed the comparison. Put together a wrapper based on `numpy.isclose` which will print out information about failed comparisons. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Implement GroupMetricSet (#250) Create a `GroupMetricSet` class for holding collections of grouped metrics. This is to help with AzureML integration. There has been some (possibly unnecessary) reorganisation of things under `fairlearn/metrics` but the public interface is unchanged. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Exclude install tests in code coverage check (#251) * ignore install tests since they'll unexpectedly work Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add python -m before pip install Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade tempeh to v0.1.12 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Replace powershell scripts with python and add Makefile (#249) * check in generated javascript files and split into package with extensions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add installation tests, move yml files to templates directory if appropriate, delete unused and broken yml file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * separate directories per package, composition with minimal fairlearn package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * script updates to get doc and wheel builds in shape Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update yml files and scripts to enable wheel upload per package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address feedback from PR by adding documentation to the pipeline definition yml files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove "templates/" as location prefix for files in the templates directory itself Signed-off-by: Roman Lutz <rolutz@microsoft.com> * first version of widget build validation script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * corrections in widget build validation Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo adjustments to completely split up packages Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * reverse code coverage build changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ignore install tests when necessary Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add macos python 3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add exceptions module back to documentaiton Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add name for job Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix characters in job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add image label Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct installation path Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo other changes to rst file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * rewrite scripts in python Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace widget build script with python script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * build_widget adjustments to make it work Signed-off-by: Roman Lutz <rolutz@microsoft.com> * build_widget finalization plus add ls commands to find yarn installation in ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * some more paths to check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * task -> script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * usr/bin/yarn check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * workingDirectory adjustment Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add ls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustment for fairlearn root dir check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add ./ Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix template Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * comment about set-variable-from-file script only being required in ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add makefile, update contributing guide, and replace remaining ps1 occurrences in pipeline ymls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add romanlutz to codeowners for scripts dir Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix comment Signed-off-by: Roman Lutz <rolutz@microsoft.com> * make process_readme a standalone script again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * delete build_docs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * makefile adjustments according to feedback Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix pypi release yaml (#260) * undo erroneous changes to yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo prior erroneous change Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace job template usage with just a step Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add CHANGES.md for v0.4.2 (#262) * add CHANGES.md for v0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add general instructions to always do that Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update CHANGES.md Adding `GroupMetricSet` to the changelog Signed-off-by: Richard Edgar <riedgar@microsoft.com> * comment out test that fails consistently only on windows Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix readme processing script by adding fairlearn dir to sys path, add second solution for issue 265 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix syntax error, flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * bump version to 0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * test with list of lists instead of single list Signed-off-by: Roman Lutz <rolutz@microsoft.com> Co-authored-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update metric keys to match dashboard (#268) The dashboard already had its own keys defined for mapping metric functions to strings. Update the `GroupMetricSet` to use the same keys. Figuring out how to unify the two implementations of this mapping is left as a issue #269 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix release blockers - widget generated files, widget validation (#267) * add built widget file updates & fix widget build validation, as well as pypi release template for empty DEV_VERSION Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo DEV_VERSION change Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add comment and link to issue Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove --assert-no-changes flag in release as well (#272) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update default metrics in GroupMetricSet (#271) Tweak the list of metrics computed by default by the `compute` method of `GroupMetricSet` to match those expected by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * set env var before installing fairlearn to correct version file name content (#274) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * MNT use sklearn's NotFittedError instead of NotFittedException (#259) * MNT use sklearn's NotFittedError instead of NotFittedException Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * add to the changelog Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Updates for GroupMetricResult and GroupMetricSet (#279) Add (in)equality operators to `GroupMetricResult` and `GroupMetricSet`, along with basic tests. These will simplify other testing in future. Change `GroupMetricSet` so that the `groups` have to be specified as sequential integers from zero. If this is not the case then the `compute()` method will remap the supplied groups to `[0, 1, 2, ….]` and put the stringified original values into the `group_names` property. Since the keys are now sequential integers, convert the `group_names` property itself from a dictionary into a list. Closes #275 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * DOC contributing: trim lines and add notes on signoff (#276) * DOC contributing: trim lines and add notes on signoff Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * hook Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * modify note to point to the right answer Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Further metric changes (#281) A number of extra changes to metrics: - `GroupMetricResult` now dynamically calculates `maximum`, `range` etc. - `GroupMetricSet` has a consistency check - `GroupMetricSet` can transform itself to and from a dictionary matching the schema used by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Preparations for v0.4.3 Release (#284) Bump version and update Markdown files Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Use kwargs in metrics (#286) Change `metric_by_group` and `make_group_metric` to understand `**kwargs`. This removes the need for lots of small wrapper functions Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * take changes from other branch that touches all modules Signed-off-by: Roman Lutz <rolutz@microsoft.com> * get all tests working again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * squeeze instead of reshape, deselect instead of skip in pytest, utility function for compression Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Co-authored-by: Richard Edgar <riedgar@microsoft.com> Co-authored-by: MiroDudik <mdudik@gmail.com> Co-authored-by: Ilya Matiach <ilmat@microsoft.com> Co-authored-by: Brandon Horn <rihorn@microsoft.com> Co-authored-by: Adrin Jalali <adrin.jalali@gmail.com>
romanlutz
pushed a commit
that referenced
this pull request
Feb 11, 2020
* take changes from other branch that touches all modules Signed-off-by: Roman Lutz <rolutz@microsoft.com> * get all tests working again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Dashboard for Census Notebook (#171) Update the existing Census notebook for grid search to use the new dashboard. The bulk of the notebook is unchanged (including the fictional motivating scenario). Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Stop installing old dashboard (#176) Have moved notebooks off the old dashboard. Remove dependency from pipelines Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update ReadMe with Yarn instructions (#177) Now that the dashboard tarball is no longer checked in, provide instructions on creating it in a cloned repo Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Standardise input convertors for test (#178) Create and use a standard set of convertors for use with our 'argument type' tests. This has required adding several `__init__.py` files to the `test` directory to enable `pytest` to find the common code. Also add an 'argument type' test to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Small fixes to get the documentation appearing (#179) Fix issues in getting documentation to appear in Sphinx. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * law school notebook (#169) * law school notebook Signed-off-by: Miro Dudik <mdudik@gmail.com> * Remove hypens from filename Some copy edit fixes Correct suspected bug in ExponentiatedGradient section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Didn't quite undo all my temporary changes Signed-off-by: Richard Edgar <riedgar@microsoft.com> * address some of the comments Signed-off-by: Miro Dudik <mdudik@gmail.com> * Fix typo in name Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Improve spacing and add a comment in expgrad section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * add AUC explanation Signed-off-by: Miro Dudik <mdudik@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Code cleanups (#181) Fix some minor things: - Make the dashboard use the same copyright notice as the rest of the code - Some renaming of `expgrad` to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Build the widget (#185) Add a job template which builds the widget to the PR-Gate, Nightly and Nightly-Fixed builds. Note that this does not run any tests, but just ensures that the widget builds successfully Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update logging to use FileHandler instead of basicConfig (#175) Signed-off-by: Ilya Matiach <ilmat@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Enable ReadTheDocs (#182) Change how the documentation is done slightly, so that our documentation can show up on ReadTheDocs. Some additional copy-editing of the in-code documentation has been done as a result of this. The docs should appear at: https://fairlearn.readthedocs.io/en/latest/ Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Pin scikit-learn (#189) The recent update to scikit-learn is causing a break in one of the Notebooks. Until this is debugged, pin the version Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add more flake8 checks (#187) Add a number of extra flake8 checks: - flake8-blind-except - flake8-builtins - flake8-docstrings - flake8-logging-format - flake8-rst-docstrings Since these create a huge number of issues, suppress a lot of these for now in `setup.cfg` (plus a handful of special cases done inline). Put in fixes for the simpler complaints, such as: - Separate summaries in docstrings - Spacing within and around docstrings - Deferring string interpolation in `logging` calls Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Rename files and update license and docs (#183) * rename files * update comment * update license * address comments Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix for Law School Notebook (#191) Tweak the Law School notebook so that it works with the latest `scikit-learn` This enables us to unpin the version of `scikit-learn` in our `requirements.txt` file Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Markdown updates based on doc bash (#186) * address feedback from doc bash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex updates Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo latex changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove commas Signed-off-by: Roman Lutz <rolutz@microsoft.com> * rephrasing postprocessing constructor requirements Signed-off-by: Roman Lutz <rolutz@microsoft.com> * feedback from Miro Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Reorganise documentation (#192) Reorganising how the documentation is presented, since the default style from `sphinx-apidoc` assumed we had lots of individual modules rather than larger packages Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Declare 0.4.0 release (#193) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Workaround Python 3.5 issue with Linux (#194) An issue with the `pip` install of `shap` has appeared on the Linux agents under Python 3.5. Reasons are currently obscure, but this is blocking a release. Since Python 3.5 continues to work on Windows, rely on that (pending further debugging) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix classification classification bug (#201) * Update readme for v0.4.0 (#196) Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Pin troublesome package (#198) During our release process, a new version of `colorama` (required by one of our dependencies) was released. This has issues with the Windows/3.7 build. Unblock the release by pinning the version Signed-off-by: Richard Edgar <riedgar@microsoft.com> * fix classification classification bug Signed-off-by: Roman Lutz <rolutz@microsoft.com> * version change to address security bug (#203) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix accidental merge (#205) Some portions of the v0.4.0 release branch were accidentally merged into master - Making the ReadMe version suitable for PyPI - Pinning the `colorama` version to unblock the release train This changeset undoes these fixes in master Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ReadMe Processor for Releases (#206) Create a python script to translate `ReadMe.md` from GitHub to PyPI. This will avoid the need to create a branch to do a release. This script is slightly dependent on the structure of the file, so if there are substantial changes to that, this script will require updating. It also assumes that a tag `v(fairlearn.__version__)` exists in the repo. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Track pip dependencies (#208) We've had trouble with our dependencies updating and breaking our builds Augment the build pipelines so that they publish the output of `pip freeze` to an artifact. This will aid debugging these issues. The name of both the artifact itself and the file therein can be specified. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Remove some flake8 global suppressions (#209) After adding more `flake8` analysers, we were obliged to put in some global suppressions to keep the number of issues manageable. Start the process of removing these with D102, D103 and D401. Some of these just move the suppression to file-level, while others tweak documentation blocks to suit. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Re-enable Linux 3.5 (#210) Roman figured out a workaround for getting `shap` installed with Linux and Python 3.5. Put this into `fairlearn` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Expand Notebook testing (#212) Increase the variety of platforms used for testing our Jupyter Notebooks. Unable to test on MacOS at present, due to some problem installing `lightgbm`. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Improvements for pinning requirements (#213) A better way of running our tests with pinned requirements. Rather than have a separate `requirements-fixed.txt` file, have a script to turn the `requirements.txt` file into the former. Update builds accordingly. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Standardise ML argument documentation (#214) Make our documentation of fit(), X, predict() etc. more standard between our various submodules. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf test through Azure ML (#180) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Documentation and flake8 (#215) Various updates for the documentation: - Remove another `flake8` global suppression - Add explanations for remaining `flake8` suppressions - Replace the `:any:` references in the documentation with appropriate ones - Make some file-level suppressions (which may have actually turned `flake8` off entirely on the file) specific to the appropriate lines Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update Release pipeline after KV move (#216) The KeyVault containing the PyPI secrets has been moved to a more appropriate subscription. As a result, the Release pipeline needs to be updated with the correct service connection Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Basic unit tests for EqualizedOdds and DemographicParity moments (#217) Some very basic unit tests for the `EqualizedOdds` and `DemographicParity` moment classes. These are 'pinning' tests two establish the behaviour of these classes. The `gamma` method is not yet included in these tests, since that requires a trained model. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add more time metrics to performance tests (#219) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add additional time-based metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustments to fix syntax errors and logical issues in the calculation of metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add oracle calls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * custom metrics for executions times, min, max, mean Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo sphinx special docs for test/perf Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update description of oracle execution time properties Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix perf tests by logging lists through log_list instead of log (#221) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add additional time-based metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustments to fix syntax errors and logical issues in the calculation of metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add oracle calls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * custom metrics for executions times, min, max, mean Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo sphinx special docs for test/perf Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update description of oracle execution time properties Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * bug fix for list logging Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * make metric logging a lot more readable and provide additional metrics to show the overhead fairlearn adds (#228) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Convert Notebook tests to papermill (#223) Rather than using `nbval`, convert our notebook tests to use `papermill`. With the help of `nteract-scrapbook` we can then examine the contents of particular variables from the notebooks to ensure that we're getting the expected results. Explicit `scrapbook` commands are required to save out values for future examination, but we don't want to include these when our users look at the notebooks. Accordingly, we include machinery for adding the necessary cells to the notebooks dynamically. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Bump dashboard npm package to match source code (#229) * publish latest version * update docs for push Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Remove unused ReST files (#233) Two of the ReST files generated by sphinx-autodoc weren't actually used. Remove them to get rid of a warning. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Basic Moments documentation (#241) Add some basic documentation of the `Moment` class and its subclasses. Also: - Turn the `n` field of the `Moment` object into a `total_samples` property - Add `intersphinx` hook for `pandas` documentation Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * [WIP] create extensions to install custom plots separately & check in generated files (#240) * check in generated javascript files and split into package with extensions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add installation tests, move yml files to templates directory if appropriate, delete unused and broken yml file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * separate directories per package, composition with minimal fairlearn package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * script updates to get doc and wheel builds in shape Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update yml files and scripts to enable wheel upload per package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address feedback from PR by adding documentation to the pipeline definition yml files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove "templates/" as location prefix for files in the templates directory itself Signed-off-by: Roman Lutz <rolutz@microsoft.com> * first version of widget build validation script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * corrections in widget build validation Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo adjustments to completely split up packages Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * reverse code coverage build changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ignore install tests when necessary Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add macos python 3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add exceptions module back to documentaiton Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add name for job Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix characters in job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add image label Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct installation path Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo other changes to rst file Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add logging variant of numpy.all_close (#246) The `numpy` package provides an `all_close` routine for comparing two arrays. Unfortunately, there's no mechanism for showing which elements failed the comparison. Put together a wrapper based on `numpy.isclose` which will print out information about failed comparisons. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Implement GroupMetricSet (#250) Create a `GroupMetricSet` class for holding collections of grouped metrics. This is to help with AzureML integration. There has been some (possibly unnecessary) reorganisation of things under `fairlearn/metrics` but the public interface is unchanged. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Exclude install tests in code coverage check (#251) * ignore install tests since they'll unexpectedly work Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add python -m before pip install Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade tempeh to v0.1.12 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Replace powershell scripts with python and add Makefile (#249) * check in generated javascript files and split into package with extensions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add installation tests, move yml files to templates directory if appropriate, delete unused and broken yml file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * separate directories per package, composition with minimal fairlearn package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * script updates to get doc and wheel builds in shape Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update yml files and scripts to enable wheel upload per package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address feedback from PR by adding documentation to the pipeline definition yml files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove "templates/" as location prefix for files in the templates directory itself Signed-off-by: Roman Lutz <rolutz@microsoft.com> * first version of widget build validation script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * corrections in widget build validation Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo adjustments to completely split up packages Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * reverse code coverage build changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ignore install tests when necessary Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add macos python 3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add exceptions module back to documentaiton Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add name for job Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix characters in job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add image label Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct installation path Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo other changes to rst file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * rewrite scripts in python Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace widget build script with python script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * build_widget adjustments to make it work Signed-off-by: Roman Lutz <rolutz@microsoft.com> * build_widget finalization plus add ls commands to find yarn installation in ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * some more paths to check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * task -> script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * usr/bin/yarn check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * workingDirectory adjustment Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add ls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustment for fairlearn root dir check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add ./ Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix template Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * comment about set-variable-from-file script only being required in ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add makefile, update contributing guide, and replace remaining ps1 occurrences in pipeline ymls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add romanlutz to codeowners for scripts dir Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix comment Signed-off-by: Roman Lutz <rolutz@microsoft.com> * make process_readme a standalone script again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * delete build_docs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * makefile adjustments according to feedback Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix pypi release yaml (#260) * undo erroneous changes to yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo prior erroneous change Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace job template usage with just a step Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add CHANGES.md for v0.4.2 (#262) * add CHANGES.md for v0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add general instructions to always do that Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update CHANGES.md Adding `GroupMetricSet` to the changelog Signed-off-by: Richard Edgar <riedgar@microsoft.com> * comment out test that fails consistently only on windows Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix readme processing script by adding fairlearn dir to sys path, add second solution for issue 265 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix syntax error, flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * bump version to 0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * test with list of lists instead of single list Signed-off-by: Roman Lutz <rolutz@microsoft.com> Co-authored-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update metric keys to match dashboard (#268) The dashboard already had its own keys defined for mapping metric functions to strings. Update the `GroupMetricSet` to use the same keys. Figuring out how to unify the two implementations of this mapping is left as a issue #269 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix release blockers - widget generated files, widget validation (#267) * add built widget file updates & fix widget build validation, as well as pypi release template for empty DEV_VERSION Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo DEV_VERSION change Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add comment and link to issue Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove --assert-no-changes flag in release as well (#272) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update default metrics in GroupMetricSet (#271) Tweak the list of metrics computed by default by the `compute` method of `GroupMetricSet` to match those expected by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * set env var before installing fairlearn to correct version file name content (#274) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * MNT use sklearn's NotFittedError instead of NotFittedException (#259) * MNT use sklearn's NotFittedError instead of NotFittedException Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * add to the changelog Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Updates for GroupMetricResult and GroupMetricSet (#279) Add (in)equality operators to `GroupMetricResult` and `GroupMetricSet`, along with basic tests. These will simplify other testing in future. Change `GroupMetricSet` so that the `groups` have to be specified as sequential integers from zero. If this is not the case then the `compute()` method will remap the supplied groups to `[0, 1, 2, ….]` and put the stringified original values into the `group_names` property. Since the keys are now sequential integers, convert the `group_names` property itself from a dictionary into a list. Closes #275 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * DOC contributing: trim lines and add notes on signoff (#276) * DOC contributing: trim lines and add notes on signoff Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * hook Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * modify note to point to the right answer Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Further metric changes (#281) A number of extra changes to metrics: - `GroupMetricResult` now dynamically calculates `maximum`, `range` etc. - `GroupMetricSet` has a consistency check - `GroupMetricSet` can transform itself to and from a dictionary matching the schema used by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Preparations for v0.4.3 Release (#284) Bump version and update Markdown files Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Use kwargs in metrics (#286) Change `metric_by_group` and `make_group_metric` to understand `**kwargs`. This removes the need for lots of small wrapper functions Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * take changes from other branch that touches all modules Signed-off-by: Roman Lutz <rolutz@microsoft.com> * get all tests working again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * squeeze instead of reshape, deselect instead of skip in pytest, utility function for compression Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Co-authored-by: Richard Edgar <riedgar@microsoft.com> Co-authored-by: MiroDudik <mdudik@gmail.com> Co-authored-by: Ilya Matiach <ilmat@microsoft.com> Co-authored-by: Brandon Horn <rihorn@microsoft.com> Co-authored-by: Adrin Jalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com>
romanlutz
pushed a commit
that referenced
this pull request
Feb 11, 2020
Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com>
romanlutz
pushed a commit
that referenced
this pull request
Feb 11, 2020
* take changes from other branch that touches all modules Signed-off-by: Roman Lutz <rolutz@microsoft.com> * get all tests working again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Dashboard for Census Notebook (#171) Update the existing Census notebook for grid search to use the new dashboard. The bulk of the notebook is unchanged (including the fictional motivating scenario). Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Stop installing old dashboard (#176) Have moved notebooks off the old dashboard. Remove dependency from pipelines Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update ReadMe with Yarn instructions (#177) Now that the dashboard tarball is no longer checked in, provide instructions on creating it in a cloned repo Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Standardise input convertors for test (#178) Create and use a standard set of convertors for use with our 'argument type' tests. This has required adding several `__init__.py` files to the `test` directory to enable `pytest` to find the common code. Also add an 'argument type' test to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Small fixes to get the documentation appearing (#179) Fix issues in getting documentation to appear in Sphinx. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * law school notebook (#169) * law school notebook Signed-off-by: Miro Dudik <mdudik@gmail.com> * Remove hypens from filename Some copy edit fixes Correct suspected bug in ExponentiatedGradient section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Didn't quite undo all my temporary changes Signed-off-by: Richard Edgar <riedgar@microsoft.com> * address some of the comments Signed-off-by: Miro Dudik <mdudik@gmail.com> * Fix typo in name Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Improve spacing and add a comment in expgrad section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * add AUC explanation Signed-off-by: Miro Dudik <mdudik@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Code cleanups (#181) Fix some minor things: - Make the dashboard use the same copyright notice as the rest of the code - Some renaming of `expgrad` to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Build the widget (#185) Add a job template which builds the widget to the PR-Gate, Nightly and Nightly-Fixed builds. Note that this does not run any tests, but just ensures that the widget builds successfully Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update logging to use FileHandler instead of basicConfig (#175) Signed-off-by: Ilya Matiach <ilmat@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Enable ReadTheDocs (#182) Change how the documentation is done slightly, so that our documentation can show up on ReadTheDocs. Some additional copy-editing of the in-code documentation has been done as a result of this. The docs should appear at: https://fairlearn.readthedocs.io/en/latest/ Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Pin scikit-learn (#189) The recent update to scikit-learn is causing a break in one of the Notebooks. Until this is debugged, pin the version Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add more flake8 checks (#187) Add a number of extra flake8 checks: - flake8-blind-except - flake8-builtins - flake8-docstrings - flake8-logging-format - flake8-rst-docstrings Since these create a huge number of issues, suppress a lot of these for now in `setup.cfg` (plus a handful of special cases done inline). Put in fixes for the simpler complaints, such as: - Separate summaries in docstrings - Spacing within and around docstrings - Deferring string interpolation in `logging` calls Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Rename files and update license and docs (#183) * rename files * update comment * update license * address comments Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix for Law School Notebook (#191) Tweak the Law School notebook so that it works with the latest `scikit-learn` This enables us to unpin the version of `scikit-learn` in our `requirements.txt` file Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Markdown updates based on doc bash (#186) * address feedback from doc bash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex updates Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo latex changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove commas Signed-off-by: Roman Lutz <rolutz@microsoft.com> * rephrasing postprocessing constructor requirements Signed-off-by: Roman Lutz <rolutz@microsoft.com> * feedback from Miro Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Reorganise documentation (#192) Reorganising how the documentation is presented, since the default style from `sphinx-apidoc` assumed we had lots of individual modules rather than larger packages Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Declare 0.4.0 release (#193) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Workaround Python 3.5 issue with Linux (#194) An issue with the `pip` install of `shap` has appeared on the Linux agents under Python 3.5. Reasons are currently obscure, but this is blocking a release. Since Python 3.5 continues to work on Windows, rely on that (pending further debugging) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix classification classification bug (#201) * Update readme for v0.4.0 (#196) Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Pin troublesome package (#198) During our release process, a new version of `colorama` (required by one of our dependencies) was released. This has issues with the Windows/3.7 build. Unblock the release by pinning the version Signed-off-by: Richard Edgar <riedgar@microsoft.com> * fix classification classification bug Signed-off-by: Roman Lutz <rolutz@microsoft.com> * version change to address security bug (#203) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix accidental merge (#205) Some portions of the v0.4.0 release branch were accidentally merged into master - Making the ReadMe version suitable for PyPI - Pinning the `colorama` version to unblock the release train This changeset undoes these fixes in master Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ReadMe Processor for Releases (#206) Create a python script to translate `ReadMe.md` from GitHub to PyPI. This will avoid the need to create a branch to do a release. This script is slightly dependent on the structure of the file, so if there are substantial changes to that, this script will require updating. It also assumes that a tag `v(fairlearn.__version__)` exists in the repo. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Track pip dependencies (#208) We've had trouble with our dependencies updating and breaking our builds Augment the build pipelines so that they publish the output of `pip freeze` to an artifact. This will aid debugging these issues. The name of both the artifact itself and the file therein can be specified. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Remove some flake8 global suppressions (#209) After adding more `flake8` analysers, we were obliged to put in some global suppressions to keep the number of issues manageable. Start the process of removing these with D102, D103 and D401. Some of these just move the suppression to file-level, while others tweak documentation blocks to suit. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Re-enable Linux 3.5 (#210) Roman figured out a workaround for getting `shap` installed with Linux and Python 3.5. Put this into `fairlearn` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Expand Notebook testing (#212) Increase the variety of platforms used for testing our Jupyter Notebooks. Unable to test on MacOS at present, due to some problem installing `lightgbm`. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Improvements for pinning requirements (#213) A better way of running our tests with pinned requirements. Rather than have a separate `requirements-fixed.txt` file, have a script to turn the `requirements.txt` file into the former. Update builds accordingly. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Standardise ML argument documentation (#214) Make our documentation of fit(), X, predict() etc. more standard between our various submodules. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf test through Azure ML (#180) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Documentation and flake8 (#215) Various updates for the documentation: - Remove another `flake8` global suppression - Add explanations for remaining `flake8` suppressions - Replace the `:any:` references in the documentation with appropriate ones - Make some file-level suppressions (which may have actually turned `flake8` off entirely on the file) specific to the appropriate lines Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update Release pipeline after KV move (#216) The KeyVault containing the PyPI secrets has been moved to a more appropriate subscription. As a result, the Release pipeline needs to be updated with the correct service connection Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Basic unit tests for EqualizedOdds and DemographicParity moments (#217) Some very basic unit tests for the `EqualizedOdds` and `DemographicParity` moment classes. These are 'pinning' tests two establish the behaviour of these classes. The `gamma` method is not yet included in these tests, since that requires a trained model. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add more time metrics to performance tests (#219) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add additional time-based metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustments to fix syntax errors and logical issues in the calculation of metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add oracle calls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * custom metrics for executions times, min, max, mean Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo sphinx special docs for test/perf Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update description of oracle execution time properties Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix perf tests by logging lists through log_list instead of log (#221) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add additional time-based metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustments to fix syntax errors and logical issues in the calculation of metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add oracle calls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * custom metrics for executions times, min, max, mean Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo sphinx special docs for test/perf Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update description of oracle execution time properties Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * bug fix for list logging Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * make metric logging a lot more readable and provide additional metrics to show the overhead fairlearn adds (#228) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Convert Notebook tests to papermill (#223) Rather than using `nbval`, convert our notebook tests to use `papermill`. With the help of `nteract-scrapbook` we can then examine the contents of particular variables from the notebooks to ensure that we're getting the expected results. Explicit `scrapbook` commands are required to save out values for future examination, but we don't want to include these when our users look at the notebooks. Accordingly, we include machinery for adding the necessary cells to the notebooks dynamically. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Bump dashboard npm package to match source code (#229) * publish latest version * update docs for push Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Remove unused ReST files (#233) Two of the ReST files generated by sphinx-autodoc weren't actually used. Remove them to get rid of a warning. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Basic Moments documentation (#241) Add some basic documentation of the `Moment` class and its subclasses. Also: - Turn the `n` field of the `Moment` object into a `total_samples` property - Add `intersphinx` hook for `pandas` documentation Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * [WIP] create extensions to install custom plots separately & check in generated files (#240) * check in generated javascript files and split into package with extensions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add installation tests, move yml files to templates directory if appropriate, delete unused and broken yml file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * separate directories per package, composition with minimal fairlearn package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * script updates to get doc and wheel builds in shape Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update yml files and scripts to enable wheel upload per package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address feedback from PR by adding documentation to the pipeline definition yml files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove "templates/" as location prefix for files in the templates directory itself Signed-off-by: Roman Lutz <rolutz@microsoft.com> * first version of widget build validation script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * corrections in widget build validation Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo adjustments to completely split up packages Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * reverse code coverage build changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ignore install tests when necessary Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add macos python 3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add exceptions module back to documentaiton Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add name for job Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix characters in job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add image label Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct installation path Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo other changes to rst file Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add logging variant of numpy.all_close (#246) The `numpy` package provides an `all_close` routine for comparing two arrays. Unfortunately, there's no mechanism for showing which elements failed the comparison. Put together a wrapper based on `numpy.isclose` which will print out information about failed comparisons. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Implement GroupMetricSet (#250) Create a `GroupMetricSet` class for holding collections of grouped metrics. This is to help with AzureML integration. There has been some (possibly unnecessary) reorganisation of things under `fairlearn/metrics` but the public interface is unchanged. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Exclude install tests in code coverage check (#251) * ignore install tests since they'll unexpectedly work Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add python -m before pip install Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade tempeh to v0.1.12 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Replace powershell scripts with python and add Makefile (#249) * check in generated javascript files and split into package with extensions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add installation tests, move yml files to templates directory if appropriate, delete unused and broken yml file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * separate directories per package, composition with minimal fairlearn package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * script updates to get doc and wheel builds in shape Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update yml files and scripts to enable wheel upload per package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address feedback from PR by adding documentation to the pipeline definition yml files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove "templates/" as location prefix for files in the templates directory itself Signed-off-by: Roman Lutz <rolutz@microsoft.com> * first version of widget build validation script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * corrections in widget build validation Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo adjustments to completely split up packages Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * reverse code coverage build changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ignore install tests when necessary Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add macos python 3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add exceptions module back to documentaiton Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add name for job Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix characters in job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add image label Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct installation path Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo other changes to rst file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * rewrite scripts in python Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace widget build script with python script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * build_widget adjustments to make it work Signed-off-by: Roman Lutz <rolutz@microsoft.com> * build_widget finalization plus add ls commands to find yarn installation in ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * some more paths to check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * task -> script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * usr/bin/yarn check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * workingDirectory adjustment Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add ls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustment for fairlearn root dir check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add ./ Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix template Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * comment about set-variable-from-file script only being required in ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add makefile, update contributing guide, and replace remaining ps1 occurrences in pipeline ymls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add romanlutz to codeowners for scripts dir Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix comment Signed-off-by: Roman Lutz <rolutz@microsoft.com> * make process_readme a standalone script again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * delete build_docs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * makefile adjustments according to feedback Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix pypi release yaml (#260) * undo erroneous changes to yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo prior erroneous change Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace job template usage with just a step Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add CHANGES.md for v0.4.2 (#262) * add CHANGES.md for v0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add general instructions to always do that Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update CHANGES.md Adding `GroupMetricSet` to the changelog Signed-off-by: Richard Edgar <riedgar@microsoft.com> * comment out test that fails consistently only on windows Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix readme processing script by adding fairlearn dir to sys path, add second solution for issue 265 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix syntax error, flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * bump version to 0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * test with list of lists instead of single list Signed-off-by: Roman Lutz <rolutz@microsoft.com> Co-authored-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update metric keys to match dashboard (#268) The dashboard already had its own keys defined for mapping metric functions to strings. Update the `GroupMetricSet` to use the same keys. Figuring out how to unify the two implementations of this mapping is left as a issue #269 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix release blockers - widget generated files, widget validation (#267) * add built widget file updates & fix widget build validation, as well as pypi release template for empty DEV_VERSION Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo DEV_VERSION change Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add comment and link to issue Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove --assert-no-changes flag in release as well (#272) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update default metrics in GroupMetricSet (#271) Tweak the list of metrics computed by default by the `compute` method of `GroupMetricSet` to match those expected by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * set env var before installing fairlearn to correct version file name content (#274) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * MNT use sklearn's NotFittedError instead of NotFittedException (#259) * MNT use sklearn's NotFittedError instead of NotFittedException Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * add to the changelog Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Updates for GroupMetricResult and GroupMetricSet (#279) Add (in)equality operators to `GroupMetricResult` and `GroupMetricSet`, along with basic tests. These will simplify other testing in future. Change `GroupMetricSet` so that the `groups` have to be specified as sequential integers from zero. If this is not the case then the `compute()` method will remap the supplied groups to `[0, 1, 2, ….]` and put the stringified original values into the `group_names` property. Since the keys are now sequential integers, convert the `group_names` property itself from a dictionary into a list. Closes #275 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * DOC contributing: trim lines and add notes on signoff (#276) * DOC contributing: trim lines and add notes on signoff Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * hook Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * modify note to point to the right answer Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Further metric changes (#281) A number of extra changes to metrics: - `GroupMetricResult` now dynamically calculates `maximum`, `range` etc. - `GroupMetricSet` has a consistency check - `GroupMetricSet` can transform itself to and from a dictionary matching the schema used by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Preparations for v0.4.3 Release (#284) Bump version and update Markdown files Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Use kwargs in metrics (#286) Change `metric_by_group` and `make_group_metric` to understand `**kwargs`. This removes the need for lots of small wrapper functions Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * take changes from other branch that touches all modules Signed-off-by: Roman Lutz <rolutz@microsoft.com> * get all tests working again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * squeeze instead of reshape, deselect instead of skip in pytest, utility function for compression Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Co-authored-by: Richard Edgar <riedgar@microsoft.com> Co-authored-by: MiroDudik <mdudik@gmail.com> Co-authored-by: Ilya Matiach <ilmat@microsoft.com> Co-authored-by: Brandon Horn <rihorn@microsoft.com> Co-authored-by: Adrin Jalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com>
romanlutz
pushed a commit
that referenced
this pull request
Feb 11, 2020
* take changes from other branch that touches all modules Signed-off-by: Roman Lutz <rolutz@microsoft.com> * get all tests working again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Dashboard for Census Notebook (#171) Update the existing Census notebook for grid search to use the new dashboard. The bulk of the notebook is unchanged (including the fictional motivating scenario). Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Stop installing old dashboard (#176) Have moved notebooks off the old dashboard. Remove dependency from pipelines Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update ReadMe with Yarn instructions (#177) Now that the dashboard tarball is no longer checked in, provide instructions on creating it in a cloned repo Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Standardise input convertors for test (#178) Create and use a standard set of convertors for use with our 'argument type' tests. This has required adding several `__init__.py` files to the `test` directory to enable `pytest` to find the common code. Also add an 'argument type' test to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Small fixes to get the documentation appearing (#179) Fix issues in getting documentation to appear in Sphinx. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * law school notebook (#169) * law school notebook Signed-off-by: Miro Dudik <mdudik@gmail.com> * Remove hypens from filename Some copy edit fixes Correct suspected bug in ExponentiatedGradient section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Didn't quite undo all my temporary changes Signed-off-by: Richard Edgar <riedgar@microsoft.com> * address some of the comments Signed-off-by: Miro Dudik <mdudik@gmail.com> * Fix typo in name Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Improve spacing and add a comment in expgrad section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * add AUC explanation Signed-off-by: Miro Dudik <mdudik@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Code cleanups (#181) Fix some minor things: - Make the dashboard use the same copyright notice as the rest of the code - Some renaming of `expgrad` to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Build the widget (#185) Add a job template which builds the widget to the PR-Gate, Nightly and Nightly-Fixed builds. Note that this does not run any tests, but just ensures that the widget builds successfully Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update logging to use FileHandler instead of basicConfig (#175) Signed-off-by: Ilya Matiach <ilmat@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Enable ReadTheDocs (#182) Change how the documentation is done slightly, so that our documentation can show up on ReadTheDocs. Some additional copy-editing of the in-code documentation has been done as a result of this. The docs should appear at: https://fairlearn.readthedocs.io/en/latest/ Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Pin scikit-learn (#189) The recent update to scikit-learn is causing a break in one of the Notebooks. Until this is debugged, pin the version Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add more flake8 checks (#187) Add a number of extra flake8 checks: - flake8-blind-except - flake8-builtins - flake8-docstrings - flake8-logging-format - flake8-rst-docstrings Since these create a huge number of issues, suppress a lot of these for now in `setup.cfg` (plus a handful of special cases done inline). Put in fixes for the simpler complaints, such as: - Separate summaries in docstrings - Spacing within and around docstrings - Deferring string interpolation in `logging` calls Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Rename files and update license and docs (#183) * rename files * update comment * update license * address comments Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix for Law School Notebook (#191) Tweak the Law School notebook so that it works with the latest `scikit-learn` This enables us to unpin the version of `scikit-learn` in our `requirements.txt` file Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Markdown updates based on doc bash (#186) * address feedback from doc bash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex updates Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo latex changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove commas Signed-off-by: Roman Lutz <rolutz@microsoft.com> * rephrasing postprocessing constructor requirements Signed-off-by: Roman Lutz <rolutz@microsoft.com> * feedback from Miro Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Reorganise documentation (#192) Reorganising how the documentation is presented, since the default style from `sphinx-apidoc` assumed we had lots of individual modules rather than larger packages Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Declare 0.4.0 release (#193) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Workaround Python 3.5 issue with Linux (#194) An issue with the `pip` install of `shap` has appeared on the Linux agents under Python 3.5. Reasons are currently obscure, but this is blocking a release. Since Python 3.5 continues to work on Windows, rely on that (pending further debugging) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix classification classification bug (#201) * Update readme for v0.4.0 (#196) Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Pin troublesome package (#198) During our release process, a new version of `colorama` (required by one of our dependencies) was released. This has issues with the Windows/3.7 build. Unblock the release by pinning the version Signed-off-by: Richard Edgar <riedgar@microsoft.com> * fix classification classification bug Signed-off-by: Roman Lutz <rolutz@microsoft.com> * version change to address security bug (#203) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix accidental merge (#205) Some portions of the v0.4.0 release branch were accidentally merged into master - Making the ReadMe version suitable for PyPI - Pinning the `colorama` version to unblock the release train This changeset undoes these fixes in master Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ReadMe Processor for Releases (#206) Create a python script to translate `ReadMe.md` from GitHub to PyPI. This will avoid the need to create a branch to do a release. This script is slightly dependent on the structure of the file, so if there are substantial changes to that, this script will require updating. It also assumes that a tag `v(fairlearn.__version__)` exists in the repo. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Track pip dependencies (#208) We've had trouble with our dependencies updating and breaking our builds Augment the build pipelines so that they publish the output of `pip freeze` to an artifact. This will aid debugging these issues. The name of both the artifact itself and the file therein can be specified. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Remove some flake8 global suppressions (#209) After adding more `flake8` analysers, we were obliged to put in some global suppressions to keep the number of issues manageable. Start the process of removing these with D102, D103 and D401. Some of these just move the suppression to file-level, while others tweak documentation blocks to suit. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Re-enable Linux 3.5 (#210) Roman figured out a workaround for getting `shap` installed with Linux and Python 3.5. Put this into `fairlearn` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Expand Notebook testing (#212) Increase the variety of platforms used for testing our Jupyter Notebooks. Unable to test on MacOS at present, due to some problem installing `lightgbm`. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Improvements for pinning requirements (#213) A better way of running our tests with pinned requirements. Rather than have a separate `requirements-fixed.txt` file, have a script to turn the `requirements.txt` file into the former. Update builds accordingly. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Standardise ML argument documentation (#214) Make our documentation of fit(), X, predict() etc. more standard between our various submodules. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf test through Azure ML (#180) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Documentation and flake8 (#215) Various updates for the documentation: - Remove another `flake8` global suppression - Add explanations for remaining `flake8` suppressions - Replace the `:any:` references in the documentation with appropriate ones - Make some file-level suppressions (which may have actually turned `flake8` off entirely on the file) specific to the appropriate lines Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update Release pipeline after KV move (#216) The KeyVault containing the PyPI secrets has been moved to a more appropriate subscription. As a result, the Release pipeline needs to be updated with the correct service connection Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Basic unit tests for EqualizedOdds and DemographicParity moments (#217) Some very basic unit tests for the `EqualizedOdds` and `DemographicParity` moment classes. These are 'pinning' tests two establish the behaviour of these classes. The `gamma` method is not yet included in these tests, since that requires a trained model. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add more time metrics to performance tests (#219) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add additional time-based metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustments to fix syntax errors and logical issues in the calculation of metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add oracle calls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * custom metrics for executions times, min, max, mean Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo sphinx special docs for test/perf Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update description of oracle execution time properties Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix perf tests by logging lists through log_list instead of log (#221) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add additional time-based metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustments to fix syntax errors and logical issues in the calculation of metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add oracle calls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * custom metrics for executions times, min, max, mean Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo sphinx special docs for test/perf Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update description of oracle execution time properties Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * bug fix for list logging Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * make metric logging a lot more readable and provide additional metrics to show the overhead fairlearn adds (#228) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Convert Notebook tests to papermill (#223) Rather than using `nbval`, convert our notebook tests to use `papermill`. With the help of `nteract-scrapbook` we can then examine the contents of particular variables from the notebooks to ensure that we're getting the expected results. Explicit `scrapbook` commands are required to save out values for future examination, but we don't want to include these when our users look at the notebooks. Accordingly, we include machinery for adding the necessary cells to the notebooks dynamically. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Bump dashboard npm package to match source code (#229) * publish latest version * update docs for push Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Remove unused ReST files (#233) Two of the ReST files generated by sphinx-autodoc weren't actually used. Remove them to get rid of a warning. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Basic Moments documentation (#241) Add some basic documentation of the `Moment` class and its subclasses. Also: - Turn the `n` field of the `Moment` object into a `total_samples` property - Add `intersphinx` hook for `pandas` documentation Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * [WIP] create extensions to install custom plots separately & check in generated files (#240) * check in generated javascript files and split into package with extensions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add installation tests, move yml files to templates directory if appropriate, delete unused and broken yml file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * separate directories per package, composition with minimal fairlearn package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * script updates to get doc and wheel builds in shape Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update yml files and scripts to enable wheel upload per package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address feedback from PR by adding documentation to the pipeline definition yml files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove "templates/" as location prefix for files in the templates directory itself Signed-off-by: Roman Lutz <rolutz@microsoft.com> * first version of widget build validation script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * corrections in widget build validation Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo adjustments to completely split up packages Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * reverse code coverage build changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ignore install tests when necessary Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add macos python 3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add exceptions module back to documentaiton Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add name for job Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix characters in job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add image label Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct installation path Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo other changes to rst file Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add logging variant of numpy.all_close (#246) The `numpy` package provides an `all_close` routine for comparing two arrays. Unfortunately, there's no mechanism for showing which elements failed the comparison. Put together a wrapper based on `numpy.isclose` which will print out information about failed comparisons. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Implement GroupMetricSet (#250) Create a `GroupMetricSet` class for holding collections of grouped metrics. This is to help with AzureML integration. There has been some (possibly unnecessary) reorganisation of things under `fairlearn/metrics` but the public interface is unchanged. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Exclude install tests in code coverage check (#251) * ignore install tests since they'll unexpectedly work Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add python -m before pip install Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade tempeh to v0.1.12 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Replace powershell scripts with python and add Makefile (#249) * check in generated javascript files and split into package with extensions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add installation tests, move yml files to templates directory if appropriate, delete unused and broken yml file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * separate directories per package, composition with minimal fairlearn package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * script updates to get doc and wheel builds in shape Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update yml files and scripts to enable wheel upload per package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address feedback from PR by adding documentation to the pipeline definition yml files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove "templates/" as location prefix for files in the templates directory itself Signed-off-by: Roman Lutz <rolutz@microsoft.com> * first version of widget build validation script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * corrections in widget build validation Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo adjustments to completely split up packages Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * reverse code coverage build changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ignore install tests when necessary Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add macos python 3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add exceptions module back to documentaiton Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add name for job Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix characters in job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add image label Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct installation path Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo other changes to rst file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * rewrite scripts in python Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace widget build script with python script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * build_widget adjustments to make it work Signed-off-by: Roman Lutz <rolutz@microsoft.com> * build_widget finalization plus add ls commands to find yarn installation in ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * some more paths to check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * task -> script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * usr/bin/yarn check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * workingDirectory adjustment Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add ls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustment for fairlearn root dir check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add ./ Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix template Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * comment about set-variable-from-file script only being required in ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add makefile, update contributing guide, and replace remaining ps1 occurrences in pipeline ymls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add romanlutz to codeowners for scripts dir Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix comment Signed-off-by: Roman Lutz <rolutz@microsoft.com> * make process_readme a standalone script again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * delete build_docs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * makefile adjustments according to feedback Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix pypi release yaml (#260) * undo erroneous changes to yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo prior erroneous change Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace job template usage with just a step Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add CHANGES.md for v0.4.2 (#262) * add CHANGES.md for v0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add general instructions to always do that Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update CHANGES.md Adding `GroupMetricSet` to the changelog Signed-off-by: Richard Edgar <riedgar@microsoft.com> * comment out test that fails consistently only on windows Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix readme processing script by adding fairlearn dir to sys path, add second solution for issue 265 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix syntax error, flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * bump version to 0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * test with list of lists instead of single list Signed-off-by: Roman Lutz <rolutz@microsoft.com> Co-authored-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update metric keys to match dashboard (#268) The dashboard already had its own keys defined for mapping metric functions to strings. Update the `GroupMetricSet` to use the same keys. Figuring out how to unify the two implementations of this mapping is left as a issue #269 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix release blockers - widget generated files, widget validation (#267) * add built widget file updates & fix widget build validation, as well as pypi release template for empty DEV_VERSION Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo DEV_VERSION change Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add comment and link to issue Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove --assert-no-changes flag in release as well (#272) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update default metrics in GroupMetricSet (#271) Tweak the list of metrics computed by default by the `compute` method of `GroupMetricSet` to match those expected by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * set env var before installing fairlearn to correct version file name content (#274) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * MNT use sklearn's NotFittedError instead of NotFittedException (#259) * MNT use sklearn's NotFittedError instead of NotFittedException Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * add to the changelog Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Updates for GroupMetricResult and GroupMetricSet (#279) Add (in)equality operators to `GroupMetricResult` and `GroupMetricSet`, along with basic tests. These will simplify other testing in future. Change `GroupMetricSet` so that the `groups` have to be specified as sequential integers from zero. If this is not the case then the `compute()` method will remap the supplied groups to `[0, 1, 2, ….]` and put the stringified original values into the `group_names` property. Since the keys are now sequential integers, convert the `group_names` property itself from a dictionary into a list. Closes #275 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * DOC contributing: trim lines and add notes on signoff (#276) * DOC contributing: trim lines and add notes on signoff Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * hook Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * modify note to point to the right answer Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Further metric changes (#281) A number of extra changes to metrics: - `GroupMetricResult` now dynamically calculates `maximum`, `range` etc. - `GroupMetricSet` has a consistency check - `GroupMetricSet` can transform itself to and from a dictionary matching the schema used by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Preparations for v0.4.3 Release (#284) Bump version and update Markdown files Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Use kwargs in metrics (#286) Change `metric_by_group` and `make_group_metric` to understand `**kwargs`. This removes the need for lots of small wrapper functions Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * take changes from other branch that touches all modules Signed-off-by: Roman Lutz <rolutz@microsoft.com> * get all tests working again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * squeeze instead of reshape, deselect instead of skip in pytest, utility function for compression Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Co-authored-by: Richard Edgar <riedgar@microsoft.com> Co-authored-by: MiroDudik <mdudik@gmail.com> Co-authored-by: Ilya Matiach <ilmat@microsoft.com> Co-authored-by: Brandon Horn <rihorn@microsoft.com> Co-authored-by: Adrin Jalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com>
romanlutz
pushed a commit
that referenced
this pull request
Feb 11, 2020
* remove exta comma Signed-off-by: Brandon Horn <rihorn@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * string changes Signed-off-by: Brandon Horn <rihorn@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * missed a few Signed-off-by: Brandon Horn <rihorn@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Developer Certificate of Origin and Code of Conduct updates (#130) Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Brandon Horn <rihorn@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update contact details (#131) The `setup.py` script was still pointing just at Miro. Signed-off-by: Brandon Horn <rihorn@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix text Signed-off-by: Brandon Horn <rihorn@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update chart sublabels Signed-off-by: Brandon Horn <rihorn@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * package update Signed-off-by: Brandon Horn <rihorn@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * with retry Signed-off-by: Brandon Horn <rihorn@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * last tweaks Signed-off-by: Brandon Horn <rihorn@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update short description as agreed (#139) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update email address (#142) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Turn off Python 3.5 for Notebooks (#143) Having trouble with Python 3.5 in the Notebooks build Since this was a recent addition, turn it back off pending debug Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * README update for v0.3.0 release (#141) Update ReadMe for release Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Miro Dudik <mdudik@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix ReadMe characters (#147) Back port some fixes for 'special' characters in the ReadMe from the release branch Can't just cherry pick the changeset since other release-only changes were made Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh upgrade to v0.1.7 to fix notebooks in python 3.5 (#145) * tempeh upgrade, should fix py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * restrict macos to 3.6+ again Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * polish and fix contributing.md (#146) * polish and fix contributing.md Signed-off-by: Miro Dudik <mdudik@gmail.com> * remove _pmf_predict and rephrase 'allow providing' Signed-off-by: Miro Dudik <mdudik@gmail.com> * Update section on creating new releases Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Link fixing Signed-off-by: Richard Edgar <riedgar@microsoft.com> * fix TestPyPI and remove conda Signed-off-by: Miro Dudik <mdudik@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add logging for GridSearch (#150) Put in some basic logging for `GridSearch` Remove annoying console output from `ExponentiatedGradient` smoke tests Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add regression metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address comments and add descriptions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Postprocessing documentation updates (#152) * lots of documentation, minor simplifications Signed-off-by: Roman Lutz <rolutz@microsoft.com> * complete documentation updates for postprocessing Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add test case for degenerate labels Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update notebook with renamed functions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * a few more corrections in the notebook from attribute -> sensitive_feature Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move changes to the correct notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * improve _pmf_predict return description based on feedback Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add authors page (#149) * authors page Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add user study Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add CODEOWNERS and get rid of detailed descriptions Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logging file initialization (#153) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Exponentiated Gradient documentation updates (#151) * first pass at expgrad related doc updates Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix typo Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Make second linprog call use simplex method (#154) Ensure second call to `linprog` uses same algorithm as the first Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh v0.1.8 (#155) Upgrade to have simpler dataset formatting for notebooks Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Single element fix for group metrics (#158) It turns out that `numpy.squeeze([1])` results in a scalar and not a single element array This was causing trouble with the group metrics when only a single prediction was passed Issues also showed up in the mean prediction metrics, which made a similar call Solution is to add a helper function (with tests) to check for this case, and ensure that a `numpy.ndarray` is always returned Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Allow a release candidate to be specified via environment variable (#159) Add logic to `__init__.py` to allow a release candidate number to be specified via an environment variable The goal of this is to allow us to push different versions to PyPI-Test without affecting the version which is ultimately pushed to PyPI itself Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tweak text Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Binning of numeric sensitive features (#157) * first pass at binning control * report and multimodel handle NaN * some styling * build with numeric bins * styled to comments * update built files * morre style * update style and tooltips * build generated files * fix styling of tile * rename to y_true * missed rename * allow dictionary ofy_pred with names * comment * patch regression to probit bug * include ipywidgets in requirements * flake comment Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Initial Pipeline for PyPI releases (#160) An initial attempt at a pipeline for performing PyPI releases. The pipeline does the following - Prevalidation - For PyPI-Test - Creates the wheels - Uploads to PyPI-Test - Fetches the package using `pip install` and runs our tests - Repeats the above for PyPI Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Version bump (#161) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix probit case (#162) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix name duplication (#163) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add balanced accuracy (#164) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Change macOS agent to be the latest (#165) Trying to work around an ongoing issue with Azure Pipelines: https://status.dev.azure.com/_event/159022677 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Change to an alpha release (#166) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Model ID and text changes (#168) * make small changes * one more string fix * the steps with updating new npm package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Improve flexibility of release versions (#170) Change the logic which adjusts the version uploaded to PyPI-Test by the release pipeline so that it appends `.dev[n]` rather than `rc[n]` .This change allows releases to be specified as alpha, beta or release candidate in the `_base_version` specified in `__init__.py`. Also update the steps for performing a release on the Contributing page. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix probit issue (#172) * fix name duplication (#163) Signed-off-by: Richard Edgar <riedgar@microsoft.com> * add balanced accuracy (#164) Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Change macOS agent to be the latest (#165) Trying to work around an ongoing issue with Azure Pipelines: https://status.dev.azure.com/_event/159022677 Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Change to an alpha release (#166) Signed-off-by: Richard Edgar <riedgar@microsoft.com> * make small changes Signed-off-by: Richard Edgar <riedgar@microsoft.com> * one more string fix Signed-off-by: Richard Edgar <riedgar@microsoft.com> * the steps with updating new npm package Signed-off-by: Richard Edgar <riedgar@microsoft.com> * fix probit Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Improve flexibility of release versions (#170) Change the logic which adjusts the version uploaded to PyPI-Test by the release pipeline so that it appends `.dev[n]` rather than `rc[n]` .This change allows releases to be specified as alpha, beta or release candidate in the `_base_version` specified in `__init__.py`. Also update the steps for performing a release on the Contributing page. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Richard Edgar <riedgar@microsoft.com> * version numbers Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Include yarn build in deployment pipeline (#174) Update the deployment pipeline so that it runs `yarn` to build the widget code This enables the removal of some automatically generated files from our repository Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Dashboard for Census Notebook (#171) Update the existing Census notebook for grid search to use the new dashboard. The bulk of the notebook is unchanged (including the fictional motivating scenario). Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Stop installing old dashboard (#176) Have moved notebooks off the old dashboard. Remove dependency from pipelines Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update ReadMe with Yarn instructions (#177) Now that the dashboard tarball is no longer checked in, provide instructions on creating it in a cloned repo Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Standardise input convertors for test (#178) Create and use a standard set of convertors for use with our 'argument type' tests. This has required adding several `__init__.py` files to the `test` directory to enable `pytest` to find the common code. Also add an 'argument type' test to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Small fixes to get the documentation appearing (#179) Fix issues in getting documentation to appear in Sphinx. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * law school notebook (#169) * law school notebook Signed-off-by: Miro Dudik <mdudik@gmail.com> * Remove hypens from filename Some copy edit fixes Correct suspected bug in ExponentiatedGradient section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Didn't quite undo all my temporary changes Signed-off-by: Richard Edgar <riedgar@microsoft.com> * address some of the comments Signed-off-by: Miro Dudik <mdudik@gmail.com> * Fix typo in name Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Improve spacing and add a comment in expgrad section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * add AUC explanation Signed-off-by: Miro Dudik <mdudik@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Code cleanups (#181) Fix some minor things: - Make the dashboard use the same copyright notice as the rest of the code - Some renaming of `expgrad` to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Build the widget (#185) Add a job template which builds the widget to the PR-Gate, Nightly and Nightly-Fixed builds. Note that this does not run any tests, but just ensures that the widget builds successfully Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update logging to use FileHandler instead of basicConfig (#175) Signed-off-by: Ilya Matiach <ilmat@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Enable ReadTheDocs (#182) Change how the documentation is done slightly, so that our documentation can show up on ReadTheDocs. Some additional copy-editing of the in-code documentation has been done as a result of this. The docs should appear at: https://fairlearn.readthedocs.io/en/latest/ Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Pin scikit-learn (#189) The recent update to scikit-learn is causing a break in one of the Notebooks. Until this is debugged, pin the version Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add more flake8 checks (#187) Add a number of extra flake8 checks: - flake8-blind-except - flake8-builtins - flake8-docstrings - flake8-logging-format - flake8-rst-docstrings Since these create a huge number of issues, suppress a lot of these for now in `setup.cfg` (plus a handful of special cases done inline). Put in fixes for the simpler complaints, such as: - Separate summaries in docstrings - Spacing within and around docstrings - Deferring string interpolation in `logging` calls Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Rename files and update license and docs (#183) * rename files * update comment * update license * address comments Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix for Law School Notebook (#191) Tweak the Law School notebook so that it works with the latest `scikit-learn` This enables us to unpin the version of `scikit-learn` in our `requirements.txt` file Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Markdown updates based on doc bash (#186) * address feedback from doc bash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex updates Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo latex changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove commas Signed-off-by: Roman Lutz <rolutz@microsoft.com> * rephrasing postprocessing constructor requirements Signed-off-by: Roman Lutz <rolutz@microsoft.com> * feedback from Miro Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Reorganise documentation (#192) Reorganising how the documentation is presented, since the default style from `sphinx-apidoc` assumed we had lots of individual modules rather than larger packages Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Declare 0.4.0 release (#193) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Workaround Python 3.5 issue with Linux (#194) An issue with the `pip` install of `shap` has appeared on the Linux agents under Python 3.5. Reasons are currently obscure, but this is blocking a release. Since Python 3.5 continues to work on Windows, rely on that (pending further debugging) Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix classification classification bug (#201) * Update readme for v0.4.0 (#196) Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Pin troublesome package (#198) During our release process, a new version of `colorama` (required by one of our dependencies) was released. This has issues with the Windows/3.7 build. Unblock the release by pinning the version Signed-off-by: Richard Edgar <riedgar@microsoft.com> * fix classification classification bug Signed-off-by: Roman Lutz <rolutz@microsoft.com> * version change to address security bug (#203) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix accidental merge (#205) Some portions of the v0.4.0 release branch were accidentally merged into master - Making the ReadMe version suitable for PyPI - Pinning the `colorama` version to unblock the release train This changeset undoes these fixes in master Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ReadMe Processor for Releases (#206) Create a python script to translate `ReadMe.md` from GitHub to PyPI. This will avoid the need to create a branch to do a release. This script is slightly dependent on the structure of the file, so if there are substantial changes to that, this script will require updating. It also assumes that a tag `v(fairlearn.__version__)` exists in the repo. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Track pip dependencies (#208) We've had trouble with our dependencies updating and breaking our builds Augment the build pipelines so that they publish the output of `pip freeze` to an artifact. This will aid debugging these issues. The name of both the artifact itself and the file therein can be specified. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Remove some flake8 global suppressions (#209) After adding more `flake8` analysers, we were obliged to put in some global suppressions to keep the number of issues manageable. Start the process of removing these with D102, D103 and D401. Some of these just move the suppression to file-level, while others tweak documentation blocks to suit. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Re-enable Linux 3.5 (#210) Roman figured out a workaround for getting `shap` installed with Linux and Python 3.5. Put this into `fairlearn` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Expand Notebook testing (#212) Increase the variety of platforms used for testing our Jupyter Notebooks. Unable to test on MacOS at present, due to some problem installing `lightgbm`. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Improvements for pinning requirements (#213) A better way of running our tests with pinned requirements. Rather than have a separate `requirements-fixed.txt` file, have a script to turn the `requirements.txt` file into the former. Update builds accordingly. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Standardise ML argument documentation (#214) Make our documentation of fit(), X, predict() etc. more standard between our various submodules. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf test through Azure ML (#180) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Documentation and flake8 (#215) Various updates for the documentation: - Remove another `flake8` global suppression - Add explanations for remaining `flake8` suppressions - Replace the `:any:` references in the documentation with appropriate ones - Make some file-level suppressions (which may have actually turned `flake8` off entirely on the file) specific to the appropriate lines Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update Release pipeline after KV move (#216) The KeyVault containing the PyPI secrets has been moved to a more appropriate subscription. As a result, the Release pipeline needs to be updated with the correct service connection Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Basic unit tests for EqualizedOdds and DemographicParity moments (#217) Some very basic unit tests for the `EqualizedOdds` and `DemographicParity` moment classes. These are 'pinning' tests two establish the behaviour of these classes. The `gamma` method is not yet included in these tests, since that requires a trained model. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add more time metrics to performance tests (#219) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add additional time-based metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustments to fix syntax errors and logical issues in the calculation of metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add oracle calls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * custom metrics for executions times, min, max, mean Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo sphinx special docs for test/perf Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update description of oracle execution time properties Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix perf tests by logging lists through log_list instead of log (#221) * perf test first version through Azure ML Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move some code to tempeh Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add missing files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * perf tests that get auth details through Azure Keyvault Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade to alpha tempeh version Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * exclude D100 and D103 for script generation python file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * move variables into nightly-perf.yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml sdk requirement for perf test Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove powershell syntax Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add cwd for tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print cwd Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove special working directory condition Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix directory handling based on ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to a2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print message for debugging Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try upper case variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove extraneous dash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * print env var names Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try explicitly adding variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use variables directly, tempeh bump Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add hardcoced data as variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use windows instead of linux because some of the UI packages aren't available in linux Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add wheel dependency Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fake dashboard files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pass parameters for perf tests via args Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * yaml fix Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove waiting for run to complete Signed-off-by: Roman Lutz <rolutz@microsoft.com> * refactor to submit all jobs without waiting for result Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove obsolete gitignore line Signed-off-by: Roman Lutz <rolutz@microsoft.com> * tempeh bump to 0.1.11 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * azureml-sdk warning Signed-off-by: Roman Lutz <rolutz@microsoft.com> * pipeline improvements to use keyvault tasks Signed-off-by: Roman Lutz <rolutz@microsoft.com> * logically separate script generation into steps Signed-off-by: Roman Lutz <rolutz@microsoft.com> * simplify writing long string of = signs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace incorrect variables in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use importerror instead of modulenotfounderror for py3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add PR trigger for changes to test/perf directory Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove output from notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * quotes for yaml variables Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct parameter in yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add additional time-based metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustments to fix syntax errors and logical issues in the calculation of metrics Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add oracle calls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * custom metrics for executions times, min, max, mean Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo sphinx special docs for test/perf Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update description of oracle execution time properties Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * bug fix for list logging Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * make metric logging a lot more readable and provide additional metrics to show the overhead fairlearn adds (#228) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Convert Notebook tests to papermill (#223) Rather than using `nbval`, convert our notebook tests to use `papermill`. With the help of `nteract-scrapbook` we can then examine the contents of particular variables from the notebooks to ensure that we're getting the expected results. Explicit `scrapbook` commands are required to save out values for future examination, but we don't want to include these when our users look at the notebooks. Accordingly, we include machinery for adding the necessary cells to the notebooks dynamically. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Bump dashboard npm package to match source code (#229) * publish latest version * update docs for push Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Remove unused ReST files (#233) Two of the ReST files generated by sphinx-autodoc weren't actually used. Remove them to get rid of a warning. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Basic Moments documentation (#241) Add some basic documentation of the `Moment` class and its subclasses. Also: - Turn the `n` field of the `Moment` object into a `total_samples` property - Add `intersphinx` hook for `pandas` documentation Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * [WIP] create extensions to install custom plots separately & check in generated files (#240) * check in generated javascript files and split into package with extensions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add installation tests, move yml files to templates directory if appropriate, delete unused and broken yml file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * separate directories per package, composition with minimal fairlearn package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * script updates to get doc and wheel builds in shape Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update yml files and scripts to enable wheel upload per package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address feedback from PR by adding documentation to the pipeline definition yml files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove "templates/" as location prefix for files in the templates directory itself Signed-off-by: Roman Lutz <rolutz@microsoft.com> * first version of widget build validation script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * corrections in widget build validation Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo adjustments to completely split up packages Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * reverse code coverage build changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ignore install tests when necessary Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add macos python 3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add exceptions module back to documentaiton Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add name for job Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix characters in job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add image label Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct installation path Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo other changes to rst file Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add logging variant of numpy.all_close (#246) The `numpy` package provides an `all_close` routine for comparing two arrays. Unfortunately, there's no mechanism for showing which elements failed the comparison. Put together a wrapper based on `numpy.isclose` which will print out information about failed comparisons. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Implement GroupMetricSet (#250) Create a `GroupMetricSet` class for holding collections of grouped metrics. This is to help with AzureML integration. There has been some (possibly unnecessary) reorganisation of things under `fairlearn/metrics` but the public interface is unchanged. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Exclude install tests in code coverage check (#251) * ignore install tests since they'll unexpectedly work Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add python -m before pip install Signed-off-by: Roman Lutz <rolutz@microsoft.com> * upgrade tempeh to v0.1.12 Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Replace powershell scripts with python and add Makefile (#249) * check in generated javascript files and split into package with extensions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add installation tests, move yml files to templates directory if appropriate, delete unused and broken yml file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * separate directories per package, composition with minimal fairlearn package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * script updates to get doc and wheel builds in shape Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update yml files and scripts to enable wheel upload per package Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address feedback from PR by adding documentation to the pipeline definition yml files Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove "templates/" as location prefix for files in the templates directory itself Signed-off-by: Roman Lutz <rolutz@microsoft.com> * first version of widget build validation script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * corrections in widget build validation Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo adjustments to completely split up packages Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * reverse code coverage build changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix yml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * ignore install tests when necessary Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add macos python 3.5 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add exceptions module back to documentaiton Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add name for job Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix characters in job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix job name Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add image label Signed-off-by: Roman Lutz <rolutz@microsoft.com> * correct installation path Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo other changes to rst file Signed-off-by: Roman Lutz <rolutz@microsoft.com> * rewrite scripts in python Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace widget build script with python script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * build_widget adjustments to make it work Signed-off-by: Roman Lutz <rolutz@microsoft.com> * build_widget finalization plus add ls commands to find yarn installation in ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * some more paths to check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * task -> script Signed-off-by: Roman Lutz <rolutz@microsoft.com> * usr/bin/yarn check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * workingDirectory adjustment Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add ls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjustment for fairlearn root dir check Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add ./ Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix template Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * comment about set-variable-from-file script only being required in ADO Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add makefile, update contributing guide, and replace remaining ps1 occurrences in pipeline ymls Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add romanlutz to codeowners for scripts dir Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix comment Signed-off-by: Roman Lutz <rolutz@microsoft.com> * make process_readme a standalone script again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * delete build_docs Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * makefile adjustments according to feedback Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * try out multiple sensitive features using new tempeh features Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix pypi release yaml (#260) * undo erroneous changes to yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo prior erroneous change Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace job template usage with just a step Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add CHANGES.md for v0.4.2 (#262) * add CHANGES.md for v0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add general instructions to always do that Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update CHANGES.md Adding `GroupMetricSet` to the changelog Signed-off-by: Richard Edgar <riedgar@microsoft.com> * comment out test that fails consistently only on windows Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix readme processing script by adding fairlearn dir to sys path, add second solution for issue 265 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix syntax error, flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * bump version to 0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * test with list of lists instead of single list Signed-off-by: Roman Lutz <rolutz@microsoft.com> Co-authored-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update metric keys to match dashboard (#268) The dashboard already had its own keys defined for mapping metric functions to strings. Update the `GroupMetricSet` to use the same keys. Figuring out how to unify the two implementations of this mapping is left as a issue #269 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix release blockers - widget generated files, widget validation (#267) * add built widget file updates & fix widget build validation, as well as pypi release template for empty DEV_VERSION Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo DEV_VERSION change Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add comment and link to issue Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * trying out notebook Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove --assert-no-changes flag in release as well (#272) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update default metrics in GroupMetricSet (#271) Tweak the list of metrics computed by default by the `compute` method of `GroupMetricSet` to match those expected by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjust notebook to work with multiple sensitive features Signed-off-by: Roman Lutz <rolutz@microsoft.com> * input transformation for reductions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * set env var before installing fairlearn to correct version file name content (#274) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * MNT use sklearn's NotFittedError instead of NotFittedException (#259) * MNT use sklearn's NotFittedError instead of NotFittedException Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * add to the changelog Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * multiple sensitive features (passing tests) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * get tests to pass again after reformatting input data Signed-off-by: Roman Lutz <rolutz@microsoft.com> * switch to commas as separators Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use sklearn input validation and get all tests to pass (grid search to be investigated) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix numerical inconsistencies due to rounding to int; fix warning from 2d arrays Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove last warnings Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo notebook changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove TODO that was fixed with removing the int-rounding earlier Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use fixtures to reduce parametrization duplication Signed-off-by: Roman Lutz <rolutz@microsoft.com> * avoid fixture data corruption by doing deepcopy Signed-off-by: Roman Lutz <rolutz@microsoft.com> * grid search tests for multiple sensitive features Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix pypi release yaml (#260) * undo erroneous changes to yaml Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo prior erroneous change Signed-off-by: Roman Lutz <rolutz@microsoft.com> * replace job template usage with just a step Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add CHANGES.md for v0.4.2 (#262) * add CHANGES.md for v0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add general instructions to always do that Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update CHANGES.md Adding `GroupMetricSet` to the changelog Signed-off-by: Richard Edgar <riedgar@microsoft.com> * comment out test that fails consistently only on windows Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix readme processing script by adding fairlearn dir to sys path, add second solution for issue 265 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix syntax error, flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * bump version to 0.4.2 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * test with list of lists instead of single list Signed-off-by: Roman Lutz <rolutz@microsoft.com> Co-authored-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove --assert-no-changes flag in release as well (#272) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update default metrics in GroupMetricSet (#271) Tweak the list of metrics computed by default by the `compute` method of `GroupMetricSet` to match those expected by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * adjust notebook to work with multiple sensitive features Signed-off-by: Roman Lutz <rolutz@microsoft.com> * MNT use sklearn's NotFittedError instead of NotFittedException (#259) * MNT use sklearn's NotFittedError instead of NotFittedException Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * add to the changelog Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * input transformation for reductions Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Updates for GroupMetricResult and GroupMetricSet (#279) Add (in)equality operators to `GroupMetricResult` and `GroupMetricSet`, along with basic tests. These will simplify other testing in future. Change `GroupMetricSet` so that the `groups` have to be specified as sequential integers from zero. If this is not the case then the `compute()` method will remap the supplied groups to `[0, 1, 2, ….]` and put the stringified original values into the `group_names` property. Since the keys are now sequential integers, convert the `group_names` property itself from a dictionary into a list. Closes #275 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * DOC contributing: trim lines and add notes on signoff (#276) * DOC contributing: trim lines and add notes on signoff Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * hook Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * modify note to point to the right answer Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * multiple sensitive features (passing tests) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Further metric changes (#281) A number of extra changes to metrics: - `GroupMetricResult` now dynamically calculates `maximum`, `range` etc. - `GroupMetricSet` has a consistency check - `GroupMetricSet` can transform itself to and from a dictionary matching the schema used by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Preparations for v0.4.3 Release (#284) Bump version and update Markdown files Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Use kwargs in metrics (#286) Change `metric_by_group` and `make_group_metric` to understand `**kwargs`. This removes the need for lots of small wrapper functions Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add expgrad tests for multiple sensitive features Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address ensure_ndarray_2d logic for clarity Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add skip reason for invalid tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Updates for GroupMetricResult and GroupMetricSet (#279) Add (in)equality operators to `GroupMetricResult` and `GroupMetricSet`, along with basic tests. These will simplify other testing in future. Change `GroupMetricSet` so that the `groups` have to be specified as sequential integers from zero. If this is not the case then the `compute()` method will remap the supplied groups to `[0, 1, 2, ….]` and put the stringified original values into the `group_names` property. Since the keys are now sequential integers, convert the `group_names` property itself from a dictionary into a list. Closes #275 Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * DOC contributing: trim lines and add notes on signoff (#276) * DOC contributing: trim lines and add notes on signoff Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * hook Signed-off-by: adrinjalali <adrin.jalali@gmail.com> * modify note to point to the right answer Signed-off-by: adrinjalali <adrin.jalali@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Further metric changes (#281) A number of extra changes to metrics: - `GroupMetricResult` now dynamically calculates `maximum`, `range` etc. - `GroupMetricSet` has a consistency check - `GroupMetricSet` can transform itself to and from a dictionary matching the schema used by the dashboard Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * get tests to pass again after reformatting input data Signed-off-by: Roman Lutz <rolutz@microsoft.com> * switch to commas as separators Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Preparations for v0.4.3 Release (#284) Bump version and update Markdown files Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use sklearn input validation and get all tests to pass (grid search to be investigated) Signed-off-by: Roman Lutz <rolutz@microsoft.com> * fix numerical inconsistencies due to rounding to int; fix warning from 2d arrays Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove last warnings Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo notebook changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove TODO that was fixed with removing the int-rounding earlier Signed-off-by: Roman Lutz <rolutz@microsoft.com> * use fixtures to reduce parametrization duplication Signed-off-by: Roman Lutz <rolutz@microsoft.com> * avoid fixture data corruption by doing deepcopy Signed-off-by: Roman Lutz <rolutz@microsoft.com> * grid search tests for multiple sensitive features Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add expgrad tests for multiple sensitive features Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * address ensure_ndarray_2d logic for clarity Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Use kwargs in metrics (#286) Change `metric_by_group` and `make_group_metric` to understand `**kwargs`. This removes the need for lots of small wrapper functions Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * add skip reason for invalid tests Signed-off-by: Roman Lutz <rolutz@microsoft.com> * flake8 Signed-off-by: Roman Lutz <rolutz@microsoft.com> * multiple sensitive features - postprocessing (#288) * take changes from other branch that touches all modules Signed-off-by: Roman Lutz <rolutz@microsoft.com> * get all tests working again Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Dashboard for Census Notebook (#171) Update the existing Census notebook for grid search to use the new dashboard. The bulk of the notebook is unchanged (including the fictional motivating scenario). Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Stop installing old dashboard (#176) Have moved notebooks off the old dashboard. Remove dependency from pipelines Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Update ReadMe with Yarn instructions (#177) Now that the dashboard tarball is no longer checked in, provide instructions on creating it in a cloned repo Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Standardise input convertors for test (#178) Create and use a standard set of convertors for use with our 'argument type' tests. This has required adding several `__init__.py` files to the `test` directory to enable `pytest` to find the common code. Also add an 'argument type' test to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Small fixes to get the documentation appearing (#179) Fix issues in getting documentation to appear in Sphinx. Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * law school notebook (#169) * law school notebook Signed-off-by: Miro Dudik <mdudik@gmail.com> * Remove hypens from filename Some copy edit fixes Correct suspected bug in ExponentiatedGradient section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Didn't quite undo all my temporary changes Signed-off-by: Richard Edgar <riedgar@microsoft.com> * address some of the comments Signed-off-by: Miro Dudik <mdudik@gmail.com> * Fix typo in name Signed-off-by: Richard Edgar <riedgar@microsoft.com> * Improve spacing and add a comment in expgrad section Signed-off-by: Richard Edgar <riedgar@microsoft.com> * add AUC explanation Signed-off-by: Miro Dudik <mdudik@gmail.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Code cleanups (#181) Fix some minor things: - Make the dashboard use the same copyright notice as the rest of the code - Some renaming of `expgrad` to `ExponentiatedGradient` Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Build the widget (#185) Add a job template which builds the widget to the PR-Gate, Nightly and Nightly-Fixed builds. Note that this does not run any tests, but just ensures that the widget builds successfully Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * update logging to use FileHandler instead of basicConfig (#175) Signed-off-by: Ilya Matiach <ilmat@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Enable ReadTheDocs (#182) Change how the documentation is done slightly, so that our documentation can show up on ReadTheDocs. Some additional copy-editing of the in-code documentation has been done as a result of this. The docs should appear at: https://fairlearn.readthedocs.io/en/latest/ Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Pin scikit-learn (#189) The recent update to scikit-learn is causing a break in one of the Notebooks. Until this is debugged, pin the version Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Add more flake8 checks (#187) Add a number of extra flake8 checks: - flake8-blind-except - flake8-builtins - flake8-docstrings - flake8-logging-format - flake8-rst-docstrings Since these create a huge number of issues, suppress a lot of these for now in `setup.cfg` (plus a handful of special cases done inline). Put in fixes for the simpler complaints, such as: - Separate summaries in docstrings - Spacing within and around docstrings - Deferring string interpolation in `logging` calls Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Rename files and update license and docs (#183) * rename files * update comment * update license * address comments Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Fix for Law School Notebook (#191) Tweak the Law School notebook so that it works with the latest `scikit-learn` This enables us to unpin the version of `scikit-learn` in our `requirements.txt` file Signed-off-by: Richard Edgar <riedgar@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Markdown updates based on doc bash (#186) * address feedback from doc bash Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex updates Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * latex update Signed-off-by: Roman Lutz <rolutz@microsoft.com> * undo latex changes Signed-off-by: Roman Lutz <rolutz@microsoft.com> * remove commas Signed-off-by: Roman Lutz <rolutz@microsoft.com> * rephrasing postprocessing constructor requirements Signed-off-by: Roman Lutz <rolutz@microsoft.com> * feedback from Miro Signed-off-by: Roman Lutz <rolutz@microsoft.com> Signed-off-by: Roman Lutz <rolutz@microsoft.com> * Reorganise documentation (#192) Reorganising how the documentation is presented, since the default style from `sphinx-apidoc` assumed we had lot…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Richard Edgar riedgar@microsoft.com