feat: Measure transformers for Data Balance Analysis #1218

ms-kashyap · 2021-10-19T08:11:39Z

As part of the Data Balance Analysis feature, I'm introducing three transformers under the exploratory namespace in synapseml-core.

At a high level:

The DistributionMeasures transformer computes data imbalance measures based on a reference distribution.
The AggregateMeasures transformer computes a set of aggregated measures that represents how balanced or imbalanced the given dataframe is along the given sensitive features.
The ParityMeasures transformer computes a set of parity measures from the given dataframe and sensitive features.

This work is based on the work of @ankit-oss.
Additionally, the parity measures were heavily influenced by this paper titled "Measuring Model Biases in the Absence of Ground Truth".

feat: ONNX model inference on Spark (microsoft#1152)

…dress PR to jasowang comments, pSensitive -> pFeature

…ence values as columns

…p-value, address memoryz PR comments, merge from master (new namespace)

…e Analysis)

mhamilton723

Awesome Contribution and very clean code! Left a few nits.

On the whole one thing I am thinking about is whether these should be Transformers or Evaluators. Perhaps we can discuss offline with everyone

core/src/main/scala/com/microsoft/azure/synapse/ml/exploratory/AggregateMeasures.scala

core/src/main/scala/com/microsoft/azure/synapse/ml/exploratory/DistributionMeasures.scala

core/src/test/scala/com/microsoft/azure/synapse/ml/exploratory/AggregateMeasuresSuite.scala

core/src/test/scala/com/microsoft/azure/synapse/ml/exploratory/DistributionMeasuresSuite.scala

core/src/main/scala/com/microsoft/azure/synapse/ml/exploratory/AggregateMeasures.scala

…ents

ms-kashyap · 2021-10-19T22:54:57Z

/azp run

azure-pipelines · 2021-10-19T22:55:03Z

Commenter does not have sufficient privileges for PR 1218 in repo microsoft/SynapseML

ms-kashyap · 2021-10-19T23:00:35Z

/azp run

azure-pipelines · 2021-10-19T23:00:45Z

Azure Pipelines successfully started running 1 pipeline(s).

ms-kashyap · 2021-10-21T22:40:04Z

/azp run

azure-pipelines · 2021-10-21T22:40:15Z

Azure Pipelines successfully started running 1 pipeline(s).

...src/test/scala/com/microsoft/azure/synapse/ml/exploratory/AggregateBalanceMeasureSuite.scala

.../test/scala/com/microsoft/azure/synapse/ml/exploratory/DistributionBalanceMeasureSuite.scala

core/src/test/scala/com/microsoft/azure/synapse/ml/exploratory/FeatureBalanceMeasureSuite.scala

ms-kashyap · 2021-10-22T01:02:11Z

/azp run

azure-pipelines · 2021-10-22T01:02:21Z

Azure Pipelines successfully started running 1 pipeline(s).

ms-kashyap and others added 30 commits August 11, 2021 15:29

Merge pull request #1 from Azure/master

8f29786

feat: ONNX model inference on Spark (microsoft#1152)

Merge remote-tracking branch 'upstream/master'

e2b68cc

Initial implementation of AssociationGapTransformer

d67aa64

Add normalized PMI

651c825

Add other association metrics

f484deb

Add names of measures

4121238

Change counts to probabilities in transformer, Add dp unit test

b321458

Add other association measure unit tests, Fix sPmi and krc bugs

d83b398

Unit tests cleanup

063e290

Merge remote-tracking branch 'upstream/master' into kapat/dataimbalance

7351aca

Refactor code

ccf318f

[AggregateMeasureTransformer] Initial implementation

25a6d39

[AggregateMeasureTransformer] Checkpoint

2a404f3

Changes by jasowang

5b5693f

[AggregateMeasureTransformer] Move agg calcs to AggregateMetrics

a582292

[AssociationGapTransformer] Quick fix

8d83710

[AggregateMeasureTransformer] Add unit tests

89b2cc1

Merge remote-tracking branch 'upstream/master' into kapat/dataimbalance

e08d54a

Add DistributionMeasures, update DataImbalance tests

68723fc

Changes by jasowang

38a762d

Update all DataImbalance transformers based on jasowang's comments

ba0ab2c

Remove use of null

aa7fa71

Make fixes to build Python package

aed785e

Move common code to DataImbalanceParams and DataImbalanceTestBase, ad…

e4a57ad

…dress PR to jasowang comments, pSensitive -> pFeature

[DistributionMeasures] Introduce uniform distribution and treat refer…

c6e81d0

…ence values as columns

AssociationGaps -> ParityMeasures

2964c68

Merge remote-tracking branch 'upstream/master'

84d1441

Merge from master branch

d319af3

Add TransformerFuzzing to all unit test suites, refactor chi-squared …

512de30

…p-value, address memoryz PR comments, merge from master (new namespace)

DataImbalance -> DataBalance (to go with official name of Data Balanc…

105b1f0

…e Analysis)

mhamilton723 requested changes Oct 19, 2021

View reviewed changes

mhamilton723 reviewed Oct 19, 2021

View reviewed changes

core/src/main/scala/com/microsoft/azure/synapse/ml/exploratory/AggregateMeasures.scala Outdated Show resolved Hide resolved

ms-kashyap added 2 commits October 19, 2021 15:52

{Refactor unit tests, introduce imbalance namespace} based on PR comm…

8acd169

…ents

Merge remote-tracking branch 'upstream/master' into kapat/dataimbalance

9b8773f

ms-kashyap requested a review from mhamilton723 October 19, 2021 22:54

ms-kashyap marked this pull request as draft October 20, 2021 03:22

ms-kashyap and others added 5 commits October 20, 2021 00:15

Change measures output from MapType to nested struct

34eb4d4

Update scaladocs based on MapType -> nested struct change

f0244aa

[ParityMeasures] Fix bug for calculating p(PositiveAndSensitive)

c03b35e

[ParityMeasures] Update unit tests

60684a8

Merge branch 'master' into kapat/dataimbalance

9a6a5c0

ms-kashyap marked this pull request as ready for review October 21, 2021 03:07

memoryz and others added 3 commits October 20, 2021 20:42

Cosmetic changes

2b9dce2

adding the experimental tag

4bf586a

[DataBalanceAnalysis] Naming changes

041142a

mhamilton723 previously approved these changes Oct 22, 2021

View reviewed changes

[DataBalanceAnalysis] Use === in unit tests

c2a759f

ms-kashyap dismissed mhamilton723’s stale review via c2a759f October 22, 2021 01:01

ms-kashyap requested a review from mhamilton723 October 22, 2021 01:02

mhamilton723 approved these changes Oct 22, 2021

View reviewed changes

ms-kashyap merged commit c5e1742 into microsoft:master Oct 22, 2021

ms-kashyap mentioned this pull request Mar 21, 2023

feat: [DistributionBalanceMeasure] Add implementation + unit tests for custom reference distribution #1885

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Measure transformers for Data Balance Analysis #1218

feat: Measure transformers for Data Balance Analysis #1218

ms-kashyap commented Oct 19, 2021

mhamilton723 left a comment

ms-kashyap commented Oct 19, 2021

azure-pipelines bot commented Oct 19, 2021

ms-kashyap commented Oct 19, 2021

azure-pipelines bot commented Oct 19, 2021

ms-kashyap commented Oct 21, 2021

azure-pipelines bot commented Oct 21, 2021

ms-kashyap commented Oct 22, 2021

azure-pipelines bot commented Oct 22, 2021

feat: Measure transformers for Data Balance Analysis #1218

feat: Measure transformers for Data Balance Analysis #1218

Conversation

ms-kashyap commented Oct 19, 2021

mhamilton723 left a comment

Choose a reason for hiding this comment

ms-kashyap commented Oct 19, 2021

azure-pipelines bot commented Oct 19, 2021

ms-kashyap commented Oct 19, 2021

azure-pipelines bot commented Oct 19, 2021

ms-kashyap commented Oct 21, 2021

azure-pipelines bot commented Oct 21, 2021

ms-kashyap commented Oct 22, 2021

azure-pipelines bot commented Oct 22, 2021