Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add experimental metrics package following the array API standard #499

Merged
merged 42 commits into from
Nov 14, 2023

Conversation

fcogidi
Copy link
Collaborator

@fcogidi fcogidi commented Nov 7, 2023

PR Type

Feature.

Short Description

Add new (experimental) metrics package with following features:

  • Cross-framework support based on the Array API Standard.
  • Distributed metric computation, including synchronization of metric state across multiple devices. Supported backend include mpi4py and torch.distributed.

Tests Added

  • tests/cyclops/evaluate/metrics/experimental/test_metric.py
  • tests/cyclops/evaluate/metrics/experimental/utils/test_ops.py
  • tests/cyclops/evaluate/metrics/experimental/utils/test_validation.py
  • tests/cyclops/evaluate/metrics/experimental/test_confusion_matrix.py

@fcogidi fcogidi added enhancement New feature or request refactor Refactor existing code, with same or similar functionality labels Nov 7, 2023
Copy link

codecov bot commented Nov 8, 2023

Codecov Report

Merging #499 (b80384a) into main (02c7e8c) will increase coverage by 1.40%.
Report is 4 commits behind head on main.
The diff coverage is 78.22%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #499      +/-   ##
==========================================
+ Coverage   62.30%   63.70%   +1.40%     
==========================================
  Files          87       96       +9     
  Lines        8454     9409     +955     
==========================================
+ Hits         5267     5994     +727     
- Misses       3187     3415     +228     
Files Coverage Δ
cyclops/utils/optional.py 100.00% <100.00%> (ø)
.../evaluate/metrics/experimental/confusion_matrix.py 98.80% <98.80%> (ø)
...perimental/distributed_backends/non_distributed.py 85.71% <85.71%> (ø)
...lops/evaluate/metrics/experimental/utils/typing.py 85.71% <85.71%> (ø)
.../metrics/experimental/distributed_backends/base.py 88.46% <88.46%> (ø)
.../evaluate/metrics/experimental/utils/validation.py 84.44% <84.44%> (ø)
...etrics/experimental/functional/confusion_matrix.py 89.47% <89.47%> (ø)
...etrics/experimental/distributed_backends/mpi4py.py 36.73% <36.73%> (ø)
...rimental/distributed_backends/torch_distributed.py 32.69% <32.69%> (ø)
cyclops/evaluate/metrics/experimental/metric.py 82.68% <82.68%> (ø)
... and 1 more

... and 9 files with indirect coverage changes

Impacted file tree graph

@fcogidi fcogidi marked this pull request as ready for review November 8, 2023 21:43
@fcogidi fcogidi requested a review from amrit110 November 8, 2023 21:44
@fcogidi fcogidi marked this pull request as draft November 8, 2023 22:13
cyclops/utils/optional.py Show resolved Hide resolved
@fcogidi fcogidi marked this pull request as ready for review November 14, 2023 22:08
@fcogidi fcogidi merged commit aee3f78 into main Nov 14, 2023
8 of 9 checks passed
@fcogidi fcogidi deleted the fco/init_array_api_adoption branch November 14, 2023 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request refactor Refactor existing code, with same or similar functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants