mean average precision meter #265
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fbshipit-source-id: c914abb26f72a87a9d156527438cf2e106b5bcdf
Summary: Pull Request resolved: facebookresearch/ClassyVision#240 As titled Reviewed By: vreis Differential Revision: D18632115 fbshipit-source-id: 3d4c0f9e1ce2cabbf24a0ff558c0cf968f38e24a
Summary: Pull Request resolved: facebookresearch/ClassyVision#205 Pull Request resolved: fairinternal/ClassyVision#34 OSS Classy Imagenet. Simple wrapper around torchvision dataset. The classy dataset wrapper provides batching, transforms, shuffling, and restricting the size of the dataset. Side-effects: Changed the name of the fb internal classy vision dataset to fb_imagenet and changed all naming schemes to match this. I also added a unittest that relies on some of the torchvision testing utilities which are not packaged with torchvision...as such, I did this via a fbcode import. Reviewed By: vreis Differential Revision: D18403673 fbshipit-source-id: 55337522667ef236dd2f4f274a5dcc40a79d0753
Summary: Pull Request resolved: fairinternal/ClassyVision#35 Pull Request resolved: facebookresearch/ClassyVision#206 OSS Classy CIFAR. Simple wrapper around torchvision dataset. The classy dataset wrapper provides batching, transforms, shuffling, and restricting the size of the dataset. Side-effects: Changed the name of the fb internal classy vision dataset to fb_cifar* and changed all naming schemes to match this. I also added a unittest that relies on some of the torchvision testing utilities which are not packaged with torchvision...as such, I did this via a fbcode import. Reviewed By: vreis Differential Revision: D18429440 fbshipit-source-id: 3ecf2ba19760e88841b08d15b121cf51444f7cb1
… to ApplyTransformToKey (#241) Summary: Pull Request resolved: facebookresearch/ClassyVision#241 Pull Request resolved: fairinternal/ClassyVision#40 Two changes are made in this diff: 1. Change the name of FieldTransform -> ApplyTransformToKey 2. Make the transform work on tuples / lists as well as dicts (so we can apply it to torchvision tuples, for example) Reviewed By: vreis Differential Revision: D18604454 fbshipit-source-id: 03b18c0d828df8cee1ec19bdc60a496175c176e3
Summary: Pull Request resolved: facebookresearch/ClassyVision#243 Pull Request resolved: fairinternal/ClassyVision#42 As titled, this registers the ApplyTransformToKey transform so that users can use it in configs. For example: { 'name': 'apply_transform_to_key', 'transform': [{'name': 'my_awesome_transform'}], 'key': 'input', } Applies my_awesome_transform to only the "input" key of the sample. { 'name': 'apply_transform_to_key', 'transform': [{'name': 'my_awesome_transform'}], 'key': 0, } If the sample is a tuple, then this applies the my_awesome_transform to the first entry in the tuple. Reviewed By: vreis Differential Revision: D18605946 fbshipit-source-id: 70a4c5bfb5a12db215b249a10826dc5f42f8b819
Summary: Pull Request resolved: facebookresearch/ClassyVision#242 Pull Request resolved: fairinternal/ClassyVision#41 Generic image transform. This is meant to be an easy default for applying transforms to torchvision datasets. It applies the transform provided to the first entry of a tuple and matches the tuple to a dict with 'input' / 'target' keys. If no transform is provided, then the split of the data needs to be provided and a default transform (currently standard random flips / crops for imagenet training) is applied. For example: config = [{"name": "generic_image_transform", "split": "train"}] Should take a sample of the form (image, target) -> {'input': transformed_image, 'target': target}. The transformed_image should have the transform defined by the ImagenetAugment transform or in torchvision terms: transforms.Compose( [ transforms.RandomResizedCrop(crop_size), transforms.RandomHorizontalFlip(), transforms.ToTensor(), transforms.Normalize(mean=mean, std=std), ] ) Reviewed By: vreis Differential Revision: D18620458 fbshipit-source-id: 4fdcd6c1c59b4b9150904736265eb65af4ae923c
Summary: Pull Request resolved: facebookresearch/ClassyVision#244 Changes - add `sync_state` method to `video_accuracy` meter. Otherwise, meter value is not consistent between gpus. Also add an unit test for synced `video_accuracy` meter. - We use `LossLrMeterLoggingHook` to log meters. We should run this hook after meters are updated. We fix this in `train_step` of `ClassificationTask` - Rename hook event `on_loss` to `on_loss_and_meter`. - Fix lint in classy_vision/classy_train.py` Reviewed By: vreis Differential Revision: D18648197 fbshipit-source-id: 06c44a5bf2a0b3fe32e4557df361b9cd15211ac1
…es (#245) Summary: Pull Request resolved: facebookresearch/ClassyVision#245 Pull Request resolved: fairinternal/ClassyVision#43 Reviewed By: vreis Differential Revision: D18660522 fbshipit-source-id: 4f39bb7eae66cbacf53f1d073e10bdcfc8a72eae
Summary: Pull Request resolved: facebookresearch/ClassyVision#229 Currently we are executing heads right after the forked block is executed, the input of the head would just be the output of the block. When we need extra input for a particular head, we would need to pass the input from the beginning of the network. This diff changes the head execution to happen after whole trunk execution: after the forked block is executed, we save the output of the block if there is heads on it, later we iterate through all the heads and feed corresponding block output to it. This allows us to pass arbitrary extra input to the head. Reviewed By: vreis Differential Revision: D18442933 fbshipit-source-id: e1eeaf1583df9c8b5e7d795aabb417cfe851e56e
Summary: Tweak a few things based on Jessica's suggestion: divide tutorial into parts, add an intro and recap sections. It looks a lot cleaner IMO Pull Request resolved: facebookresearch/ClassyVision#247 Differential Revision: D18673187 Pulled By: vreis fbshipit-source-id: 3255699f274b0a33da45b94b063b08397d9516cf
Summary: Pull Request resolved: facebookresearch/ClassyVision#215 Added a tutorial for creating a model. This is a very basic tutorial which skips the following concepts- - Attachable blocks - Heads - Optimizer params The primary reason is that the API for these isn't the best and we will probably change it in the next iteration Pull Request resolved: facebookresearch/ClassyVision#248 Test Plan: {F222475868} Differential Revision: D18511058 Pulled By: vreis fbshipit-source-id: b0516ea1fcf9befc1d34cb9f0ae24d91ef4c0ef2
Summary: - Improve layout on sphinx docs - Remove nonsense from index page - List out undocumented members - Add a few of transforms that were undocumented - Add a small blurb to the top of the transforms doc Pull Request resolved: facebookresearch/ClassyVision#250 Differential Revision: D18690886 Pulled By: vreis fbshipit-source-id: 9cae5f020c7aa4ee714fa7fbf5495a1d888266e5
Summary: Change the losses tutorial to be more structured -- following Jessica's feedback. I think this looks much cleaner. Pull Request resolved: facebookresearch/ClassyVision#251 Differential Revision: D18697317 Pulled By: vreis fbshipit-source-id: 84df08436ce9328f5db4a389d399f4f023ebc0b6
Summary: Pull Request resolved: pytorch/elastic#4 Makes state API's persistence (rollback and serialization) more coherent, consistent, and natural. Does the following: * Renames `deep_copy` and `rollback` to `snapshot` and `apply` * The semantics of `snapshot` and `apply` is that the state is recoverable by: ``` any_user_defined_snapshot_obj = state.snapshot() modify_state(state) state.apply(any_user_defined_snapshot_obj) state.sync() ``` * Renames `serialize` and `deserialize` to `save` and `load` (to be consistent with torch) * `State` provides a default implementation of `save` and `load` using `snapshot` and `apply`. * Removes the redundant `supports_rollback()` method from `State`. By not implementing `snapshot/apply` the user indicates that rollback is not supported on the `State` object. If the user wants to checkpoint but not rollback they can implement the `save/load` and not implement `snapshot/apply`. If the user wants rollback support, they lose no performance (in comparison) in doing checkpoints so they might as well get checkpoint for free. * Makes changes to the `test_mock` and `elastic classy_vision` code to be compliant with the new API. * Makes imagenet example compliant with the new API. NOTE: This change renders the imagenet example under `//fblearner/flow/projects/pytorch/elastic/imagenet` broken. However this example was already broken and has zero users. The task to fix this is T57831531. (Note: this ignores all push blocking failures!) Reviewed By: vreis Differential Revision: D18672302 fbshipit-source-id: 88718b790f8f0fe7ae0381a2e40455af7e2ba0ce
…ining (#246) Summary: Pull Request resolved: facebookresearch/ClassyVision#246 Pull Request resolved: fairinternal/ClassyVision#44 Reviewed By: mannatsingh Differential Revision: D18660801 fbshipit-source-id: 748935c03a3ef1a2cc682c6d78852db07c6a60ea
Summary: Pull Request resolved: fairinternal/ClassyVision#45 Pull Request resolved: facebookresearch/ClassyVision#249 Update readme blurb with some verbage from blog post. Reviewed By: vreis Differential Revision: D18675503 fbshipit-source-id: d74211914df1d31574a104365cb651dcad05e089
Summary: A bunch of changes here: - Add tensorboard instructions; - Add distributed training instructions; - More structured steps; Pull Request resolved: facebookresearch/ClassyVision#253 Differential Revision: D18710956 Pulled By: vreis fbshipit-source-id: 29d55e73103d189208f8f28c3bf4641e1b96b9a9
Summary: D18660801 made it so that `load_checkpoint` loads checkpoints to the cpu by default. Changing the code so that we always load checkpoints to cpu for both gluster and modelstore. Added a few tests to test checkpointing works correctly. It turns out that https://pytorch.org/tutorials/beginner/saving_loading_models.html isn't up to date. `torch.nn.Module.load_state_dict()` and `torch.optim.Optimizer.load_state_dict()` now move the tensors to the appropriate device during the load, so we don't need to do anything. Reviewed By: vreis Differential Revision: D18707769 fbshipit-source-id: 718fecefa9425547fd07e3931bcecea2b55242d3
Summary: Add a tutorial showing how to train PET on AWS. Pull Request resolved: facebookresearch/ClassyVision#254 Differential Revision: D18713897 Pulled By: vreis fbshipit-source-id: 387c8d032432d7ffacb576519a408efd0aa6c173
Summary: Add a test using our public API as recommended in the Getting started tutorial. This tutorial was broken by D18604454 but no tests caught it because we mostly use the `from_config` API instead of calling constructors directly. Fix the tutorial as well. Pull Request resolved: facebookresearch/ClassyVision#255 Test Plan: New unit test Differential Revision: D18717267 Pulled By: vreis fbshipit-source-id: 32cc0dc63f0b429967f173f7c44c3698a637dbba
Summary: Based on Kai's notebook. Pull Request resolved: facebookresearch/ClassyVision#257 Differential Revision: D18727667 Pulled By: vreis fbshipit-source-id: 43a7092fbb1f4cc245995b7cd3380b6cf3b8771c
Summary: Change video tutorial based on Jessica's feedback. Pull Request resolved: facebookresearch/ClassyVision#258 Differential Revision: D18729256 Pulled By: vreis fbshipit-source-id: 43643caafcaeb699246482f8a4ed96d6e78e9e4b
Summary: Pull Request resolved: facebookresearch/ClassyVision#252 As titled Reviewed By: mannatsingh, vreis Differential Revision: D18517906 fbshipit-source-id: de0a6a76eb872d60e757ac0ba7d0b2533fcc306b
Summary: Pull Request resolved: facebookresearch/ClassyVision#259 Fixes two issues: - os.makedirs is racy when doing distributed training, which was making our test flaky; - config was not defined within configure_hooks; Reviewed By: mannatsingh Differential Revision: D18736840 fbshipit-source-id: 033639264c8c5aeaa0e71adfddc7ed30d05e29e9
facebook-github-bot
added
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D18715190 |
stephenyan1231
force-pushed
the
export-D18715190
branch
from
774fee5
to
cdb6415
Compare
|
This pull request was exported from Phabricator. Differential Revision: D18715190 |
Summary: Pull Request resolved: facebookresearch/ClassyVision#270 Test Plan: Verified the issue templates on my fork Reviewed By: vreis Differential Revision: D18774172 Pulled By: mannatsingh fbshipit-source-id: bdd28c8c71c101a81fb5eb1ccfc5908a98451dd0
Summary: Pull Request resolved: fairinternal/ClassyVision#50 Tutorial on how to use the Classy Vision dataset API Pull Request resolved: facebookresearch/ClassyVision#268 Test Plan: Published notebook for viewing: n179685 Reviewed By: vreis Differential Revision: D18758543 Pulled By: aadcock fbshipit-source-id: f0524a2d0b79c0d0854538442d8b5783adcfd921
Summary: Pull Request resolved: facebookresearch/ClassyVision#277 Differential Revision: D18807581 Pulled By: vreis fbshipit-source-id: 37e5598ea839f05ef067344ba27accf6e1aad380
Summary: - Added links (internally, externally to pytorch references) to docs - Fixed some formatting issues - Minor rewarding of potentially confusing docs Only for Dataset, Transforms, and Models (through the base class for model) Pull Request resolved: facebookresearch/ClassyVision#276 Test Plan: sandcastleit Reviewed By: aadcock, stephenyan1231 Differential Revision: D18798520 Pulled By: lauragustafson fbshipit-source-id: 09a302b7f1007f18808eacfe5f55d2a6416bd4e9
Summary: Change PET tutorial to incorporate the changes made in D18779469. It's a lot easier to use now. Pull Request resolved: facebookresearch/ClassyVision#278 Differential Revision: D18814121 Pulled By: vreis fbshipit-source-id: 0cd4bc183c0f9b63b523d96f6ab5cabdab11214a
Summary: - Change `! cd` to `% cd` - Make it clear what's running in a terminal; - Add flags to ray commands that require user input Pull Request resolved: facebookresearch/ClassyVision#279 Differential Revision: D18814526 Pulled By: vreis fbshipit-source-id: d29e72a5a75e7bd4b7d296a7a539d76369f81c8a
stephenyan1231
force-pushed
the
export-D18715190
branch
from
b35cb67
to
6f3f907
Compare
|
This pull request was exported from Phabricator. Differential Revision: D18715190 |
stephenyan1231
force-pushed
the
export-D18715190
branch
from
6f3f907
to
15264c3
Compare
|
This pull request was exported from Phabricator. Differential Revision: D18715190 |
Summary: I meant to do 1 hour instead of 6 minutes. Pull Request resolved: facebookresearch/ClassyVision#283 Differential Revision: D18825613 Pulled By: vreis fbshipit-source-id: b96ad624871750d5781abed2430e8a0d09276dc8
Summary: Pull Request resolved: facebookresearch/ClassyVision#282 Differential Revision: D18825492 Pulled By: vreis fbshipit-source-id: 60e51888e5e03fdfc84a702068fc33f9fe385db2
Summary: Add a `cd` command to the tutorial so we create the classy project in the EFS volume. Pull Request resolved: facebookresearch/ClassyVision#284 Differential Revision: D18826240 Pulled By: vreis fbshipit-source-id: 328ff3c29cf51e98f16578e8e18dd1db76748445
Summary: Fix a few more issues in the PET tutorial now that I've ran it myself Pull Request resolved: facebookresearch/ClassyVision#285 Differential Revision: D18827625 Pulled By: vreis fbshipit-source-id: ee8d4b21a45f8d6229ee01698391d2fae2331824
Summary: These configs are not useful for the OSS release Pull Request resolved: facebookresearch/ClassyVision#286 Differential Revision: D18827676 Pulled By: vreis fbshipit-source-id: bcf90edbd2c9857e92c0956930cb0d50492649a2
Summary: Updated docs based on how the look on the site for Models, Heads, Hooks, Losses, Meters, Optimizers, Param Schedulers, Tasks, Trainer. Fixed some internal/external links to code/sources, reworded docs for clarity, formatting changes for sphinx, ensure docs for register/build param_scheduler are picked up by sphinx. Pull Request resolved: facebookresearch/ClassyVision#281 Test Plan: sandcastleit Reviewed By: mannatsingh Differential Revision: D18818978 Pulled By: lauragustafson fbshipit-source-id: 98105d9a9f9751489e6fa2fd3a6c7382987288ed
Summary: Pull Request resolved: facebookresearch/ClassyVision#275 - Updated the OSS Imagenet dataset to remove the `download` arg. Instead, the dataset now uses `ImageFolder` and expects the user to extract the data in a compatible format. - Deleted `dataset_classy_oss_imagenet_test.py` (fb folder) and created `dataset_classy_imagenet_test.py` (oss folder). This test writes test images to a dir and tests that the dataset can load them - Updated the imagenet configs to use `classy_imagenet` instead of `imagenet` and added the transforms - Added a short section in the getting started tutorial with instructions to train on imagenet and a 90 epoch config that reaches 76.2% accuracy Reviewed By: vreis Differential Revision: D18796349 fbshipit-source-id: cb2d3b505f54ad33d21c6ade1c8a8ccf063337bb
Summary: Pull Request resolved: facebookresearch/ClassyVision#292 Added an arg to set the image backend (like the video backend) and set the default to PIL. Note that I have not added this to the flow trainer. Also updated the `__init__` for the datasets to add imagenet, cifar and the image path dataset (Note: this ignores all push blocking failures!) Reviewed By: vreis Differential Revision: D18834469 fbshipit-source-id: 7dd51c771614ce2e202f3c206124e62809a79d2b
stephenyan1231
force-pushed
the
export-D18715190
branch
from
15264c3
to
2d33c03
Compare
|
This pull request was exported from Phabricator. Differential Revision: D18715190 |
stephenyan1231
force-pushed
the
export-D18715190
branch
from
2d33c03
to
f7ed130
Compare
|
This pull request was exported from Phabricator. Differential Revision: D18715190 |
Summary: Pull Request resolved: facebookresearch/ClassyVision#265 Average Precision (AP) summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight. See AP formal definition (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html). Mean Average Precision is an average of AP over different classes. In this diff, we implement Mean Average Precision meter that is useful for multi-label classification task. We implement a simple `SparseBinaryMatrix` to store multi-hot groundtruth label to save memory. It also supports `max_capacity` argument to limit the memory footprint of the meter because model predictions, which is stored as a dense matrix of size N x K ( N is number of samples, and K number of classes), can be quite large specially for training set. For example, SSVP has a training set of size 2.6M, and has 7K+ classes. Reviewed By: aadcock Differential Revision: D18715190 fbshipit-source-id: a1ddce5f049c67c335b4c1bc4216b2aaa6ae87dd
stephenyan1231
force-pushed
the
export-D18715190
branch
from
f7ed130
to
8e0b111
Compare
|
This pull request was exported from Phabricator. Differential Revision: D18715190 |
|
This pull request was exported from Phabricator. Differential Revision: D18715190 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D18715190 |
facebook-github-bot
pushed a commit
that referenced
this pull request
Dec 6, 2019
Summary: Pull Request resolved: #265 Average Precision (AP) summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight. See AP formal definition (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html). Mean Average Precision is an average of AP over different classes. In this diff, we implement Mean Average Precision meter that is useful for multi-label classification task. We implement a simple `SparseBinaryMatrix` to store multi-hot groundtruth label to save memory. It also supports `max_capacity` argument to limit the memory footprint of the meter because model predictions, which is stored as a dense matrix of size N x K ( N is number of samples, and K number of classes), can be quite large specially for training set. For example, SSVP has a training set of size 2.6M, and has 7K+ classes. Reviewed By: aadcock Differential Revision: D18715190 fbshipit-source-id: f352472445c1a9ab96b6fac1f00a5f22ab735957
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Average Precision (AP) summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight.
See AP formal definition (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html).
Mean Average Precision is an average of AP over different classes.
In this diff, we implement Mean Average Precision meter that is useful for multi-label classification task.
Differential Revision: D18715190