fbshipit-source-id: b4001a6fb460b8b35bc862a5564fad55d86f5bc3

Summary:
Pull Request resolved: fairinternal/ClassyVision#1

Pull Request resolved: facebookresearch/ClassyVision#1

We got approved for MIT license.

Reviewed By: mannatsingh

Differential Revision: D17453010

fbshipit-source-id: d6b29db5c142b9a905e821864b8fc15597f134a6

Summary:
Add template files usually available in our OSS releases. Will work on
updating the content from them as we go.

Reviewed By: mannatsingh

Differential Revision: D17454683

fbshipit-source-id: c8ea8f9b3a7f0746d73c1f2062114623d0fc2c88

Summary:
Created a `multi_output_sum_loss`. It takes a registered loss's config as input and applies it to all the outputs. See the associated test for usage.

This loss is supposed to replace the "sum" functionality of `sum_bce_with_logits_loss` and `sum_cross_entropy_loss` so that we don't have code duplication. The updates to these two losses will happen in a follow up diff.

Reviewed By: aadcock, vreis

Differential Revision: D17438898

fbshipit-source-id: 20434cfc340a29a5058b2fdd51f067aa77cb82c9

Summary:
`build_criterion` now searches for a criterion in `torch.nn.modules.loss` if it is not found in the registry. This is similar to the logic used for transforms.
- We don't need `sum_bce_with_logits`, `sum_cross_entropy` anymore - users can use `BCEWithLogitsLoss` and `CrossEntropyLoss` in combination with `multi_output_sum_loss` (D17438898) to get the same functionality
- There was a bug wherein passing `weights` wouldn't work to the losses since `weights` is expected to be a tensor, not a list. `weights` is now updated to a tensor inside `build_criterion`
- `criterions_test.py` is no longer a misnomer - it tests that `build_criterion` can build a criterion from torch correctly
- Changed `sum_cross_entropy` to `CrossEntropyLoss` in all the configs I found using fbgs

Reviewed By: aadcock, vreis

Differential Revision: D17439082

fbshipit-source-id: b075fdbdbbc879a04fea397628385fdb0a3af5a1

Summary:
Use `DataloaderLimitWrapper` in the koski on box datasets as well (`on_box_uru` and `multimodal_video`)
- Instead of having Koski apply a limit, the wrapper does that for us. This will make sure we wrap around if one of the instances runs out of samples before others and the code will not stay hung
- `ClassyState.num_batches_per_phase` doesn't have a special branch for these datasets anymore
- The datasets use `num_samples` instead of `max_samples` like the other streaming datasets
- Updated `DataloaderLimitWrapper` to pass all `getattr` / `setattr` calls to the wrapped data loader
- Got rid of all the unused special methods in the datasets to simplify the code

Reviewed By: aadcock

Differential Revision: D17372111

fbshipit-source-id: d3e3a634eb20eb82db3dc587fab576c5cc596d30

Summary:
Add basic support for packaging the project. This will be used to run ours tests with pip as well.
Pull Request resolved: facebookresearch/ClassyVision#2

Test Plan: python3 setup.py sdist bdist_wheel

Differential Revision: D17490325

Pulled By: vreis

fbshipit-source-id: a3fe2a4cfe8b943bfc0922b1ad1eb2daa5802584

Summary:
Add a configuration file for CircleCI. Not all tests are passing yet, but I'll work on fixing that in follow-up diffs.
Pull Request resolved: facebookresearch/ClassyVision#3

Differential Revision: D17494024

Pulled By: vreis

fbshipit-source-id: 53951f3e853c0c4ca48d3301ec9064142674f9f1

Summary:
Pull Request resolved: facebookresearch/ClassyVision#4

Needed for the OSS release

Reviewed By: mannatsingh

Differential Revision: D17504533

fbshipit-source-id: ea2d9c9579623bc8776f89493aaaad2542351802

Summary:
Pull Request resolved: facebookresearch/ClassyVision#6

I missed the test directory and files in the root directory :(

Reviewed By: mannatsingh

Differential Revision: D17507489

fbshipit-source-id: 0564ffa81df35443f89422fe1813d68c8682652a

Summary:
In D17349805 we had introduced `DataloaderLimitWrapper`, but I forgot to scale the limit by the world size.

- Added a comment about `getattr` in the `Dataset` class (it's easy to miss that there's a `__getattr__` defined which passes on the search to `self.dataset`)
- Added `get_batchsize_per_replica()` and `get_global_batchsize()` to  `ClassyDataset` and updated `ClassyState.get_batchsize_per_replica()` to use the dataset function
- The `multimodal_video` dataset used inconsistent names for the batch size in the config and in the class which has been fixed

Reviewed By: vreis

Differential Revision: D17481832

fbshipit-source-id: 5642799d3b708cc0aa58d9b0279cb49590b01e88

Summary: add an utility function `is_pos_int_list`

Reviewed By: mannatsingh

Differential Revision: D17496207

fbshipit-source-id: 8c3e22e290fc85728208bca55dd0b1cbe0aa29c1

Reviewed By: vreis

Differential Revision: D17472655

fbshipit-source-id: 087f2a3a7e66efa76723109eb41097c13ea8ef1b

Summary:
Pull Request resolved: facebookresearch/ClassyVision#5

Three fixes:
1) Some tests require external dependencies (progressbar, visdom, tensorboard).
For now let's just test them on the FB environment.

2) Some tests had a wrong relative imports (.generic);

3) One test was checking the exact package name of the test class. That's
pretty fragile and I don't see the need for it.

Reviewed By: mannatsingh

Differential Revision: D17506164

fbshipit-source-id: 4e35a4c3860321d778434049cc03c28fe30e0c2d

Summary:
Pull Request resolved: facebookresearch/ClassyVision#7

While moving everything to the classy_vision module I missed this file. It
doesn't make sense to have it in the root directory, move it where it belongs.

Reviewed By: mannatsingh

Differential Revision: D17515540

fbshipit-source-id: b33ae5a04d199cbddb64838dd071ae383090d8bc

Summary:
Pull Request resolved: facebookresearch/ClassyVision#8

This will make more sense after you review D17515697: I'm hiding ClassyState
from the public API, but in some cases we access state.optimizer, state.model
etc. This diff adds these members to ClassyTask so it can replace the state.
Eventually I want to move these from the state to the task, but for now we
need them in both.

Reviewed By: mannatsingh

Differential Revision: D17515732

fbshipit-source-id: e00820cf608dfc86a658e293e8446b6ae10b36a0

Summary:
Pull Request resolved: facebookresearch/ClassyVision#9

Move things around a bit more to get closer to our goal. Introduce a
ClassyTrainer.train method, which only takes a task in. The state is hidden
from the public API that way. Other parameters are passed in the constructor
instead.

Reviewed By: mannatsingh

Differential Revision: D17515697

fbshipit-source-id: 1d4163f1c49a0a29c5bbf8fc01eda88e810dbed3

Summary:
Pull Request resolved: facebookresearch/ClassyVision#10

The name "optimizer_config" is confusing since it does not come from the
config used to build the optimizer. I think parameters makes that purpose
more clear.

Reviewed By: mannatsingh

Differential Revision: D17517990

fbshipit-source-id: f2906668a9ea571bc00497b5f898d14755066bd8

Summary:
Pull Request resolved: facebookresearch/ClassyVision#11

We want to be able to instantiate classy abstractions without a configuration
file. Instead of taking a config in the constructor for the param schedulers,
make a static method to instantiate a scheduler from a config.

Reviewed By: mannatsingh

Differential Revision: D17518149

fbshipit-source-id: 84f8f0d4cf4fe0249c14895debf4d93d9b35b5ec

Summary:
Pull Request resolved: facebookresearch/ClassyVision#12

Create a .from_config method to instantiate a criterion from a configuration
dictionary. That means we can have actual arguments in the constructors,
instead of passing a single config blob.

Reviewed By: mannatsingh

Differential Revision: D17528154

fbshipit-source-id: dd94002069d56c9933e3fb361a89b6ebcb6b9967

Summary:
Pull Request resolved: facebookresearch/ClassyVision#13

Two changes:
1) Create a .from_config method to instantiate a meter from a configuration
dictionary.
2) Change the build_meter API to be consistent with the other build_* APIs.
That means taking a single config argument (instead of name, config);

Reviewed By: mannatsingh

Differential Revision: D17531961

fbshipit-source-id: 3e511615343061d7a06c1acce980c401d49c9553

Summary: we need to transfer the tensors to GPU and back when using NCCL backend.

Reviewed By: mannatsingh

Differential Revision: D17473796

fbshipit-source-id: d066aa79e32cb2c91927121e3e8049c53b15c4fb

Summary:
Pull Request resolved: facebookresearch/ClassyVision#15

That is not being used AFAICT and it's polluting the API.

Reviewed By: mannatsingh

Differential Revision: D17547887

fbshipit-source-id: c5ebde37932429801f7034635dd73ee85c16a292

Summary:
Pull Request resolved: facebookresearch/ClassyVision#17

The compute approximate PR curve utility in classy vision was created for a mAP meter which pre-existed classy vision. Zheng wants to reimplement the mAP meter but he found that our PR curve was different from other definitions in SKLearn in that it computed the convex hull of the PR curve (which is fine if you are choosing a threshold but not the standard for mAP).

This diff converts it to use the standard PR curve (i.e. no convex hull).

Reviewed By: vreis, zhengshou

Differential Revision: D17584834

fbshipit-source-id: ca99ad257e82c5a0fc490d7cdec2d7ff0f5af1cd

Summary:
Pull Request resolved: facebookresearch/ClassyVision#18

I was originally against a ClassyTransform abstraction, but I've come around.
All other Classy things implement a from_config method, so we need a
ClassyTransform as well. That gives us the flexibility of implementing
transforms where the dictionary can't be mapped directly to arguments in the
constructor.

Reviewed By: mannatsingh

Differential Revision: D17578366

fbshipit-source-id: 7cd4addc7cba79054364939947bd34f3325efc02

Summary:
Pull Request resolved: facebookresearch/ClassyVision#21

In preparation to switch datasets to a .from_config constructor, make sure no
datasets store the config itself as a member of the class. This makes things
easier later.

Reviewed By: aadcock

Differential Revision: D17578858

fbshipit-source-id: 33137eee8b90e47b0bcd2a0d060ae24e927087cb

Summary:
Pull Request resolved: facebookresearch/ClassyVision#22

`DenseNet.input_shape` calls `self.small_input`, which isn't defined currently.

Reviewed By: vreis

Differential Revision: D17628854

fbshipit-source-id: fff8b4e138ea68b2dc50ac60572dadc60b935ce2

Summary:
Pull Request resolved: facebookresearch/ClassyVision#20

Implemented fine-tuning for loading state from checkpoint state and tests for it

Reviewed By: mannatsingh

Differential Revision: D17481209

fbshipit-source-id: 288e597f42453afd7a50ebf5e4d0508402b8ee69

Summary:
Pull Request resolved: facebookresearch/ClassyVision#23

Pull Request resolved: facebookresearch/ClassyVision#16

Pull Request resolved: fairinternal/ClassyVision#2

As I tried to work on the downstream diffs some of the internal datasets which have sample fetch failures had a different API and this was causing me headaches. In particular, most of the dataset API is reimplemented for the everstore dataset (particularly the batching and transformation logic). This made the downstream work to refactor and ultimately delete this logic twice as complicated since I effectively had to do it twice (one for canonical datasets and once for the everstore datasets).

This diff migrates the API for those datasets to fetch single samples and use the dataset wrappers like all of the other Classy datasets (so future migrations happen only once). This has performance implications since the original goal of separate batching logic was to issue multiple everstore requests simultaneously, so I also had to make a change to prefetch the upcoming samples to make sure performance was still good.

This involved:

1. Everstore batching / transformation logic dealt with missing samples (None's), so I had to move some of that logic to the "is_not_none" filter function and add a ignore none's option to the transforms.
2. Modifying all downstream datasets (e.g. the visual relevance team's datasets, also made max_samples -> num_samples to match all other datasets)
3. Adding prefetching to the Everstore dataset for performance.
4. Found some small CLI changes to the internal benchmark to make it easier to use by displaying samples when needed.

I then verified that performance was fine. The performance turned out to be faster than I measured on the previous benchmark, I'm still not sure why this is, but I added functionality to the benchmark to print out a tensor for visual inspection and the data appears to be fine.

As a note, I was trying to figure out if we can switch to spawn here and ran into a host of problems which was what was blocking this work, so I will save that for a later diff.

Reviewed By: vreis

Differential Revision: D17266435

fbshipit-source-id: 337e8dcd4558d58fc28c5c47f99ff681addccc8b

Summary:
Pull Request resolved: facebookresearch/ClassyVision#25

Remove criterion_config from ClassyTask constructor. Instead, use a set_criterion method
and instantiate the criterion from the config. This looks weird now cause I haven't migrated
the other modules, but eventually ClassyTask will take no arguments in the constructor and
this will look like n128788. It needs to get worse before it gets better though.

Reviewed By: mannatsingh

Differential Revision: D17581037

fbshipit-source-id: 87c6ae9b07f3cf03b71180e0fcd56b660036948d

Summary:
Pull Request resolved: facebookresearch/ClassyVision#27

Fixed classy vision head imports

Reviewed By: vreis

Differential Revision: D17635517

fbshipit-source-id: 0dd0eb73bb3d03cd442de35ba893d78a7a9180d5

Summary:
Pull Request resolved: facebookresearch/ClassyVision#24

Import all the criterions in `__init__.py` and add them to `__all__`

Removed `import_all_modules` since it is not needed anyway and can cause people to forget importing future modules.

Reviewed By: vreis

Differential Revision: D17632979

fbshipit-source-id: 7a6c9dcb0d53d86f8f339b547e7859f88be60515

Summary:
Pull Request resolved: facebookresearch/ClassyVision#26

Import `LightingTransform` in `__init__.py`

Reviewed By: vreis

Differential Revision: D17634829

fbshipit-source-id: ce5999255c850789260c03c3f51049c9a0d19ed2

Summary:
Pull Request resolved: facebookresearch/ClassyVision#28

pin_memory is a setting that only makes sense when training on GPUs. So move
that to build_initial_state which happens to get called whenever a Trainer is
instantiated (the Trainer keeps track of whether to use gpu).

Reviewed By: mannatsingh

Differential Revision: D17658720

fbshipit-source-id: 2d2cab315532253bf36a38f8d0e03a5a46e0aef7

Summary:
Pull Request resolved: facebookresearch/ClassyVision#29

ClassyTrainer decides whether to train on the GPU or not. Remove device_type from ClassyTask
now that it exists in the trainer.

Reviewed By: mannatsingh

Differential Revision: D17636861

fbshipit-source-id: 78b25e940ed5a2cfbb9c43902b7c0b98f7fba2c4

Summary:
Pull Request resolved: facebookresearch/ClassyVision#30

Renamed `LossLrTensorboardHook` to `TensorboardPlotHook` and plot the meters to tensorboard as well (apart from the loss and lr).

Reviewed By: vreis

Differential Revision: D17646340

fbshipit-source-id: 32c8915d7bf4ae12ee39bd61be5246a04159a604

Summary:
Pull Request resolved: facebookresearch/ClassyVision#32

test_only is a feature rarely used. Instead of taking it in the constructor, set a default and create a
setter method.

Reviewed By: mannatsingh

Differential Revision: D17659502

fbshipit-source-id: 5a51c7c3c6490201a1e075897bb7913145808af8

Summary:
Pull Request resolved: facebookresearch/ClassyVision#31

Create a DistributedTrainer class to take care of the initial setup of
environment variables for distributed training. This greatly simplifies the
code in classy_train.py, and moves us a bit closer to the vision in n128788

Reviewed By: mannatsingh

Differential Revision: D17661068

fbshipit-source-id: 5cdf966dc3f8d78db99c141441a988ce946e1f5d

Summary:
Pull Request resolved: facebookresearch/ClassyVision#34

Datasets are not for the faint of heart. Unlike the other
abstractions, I cannot do the migration to .from_config with a single diff.
Instead, I'm adding a .from_config method that passes the config to the
constructor. That way each class can be migrated on its own.

Reviewed By: mannatsingh

Differential Revision: D17689190

fbshipit-source-id: 082d3dfa97954a8ae7548a84a06edc9d7c46c741

Summary:
Pull Request resolved: facebookresearch/ClassyVision#33

Instead of taking the meters config in the constructor, store the meter
instances themselves in ClassyTask. Meters can be added with an `add_meter`
method.

Reviewed By: mannatsingh

Differential Revision: D17659504

fbshipit-source-id: 08d6113802a1e4a21b84e21a3aed1732277badc3

Summary: Pull Request resolved: facebookresearch/ClassyVision#36

Reviewed By: kazhang

Differential Revision: D17726216

fbshipit-source-id: 374b839b333a45f6c9f59e39e04b29975bb6074a

Summary:
Pull Request resolved: facebookresearch/ClassyVision#39

Changed the ModelTensorboardHook API to be the same as TensorboardPlotHook (they take summary writers as inputs). Plugged ModelTensorboardHook into the fblearner workflows. Since I was making the change anyway, added TensorboardPlotHook to all the local runs which I changed as well. Also, pyfakefs isn't a dependency anymore.

Reviewed By: vreis

Differential Revision: D17743822

fbshipit-source-id: 56966cb0cca6d2cd291040f44084296f83d1c63a

Summary:
Pull Request resolved: facebookresearch/ClassyVision#37

Implemented an `ImageDataset` in `dataset/generic` and added an `__init__.py` file. The dataset creates a classy dataset using images from local paths. Look at the docstring for more information. This will be used by the torchhub interface in the following diff.

Ideally, there should have been a class/function which returned a class object, and the returned class would expect the config as an input. But that would mean the class wouldn't be serializable. This is a bit of a compromise - the class takes all the args along with the config in `__init__`. The non-config args are named so that we can apply partial to them in the future - the returned object would just then take the config as an argument.

Please let me know if you have any suggestions to improve the interface!

Reviewed By: vreis

Differential Revision: D17728549

fbshipit-source-id: 47828d8321c34f7ef1aeeee67ac255e10b021217

Summary:
Pull Request resolved: facebookresearch/ClassyVision#38

Implemented a `ClassyHubInterface` class. To see its usage look at the associated test. This is a first version and will need further improvements. I wanted to get feedback on the API to make sure it makes sense.

Added an `extract_features` method to `ClassyVisionModel`, which defaults to `forward`.

Note that transforms aren't picked up from self.task's datasets, but from the task's config. This means that a dataset's default transform will not be picked up. Added a TODO for this.

Reviewed By: vreis

Differential Revision: D17729140

fbshipit-source-id: 1dd8dfcfbb95cd57873f460b42a01d4cf8adeec0

Summary:
Pull Request resolved: facebookresearch/ClassyVision#42

Created a `ClassyModelWrapper` which wraps an `nn.Module` by a `ClassyVisionModel`, so that the model can be used in Classy Vision.

This wrapper is used in `ClassyHubInterface` so that `from_model()` can support any general models.

Reviewed By: vreis

Differential Revision: D17767522

fbshipit-source-id: c80d7a0ee5523b313491a8c3e158ce49c348c5f3

Summary: Pull Request resolved: facebookresearch/ClassyVision#35

Reviewed By: vreis

Differential Revision: D17708861

fbshipit-source-id: 8c9579de7af35efb63c2be7eca6e37b57ce90e48

Summary:
Pull Request resolved: facebookresearch/ClassyVision#41

1. Moved FC head from resnext.py to classy_vision/heads
2. Changed corresponding imports
3. Changed head name to "fully_connected" and corresponding json files

Reviewed By: mannatsingh, vreis

Differential Revision: D17730080

fbshipit-source-id: d346fdf59e3fc0aa2338efcb12c2c0c35337cd9e

Summary:
Added support to store state within the hooks. Using the functions names `load_state_dict` and `state_dict` - we will be updating all the state functions to have these names.

This will be followed by a diff to migrate the hooks to `ClassyState`, so that hooks can resume from checkpoints. This will also allow the hooks to work seamlessly with PET.

Reviewed By: aadcock

Differential Revision: D17772753

fbshipit-source-id: 01967e7028a8d0558f285de8d42943699a31dd4e

Summary:
Pull Request resolved: facebookresearch/ClassyVision#44

Moved hooks to `ClassyState` - this means that the hooks' state can be checkpointed and synced (PET). Also moved the `run_hooks` function inside the state. The repetition of the `set_hooks` function in the task and the state is only there temporarily until the task absorbs the state.

Reviewed By: aadcock

Differential Revision: D17795516

fbshipit-source-id: 2cef4a0a8fed805a21370d0c5a9fb7db24565972

Summary:
Pull Request resolved: facebookresearch/ClassyVision#45

Pull Request resolved: facebookresearch/ClassyVision#40

Instead of forcing heads to take a config in the constructor, change ClassyHead
to use the .from_config idiom.

Reviewed By: aadcock

Differential Revision: D17723558

fbshipit-source-id: 2a3a3670a55355fde6c496eb9d149f81e3bd1544

Summary:
Pull Request resolved: facebookresearch/ClassyVision#46

Added argument for log freq and set default to 5. It is very confusing that there is no output by default.

Reviewed By: aadcock

Differential Revision: D17812560

fbshipit-source-id: 93f057148140a0aae79869d94c5e05d8f569d431

Summary:
Pull Request resolved: facebookresearch/ClassyVision#49

This removes the Classy Dataloaders from the OSS code.

THe classy dataloaders are still used by the everstore datasets (planned to be deprecated), so I moved the associated files to the fb/datasets folder and verified the tests. I will delete them in a later diff.

All unittests pass and the run I kicked off yesterday trained well.

Reviewed By: mannatsingh

Differential Revision: D17842785

fbshipit-source-id: 82661d4a51965f6d268b6ba54188e74b0f317b7a

Summary:
Pull Request resolved: facebookresearch/ClassyVision#50

D17481209 updated `update_classy_state` to support fine tuning, but introduced a bug wherein we only loaded the model state from the checkpoint instead of the whole state. It wasn't caught because the test which was added only compared the model state and not the whole state.

Reviewed By: aadcock

Differential Revision: D17855725

fbshipit-source-id: a85e85a8be0000fa6e33019c03b2229b101e6568

Summary:
Pull Request resolved: facebookresearch/ClassyVision#48

Instead of taking the model and optimizer config in the constructor, store the model and optimizer
instances themselves in ClassyTask. These can be set using the `set_model` and `set_optimizer` methods.

Also, moved `rest_heads` to the task config instead of the `model_config`.

Updated the name of the config variable from `model_config` to `config` in `build_model` since it is obvious from the context.

Reviewed By: aadcock

Differential Revision: D17770367

fbshipit-source-id: af10ef6507cd5ef6df32fd0aa5bd95bb9f309747

Summary:
Pull Request resolved: facebookresearch/ClassyVision#52

Need this so that phases aren't built before `set_test_only()` is called.

Reviewed By: aadcock

Differential Revision: D17880326

fbshipit-source-id: e3d8b49e9c79dcd111337f54c10d8b1847032bc6

Summary:
Pull Request resolved: facebookresearch/ClassyVision#51

Currently, for all datasets, the initialization happens through the config. This diff changes it so that all the datasets take kwargs instead. `ClassyDataset` now expects the following args to be provided during initialization - `split, batchsize_per_replica, shuffle, transform, num_samples`. Converted `parse_config` and `wrap_dataset` to `classmethod`s (In a strict sense, they shoud be `staticmethod`s, but that conversion will do more harm than good).

There are a lot of other things which need to be cleaned up in datasets, but this is a start!

Reviewed By: aadcock

Differential Revision: D17869562

fbshipit-source-id: d69219795df7d12169c6ea22e2b05e68941a609d

Summary:
Pull Request resolved: facebookresearch/ClassyVision#54

For video model evaluation, we sample N clips from a video, and average clip predictions to get a video-level prediction.
Assume, we sample 2 clips per video. The test dataset, which has 4 videos {A,B,C,D} is illustrated below.

    [A_0, A_1, B_0, B_1, C_0, C_1, D_0, D_1]

Assume we have 2 gpus. The existing ShardDataset will distribute clips from the same video to different gpus, and make it difficult to average clip predictions.
    GPU 0: [A_0, B_0, C_0, D_0]
    GPU 1: [A_1, B_1, C_1, D_1]

This is called Stride Shading.

We extend ShardDataset to support Sequential Sharding, which will shard clips below.
    GPU 0: [A_0, A_1, B_0, B_1]
    GPU 1: [C_0, C_1, D_0, D_1]

This facilitates the averaging of clip predictions.

Reviewed By: mannatsingh

Differential Revision: D17662575

fbshipit-source-id: 1639e9b98a6147a0a19399d6de764f1566852a08

Summary:
Pull Request resolved: facebookresearch/ClassyVision#47

Instead of taking the config in the models' constructor, added a `from_config` classmethod. All the OSS models don't take the config as an arg in their `__init__`s.

Reviewed By: aadcock

Differential Revision: D17724370

fbshipit-source-id: fc1317a4fd78509953f15e7800ed2cc1835ca854

Summary:
Instead of taking the dataset config in the constructor, store the dataset instances in ClassyTask. A dataset for a split can be set using the `set_dataset` method.

Since tasks can be built without config files, datasets can be provided using `task.set_dataset()`. `ClassyElasticState` used to call `task.rebuild_datasets()` to re-shard the datasets properly. I had to disable this code since we won't have access to the config to re-build the datasets. This is only needed when CV starts using elasticity with PET. Created a task for it - T55691442.

Reviewed By: vreis

Differential Revision: D17880629

fbshipit-source-id: b02213139414b6ccca98f93e1063278c7b89ac78

Summary:
Pull Request resolved: facebookresearch/ClassyVision#55

Build `ResNeXt3DStem` as a thin wrapper on top of `VideoModelStem` class from SF codebase.

Reviewed By: takatosp1

Differential Revision: D17542967

fbshipit-source-id: d5c612c3f62c3c9edaca49074a100768538ab449

Summary:
Pull Request resolved: facebookresearch/ClassyVision#56

Implement `BasicTransformation` and `BottleneckTransformation` for ResNeXt3D family models by sharing implementation from SF codebase.

Reviewed By: takatosp1

Differential Revision: D17543143

fbshipit-source-id: 82c9855bd3970712e6fe9333c818aef938d8855d

Summary:
Pull Request resolved: facebookresearch/ClassyVision#58

Implement ResNeXt3D model on top of SlowFast codebase.

Reviewed By: takatosp1

Differential Revision: D17311655

fbshipit-source-id: ee8229be0bb6eba8445c63eba6f4bed1412634ba

Summary:
Pull Request resolved: facebookresearch/ClassyVision#59

Warmup is also useful for cosine learning rate scheduling. We extend existing `CosineDecayParamScheduler` to support it.

Reviewed By: vreis

Differential Revision: D17899790

fbshipit-source-id: 82aa8687984a8159b716151ccb5b5b737b23b606

…rward method (#60)

Summary:
Pull Request resolved: facebookresearch/ClassyVision#60

In `forward` method of video models, we take input of data type dictionary where key is modality such as 'video' and `audio`.
To support this, we extend profiler and `plot_model` to prepare dictionary input as needed.

Reviewed By: mannatsingh

Differential Revision: D17770657

fbshipit-source-id: 723f164f5359a60b9208689d7a8545b8f9909287

Summary:
Pull Request resolved: facebookresearch/ClassyVision#57

Now that a dataset's `transform` is part of the `ClassyDataset` API, we can pick up the correct transform in `ClassyHubInterface().create_image_dataset()`.

Reviewed By: vreis

Differential Revision: D17912094

fbshipit-source-id: c8f13d6b957198eba816bbfdeed08d5d6fdbfae5

Summary:
Pull Request resolved: facebookresearch/ClassyVision#62

Current dependency torchvision 0.4.0 was released in August.
It missed quite a few PRs that are merged after that, and that are needed for video classification, such as

- pytorch/vision#1437
- pytorch/vision#1431
- pytorch/vision#1423
- pytorch/vision#1418
- pytorch/vision#1408
- pytorch/vision#1376
- pytorch/vision#1363
- pytorch/vision#1353
- pytorch/vision#1303

This will fail the CI test when a diff uses changes made in those PRs.
Before a new official version of TorchVision is released, we can temporarily use the nightly torchvision to get all the recent PRs, and unblock the PR merging.
We plan to use a fixed version of TorchVision later.

Reviewed By: vreis

Differential Revision: D17944239

fbshipit-source-id: 86ff540e3fc4f08ef767e84ef103525db5158201

Summary:
Changes
- `CsvFbDataset` represents image classification dataset stored in csv file.
- `CsvFbDataset` is broken by previous diffs. Fix it in this diff
-  Fix a bug In `EverstoreDataset`. It always tried to fetch the 1st sample. This is not correct when meaningful world_size and rank is provided.

Reviewed By: mannatsingh

Differential Revision: D17772040

fbshipit-source-id: 48762da9e6539c1ded1aa9c8ed96bfc5f6830e37

Summary:
Pull Request resolved: facebookresearch/ClassyVision#64

In D17944239, we try to get a more recent version of Torchvision by pip installing from nightly page.
After a close look, we find for linux platform, we can only get version `0.4.1`.
We can achieve this by noting `torchvision==0.4.1` in the `requirements.txt` file.
Thus, we don't need to use nightly page.

Reviewed By: simran2905

Differential Revision: D17949130

fbshipit-source-id: b6092bff39ab6d903d4b69583037b7463bed094b

Summary:
Pull Request resolved: facebookresearch/ClassyVision#65

The latest torchvision version `0.5.0` is recently uploaded to nightly page, and supports all linux, osx and win platforms.

Upgrade ClassyVision to use nightly version and get access to many new features that are needed for video classification.

Reviewed By: vreis

Differential Revision: D17964239

fbshipit-source-id: 26a84ea0b3f930a4d8ccd49d56359bc5124d5b0c

Summary: Correct Fwdproxy definition.

Reviewed By: dddmello

Differential Revision: D17917027

fbshipit-source-id: 1026007f28851b3ec316f3230dce8c2d84d7b8cb

Summary:
Pull Request resolved: facebookresearch/ClassyVision#66

TorchVision on CI hosts remains version `0.4.1` even we choose to use nightly version `0.5.0` in D17964239.
The reason is the hash key of cache remains the same, and the cached TV installation is used.
Change to hash key, and force to update cache to use TV `0.5.0`.

Reviewed By: vreis

Differential Revision: D17970613

fbshipit-source-id: fb58bc0af85228ffc20f280238239c8eef9a1418

Summary:
Pull Request resolved: facebookresearch/ClassyVision#53

- We add video transforms in TorchVision: pytorch/vision#1306
- In ClassyVision, we add default transforms for training / test stage. Alternatively, user can also explicitly provide transform config in json config input. See an example in the unit test.
-  Video data transforms supports audio modality in the video dataset.
-  Compared with image transforms which only returns a torch.Tensor, video transforms return a dict where key is the modality name (e.g. {"video", "audio"}) and value is a torch.Tensor for the modality data.

Reviewed By: taylorgordon20

Differential Revision: D16999453

fbshipit-source-id: 112b66a3965cba4201bbb12c99f3fdd2f1fce86f

Summary: `convert_to_one_hot()` takes a target tensor containing just the target index and returns a one hot tensor. It used to check if cuda was available to determine the device of the returned tensor. Instead, the returned tensor now uses the same device as the input tensor.

Reviewed By: vreis

Differential Revision: D17973726

fbshipit-source-id: 23a63e8ae5f4c5f6c18a93eb8c67aa9a7623ef3e

Summary:
Pull Request resolved: facebookresearch/ClassyVision#67

There has never been a clear distiction between ClassyState and ClassyTask.
Merge the two abstractions to simplify things. Please note we still keep the
ClassyElasticState, since that's an abstraction required by PET.

Reviewed By: mannatsingh

Differential Revision: D17922207

fbshipit-source-id: b04378abb9759c5ebe79b25eae625c8093053a9b

Summary:
Pull Request resolved: facebookresearch/ClassyVision#68

Implement a new base class `ClassyVideoDataset`, which abstracts out common API for video datasets.

Reviewed By: taylorgordon20

Differential Revision: D17414112

fbshipit-source-id: 13e2508293ab142353e9d152acaa1951504133cd

Summary:
TorchVision supports 2 types of video backend
- pyav
- video_reader

`video_reader` is generally faster for video decoding but has dependency on ffmpeg.  However, it has ffmpeg dependency, and is hard to be automatically resolved in TorchVision release with prebuilt package.
Therefore, by default, for OSS ClassyVision, we choose to use `pyav` backend.

Reviewed By: taylorgordon20, kazhang

Differential Revision: D17587685

fbshipit-source-id: 29835e53dd17f741cb37fdaf48c44f91f6f11847

Summary:
Pull Request resolved: facebookresearch/ClassyVision#61

Added a method - `init_pytorch_optimizer()` to `ClassyOptimizer`. Instead of initializing the pytorch optimizers in `__init__()`, they are initialized here. This is needed because we create the optimizer before moving the model to the gpu.

 `ClassyTask.build_initial_state()` now takes `use_gpu` as an arg, and copies the model and criterion to the gpu (if needed), and then calls `init_pytorch_optimizer()`.

Reviewed By: vreis

Differential Revision: D17936851

fbshipit-source-id: 6306ca8b022fe9369af7a6a83e0d08dbe49c4d94

Summary:
Pull Request resolved: facebookresearch/ClassyVision#70

Since the pytorch optimizer isn't initialized in `__init__()`, `build_optimizer()` doesn't need to take the model as an argument. Instead, the model is passed in `build_initial_state()`.
Tagged on a couple of small changes as well -
Added `from_config` to the base `ClassyOptimizer` class. Also, initialize `self._optimizer` to `None` instead of checking for `hasattr()` since that is more natural.

Reviewed By: vreis

Differential Revision: D17964466

fbshipit-source-id: fd0f2754db6ceb2b16e6ffd3d8bed1ca923feeb4

Summary:
Pull Request resolved: facebookresearch/ClassyVision#72

Added a simple `hubconf.py` file which creates an interface for the WSL models by downloading them from torchhub. Created an example notebook where we evaluate an image using one of these models. Note that the example cannot call torchhub right now since the repo isn't public right now, so I had to call the function manually, this will have to be fixed.

Reviewed By: vreis

Differential Revision: D17990569

fbshipit-source-id: 8326200ddffd77f6d7205ace84e07d8a26dfa456

Summary: Pull Request resolved: facebookresearch/ClassyVision#78

Reviewed By: vreis

Differential Revision: D18018163

fbshipit-source-id: 8542cd8d1162c2f5e7f9691b0a3ae3e50a4720fd

Summary: Pull Request resolved: facebookresearch/ClassyVision#79

Reviewed By: mannatsingh

Differential Revision: D18019618

fbshipit-source-id: 9bb802e8ba3002b0ab4ccd1a4c944dab3d6b9466

Summary:
Pull Request resolved: facebookresearch/ClassyVision#76

Since we are creating a separate task for fine tuning, removing `reset_heads` as a parameter from `ClassyVisionTask`

This is my plans for the diffs in this stack -
- Remove reset_heads from base task
- Refactor tests
- Create a fine tuning task
- Add support for this task in fblearner and actually run fine tuning in the test plan

Reviewed By: vreis

Differential Revision: D18007666

fbshipit-source-id: c88d0f0dfc35d46886fec6b629fd96bcf8bd96f2

Summary:
The Manifold checkpoint hook didn't checkpoint the input args / params. This diff makes it so that both the checkpoint hooks save the same data.

Also, updated one of the integration tests to use the manifold hook so that we have better coverage.

Reviewed By: vreis

Differential Revision: D18013407

fbshipit-source-id: 3d5533be4bfec6cbdf5f142bc99ea30d67954af1

Summary: Pull Request resolved: facebookresearch/ClassyVision#80

Reviewed By: vreis

Differential Revision: D18020540

fbshipit-source-id: 75a086597365ce00d01ccaabc394677d3bc30d3b

Summary:
Pull Request resolved: facebookresearch/ClassyVision#75

Moved a test in `state_classy_state_test.py` which didn't actually test the state (we need to rename the test since state doesn't exist anymore, but that's not what I'm focusing on here). Changed the test to use the public trainer API instead of referring to internals. Also, factored out a common configuration and test so that it they be reused by multiple tests.

Reviewed By: vreis

Differential Revision: D18007698

fbshipit-source-id: 91eeb08d6e59882254ecdf52a106d94966fd3704

Summary:
Pull Request resolved: facebookresearch/ClassyVision#81

Pull Request resolved: fairinternal/ClassyVision#3

To be consistent with PyTorch, this diff renames `ClassyCriterion` into `ClassyLoss` and adapts the configuration file accordingly.

Reviewed By: mannatsingh

Differential Revision: D18029164

fbshipit-source-id: 42b93ffa58a12741f0d8685edc9bccc628036e0b

Summary:
Pull Request resolved: facebookresearch/ClassyVision#82

Compared to `nn.Module`, the `nn.loss._WeightedLoss` only registers a `weight` buffer. As far as I can tell, none of the current `ClassyLoss`es use this `weight` buffer though. So I think extending the public `nn.Module` class should be fine.

Reviewed By: mannatsingh

Differential Revision: D18032147

fbshipit-source-id: e86e0bb0b69020bdbcc1594cd0fa19b650b2a205

Summary:
Pull Request resolved: facebookresearch/ClassyVision#87

This diff has the goal of removing the custom batching logic from the classy datasets and instead directly using the PyTorch DataLoader's batching logic.

It became slightly more complicated because there are several blockers to migrating the EverstoreDatasets to the PyTorch DataLoaders (mainly backfilling logic and my attempts to serialize the EverstoreDataset have repeatedly run into issues).

So, instead, I do the following:

1. Isolate the EverstoreDataset with its own ClassyEverstoreDataset class. This class still uses the BatchDataset, etc. and the Classy Dataloader logic, etc. Goal is to migrate the custom code to the FB folder / isolate the usage in the EverstoreDatasets so that it can all be deleted once we migrate all users to Koski datasets.
2. Move OSS batching to PyTorch Dataloader.

At the end of this stack I want all dataset wrappers to be deleted or in the fb folder, the Dataset object will be deleted, and I will move the everstore functions to a folder marked as deprecated.

NOTE: I'm aware I conflated two changes in this diff. I thought I had successfully isolated the everstore and then realized afterwards that I needed a separate class and it's a bit annoying to split the diff. That being said, I'm happy to split the diff if it's difficult to review.

Reviewed By: mannatsingh

Differential Revision: D17896229

fbshipit-source-id: 8a3b99b63634e0e6ac9827121ac75cf387555aa6

Summary:
Pull Request resolved: fairinternal/ClassyVision#7

Pull Request resolved: facebookresearch/ClassyVision#88

Removing sharding logic from OSS folder. It still exists for the Everstore dataset since it is not yet compatible with OSS PyTorch loaders, but that dataset is going to be deprecated by end of stack.

The sharding logic is replaced by the PyTorch distributed sampler. This sampler has two differences from previous logic which are:
1. is that it silently replaces samples rather than marking them as dummy samples (so samples at the front of the dataset will get used slightly more often, should be fine if we shuffle regularly and is default behavior in the wild).
2. It chunks data for each process, we previously used mod to distribute data. Should not affect things if shuffling is used.

Reviewed By: mannatsingh, vreis

Differential Revision: D17897080

fbshipit-source-id: 4a7f3c7bc1754dd4554681482dd5265350b3ab93

Summary:
Pull Request resolved: facebookresearch/ClassyVision#89

Replace custom shuffling / resampling logic with logic from OSS PyTorch samplers.

Reviewed By: mannatsingh

Differential Revision: D17897767

fbshipit-source-id: 5e760888c70aa299c1feae2c40b16e6465cb4c2d

Summary:
Pull Request resolved: facebookresearch/ClassyVision#90

Pull Request resolved: fairinternal/ClassyVision#8

Transform logic stays in the classy vision dataset, but we are no longer going to use it as a wrapper, just default logic the user can make use of in the classy_dataset wrapper.

A lot of the work in this diff is reworking the tests which made heavy use of TransformDataset in various places and I ended up moving some of the logic from test/generic/utils.py to fb/test/generic/dataset_utils.py.

Reviewed By: mannatsingh

Differential Revision: D17907249

fbshipit-source-id: fe5d625b7328b43607473d11aa14cc12c88a700a

Summary:
Pull Request resolved: fairinternal/ClassyVision#5

Pull Request resolved: facebookresearch/ClassyVision#83

When removing the wrap dataset, I realized it is used quite a bit by the Tyrell datasets...I then realized that the Tyrell datasets were all broken but that the dataset tests hadn't been running for quite a while (at least a month).

AFAIK the tyrell codebase hasn't been extensively used since Matthew has been out of the office, so I just fixed the datasets and made sure all of the tests pass locally on my devbox (I also added a couple of devbox only tests).

The first three points below are the bulk of the changes

Changes required:
* Made the Tyrell datasets use explicit passes to classy dataset
* Made use of the parse config call to make code reuse better
* Removed unused functions that were added for matching classy API (simplified datasets)
* Changed batchsize -> batchsize_per_replica
* Changed the num_sample default in the classy_dataset parse config function to be None.
* Tyrell overwrote the getitem function in classy dataset several times, this causes issues if we don't call the super first, so I updated this where necessary
* Batching is owned by the dataloader now, so length checks that take into account batching must be on the dataloader.

Reviewed By: vreis

Differential Revision: D18033214

fbshipit-source-id: 22689a16c79fade52f7d7eddc349ba2f3e6019d8