-
Notifications
You must be signed in to change notification settings - Fork 225
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Question about the MMN network design #24
Comments
Hi, Thanks for your interest in our project! MMN aims to calculate the matching degree of student and teacher subnets. One image is not abundant enough to reflect their matching degree, thus we use a batch of data containing several images to reflect. So the size of MMN depends on the slice size. Best, |
Hi, thanks for the quick response! n = num classes the MMN net takes a [s, n] shaped vector, reshapes it to [s * n], and then uses a fully connected layer to get an output of size [1] in the second case, neither slice size nor the order of the images matters, and it is also not limited to a single image. Best, |
Hi, Thanks for your interest in our project! We design the MMN to predict the matching degree of two subnets over mini-batch data. Firstly, it's ok for the second case. However, in the second case, the weight tensor of MMN is only Nx1, theoretically worse than (SxN)x1. Besides, the updating batch size of MMN is fixed during supernet training, so we take the first case. Best, |
Hi, thanks for the nice work and code!
The MMN is a single fully-connected layed defined and used in
https://github.com/microsoft/Cream/blob/main/lib/models/structures/supernet.py#L92
https://github.com/microsoft/Cream/blob/main/lib/models/structures/supernet.py#L127
and then used by the board in
https://github.com/microsoft/Cream/blob/main/lib/models/PrioritizedBoard.py#L32
I do wonder why the MMN size depends on the slice size (i.e. the number of images) and concatenates their class probabilities, instead of using the same weights for each of image?
The text was updated successfully, but these errors were encountered: