Swin S3-Small is different from original repo?

[gau-nernst]

In timm

pytorch-image-models/timm/models/swin_transformer.py

Lines 776 to 782 in f677190

	@register_model
	def swin_s3_small_224(pretrained=False, **kwargs) -> SwinTransformer:
	""" Swin-S3-S @ 224x224, https://arxiv.org/abs/2111.14725
	"""
	model_args = dict(
	patch_size=4, window_size=(14, 14, 14, 7), embed_dim=96, depths=(2, 2, 18, 2), num_heads=(3, 6, 12, 24))
	return _create_swin_transformer('swin_s3_small_224', pretrained=pretrained, **dict(model_args, **kwargs))

S3-Small uses window size (14, 14, 14, 7).

However, from what I see in https://github.com/microsoft/Cream/blob/main/AutoFormerV2/configs/S3-S.yaml, the original repo uses (14, 14, 14, 14). Is it a mistake?

I also see that timm has pre-trained weights for S3-Small. Was it trained by @rwightman or converted from microsoft/Cream repo?

[rwightman]

@gau-nernst I did take the checkpoints from the Cream repo, yes. It was a while ago now, but I think the actual weights didn't match their config? I did what was needed model config wise to load the weights.

at 224x224 a window size of 14 doesn't make any sense in any case, the feature map is 7x7 for the final stage, you'd need an input size of 448x448 to support 14,14,14,14. The resulting model in timm matches the reported S3-Small accuracy (83.758 in timm) so I think their configs are just out of sync with weights...

[gau-nernst]

Thanks @rwightman. That makes sense. The original Swin code has a logic to not perform window partition when input size is smaller than window size, so it still works, but not as expected I suppose.

pytorch-image-models/timm/models/swin_transformer.py

Lines 224 to 227 in c692715

	if min(self.input_resolution) <= self.window_size:
	# if window size is larger than input resolution, we don't partition windows
	self.shift_size = 0
	self.window_size = min(self.input_resolution)